Skip to content

Lake

For data engineers who need related tables. It teaches pk and fk: orders point at customers, and a join proves every order finds its customer.

Prerequisite and run

Install rand-engine, then scaffold and run the file from the directory where it was written:

Get the file:

pip install rand-engine
rand-engine get-started lake
python lake-example.py

It writes lake-example.py to the current directory. --out DIR writes it to DIR instead, created when missing. An existing file is never overwritten. Running it writes Parquet files below lake/, reads them back, and prints customer, order, and joined-row counts.

# Databricks notebook source
# Lake: two related tables written to a Parquet lake and read back. Runs as-is with plain python.

# COMMAND ----------
# Describe the keys: customers own a pk, orders point at it through an fk.

from pathlib import Path

from rand_engine import DataGenerator

customer_pk = {"method": "pk", "kwargs": {"start": 1, "format": "C{:05d}"}}
customers_spec = {
  "customer_id": customer_pk,
  "city": {"method": "distincts", "kwargs": {"distincts": ["Rio", "Recife", "Porto Alegre"]}},
}
orders_spec = {
  "order_id": {"method": "pk", "kwargs": {"start": 1}},
  "customer_id": {"method": "fk", "kwargs": {"parent": customer_pk, "parent_size": 500, "skew": 1.0}},
  "amount": {"method": "floats", "kwargs": {"min": 5, "max": 500, "decimals": 2}},
}

# COMMAND ----------
# Write each table to the lake as one Parquet file under lake/, and record the relation.

tables = {"customers": Path("lake/customers.parquet"), "orders": Path("lake/orders.parquet")}
DataGenerator(customers_spec, seed=1).size(500).write.format("parquet").save(str(tables["customers"]))
DataGenerator(orders_spec, seed=2).size(2_000).write.format("parquet").save(str(tables["orders"]))
relations = [("orders", "customer_id", "customers", "customer_id")]

# COMMAND ----------
# Read the lake back, then join: every order must find its customer.

import pandas as pd

frames = {name: pd.read_parquet(path) for name, path in tables.items()}
child, fk, parent, pk = relations[0]
joined = frames[child].merge(frames[parent], left_on=fk, right_on=pk)
print("customers:", len(frames["customers"]), "orders:", len(frames["orders"]))
print("orders joined to a customer:", len(joined))

What it shows

  • Describe: the customers own a pk; the orders carry an fk that points at it.
  • Write: each table lands as one Parquet file under lake/, in the directory the file runs in, and the relation is recorded.
  • Read back: both tables are read from the lake and joined on the fk, then the counts are printed.

Next steps