Skip to content

Test data

For pipeline tests that need a DataFrame with nulls and extreme values. It teaches null_rate and anomaly_rate on a column.

Prerequisite and run

Install rand-engine, then scaffold and run the file from the directory where it was written:

Get the file:

pip install rand-engine
rand-engine get-started test-data
python test-data-example.py

It writes test-data-example.py to the current directory. --out DIR writes it to DIR instead, created when missing. An existing file is never overwritten. Running it prints the first rows of a 1,000-row DataFrame, the null count for amount, and rows carrying the configured anomaly values.

# Databricks notebook source
# Test data: a DataFrame with nulls and anomalies for pipeline tests. Runs as-is with plain python.

# COMMAND ----------
# Describe the data: order rows, with nulls and extreme values on amount.

from rand_engine import DataGenerator

spec = {
  "order_id": {"method": "pk", "kwargs": {"start": 1}},
  "amount": {"method": "floats", "kwargs": {"min": 5, "max": 200, "decimals": 2},
             "null_rate": 0.05, "anomaly_rate": 0.02, "anomaly_values": [-1.0, 9999.0]},
  "category": {"method": "distincts", "kwargs": {"distincts": ["books", "games", "food"]}},
}

# COMMAND ----------
# Generate 1 000 rows into df. The seed makes every run produce the same rows.

df = DataGenerator(spec, seed=42).size(1_000).get_df()

# COMMAND ----------
# Look at the first rows, then count the nulls and list the anomalous rows.

print(df.head())
print("nulls in amount:", df["amount"].isna().sum())
print(df[df["amount"].isin([-1.0, 9999.0])])

What it shows

  • Describe: amount carries null_rate and anomaly_rate, and anomaly_values lists the extreme values drawn in place of some rows.
  • Generate: DataGenerator with a seed builds 1 000 rows into df; the same seed gives the same rows on every run.
  • Look: the first rows, the count of nulls in amount, and the anomalous rows.

Next steps