Compatibility, migration and recovery¶

Choose an environment¶

rand-engine requires Python 3.11 or newer. Continuous integration tests Python 3.11 through 3.14; later Python releases may work, but are not part of that tested set. The supported dependency bounds are pandas >=2.2,<4, pyarrow >=19, numpy >=1.26; Faker is uncapped.

pip install rand-engine
pip install "rand-engine[faker]"  # only for RandSpecs.faker_pool

The core install includes Python, NumPy and pandas generation. PySpark is deliberately a caller-supplied dependency: use SparkGenerator only when your application already has a Spark session and pyspark.sql.functions. Spark CI runs Spark 3.5 and 4.x: Spark 3.5 on Python 3.11 and Spark 4.x on Python 3.12 through 3.14, on Linux, macOS and Windows. It uses Java 17. The Windows Spark skip applies from Python 3.12: those Spark tests skip on Windows, so use the pandas generator or a supported Spark/Python partition there instead.

Pick the generator¶

Use DataGenerator for seeded pandas DataFrames, record streams and files. The same spec, seed and size reproduce its NumPy-generated frame; generated values still depend on the installed NumPy version, RandSpec and requested batch/row bounds. Use SparkGenerator for ordinary columns in an existing Spark job. Its seed controls only pk and fk key columns; ordinary Spark columns remain unseeded.

SparkGenerator accepts ordinary methods, stateless keys, and int_zfilled lengths through 15. It refuses uint64, inclusive integer domains wider than 2**53, NumPy-only distributions such as poisson, and modifiers such as null_rate. The four correlated methods distincts_map, distincts_map_prop, distincts_multi_map, and complex_distincts are refused too: Spark raises SpecValidationError and points at DataGenerator. See the Spark method table for the supported surface rather than copying an exhaustive grammar into a job.

int_zfilled accepts an integer length from 1 through 18 in DataGenerator; invalid lengths fail validation before generation. SQLite is one batch output table through write.format("sqlite"); it is not a database lookup or a production database integration. See Writing files for its table, append and overwrite semantics.

Move from 0.7 to 0.8¶

Keep the supported imports:

from rand_engine import DataGenerator, RandSpecs, SparkGenerator

Then run your existing RandSpecs through construction before a production job. Version 0.8 makes several invalid inputs fail earlier, including int_zfilled lengths outside 1..18. Use the errors guide to correct a complete validation report rather than retrying one message at a time.

For new work, use the CLI starters and rendered reference: rand-engine list shows six starters, of which five are runnable Python tutorials; rand-engine get-started agent --out PROJECT writes an opt-in project context in 0.8.0. Spark pk and fk definitions now work through the shared stateless key path. SQLite output is available through the batch writer, and normalized stream microbatches convert to pandas records that retain their public Python values. Read the changelog for the complete version history, including the distinct 0.7 migration changes.

Recover from common symptoms¶