DataForge V1

The Deterministic Synthetic Dataset Generator for Machine Learning and Data Engineering.

---

⚡ The "Cold Start" Data Problem

Machine Learning Engineers and Data Engineers frequently face the Cold Start Data Problem: building models, ETL pipelines, and dashboards before the real production data is available.

Traditional mocking tools (like Faker) generate independent, flat, and uncorrelated columns. They fail to capture the complex statistical nuances of the real world. If you use basic mock data, your ML models learn nothing, and your data quality tests pass blindly.

🛠 Enter DataForge

DataForge is a fast, configuration-driven command-line engine built to generate complex, highly realistic synthetic datasets. It allows you to define precise statistical distributions, engineer cross-column mathematical relationships, and simulate real-world data corruption—all while guaranteeing 100% reproducibility.

Why Choose DataForge?

• 🎯 100% Deterministic Architecture: Powered by a robust NumPy seed-derivation tree. The exact same configuration and root seed will always yield the exact same dataset, down to the last float, regardless of the order of execution.
• 🔗 Advanced Relationship Engine: Data isn't random. Define dependencies where columns are conditionally modified, algebraically derived, scaled proportionally, or mathematically correlated using Gaussian Copulas (to preserve marginal distributions).
• 🐛 Controlled Defect Injection: Real data is messy. Test your pipelines' resilience by safely injecting noise (missingness, outliers, duplicates, label noise, and typos). DataForge exports both a clean and corrupted dataset for A/B benchmarking.
• 🚀 High Performance: Built on top of vectorized pandas and numpy operations. DataForge can generate millions of rows with complex relationships in mere seconds.

---

📦 Installation

Clone the repository and install the package locally in editable mode:

git clone https://github.com/yourusername/DataForge.git
cd DataForge
pip install -e .

---

🚀 Quickstart

DataForge uses declarative YAML or JSON specifications to define the structure of your dataset.

1. Create a Specification (spec.yaml)

name: "financial_transactions"
description: "Simulated credit card transactions with fraud markers."
num_rows: 100000
seed: 42

columns:
  - name: "transaction_id"
    type: "uuid"
    
  - name: "merchant_category"
    type: "categorical"
    constraints:
      categories: ["Retail", "Travel", "Dining", "Online"]
      weights: [0.4, 0.2, 0.2, 0.2]
      
  - name: "amount"
    type: "float"
    distribution:
      type: "log_normal"
      mean: 50.0
      sigma: 1.5
    constraints:
      positive: true

  - name: "is_fraud"
    type: "boolean"
    constraints:
      categories: ["False", "True"]
      weights: [0.98, 0.02]

relationships:
  # Fraudulent transactions tend to be significantly larger
  - type: "conditional"
    columns: ["is_fraud", "amount"]
    conditions:
      "True":
        factor: 5.0  # Multiply amount by 5 if it's fraud

defects:
  # Inject 2% missing values into amounts to test ETL resilience
  - type: "missingness"
    columns: ["amount"]
    rate: 0.02

output:
  formats: ["csv", "parquet"]
  directory: "./output"
  save_clean: true

2. Run the Generator

Execute the CLI command against your spec file:

dataforge generate spec.yaml

3. Review the Output

Check the ./output/ directory! DataForge automatically exports:

1. financial_transactions_clean.csv (The perfect ground-truth dataset).
2. financial_transactions_corrupted.csv (The dataset with 2% missing amounts).
3. financial_transactions_corrupted_manifest.json (A rich metadata provenance file recording exact generation time, spec snapshots, and column summary statistics like null counts and min/max boundaries).

---

📖 Feature Matrix

Column Types & Distributions

Type	Support	Distributions / Constraints
Numeric	integer, float	normal, uniform, log_normal, exponential, poisson, min/max, positive
Categorical	categorical	categories, weights (Probability targets)
Boolean	boolean	categories (e.g. Yes/No), weights
Datetime	datetime	Bounded within min and max timestamps
Text	text, uuid	Fixed pools, standard UUIDv4

Cross-Column Relationships

Type	Description
correlation	Mathematically induce Spearman rank correlation between columns without destroying their original marginal distributions.
conditional	Modify a target column (shift, scale, or overwrite) based on the specific value of a source column.
derived	Use algebraic expressions (e.g., col_A * col_B) and optional Gaussian noise to compute a new feature.
proportional	Scale one column based on another (supports inverse proportionality with divide-by-zero protection).

Defect Injection Types

Defect	Description
missingness	Safely nullify a percentage of values (np.nan).
outliers	Force extreme values outside of standard deviation bounds.
duplicates	Duplicate rows probabilistically while maintaining the dataset's target row size.
label_noise	Randomly flip categorical values or boolean labels to test model robustness.
typos	Introduce character-level noise (swaps, insertions, deletions) in text strings.

---

🧠 Architecture Overview

The DataForge V1 pipeline executes in five strict phases to guarantee determinism:

1. Spec Parser & Validator: Uses Pydantic V2 to rigorously validate the schema and constraints.
2. Column Generators: Base columns are generated independently using derived seeds (RootSeed + ColIndex).
3. Relationship Engine: Cross-column relationships are applied sequentially in-place.
4. Defect Engine: A copy of the clean dataset is made, and probabilistic noise is injected using a separate deterministic seed stream.
5. Exporter: Data is flushed to disk in columnar (Parquet) and flat (CSV) formats alongside JSON metadata manifests.

---

🤝 Contributing

DataForge is open-source and community-driven. If you want to contribute a new statistical distribution, relationship rule, or output format, please read our Contributing Guidelines.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Run the Test Suite (pytest tests/)
4. Commit your Changes (git commit -m 'Add some AmazingFeature')
5. Push to the Branch (git push origin feature/AmazingFeature)
6. Open a Pull Request

📄 License

Distributed under the MIT License. See LICENSE for more information.