Skip to content

Add group-consistent generation plugin - #86

Closed
andreatnvidia wants to merge 2 commits into
mainfrom
andreatnvidia/feat/group-scoped-consistent-generation
Closed

Add group-consistent generation plugin#86
andreatnvidia wants to merge 2 commits into
mainfrom
andreatnvidia/feat/group-scoped-consistent-generation

Conversation

@andreatnvidia

@andreatnvidia andreatnvidia commented Aug 21, 2026

Copy link
Copy Markdown

Summary

Adds the data-designer-group-consistent column generator plugin. It selects one candidate record deterministically for each logical group and emits correlated output fields that remain stable across row order, batches, retries, and resumed runs with the same configuration.

Why

Datasets often contain multiple rows for the same logical entity. Independent row generation can assign contradictory attributes to those rows. This plugin provides group-scoped consistency without shared mutable state or a persistent cache.

Usage

Demo: keep a synthetic contact consistent across related case events
import pandas as pd

from data_designer.config.config_builder import DataDesignerConfigBuilder
from data_designer.config.seed_source_dataframe import DataFrameSeedSource
from data_designer_group_consistent.config import GroupConsistentColumnConfig

events = pd.DataFrame(
    {
        "case_id": ["case-001", "case-002", "case-001"],
        "event": ["opened", "opened", "updated"],
    }
)

builder = DataDesignerConfigBuilder()
builder.with_seed_dataset(DataFrameSeedSource(df=events))
builder.add_column(
    GroupConsistentColumnConfig(
        name="synthetic_first_name",
        group_by=["case_id"],
        role="contact",
        seed=7,
        records=[
            {"first_name": "Amina", "last_name": "Diallo", "email": "amina@example.test"},
            {"first_name": "Carlos", "last_name": "Silva", "email": "carlos@example.test"},
            {"first_name": "Mei", "last_name": "Chen", "email": "mei@example.test"},
        ],
        field_mapping={
            "synthetic_first_name": "first_name",
            "synthetic_last_name": "last_name",
            "synthetic_email": "email",
        },
    ),
)

Both case-001 rows receive the same first name, last name, and email even if they are processed in different batches. A different role creates an independent selection namespace within the same case.

How

  • Canonicalizes the configured group-column values.
  • Hashes the group values with role, seed, and an algorithm version using SHA-256.
  • Selects one candidate record from the configured pool.
  • Writes the primary output and correlated side-effect columns from that record.
  • Validates group keys, output mappings, roles, and candidate fields at configuration time.

Validation

  • .venv/bin/ruff check .
  • .venv/bin/ruff format --check .
  • .venv/bin/pytest plugins/data-designer-group-consistent/tests/ -q - 10 passed
  • make test-plugin PLUGIN=data-designer-group-consistent - 10 passed in an isolated environment
  • .venv/bin/ddp validate
  • .venv/bin/ddp plugin-docs --check
  • .venv/bin/ddp license-headers --check
  • .venv/bin/ddp catalog check
  • ddp package-index check in the activated project environment
  • .venv/bin/zensical build --clean --strict

Checklist

  • Follows the template structure (config.py, impl.py, plugin.py) if adding a plugin
  • assert_valid_plugin(plugin) passes
  • Unit tests included and passing (make test-plugin PLUGIN=<name>)
  • Plugin installs standalone (uv pip install -e plugins/<plugin-dir>)
  • Plugin docs regenerated if plugin docs, list, or metadata changed (make plugin-docs)
  • Documentation builds if docs changed (make docs)
  • catalog/plugins.json updated only if preparing a first release (make catalog PLUGIN=<name>)
  • .github/CODEOWNERS regenerated if ownership changed (make codeowners)
  • Per-plugin CODEOWNERS file included (auto-created by ddp new)
  • NVIDIA SPDX headers on all files (make check-license-headers)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant