Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,6 @@

# Plugins
/plugins/data-designer-github/ @NVIDIA-NeMo/data_designer_reviewers @eric-tramel
/plugins/data-designer-group-consistent/ @andreatnvidia
/plugins/data-designer-retrieval-sdg/ @NVIDIA-NeMo/data_designer_reviewers @shan-nvidia @oliverholworthy
/plugins/data-designer-template/ @NVIDIA-NeMo/data_designer_reviewers
57 changes: 57 additions & 0 deletions docs/plugins/data-designer-group-consistent/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Group-consistent generation

Use `group-consistent` to select one correlated candidate record for each
logical group. The selection is deterministic for a given combination of group
values, role, seed, and candidate pool.

## Installation

```bash
uv add data-designer data-designer-group-consistent
```

## Column type

```python
from data_designer_group_consistent.config import GroupConsistentColumnConfig

builder.add_column(
GroupConsistentColumnConfig(
name="synthetic_first_name",
group_by=["patient_id"],
role="patient",
seed=7,
records=[
{"first_name": "Amina", "last_name": "Diallo", "email": "amina@example.test"},
{"first_name": "Carlos", "last_name": "Silva", "email": "carlos@example.test"},
],
field_mapping={
"synthetic_first_name": "first_name",
"synthetic_last_name": "last_name",
"synthetic_email": "email",
},
),
)
```

All mapped fields come from the same candidate record. Rows with the same
`patient_id` therefore receive a coherent first name, last name, and email even
when those rows are generated in different batches.

| Field | Required | Description |
| --- | --- | --- |
| `name` | Yes | Primary output column. It must be a key in `field_mapping`. |
| `group_by` | Yes | Ordered list of upstream columns defining a logical group. |
| `records` | Yes | Non-empty candidate record pool. |
| `field_mapping` | Yes | Mapping from output column names to candidate record fields. |
| `role` | No | Namespace for independent entities in one group. Defaults to `default`. |
| `seed` | No | Deterministic selection seed. Defaults to `0`. |

## Implementation notes

The plugin uses SHA-256 rather than Python's process-dependent `hash()` function.
It does not persist a mapping table or call an LLM. Candidate order and pool size
are part of the effective configuration: changing them may change prior choices.

For the full plugin authoring guide, see the
[main repository docs](https://nvidia-nemo.github.io/DataDesignerPlugins/authoring/).
11 changes: 11 additions & 0 deletions docs/plugins/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,17 @@ Browse available Data Designer plugins by what they add to your data generation
<span class="plugin-doc-card__chips"><span class="plugin-doc-chip">github</span></span>
</span>
</a>
<a class="plugin-doc-card" href="data-designer-group-consistent/" aria-label="Open data-designer-group-consistent documentation">
<span class="plugin-doc-card__header">
<span class="plugin-doc-card__title">data-designer-group-consistent</span>
<span class="plugin-doc-card__version">v0.1.0</span>
</span>
<span class="plugin-doc-card__description">Deterministic group-scoped record generation for Data Designer</span>
<span class="plugin-doc-card__section">
<span class="plugin-doc-card__label">Entry points</span>
<span class="plugin-doc-card__chips"><span class="plugin-doc-chip">group-consistent</span></span>
</span>
</a>
<a class="plugin-doc-card" href="data-designer-retrieval-sdg/" aria-label="Open data-designer-retrieval-sdg documentation">
<span class="plugin-doc-card__header">
<span class="plugin-doc-card__title">data-designer-retrieval-sdg</span>
Expand Down
3 changes: 3 additions & 0 deletions plugins/data-designer-group-consistent/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Owner(s) of this plugin — used to generate the root CODEOWNERS file.
# GitHub accepts @username, @org/team, or email format.
* @andreatnvidia
50 changes: 50 additions & 0 deletions plugins/data-designer-group-consistent/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# data-designer-group-consistent

Deterministic group-scoped record generation for Data Designer.

## Installation

```bash
uv add data-designer data-designer-group-consistent
```

## Usage

The `group-consistent` column type selects one candidate record for each logical
group and writes one or more correlated output columns. Selection is derived from
the group values, role, and seed, so it remains stable across row order, batches,
retries, and resumed runs with the same configuration.

```python
from data_designer.config import DataDesignerConfigBuilder
from data_designer_group_consistent.config import GroupConsistentColumnConfig

builder = DataDesignerConfigBuilder()
builder.add_column(
GroupConsistentColumnConfig(
name="synthetic_first_name",
group_by=["patient_id"],
role="patient",
seed=7,
records=[
{"first_name": "Amina", "last_name": "Diallo", "email": "amina@example.test"},
{"first_name": "Carlos", "last_name": "Silva", "email": "carlos@example.test"},
],
field_mapping={
"synthetic_first_name": "first_name",
"synthetic_last_name": "last_name",
"synthetic_email": "email",
},
),
)
```

Every row with the same `patient_id` receives fields from the same candidate
record. Use a different `role` to create an independent identity, such as a
doctor or emergency contact, within the same group.

For the full plugin authoring guide, see the
[main repository docs](https://nvidia-nemo.github.io/DataDesignerPlugins/authoring/).

Plugin documentation for the repository site lives in this package's `docs/`
directory.
57 changes: 57 additions & 0 deletions plugins/data-designer-group-consistent/docs/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Group-consistent generation

Use `group-consistent` to select one correlated candidate record for each
logical group. The selection is deterministic for a given combination of group
values, role, seed, and candidate pool.

## Installation

```bash
uv add data-designer data-designer-group-consistent
```

## Column type

```python
from data_designer_group_consistent.config import GroupConsistentColumnConfig

builder.add_column(
GroupConsistentColumnConfig(
name="synthetic_first_name",
group_by=["patient_id"],
role="patient",
seed=7,
records=[
{"first_name": "Amina", "last_name": "Diallo", "email": "amina@example.test"},
{"first_name": "Carlos", "last_name": "Silva", "email": "carlos@example.test"},
],
field_mapping={
"synthetic_first_name": "first_name",
"synthetic_last_name": "last_name",
"synthetic_email": "email",
},
),
)
```

All mapped fields come from the same candidate record. Rows with the same
`patient_id` therefore receive a coherent first name, last name, and email even
when those rows are generated in different batches.

| Field | Required | Description |
| --- | --- | --- |
| `name` | Yes | Primary output column. It must be a key in `field_mapping`. |
| `group_by` | Yes | Ordered list of upstream columns defining a logical group. |
| `records` | Yes | Non-empty candidate record pool. |
| `field_mapping` | Yes | Mapping from output column names to candidate record fields. |
| `role` | No | Namespace for independent entities in one group. Defaults to `default`. |
| `seed` | No | Deterministic selection seed. Defaults to `0`. |

## Implementation notes

The plugin uses SHA-256 rather than Python's process-dependent `hash()` function.
It does not persist a mapping table or call an LLM. Candidate order and pool size
are part of the effective configuration: changing them may change prior choices.

For the full plugin authoring guide, see the
[main repository docs](https://nvidia-nemo.github.io/DataDesignerPlugins/authoring/).
36 changes: 36 additions & 0 deletions plugins/data-designer-group-consistent/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

[project]
name = "data-designer-group-consistent"
version = "0.1.0"
description = "Deterministic group-scoped record generation for Data Designer"
requires-python = ">=3.10"
dependencies = [
"data-designer>=0.5.7",
]
license = "Apache-2.0"
readme = "README.md"
authors = [
{name = "NVIDIA Corporation"},
]
classifiers = [
"Development Status :: 3 - Alpha",
"Programming Language :: Python :: 3",
]

[project.entry-points."data_designer.plugins"]
group-consistent = "data_designer_group_consistent.plugin:plugin"

[project.urls]
Repository = "https://github.com/NVIDIA-NeMo/DataDesignerPlugins"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["src/data_designer_group_consistent"]

[tool.ruff]
extend = "../../pyproject.toml"
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

from typing import Literal

from data_designer.config.base import SingleColumnConfig
from pydantic import Field, JsonValue, model_validator
from typing_extensions import Self


class GroupConsistentColumnConfig(SingleColumnConfig):
"""Selects one correlated record deterministically for each logical group."""

column_type: Literal["group-consistent"] = "group-consistent"
group_by: list[str] = Field(min_length=1)
records: list[dict[str, JsonValue]] = Field(min_length=1)
field_mapping: dict[str, str] = Field(min_length=1)
role: str = Field(default="default", min_length=1)
seed: int = 0

@model_validator(mode="after")
def validate_generation_contract(self) -> Self:
"""Validate group keys, output columns, and candidate record fields.

Returns:
The validated configuration.

Raises:
ValueError: If the configuration cannot generate every declared output.
"""
if len(set(self.group_by)) != len(self.group_by):
raise ValueError("group_by columns must be unique")
if self.name not in self.field_mapping:
raise ValueError(f"field_mapping must include the primary output column {self.name!r}")
if not self.role.strip():
raise ValueError("role must not be blank")

required_fields = set(self.field_mapping.values())
for index, record in enumerate(self.records):
missing_fields = required_fields - record.keys()
if missing_fields:
raise ValueError(f"records[{index}] is missing mapped fields: {sorted(missing_fields)}")
return self

@staticmethod
def get_column_emoji() -> str:
return "🔗"

@property
def required_columns(self) -> list[str]:
return self.group_by

@property
def side_effect_columns(self) -> list[str]:
return [column for column in self.field_mapping if column != self.name]
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

from __future__ import annotations

import hashlib
import json
import math
from datetime import date, datetime, time
from typing import TYPE_CHECKING

from data_designer.engine.column_generators.generators.base import ColumnGeneratorFullColumn
from pydantic import JsonValue

from data_designer_group_consistent.config import GroupConsistentColumnConfig

if TYPE_CHECKING:
import pandas as pd


def normalize_key_component(value: object) -> dict[str, JsonValue]:
"""Convert a group-key value to a stable JSON representation.

Args:
value: Scalar value from a group-key column.

Returns:
Type-tagged JSON data suitable for deterministic hashing.
"""
item_method = getattr(value, "item", None)
if callable(item_method):
value = item_method()

if value is None:
return {"type": "null", "value": None}
if isinstance(value, float) and math.isnan(value):
return {"type": "null", "value": None}
if isinstance(value, float) and math.isinf(value):
return {"type": "float", "value": "infinity" if value > 0 else "-infinity"}
if isinstance(value, (date, datetime, time)):
return {"type": type(value).__name__, "value": value.isoformat()}
if isinstance(value, (bool, int, float, str)):
return {"type": type(value).__name__, "value": value}
return {
"type": f"{type(value).__module__}.{type(value).__qualname__}",
"value": str(value),
}


def select_record_index(group_values: tuple[object, ...], role: str, seed: int, record_count: int) -> int:
"""Select a deterministic candidate record for a logical group.

Args:
group_values: Ordered values from the configured group-key columns.
role: Namespace for independently generated entities in the same group.
seed: User-controlled generation seed.
record_count: Number of candidate records.

Returns:
An index into the configured candidate record list.
"""
payload = {
"version": 1,
"group": [normalize_key_component(value) for value in group_values],
"role": role,
"seed": seed,
}
encoded = json.dumps(payload, ensure_ascii=False, separators=(",", ":"), sort_keys=True).encode("utf-8")
return int.from_bytes(hashlib.sha256(encoded).digest(), byteorder="big") % record_count


class GroupConsistentColumnGenerator(ColumnGeneratorFullColumn[GroupConsistentColumnConfig]):
"""Generate correlated fields that remain stable within each logical group."""

def generate(self, data: pd.DataFrame) -> pd.DataFrame:
"""Add fields from one deterministic candidate record per group.

Args:
data: Batch containing the configured group-key columns.

Returns:
The batch with the configured output columns added.
"""
records = self.config.records
indexes = [
select_record_index(group_values, self.config.role, self.config.seed, len(records))
for group_values in data[self.config.group_by].itertuples(index=False, name=None)
]
selected_records = [records[index] for index in indexes]
for output_column, record_field in self.config.field_mapping.items():
data[output_column] = [record[record_field] for record in selected_records]
return data
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

from data_designer.plugins.plugin import Plugin, PluginType

plugin = Plugin(
config_qualified_name="data_designer_group_consistent.config.GroupConsistentColumnConfig",
impl_qualified_name="data_designer_group_consistent.impl.GroupConsistentColumnGenerator",
plugin_type=PluginType.COLUMN_GENERATOR,
)
Loading
Loading