feat: add optional Slurm package

[andreatnvidia]

📋 Summary

Adds the optional data-designer-slurm distribution and exposes it through the exact-version data-designer[slurm] extra. This establishes package isolation and shared-namespace behavior before Slurm runtime features are added.

🔗 Related Issue

Closes #852

Part of #850

🔄 Changes

• Add data-designer-slurm as a publishable workspace package under data_designer.slurm.
• Generate exact same-version dependencies in both directions while keeping the leaf optional for base installs.
• Extend dependency auditing to evaluate explicitly selected static and dynamic extras.
• Add clean built-wheel checks for base-only, extra-selected, direct-leaf, missing-counterpart, metadata, and namespace behavior.
• Wire the package into development, test, lint, build, and CI workflows.

🧪 Testing

• .venv/bin/pytest packages/data-designer/tests packages/data-designer-slurm/tests - 1121 passed, 1 skipped
• make test-slurm-wheel-install
• .venv/bin/ruff check --fix .
• .venv/bin/ruff format .
• .venv/bin/python scripts/update_license_headers.py --check
• uv lock --check
• Unit tests added/updated
• Built-wheel installation tests added

✅ Checklist

• Follows commit message conventions
• Commits are signed off (DCO)
• Architecture docs updated

[greptile-apps]

Greptile Summary

The PR adds an independently publishable data-designer-slurm distribution and exposes it through the exact-version data-designer[slurm] extra.

• Integrates the Slurm package into the workspace, builds, tests, linting, CI, documentation, and dependency auditing.
• Adds clean-wheel installation and namespace checks for base-only, extra-selected, and direct-leaf installations.
• Updates publishing to build, validate, and upload the Slurm distribution before the base package that advertises its extra.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
packages/data-designer-slurm/pyproject.toml	Defines the optional Slurm distribution with synchronized dynamic versioning and an exact dependency on the base package.
packages/data-designer/pyproject.toml	Adds the exact-version Slurm optional dependency while keeping it absent from base installations.
scripts/publish.sh	Includes the Slurm artifact in synchronized releases and publishes it before the package advertising the extra.
scripts/test_slurm_package_install.py	Exercises clean built-wheel metadata, installation isolation, dependency resolution, and shared namespace behavior.
scripts/audit_package_dependencies.py	Extends dependency auditing to account for explicitly selected static and dynamic extras.
uv.lock	Updates the workspace lock to include the new Slurm package and optional dependency relationship.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    User[User installation] --> Base[data-designer]
    Base --> Config[data-designer-config]
    Base --> Engine[data-designer-engine]
    Extra[data-designer slurm extra] --> Base
    Extra --> Slurm[data-designer-slurm]
    Slurm --> Base
    Publish[publish.sh] --> Config
    Publish --> Engine
    Publish --> Slurm
    Publish --> Base

Loading

_{Reviews (5): Last reviewed commit: "docs: generalize Slurm package wording" | Re-trigger Greptile}

[github-actions]

Thanks for putting this together, @andreatnvidia — this is a clean, well-scoped foundation for the Slurm work.

Summary

This PR adds the optional data-designer-slurm workspace package (namespace data_designer.slurm) and exposes it via the exact-version data-designer[slurm] extra, establishing package isolation and shared-namespace behavior ahead of any Slurm runtime features. It wires the new package into the build/test/lint/CI tooling and extends the dependency auditor to evaluate selected extras. The implementation matches the stated intent: the package is intentionally a near-empty shell, with the substance being packaging correctness, which is validated by an impressively thorough built-wheel test.

Findings

Warnings — Worth addressing

scripts/publish.sh:31 — New package is not wired into the publish pipeline

• What: PACKAGE_DIRS still lists only data-designer-config, data-designer-engine, and data-designer. data-designer-slurm is added to install, build, test, lint, and CI, but not to the publisher. The post-publish summary URLs (publish.sh:544-546, 564-566) are also hardcoded to the three existing packages.
• Why: The base data-designer wheel now declares data-designer-slurm=={{ version }} under the slurm extra, and the README advertises pip install "data-designer[slurm]". Once a release goes out, data-designer[slurm] will resolve to a data-designer-slurm version that was never uploaded to PyPI, so the extra install will fail for real users. The excellent missing-counterpart case in test_slurm_package_install.py:799-819 actually demonstrates exactly this failure mode.
• Suggestion: Add packages/data-designer-slurm to PACKAGE_DIRS (and a corresponding project URL in both summary blocks). If publishing is intentionally deferred to a later PR in the Epic: Slurm batch execution v1 #850 series, a one-line note in the PR description or a # TODO(#850): publish slurm marker would make that explicit so it isn't forgotten at release time.

Suggestions — Take it or leave it

Makefile:209 — test-slurm uses .venv/bin/pytest while sibling targets use uv run --group dev

• What: test-config/test-engine/test-interface all invoke uv run --group dev pytest ..., which auto-syncs the environment. The new test-slurm (and test-slurm-wheel-install, verify-imports, format-slurm, etc.) shell out to .venv/bin/... directly.
• Why: make test now depends on test-slurm, so running make test on a machine without a populated .venv would fail on the Slurm leg where the other legs would have self-provisioned. It's a minor consistency wrinkle rather than a bug (CI runs make install-dev first).
• Suggestion: Consider uv run --group dev pytest $(SLURM_TESTS) for test-slurm to match the neighbors, unless the .venv/bin form is a deliberate choice worth a short comment.

Makefile:220 — test-isolated doesn't cover the Slurm package

• What: The isolated-test suite (test-isolated → config/engine/interface) verifies dependency boundaries per package but omits slurm.
• Why: The new package's whole reason to exist is its packaging/dependency boundary, and the built-wheel test already covers isolation well — so this is low priority. But an isolated target would keep the pattern uniform if the package later gains code.
• Suggestion: Optional — add test-slurm-isolated if/when the package has importable logic worth isolating. Fine to leave as-is for now given the wheel test.

What Looks Good

• scripts/test_slurm_package_install.py is genuinely excellent — it validates version pinning in both directions, the extra marker, Provides-Extra, the absence of a top-level __init__.py in the leaf wheel, and all four install permutations (base-only, extra, direct-leaf, missing-counterpart). That last case proving the install fails without the counterpart is exactly the right kind of negative test.
• The audit extension (audit_package_dependencies.py) is a clean, backward-compatible addition: selected_extras defaults to None, the PACKAGE:EXTRA CLI parsing validates its format, and both static and dynamic optional-dependency sources are handled symmetrically. The accompanying test_includes_selected_dynamic_optional_dependencies test exercises the real dynamic-hook path.
• Docs and invariants stay honest: AGENTS.md is updated to four packages and, nicely, reframes the layering as "packaging-only reverse edge; no code may import against this flow" — which correctly captures the subtlety that the [slurm] extra points "backward" without violating import direction.

Structural Impact

(graphify, 3.5s)

Risk: LOW (localized change)

• 5 Python files, 0 AST entities, 0/78 clusters

• Note: changes touch unknown package(s) (data-designer-slurm); analysis may be incomplete - update _PACKAGE_SUBDIRS in structural_impact.py.

Verdict

Needs changes — The publish-pipeline gap (publish.sh) is the one item worth resolving (or explicitly deferring in the PR description) before this ships, since the advertised data-designer[slurm] extra would otherwise break at install time once released. The two suggestions are optional polish.

---

This review was generated by an AI assistant.

[nabinchha]

Thanks for putting this together, @andreatnvidia — the built-wheel coverage gives the new package boundary a strong foundation.

Summary

This PR adds the optional data-designer-slurm distribution, exposes it through the exact-version data-designer[slurm] extra, and integrates it with the workspace, CI, dependency audit, build, and release tooling. The packaging behavior matches the stated scope, but the extra-aware audit currently misclassifies some dependency gaps and the operational guidance still assumes a three-package repository in several important places.

Findings

Warnings — Worth addressing

scripts/audit_package_dependencies.py:173 — Selected project extras leak into unrelated dependency closures

• What: Each workspace project's declarations are precomputed with selected_extras_by_project, then requirements_for() returns that expanded declaration set regardless of the extras requested on the current dependency edge. I reproduced this with a consumer of plain base (not base[leaf]): selecting base:leaf globally made an import from leaf appear guaranteed_by: ["base"] and downgraded it from high to low severity. The current repository output shows the same pollution: adding --extra data-designer:slurm makes data-designer-slurm appear to guarantee click for data-designer through the base↔leaf cycle.
• Why: The dependency recipe explicitly treats low-severity entries as safely guaranteed by mandatory dependencies. This false downgrade can therefore hide a real standalone-installation gap from both the report and automated fix prioritization.
• Suggestion: Cache workspace declarations by (project_name, frozenset(edge_extras)) and call declared_dependencies() with the extras carried by that specific dependency edge. Use selected_extras_by_project only when constructing the selected root project's own declarations. Please add regression cases for a consumer of plain base and a consumer of base[leaf].

.github/workflows/agentic-ci-daily.yml:135 — The four-package change is not reflected in downstream audit instructions and canonical guides

• What: This workflow now generates a four-package inventory, but .agents/recipes/dependencies/recipe.md:14-46 still instructs the audit agent to inspect “all three packages” and explicitly reads only the config, engine, and interface manifests. architecture/overview.md:3-22,53 likewise documents three distributions and the old dependency direction, while DEVELOPMENT.md:108-117 says make test runs only three suites. Even make help still describes installation and isolated testing as config → engine → interface at Makefile:47,63.
• Why: The scheduled audit may omit manual inspection of the leaf manifest, and maintainers now receive contradictory guidance about the repository's package graph and standard test surface.
• Suggestion: Update the dependency recipe, architecture/overview.md, DEVELOPMENT.md, and the remaining Makefile help text to include the Slurm leaf and distinguish the import direction from the packaging-only reverse extra.

What Looks Good

• The integration script validates actual wheel metadata and clean environments for base-only, extra-selected, direct-leaf, and missing-counterpart installs.
• Namespace composition is tested from both source and built wheels, including the absence of a top-level data_designer/__init__.py.
• The exact-version metadata, workspace lock, publishing list, CI wiring, formatting, and focused tests all validate successfully; the full GitHub check suite is green.

Verdict

Needs changes — preserve extra selection per dependency edge and align the scheduled audit/canonical package guidance with the new four-package structure before merge.

---

This review was generated by an AI assistant.

[andreatnvidia]

@nabinchha Thanks, good catches. Addressed both in 9290cd12:

• Workspace dependency closures now respect extras selected on each dependency edge and stop cycles through the package under audit. I added regressions for base vs. base[leaf] and cyclic workspace dependencies.
• Updated the dependency, structure, and test-health recipes, architecture overview, development guide, and Makefile help for all four packages.

The live audit no longer lists data-designer-slurm as guaranteeing core dependencies. All affected tests and package installation checks pass. Could you take another look?

[nabinchha]

Thanks for the follow-up, @andreatnvidia — the latest publish-order change is well targeted.

Summary

The new commit publishes data-designer-slurm before data-designer and adds a regression test for that ordering, so the documented data-designer[slurm] install path is resolvable as soon as the base release appears. The implementation still matches #852's packaging-only scope, and I found no new correctness issues.

Findings

No new findings.

The new Greptile comment identifies the symmetric partial-release window: if the Slurm upload succeeds and the base upload fails, direct leaf installs are temporarily unresolvable. That is technically accurate, but the two distributions have exact-version dependencies in both directions, so no sequential upload order can eliminate both failure windows. This order protects the documented extra-based entry point, and #852 explicitly defers complete publication sequencing to the later release workstream.

What Looks Good

• The order now follows the user-facing dependency path: config and engine first, then the Slurm provider, then the base package that advertises the extra.
• The focused regression test clearly preserves that release invariant.
• The dependency-audit fixes from 9290cd12 remain sound, and the clean built-wheel installation suite passes on both the PR head and the merged target tree.

Verdict

Ship it — the latest commit addresses the prior publish-order concern without introducing a new blocker.

---

This review was generated by an AI assistant.