Create experiments/ and move all experiment code, tests, docs, CI and guards - #37
Merged
cirsteve merged 9 commits intoSep 21, 2026
Conversation
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
There are confirmed runtime-breaking issues in the migrated experiments codepath (Pier qualification script import path and invalid scikit-learn LogisticRegression parameters) that should be fixed before merge.
Review effort: Lite
Findings: None
What changed in this PR
This PR extracts non-shipping investigation/qualification code (consistency pilot, Pier qualification, and Typesafe relevance) into a standalone experiments/ uv project, and adds CI + test guards to keep experimental artifacts and imports from leaking into the shipping assay framework.
Changes:
- Introduces
experiments/as an isolated uv project (its ownpyproject.toml/lock) and moves experiment runners, tests, and docs under it. - Adds repository guards: a framework-edge import scanner and a results/receipts ownership test to prevent committed generated artifacts and out-of-scope receipt references.
- Updates CI and documentation to lint/test
experiments/, validate generated frontend bundle drift, and point Pier qualification docs/tests at the new location.
| File | Description |
|---|---|
| tests/test_results_ownership.py | Adds a git-tracked scan to block committed result/receipt material outside allowlisted evidence bundles and prose. |
| tests/test_optional_jig.py | Removes core optional-jig assertions for modules moved out of the wheel. |
| tests/test_framework_edge.py | Adds an AST-based guard preventing src/assay from importing experiment packages. |
| src/assay/investigations/relevance/__init__.py | Removes the shipping Typesafe relevance package entrypoint after migration to experiments. |
| README.md | Updates install/check commands and receipt-backed test instructions for the new experiments/ layout. |
| integrations/pier/scripts/qualify_local.py | Switches to importing Pier qualification constants from the experiments package. |
| experiments/typesafe_relevance/tests/test_relevance_serve.py | Updates imports/paths to the migrated typesafe_relevance package and its catalogues. |
| experiments/typesafe_relevance/tests/test_relevance_primary.py | Updates imports and catalogue path resolution after migration. |
| experiments/typesafe_relevance/tests/test_relevance_packet.py | Updates imports and repo root/catalogue path resolution after migration. |
| experiments/typesafe_relevance/tests/test_relevance_fitted_rerun.py | Updates import path for the fitted rerun builder after migration. |
| experiments/typesafe_relevance/tests/test_relevance_arms.py | Updates imports and catalogue root after migration. |
| experiments/typesafe_relevance/tests/test_evidence_manifests.py | Fixes repository root resolution for evidence when running under experiments/. |
| experiments/typesafe_relevance/state.py | Adds the extracted pure state projection used by the relevance study. |
| experiments/typesafe_relevance/serve_packet.py | Updates imports to the migrated typesafe_relevance modules. |
| experiments/typesafe_relevance/run_primary.py | Updates imports to the migrated typesafe_relevance modules. |
| experiments/typesafe_relevance/run_packet.py | Updates imports to the migrated typesafe_relevance modules. |
| experiments/typesafe_relevance/run_arms.py | Updates imports to the migrated typesafe_relevance modules. |
| experiments/typesafe_relevance/report_arms.py | Adds report rendering + public export for multi-arm relevance grid results. |
| experiments/typesafe_relevance/render.py | Adds prompt rendering + strict schema + normalization for LLM backends. |
| experiments/typesafe_relevance/README.md | Updates documentation to point at the migrated package and receipts repository gating. |
| experiments/typesafe_relevance/packet.py | Updates documentation wording reference away from the old Scout path. |
| experiments/typesafe_relevance/packet_html.py | Updates documentation wording reference away from the old Scout path. |
| experiments/typesafe_relevance/merge_census.py | Updates imports to the migrated typesafe_relevance package. |
| experiments/typesafe_relevance/mappings.py | Adds the extracted pure decision mappings for the experiment. |
| experiments/typesafe_relevance/fitted_rerun.py | Adds fitted replay/report generation over retained answer vectors (scikit-learn). |
| experiments/typesafe_relevance/catalogues/agent-ops-relevance.v3.yaml | Adds the v3 catalogue document under the experiments package. |
| experiments/typesafe_relevance/catalogues/agent-ops-relevance.v2.yaml | Adds the v2 catalogue document under the experiments package. |
| experiments/typesafe_relevance/catalogues/agent-ops-relevance.v1.yaml | Adds the v1 catalogue document under the experiments package. |
| experiments/typesafe_relevance/catalogue.py | Adds dependency-light catalogue loading and content-addressed versioning. |
| experiments/typesafe_relevance/backends.py | Updates backend implementation to use the migrated render utilities. |
| experiments/typesafe_relevance/__init__.py | Reintroduces a package entrypoint for the migrated relevance implementation. |
| experiments/README.md | Documents the standalone experiments uv project workflow and CI behavior. |
| experiments/pyproject.toml | Adds isolated uv project config and hatch build configuration for experiment packages. |
| experiments/pier_qualification/tests/test_pier_run.py | Updates Pier test imports to the migrated pier_qualification package. |
| experiments/pier_qualification/tests/test_pier_profiles.py | Updates Pier profile tests to import from the migrated pier_qualification package. |
| experiments/pier_qualification/tests/test_pier_prepare.py | Updates Pier preparation tests to import from the migrated pier_qualification package. |
| experiments/pier_qualification/tests/test_pier_final_rehearsal.py | Updates imports and repo-root resolution for rehearsal after migration. |
| experiments/pier_qualification/tests/test_pier_acceptance.py | Updates imports/repo-root resolution and continues exercising qualify_local.py. |
| experiments/pier_qualification/README.md | Adds the full operational guide for qualification, paid gating, verification, and rehearsal. |
| experiments/pier_qualification/pier_experiment.py | Adds Pier profiles + gating + prepare/run implementation within experiments. |
| experiments/pier_qualification/__init__.py | Adds a package marker for the migrated Pier qualification runner. |
| experiments/consistency_pilot/tests/test_realistic_fixtures.py | Updates tests to import experiment modules from consistency_pilot instead of shipping paths. |
| experiments/consistency_pilot/tests/test_optional_jig.py | Adds a local guard ensuring legacy stack failures remain actionable without optional deps. |
| experiments/consistency_pilot/tests/test_openrouter.py | Updates tests to import from migrated consistency pilot modules. |
| experiments/consistency_pilot/tests/test_dry_experiment.py | Updates tests/patch targets to the migrated dry experiment module. |
| experiments/consistency_pilot/tests/test_consistency_pilot.py | Updates tests/patch targets to the migrated pilot module. |
| experiments/consistency_pilot/realistic-pilot.md | Updates docs/examples to run the new module path from experiments/. |
| experiments/consistency_pilot/realistic_pilot.py | Moves realistic pilot runner to experiments and updates legacy import error path string. |
| experiments/consistency_pilot/pilot.py | Updates legacy import error path string for the migrated pilot module. |
| experiments/consistency_pilot/openrouter-smoke.md | Updates docs/examples to run the new module path from experiments/. |
| experiments/consistency_pilot/openrouter_smoke.py | Updates legacy import error path string for the migrated smoke module. |
| experiments/consistency_pilot/dry-experiment.md | Updates docs/examples to run the new module path from experiments/. |
| experiments/consistency_pilot/dry_experiment.py | Moves dry experiment runner to experiments and updates legacy import error path string. |
| experiments/consistency_pilot/consistency-pilot.md | Updates docs/examples to run the new module path from experiments/. |
| experiments/consistency_pilot/__init__.py | Adds a package marker for the migrated consistency pilot. |
| experiments/conftest.py | Adds shared fixtures and test-path wiring for experiments to reuse core test fixtures. |
| docs/pier-integration.md | Refactors Pier docs to focus on integration + qualification and links to the new experiments runner guide. |
| .github/workflows/ci.yml | Adds an experiments job, adds static bundle drift check, and updates Pier qualification job paths. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+116
to
+117
| - name: Lint experiments with the root Ruff configuration | ||
| run: uv run ruff check experiments |
cirsteve
deleted the
cohort/4297cdf5-ad9d-41b1-8556-03a3ef5f2b94/experiments-tree
branch
September 21, 2026 17:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves all non-shipping consistency, Pier qualification, and TypeSafe relevance runners, tests, catalogues, and operator docs into a standalone
experiments/uv project. Adds isolated CI, framework/result-ownership guards, and frontend generated-bundle drift detection.Dependency parity with the root lock:
5932f7237a369fe705512915ef4033da0ca27c74in both locks1e054d02a36d1da832e8a1c9cf10974131b09f5din both locks1.9.0in both locks0.6.0in both locks0.21.1in both locksValidation:
src/assay/review/staticclean; local browser execution could not launch Chromium because this container denies its sandbox host operationThe offline
run_arms --dry-runrequires the private 79-record population, which is not present withoutASSAY_RUN_RECEIPTS; its CLI/help path and all ungated relevance tests passed.