Release v0.3.1#712
Open
joeharris76 wants to merge 2223 commits into
Open
Conversation
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
…#504) Add reference hooks so the plan-deepening framework fires at the generative/planning moments it was built for, not only in reactive review: - code: perf (L3 before optimizing), to-spec (L3 boundary check) - blog: plan + research (L3 thesis reframe); critique/editorial/audit (L2 audit after findings) - todo: ideate/spec (L3), from-spec (L3, conditional), review (L2 after findings) - todo/blog skill.yaml: declare review-protocol + plan-deepening-framework deps (already referenced in body) - framework: L2 routes through review-protocol when reviewing; generative actions apply the question inline (no capture) Skill sources live in ~/.skill-sync/skills (shared); .claude/skills is gitignored, so the only tracked change is skill-sync.lock. Verified via `make skill-sync-lock-audit BASE=origin/develop CHECK=1` (6 files OK). Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Surface/module shrunk: TPC-DS DataFrame query registry metadata wiring in benchbox/core/tpcds/dataframe_queries/queries.py Original line count: 10,885 cloc Python code lines for benchbox/core/tpcds/dataframe_queries/queries.py New line count: 9,624 cloc Python code lines Lines removed: 1,261 cloc code lines from the touched file Reduction percentage: 11.6% for the touched file; current benchbox/ total is 247,438 cloc code lines, down 1,238 from the provided baseline of 248,676 Files changed: - benchbox/core/tpcds/dataframe_queries/queries.py Simplification type: Replaced 99 repeated register_query(DataFrameQuery(...)) constructor blocks with a compact CSV-parsed metadata table, category-code map, and existing qN_expression_impl/qN_pandas_impl name lookup. Query implementations were not changed. Behavior preserved: - Saved pre-change registry metadata for all 99 TPC-DS DataFrame queries, including query id, name, description, categories, implementation callable names, SQL-equivalent fields, expected-row fields, timeout fields, and skip-platform fields. - Regenerated the post-change registry metadata and diffed it against the baseline snapshot with no differences. - Import-time failure behavior remains strict: duplicate query ids still fail in QueryRegistry.register, invalid category codes fail during registration, and missing implementation functions fail during name lookup. Verification commands run: - cloc --by-file benchbox/core/tpcds/dataframe_queries/queries.py - uv run -- ruff format benchbox/core/tpcds/dataframe_queries/queries.py - git diff --check -- benchbox/core/tpcds/dataframe_queries/queries.py - uv run -- ruff check benchbox/core/tpcds/dataframe_queries/queries.py - uv run -- python -m pytest tests/unit/core/tpcds/test_tpcds_dataframe_queries.py -q - uv run -- python <registry metadata snapshot and comparison script>; diff -u /tmp/tpcds_dataframe_registry_before.json /tmp/tpcds_dataframe_registry_after.json - uv run -- python <registry callable/name invariant check> - cloc benchbox/ Residual risk: Low. This changes registration construction only, not query execution logic. The remaining risk is future-maintainer readability of compact category codes; the local _CATEGORY_CODES map and metadata equivalence check mitigate that. Next recommended shrink target: benchbox/core/read_primitives/dataframe_queries.py, which has 4,247 cloc Python code lines and repeated dataframe operation patterns that should be reviewed separately from this PR. Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Surface/module shrunk: Read Primitives DataFrame query registry metadata wiring in benchbox/core/read_primitives/dataframe_queries.py Original line count: 4,247 cloc Python code lines for benchbox/core/read_primitives/dataframe_queries.py New line count: 3,056 cloc Python code lines Lines removed: 1,191 cloc code lines from the touched file Reduction percentage: 28.0% for the touched file; current branch benchbox/ total is 247,508 cloc code lines, down 1,168 from the provided baseline of 248,676 Files changed: - benchbox/core/read_primitives/dataframe_queries.py Simplification type: Replaced 152 repeated DataFrameQuery constructor blocks with a compact CSV-parsed metadata table, category-code map, implementation stem lookup, and query factory. The existing expression and pandas implementations were not changed, and _QUERIES remains a list of DataFrameQuery objects. Behavior preserved: - Saved pre-change registry metadata for all 152 Read Primitives DataFrame queries, including query id, name, description, categories, implementation callable names, SQL equivalents, expected-row fields, timeout fields, and skip-platform fields. - Regenerated post-change registry metadata and diffed it against the baseline snapshot with no differences. - Compared the ordered metadata table against origin/develop's _QUERIES assignment, including implementation names and SQL equivalents, with exact equivalence. - Import-time strictness remains: duplicate query ids still fail in QueryRegistry.register, invalid category codes fail during query construction, and missing implementation functions fail during name lookup. - This PR is independent of pending shrink PR #508 and touches a disjoint file. Verification commands run: - cloc --by-file benchbox/core/read_primitives/dataframe_queries.py - uv run -- ruff format benchbox/core/read_primitives/dataframe_queries.py - git diff --check -- benchbox/core/read_primitives/dataframe_queries.py - uv run -- ruff check benchbox/core/read_primitives/dataframe_queries.py - uv run -- python -m pytest tests/unit/benchmarks/test_read_primitives_dataframe_queries.py -q - uv run -- python -m pytest tests/unit/benchmarks/test_read_primitives_polars_execution.py -q - uv run -- python <registry metadata snapshot and comparison script>; diff -u /tmp/read_primitives_registry_before.json /tmp/read_primitives_registry_after.json - uv run -- python <ordered origin/develop metadata equivalence check> - uv run -- python <registry callable/name invariant check> - cloc benchbox/ Residual risk: Low. The change is limited to registry construction, not execution logic. The main residual risk is maintainability of compact category codes; the local _CATEGORY_CODES map, explicit implementation-stem column for naming exceptions, and ordered equivalence check mitigate that. Next recommended shrink target: benchbox/core/joinorder/dataframe_queries.py or benchbox/core/clickbench/dataframe_queries/queries.py, both of which have repeated DataFrameQuery metadata patterns and do not overlap this PR. Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Surface/module shrunk: Canonical JoinOrder DataFrame registry and generated-query wiring in benchbox/core/joinorder/dataframe_queries.py
Original line count: 2,511 cloc Python code lines for benchbox/core/joinorder/dataframe_queries.py
New line count: 1,235 cloc Python code lines
Lines removed: 1,276 cloc code lines from the touched file
Reduction percentage: 50.8% for the touched file; current branch benchbox/ total is 246,162 cloc code lines, down 2,514 from the provided baseline of 248,676
Files changed:
- benchbox/core/joinorder/dataframe_queries.py
Simplification type: Removed generated-query placeholder functions that were overwritten at import time by the existing generated implementation loop, then replaced repeated DataFrameQuery constructor blocks with compact query-id and hand-query metadata tables plus a registry factory. Hand-written query implementations and the generated SQL-to-DataFrame translator were not changed.
Behavior preserved:
- Saved pre-change registry metadata for all 113 JoinOrder DataFrame queries, including query id, name, description, categories, callable names, SQL-equivalent fields, expected-row fields, timeout fields, and skip-platform fields.
- Regenerated post-change registry metadata and diffed it against the baseline snapshot with no differences.
- Saved and compared _QUERIES order before and after with no differences.
- Confirmed every registry callable still resolves to q{query_id}_expression_impl and q{query_id}_pandas_impl.
- Generated-query descriptions still end in generated DataFrame translation, matching the pre-change post-registration mutation.
- This PR is independent of pending shrink PRs #508 and #510 and touches a disjoint file.
Verification commands run:
- cloc --by-file benchbox/core/joinorder/dataframe_queries.py
- uv run -- ruff format benchbox/core/joinorder/dataframe_queries.py
- uv run -- ruff check benchbox/core/joinorder/dataframe_queries.py
- git diff --check -- benchbox/core/joinorder/dataframe_queries.py
- uv run -- python -m pytest tests/unit/core/joinorder/test_dataframe_query_capabilities.py -q
- uv run -- python -m pytest tests/unit/core/joinorder/test_query_coverage.py -q
- uv run -- python <registry metadata snapshot and comparison script>; diff -u /tmp/joinorder_dataframe_registry_before.json /tmp/joinorder_dataframe_registry_after.json
- diff -u /tmp/joinorder_dataframe_query_order_before.txt /tmp/joinorder_dataframe_query_order_after.txt
- uv run -- python <registry callable/name invariant check>
- cloc benchbox/
Residual risk: Low. The main removed code was unreachable after import because generated functions overwrite the same globals. Remaining risk is compact metadata readability; the query-id table is local, hand-query metadata stays explicit, and metadata/order equivalence checks cover drift.
Next recommended shrink target: benchbox/core/clickbench/dataframe_queries/queries.py or benchbox/core/datavault/dataframe_queries/queries.py, both with repeated DataFrameQuery registration metadata and disjoint ownership from the current shrink PRs.
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Surface/module shrunk: ClickBench DataFrame query registry. Files changed: benchbox/core/clickbench/dataframe_queries/queries.py. Shrink ledger: - Original code lines: 1,260 - New code lines: 849 - Lines removed: 411 - Reduction: 32.6% for the changed file - Simplification type: replaced repeated DataFrameQuery registration boilerplate with a compact metadata table and constructor loop; query implementations are unchanged. Behavior preserved: - Public ClickBench DataFrame imports and registry helpers are unchanged. - All 43 query IDs remain registered with identical names, descriptions, categories, implementation function names, insertion order, and category counts. - Expression and pandas query implementation functions were not modified. Verification commands run: - diff -u /tmp/clickbench_registry_before.json /tmp/clickbench_registry_after.json - uv run -- ruff format benchbox/core/clickbench/dataframe_queries/queries.py - uv run -- ruff check benchbox/core/clickbench/dataframe_queries/queries.py - git diff --check -- benchbox/core/clickbench/dataframe_queries/queries.py - uv run -- python -m pytest tests/unit/core/clickbench/test_clickbench_dataframe_queries.py -q (57 passed) - custom registry invariant check (ok 43) - make pr-preflight (22710 passed, 5 skipped, 47 warnings, 4 subtests passed in 99.87s) Residual risk: low; future edits to the compact metadata table require the same registry snapshot/invariant checks to avoid typo drift. Next recommended shrink target: Data Vault, NYC Taxi, or FlightData DataFrame registry boilerplate, avoiding JoinOrder while PR #511 is open. Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Surface/module shrunk: Data Vault DataFrame query registry. Files changed: benchbox/core/datavault/dataframe_queries/queries.py. Shrink ledger: - Original code lines: 1,353 - New code lines: 1,191 - Lines removed: 162 - Reduction: 12.0% for the changed file - Simplification type: replaced repeated DataFrameQuery registry entries with a compact metadata table and constructor loop, preserving optional expected row counts. Behavior preserved: - Public Data Vault DataFrame imports and registry helpers are unchanged. - All 22 query IDs remain registered with identical names, descriptions, categories, expected row counts, implementation function names, insertion order, and category counts. - Expression and pandas query implementation functions were not modified. Verification commands run: - diff -u /tmp/datavault_registry_before.json /tmp/datavault_registry_after.json - uv run -- ruff format benchbox/core/datavault/dataframe_queries/queries.py - uv run -- ruff check benchbox/core/datavault/dataframe_queries/queries.py - git diff --check -- benchbox/core/datavault/dataframe_queries/queries.py - uv run -- python -m pytest tests/unit/core/datavault/test_datavault_dataframe_queries.py -q (64 passed) - custom registry invariant check (ok 22) - make pr-preflight (22710 passed, 5 skipped, 47 warnings, 4 subtests passed in 96.16s) Residual risk: low; future edits to the compact metadata table require snapshot/invariant checks so expected-row-count fields do not drift. Next recommended shrink target: NYC Taxi or FlightData DataFrame registry boilerplate; ClickBench PR #512 is still open, so avoid overlapping that file. Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Surface/module shrunk: NYC Taxi DataFrame query registry. Files changed: benchbox/core/nyctaxi/dataframe_queries/queries.py. Shrink ledger: - Original code lines: 1,269 - New code lines: 984 - Lines removed: 285 - Reduction: 22.5% for the changed file - Simplification type: replaced repeated DataFrameQuery registration boilerplate with a compact metadata table and constructor loop. Behavior preserved: - Public NYC Taxi DataFrame imports and registry helpers are unchanged. - All 25 query IDs remain registered with identical names, descriptions, categories, implementation function names, insertion order, and category counts. - Expression and pandas query implementation functions were not modified. Verification commands run: - diff -u /tmp/nyctaxi_registry_before.json /tmp/nyctaxi_registry_after.json - uv run -- ruff format benchbox/core/nyctaxi/dataframe_queries/queries.py - uv run -- ruff check benchbox/core/nyctaxi/dataframe_queries/queries.py - git diff --check -- benchbox/core/nyctaxi/dataframe_queries/queries.py - uv run -- python -m pytest tests/unit/core/nyctaxi/test_nyctaxi_dataframe_queries.py -q (74 passed) - custom registry invariant check (ok 25) - make pr-preflight (22710 passed, 5 skipped, 47 warnings, 4 subtests passed in 95.65s) Residual risk: low; future edits to the compact metadata table require the same registry snapshot/invariant checks to avoid typo drift. Next recommended shrink target: FlightData DataFrame registry boilerplate, or wait for ClickBench/Data Vault PRs to merge before touching related registry surfaces. Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Surface/module shrunk: FlightData DataFrame query registry. Files changed: benchbox/core/flightdata/dataframe_queries.py. Shrink ledger: - Original code lines: 1,321 - New code lines: 1,181 - Lines removed: 140 - Reduction: 10.6% for the changed file - Simplification type: preserved the _QUERIES list while replacing repeated DataFrameQuery constructor literals with a compact metadata table and constructor helper. Behavior preserved: - Public get_dataframe_queries(), parameter override helpers, and registry benchmark name are unchanged. - All 20 query IDs remain registered with identical names, descriptions, categories, implementation function names, insertion order, and category counts. - _QUERIES remains a list of DataFrameQuery objects. - Expression and pandas query implementation functions were not modified. Verification commands run: - diff -u /tmp/flightdata_registry_before.json /tmp/flightdata_registry_after.json - uv run -- ruff format benchbox/core/flightdata/dataframe_queries.py - uv run -- ruff check benchbox/core/flightdata/dataframe_queries.py - git diff --check -- benchbox/core/flightdata/dataframe_queries.py - uv run -- python -m pytest tests/unit/core/flightdata/test_dataframe_queries.py -q (33 passed) - custom registry invariant check (ok 20) - make pr-preflight (22710 passed, 5 skipped, 47 warnings, 4 subtests passed in 96.58s) Residual risk: low; future edits to the compact metadata table require the same registry snapshot/invariant checks to avoid typo drift. Next recommended shrink target: SSB, TPCH skew, or smaller dataframe registry modules once open shrink PRs have merged. Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Surface/module shrunk: SSB DataFrame query registry. Files changed: benchbox/core/ssb/dataframe_queries/queries.py. Original code lines: 777. New code lines: 672. Lines removed: 105. Reduction: 13.5%. Simplification type: replaced repeated DataFrameQuery registration boilerplate with compact metadata rows and one constructor loop. Behavior preserved: all 13 query IDs remain registered with identical names, descriptions, categories, implementation function names, order, and category counts; expression and pandas implementations are unchanged. Verification: diff -u /tmp/ssb_registry_before.json /tmp/ssb_registry_after.json; uv run -- ruff format benchbox/core/ssb/dataframe_queries/queries.py; uv run -- ruff check benchbox/core/ssb/dataframe_queries/queries.py; git diff --check -- benchbox/core/ssb/dataframe_queries/queries.py; custom registry invariant returned ok 13; uv run -- python -m pytest tests/unit/core/ssb/test_ssb_dataframe_queries.py -q (60 passed); make pr-preflight log ended with 22710 passed, 5 skipped, 47 warnings, 4 subtests passed in 98.36s. Residual risk: low; future metadata edits should keep snapshot and invariant checks in the review path. Next recommended shrink target: TPCH skew DataFrame registry or another non-overlapping registry module after pending shrink PRs merge. Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Surface/module shrunk: TPC-H Skew DataFrame query registry. Files changed: benchbox/core/tpch_skew/dataframe_queries/queries.py. Original code lines: 21. New code lines: 10. Lines removed: 11. Reduction: 52.4%. Simplification type: replaced repeated dataclass field copy with dataclasses.replace while retaining the DataFrameQuery import that preserves the existing import-order behavior. Behavior preserved: the TPCH skew registry still creates separate query objects, keeps all 22 query IDs and fields identical to baseline, preserves shared expression/pandas implementation references, and preserves categories/skip_platform list references. Verification: diff -u /tmp/tpch_skew_registry_before.json /tmp/tpch_skew_registry_after.json; uv run -- ruff format benchbox/core/tpch_skew/dataframe_queries/queries.py; uv run -- ruff check benchbox/core/tpch_skew/dataframe_queries/queries.py; git diff --check -- benchbox/core/tpch_skew/dataframe_queries/queries.py; uv run -- python -m pytest tests/unit/core/tpch_skew/test_tpch_skew_dataframe_queries.py -q (40 passed); make pr-preflight (22707 passed, 5 skipped, 47 warnings, 4 subtests passed in 98.93s). Residual risk: low; the preserved DataFrameQuery import is intentional because removing it changed fresh-interpreter import behavior during review. Next recommended shrink target: a larger non-overlapping DataFrame query registry or adapter boilerplate cluster after pending shrink PRs merge. Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
* fix(uat): capture diagnostics when --quiet hides cell failures UAT cells run `benchbox run --quiet` so the final stdout line is the result-JSON path parsed by the runner. But --quiet also suppresses benchbox's own error output, so a non-zero exit left the per-cell log empty and every distinct failure collapsed to the same opaque signature (no_json_nonzero / submit=missing_manifest / failure_tail=""). When a cell exits non-zero with no captured output, re-run it once verbosely (quiet=False, stderr merged into stdout, capped at 180s) and append the transcript as plain lines so `_cell_log_tail` surfaces it into `failure_tail`. Add a `quiet` parameter to `benchbox_run_argv` to build the verbose argv; the default stays True to preserve result-path parsing and existing argv ordering. This unblocks root-causing the fast-platform sweep failures (clickhouse-server, scattered duckdb/datafusion/lakesail cells) by making the next run self-describing instead of opaque. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(uat): wire the verbose diagnostic re-run into run_cell Completes the diagnostics fix. The prior commit added the `quiet` parameter to `benchbox_run_argv` but nothing invoked it, leaving it inert. This wires it up in `run_cell`: when a `--quiet` cell exits non-zero with no captured output (and did not time out), re-run it once verbosely via `_append_diagnostic_rerun` (stderr merged into stdout, capped at DIAGNOSTIC_RERUN_TIMEOUT_S=180s) and append the transcript as plain lines so `_cell_log_tail` surfaces the real error into `failure_tail`. Best-effort try/except so diagnostics never mask the original failure. Verified: tests/uat/test_runner.py (24) and test_matrix.py (185) pass; ruff format/check and ty clean; py_compile clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
… mechanics (#723) The explorer-smoke release-gate phase ships on main but shells out to _project/scripts/explorer_publish.py and results-explorer/, both of which live only on develop. On a clean main checkout the phase was structurally unable to pass — a false gate. (uat-explorer-smoke-release-hygiene) - Always run the cheap, no-Node corpus contract over packaged bundles as the minimal always-on gate. - Guard the heavy build+Playwright path on explorer_present(); skip-with-reason (never hard-fail) when the explorer assets are absent on the branch. - Stop owning npm/build/playwright: delegate to a single Results Explorer `uat-external-corpus-smoke` npm script, forwarding browser --project flags via npm `--` passthrough. Add a drift-guard test pinning the script to the @uat-external-corpus tag. - docs.yml: document the explorer steps' main-vs-develop dormancy as intentional. - Bundle the UAT charter pointer (tests/uat/CHARTER.md) so the gate and its develop-only charter co-locate on the release branch. (uat-charter-visibility-on-main) Completes uat-explorer-smoke-release-hygiene and uat-charter-visibility-on-main. Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…UAT (#724) UAT forked the submit-classification policy: tests/uat/runner.py re-expressed the result -> terminal-state decision tree that benchbox submit owns, and kept its own copy of the refused-compliance frozenset. The two surfaces could drift, so UAT could silently report a different submittability verdict than `benchbox submit` enforces. (uat-shared-submit-classifier) - New benchbox/core/results/submit_classification.py owns the policy as a pure, side-effect-free function and imports the canonical CLI_REFUSED_COMPLIANCE_CLASSES from benchbox/validation/bundle.py rather than re-listing literals. SubmitTerminalState lives here now. - submit.py consumes classify_loaded_result; CLI exit codes and refusal messages are byte-identical per state. - runner.py: classify_for_submit is a thin adapter over classify_result_path; the local frozenset and SubmitTerminalState class are deleted (re-exported for existing UAT consumers). Canonical status-predicate imports are unchanged. - Contract test drives a fixture matrix (clean/query_failure/schema_violation/ unofficial/missing/malformed) through the shared policy, the UAT adapter, and the real submit CLI, asserting the verdicts cannot diverge. Completes uat-shared-submit-classifier. Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pr-followup): PR #718 comment 3330314459 — **<sub><sub> Path: scripts/generate_changelog_entry.py * fix(pr-followup): PR #722 comment 3331139468 — **<sub><sub> Path: tests/uat/runner.py --------- Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
…oding ports (#726) matrix.py kept its own PLATFORM_PORTS / PLATFORM_EXTRA_OPTS / LOCAL_MANAGED_PLATFORM_EXTRA_OPTS tables, duplicating connection truth that docker-compose already owns, while docker_assets kept a second registry. The SingleStore case proved the hazard: UAT hardcoded host 13306 while compose maps ${SINGLESTORE_HOST_PORT:-13306}->3306, so an env remap silently made UAT probe the wrong host port — a phantom skip indistinguishable from a down platform. (uat-single-connection-registry) - docker_assets.py is now the single connection registry. Host reachability ports are DERIVED from each platform's compose `ports:` mapping, matched by the in-container service port (PLATFORM_SERVICE_PORT) so sidecar ports (databend MinIO, questdb REST) don't masquerade as the endpoint. Endpoint roles stay explicit: host port vs service/container port. - matrix.py no longer declares any connection table; it consumes host_reachability_endpoint / platform_extra_opts / local_managed_platform_extra_opts. platform_is_reachable re-resolves per probe so SINGLESTORE_HOST_PORT flows through to both the probe and the adapter `port=` option. All 16 derived host ports are byte-identical to the old table. - Drift test parses compose `ports:` and asserts the resolved facts match, and that no platform loses its probe (the must_preserve 'no new skips'). Scope: ports + registry collapse. Credential/secondary-port literals (password=, http_port=) are relocated into the single registry but not yet derived — tracked as a deferred follow-up in the TODO. Completes uat-single-connection-registry. Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The connection-registry refactor (prior commit) lands the work this TODO describes: matrix.py now sources reachability host:port and platform options from tests.uat.docker_assets (compose-derived) instead of hardcoded tables. Flip status to Completed and stamp completed_date. Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…shards (#728) tests/uat/configs/ mixed 17 generated rerun shards (the 2026-05-05 tuned follow-up sweep, one aggregate + one per platform) in with reusable templates — and the shards even carried a `# TEMPLATE` header, so they masqueraded as editable starting points. (uat-resume-shard-config-sprawl) - Document a four-class config lifecycle in docs/operations/uat-framework.md: editable template / historical evidence / generated rerun shard / ephemeral resume state (resume.json), making clear a "resume config" is not a reusable class. - Relocate the 17 shards to tests/uat/configs/generated-rerun-shards/ (out of the reusable top level; retained as evidence, not deleted) and re-header them as generated/frozen-historical. Add a README explaining the class. - No code references the shards; full UAT suite + preflight green. Completes uat-resume-shard-config-sprawl. Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ication" (Path A) (#729) The approving spec never charters "certification" (zero "certif" hits); the term lives only in develop-only docs/config/function names. Decision (Path A): the chartered name is "release-gate orchestration"; do not charter a standing cert-operations surface beyond the named evidence artifacts. (uat-certification-wording-charter-decision) - Record the decision authoritatively in _project/specs/uat-framework.md §10. - Annotate the In-Progress uat-certification-rerun-ordering-and-gate (which owns the certification-named configs, report.py:certification_ordering_violations, and tests/uat/test_config.py) with the rename mandate. The mechanical rename is DELEGATED to that work rather than executed here: the cert surface lives in tests/uat/*.py, which this item's scope_limit forbids, and a standalone rename would break test_config.py and collide with active work. So the literal zero-"certif"-in-docs/configs verification is satisfied when that rename lands; this PR settles and records the charter. Path B (charter the term) is recorded as deferred. Completes uat-certification-wording-charter-decision (decision level). Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
…ot standalone (#739) The release-gate Docker re-runs (uat-docker-stack-recovery w3/w4) are executed only inside the four-stage release-gate sweep (uat-certification-rerun-ordering-and-gate w5), never as a standalone Docker-only rerun -- doing so would reproduce the native-before-docker ordering contamination w5's anti-pattern forbids. The ready queue surfaced w3/w4 as independently actionable, which misdirected a session into starting a Docker-only path. Front-load the guard into the work-unit summaries (shown by todo ready/next/show) and reconcile the reciprocal pointer in cert-rerun w5. Schema allows only pending/in_progress/done, so status stays pending; the summary carries the NOT-standalone marker. Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Joe Harris <57046+joeharris76@users.noreply.github.com>
3 tasks
github-actions Bot
pushed a commit
that referenced
this pull request
Jun 4, 2026
…es (#750) The release-cut curation was removing _benchbox_pytest_xdist_safety.py, but pytest.ini and pytest-ci.ini both load it with -p, causing an ImportError that prevented test collection on release branches. Move the plugin to the main/release tree in the A3 doc and remove it from the Makefile curation list so future release-cut runs no longer drop it. Addresses review comment on PR #712. https://claude.ai/code/session_01QooVqLTXzUJj529YMh7uJb Co-authored-by: Claude <noreply@anthropic.com>
joeharris76
added a commit
that referenced
this pull request
Jun 8, 2026
…es (#750) The release-cut curation was removing _benchbox_pytest_xdist_safety.py, but pytest.ini and pytest-ci.ini both load it with -p, causing an ImportError that prevented test collection on release branches. Move the plugin to the main/release tree in the A3 doc and remove it from the Makefile curation list so future release-cut runs no longer drop it. Addresses review comment on PR #712. https://claude.ai/code/session_01QooVqLTXzUJj529YMh7uJb Co-authored-by: Claude <noreply@anthropic.com>
17 tasks
joeharris76
added a commit
that referenced
this pull request
Jul 3, 2026
…s (sweep 9) (#941) * fix: address late-landing bot review findings across pass-9 merged PRs (sweep 9) Fixes issues raised by chatgpt-codex-connector after merge on #933, #936, #934, #931, #924, #923, #930, #926, #921, #922, #925, #927: - query_plan_models literal normalization: correct sign/escaped-quote handling so seed-varied numeric/string literals collapse to the same fingerprint (#933). - ruleset_review_enforcement: drop required_approving_review_count check (branch-wide, not CODEOWNERS-scoped) and reword the runbook/TODO to match (#934, #925-related admin-enforcement TODO). - write_primitives: guard against false-success when an optional staging table is empty because its TPC-H source table is absent from a minimal fixture (#931). - Plan-capture key propagation: thread _plan_capture_key and sibling fields through every TPC-H/TPC-DS power+throughput hop so captured plans attach correctly across combined power/throughput runs (#924). - write_primitives README: correct DataFusion's stated skip count from 64 to 62 — two BULK_LOAD ops carry a vestigial platform_override that is never consulted at runtime (#923). - adopt-apple-container-ci-parity TODO: fix w1/w2 ordering (machine creation needs the built image) and flag the shared-home-mount venv clobber risk for uv sync (#930). - track2-joinorder-stats-as-phase design doc + scaling-strategy TODO: correct the Redshift ANALYZE-during-load misattribution and gate scaled prototype implementation on the stats-phase TODO (#926). - joinorder-scale-stress decision: carve lookup tables out of the SF=N replication multiplier (single-copy, matching the prototype's must_preserve) and time DuckDB's explicit ANALYZE per the decision's own rule instead of zeroing it (#921). - apply-changes-into-vs-standard-sql blog draft: stop claiming the SCD2 SQL runs unchanged across every BenchBox-target engine — DataFusion is a documented platform_overrides exception (#922). - dataframe-csv-empty-string-parity-coverage TODO: reword the "uniform" contract to be explicit that it's null-marker-conditioned, not a single always-null outcome (#925 finding 2). - cloud-claude-setup.sh + cloud-ownership decision doc + TODO: document the environment-build caching gap (setup script doesn't re-run per-session) and the manual rebuild remedy, since project hooks are unreliable in cloud web execution (#927). Continuing to ignore #712 per standing instruction. Co-Authored-By: Claude <noreply@anthropic.com> * fix(lint): apply ruff format to write_primitives benchmark files CI lint job was failing on unformatted code in benchmark.py and its test file; run ruff format to fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
17 tasks
joeharris76
added a commit
that referenced
this pull request
Jul 4, 2026
…Rs (sweep 10) (#955) Fixes issues raised by chatgpt-codex-connector after merge on #940, #943, #946, #950, #952, #954: - joinorder-canonical-validation-test-coverage TODO: replace a --collect-only verification command (never executes tests, so a missing opt-in marker would still "pass") with a plain pytest run asserting the new full-archive tests are deselected, not merely collected (#940). - joinorder-scale-stress decision framework: align joinorder-replicated-imdb-scale-prototype's w4 validation notes with the corrected oracle description (unchanged canonical final-result oracle; underlying/join-subgraph cardinalities scale by replica count, not final query-result cardinalities scaling N times) (#943). - validation.py: restrict the JoinOrder row-count-strategy branch to an exact `benchmark_name == "joinorderbenchmark"` match instead of a substring check, so JoinOrderSyntheticBenchmark (whose lookup tables stay fixed-size regardless of scale factor) no longer gets incorrectly validated against the canonical exact-count strategy (#946). - coverage-blindspot-burndown / pr-postgres-sample-antiskip-guard DONE TODOs: fix a grep verification command that exits non-zero on the desired no-match outcome, and replace a verification step that told the implementer to rename a file the same TODO's scope_limit forbids modifying (#950). - .github/workflows/pr.yml: expose the classifier's `packaging-needed` / `viz-needed` step outputs as job-level outputs on `ci-paths` so `package-smoke`, `dependency-audit`, and the viz job actually gate on them instead of silently always-skipping; run `pip-audit` via `uv run --with` against the project's synced environment instead of `uvx`'s isolated tool environment, so it audits BenchBox's own dependency graph (#952). - Spark AQE deep-dive outline: correct the overstated claim that BenchBox's plan capture can show AQE's actual switched join operators and coalesced partition counts -- it's a post-measurement `EXPLAIN EXTENDED`, not the executed/adaptive plan, so it can only show static pre-adaptive plan diffs; added this caveat to the deliverable framing and Limitations (#954). Continuing to ignore #712 per standing instruction. Co-authored-by: Claude <noreply@anthropic.com>
16 tasks
joeharris76
added a commit
that referenced
this pull request
Jul 5, 2026
…Rs (sweep 12) (#969) * fix: address late-landing bot review findings across pass-12 merged PRs (sweep 12) Fixes issues raised by chatgpt-codex-connector after merge on #959, #964, #960, #963: - remediate-submission-trust-label-enforcement TODO: bring scripts/validate_submission.py into scope_limit and note that the new missing-sidecar hard error must be scoped to the submission-corpus CI path, not validate_bundles() unconditionally -- benchbox submit --dry-run calls the same shared validator against a source bundle before its sidecar exists, so a blanket rule would break local dry-runs (#959). - remediate-published-results-submission-ci TODO: the develop-side PR that implements w1-w4 does not itself update origin/published-results -- sync-results-data-to-published.yml only opens a draft mirror PR that a maintainer must review and merge. Reworded the verification section to check the develop-side wiring first and moved the published-results branch check to an explicit new post-merge work item (w5) (#959). - results-explorer-qa.md: corrected the stat-card counts (11 results, 2 benchmarks, 7 platforms, not the stale 12/2/4), replaced a sqlite-based empty-filter example and "test all four platforms" instruction (there is no sqlite bundle in this corpus), and replaced the stale source-bundle- filename result IDs with the actual regenerated result_ids read from a fresh fixture build -- all reconfirmed by actually running `npm run test:e2e:fixtures` and querying the generated results.duckdb (#964). - test_pr_workflow_path_classifier.py: the ci-paths output lockstep regression test's regex only matched `.outputs` (dot form), missing GitHub Actions' fully bracketed `needs['ci-paths']['outputs']['x']` index syntax -- an undeclared output referenced only that way could still silently bypass the guard. Extended the regex to cover all dot/bracket combinations across every segment and added a dedicated regression test exercising the bracketed form directly, confirmed to fail against the old regex (#960). - phase3-promotion-review.yml: added a PHASE3_REVIEW_REF=develop env var and used it in the review job's checkout, mirroring release-canary.yml's RELEASE_CANARY_REF pattern -- without it, landing this workflow on `main` (required for its schedule to ever fire) would make a scheduled run check out main's stripped tree (no _project/ or phase2_metrics.py) by default instead of develop's. Documented the fix in the admin runbook (#963). Continuing to ignore #712 per standing instruction. Co-Authored-By: Claude <noreply@anthropic.com> * fix: apply ruff format to pass-12 sweep test file CI caught a formatting drift in the regex constant introduced in the previous commit; ruff format wraps the long concatenated string literal. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v
This PR was opened by
make release-cut VERSION=X.Y.Zfromdevelop.It cuts the release branch, bumps the version sources, generates the
CHANGELOG entry from
origin/main..HEAD, drops maintainer paths (_project/,_blog/, agent configs, dev-tooling root files), and queues the change for asquash-merge into
main.Reviewer checklist
## [X.Y.Z] - YYYY-MM-DDentrypyproject.tomlandbenchbox/__init__.pyshow the new versionlanding/show the new version_project/,_blog/, agent config dirs, and dev-tooling root fileswere removed from this branch (verify with
git diff main...HEAD --stat— the diff should look like a curated subtree, not arbitrary edits)
validate-baseis green for this release branchrelease-required-resultis green on this branchrelease-canary.ymlrun issuccessful, within 48h, and its checked
developSHA is an ancestor ofthis release head)
Release-required guarantee
release-required-resultmeans this branch passed the fast test lane,credential-free integration-not-slow suite, isolated exact-one-wheel package
install smoke, dependency upper-bound check, and release-branch curation check.
It does not cover live cloud credentials, stress suites, or long-running UAT.
Slow/resource-heavy coverage is enforced by the 48h release canary freshness
gate in
validate-base, not by rerunning that suite on every release PR.release-canary.ymlruns the credential-free non-fast canary(slow or resource_heavy) and not (stress or live_integration)and theruleset drift check against
docs/operations/repo-admin-settings.md.Scheduled runs execute from the default branch, then check out
developandrecord the checked SHA in
release-canary-summary.jsonfor release-readiness.The first release that introduces the workflow may run the same evidence inline
from
validate-baseuntil GitHub can run the canary from the default branch.Emergency override requires repository variables
RELEASE_READINESS_OVERRIDE_SHA(exact release head SHA) andRELEASE_READINESS_OVERRIDE_REASON(incident/approval record).After release-required checks are green
release-finalizere-checks that the requiredvalidate-baseandrelease-required-resultcontexts are present and green before merge. If eithercontext is missing, pending, skipped, failed, or canceled, stop and fix the
release PR or ruleset/workflow contract before rerunning the command.
After that pre-merge check passes,
release-finalizesquash-merges this PR,fast-forwards
main, tagsvX.Y.Z, and pushes the tag — which fires.github/workflows/release.yml:dependency-bounds→build(withSOURCE_DATE_EPOCHfrom the tag commit) →publish(PyPI trustedpublisher) →
github-release→test-installation(cross-platform pipinstall verification).
Push-to-main jobs are post-merge signals. They can fail after the tag has
already started publication; recovery is a patch release or incident process,
not treating the public tag as if it had been blocked.
developis intentionally NOT modified byrelease-finalize. Dev-onlypaths (
_project/,_blog/, agent configs, etc.) live only on developby design (per A3 in
_project/decisions/single-repo-migration.md).If anything fails downstream, fix on a new branch, PR to
main, squash-merge,and bump to the next patch version (PyPI rejects re-uploads of an existing
version).
See
docs/operations/release-guide.mdfor the full flow.