feat(xtest): compare up to four benchmark arms in one run - #621
dmihalcik-virtru wants to merge 2 commits into
Conversation
A branch ref like 'feat/DSPX-2604-createtdf-chunked' resolved by name (not by SHA) kept its slash in the tag, nesting dist/<tag>/ and src/<tag>/ one level deeper than every consumer expects: xtest's all_versions_of() lists dist/*/ and the Go Makefile finds src/*/, so the build was silently discovered as a bare 'feat' directory with no cli.sh in it. Flatten the tag the same way _classify_sha_match already flattens a branch reached by SHA.
Generalizes the paired A/B benchmark from a fixed baseline/candidate pair to 2-4 named arms, so a bake-off between competing implementations of the same feature is one measurement on one runner instead of two separate two-arm dispatches whose ratios share no denominator. - perf/runner.py: Arm/CellResult keyed by arm_ids and an explicit reference, replacing the baseline/candidate pair. The A/A control now runs K copies of the reference (not a cheap pair), since the last arm in a K-arm round carries more within-round drift than an adjacent pair does. Also folds in trunk's independent refinements: per-arm deadline capping so one slow arm can't run past the shared budget, and a strict configured-minimum-rounds check (previously only the statistical hard floor was enforced). - perf/stats.py: every non-reference arm gets its own gated contrast against the reference; every pair of non-reference arms gets a symmetric FASTER/SLOWER/TIED head-to-head that never gates the build. Three BH-corrected families (gated / symmetric / ungated) so a bake-off costs the regression gate no power. Also folds in trunk's has_candidate_comparisons fix, which catches a control-only run that trunk's simpler nothing_measured check would have missed. - perf/report.py, fixtures/bench.py, conftest.py: --bench-refs (with --bench-baseline/--bench-candidate kept as two-arm shorthand), arm resolution and comparability checks generalized to K arms, and bake-off results rendered in the job summary and terminal output. - xtest.yml: bench-refs/bench-budget-seconds/bench-max-rounds inputs, a validate-and-normalize step, and a per-arm resolution step in the bench job. Budget defaults scale by arm-count/2 since a K-arm round costs K invocations, and the job timeout goes to 240m to cover it. Depends on the otdf-sdk-mgr slash-flattening fix: named-ref arms can be arbitrary branches, and an unflattened tag breaks build discovery.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
|
Superseded, not abandoned. This draft does six unrelated things in 3211 lines — fixes a statistical bug, converts two arms to K, adds head-to-head conclusions, rewrites the report, and reimplements benchmark input parsing in workflow shell — and is not reviewable as one unit. Replaced by the six-stage stack described in The |
|
Stack progress: #625 (stage 1, statistical fixes, CI green) and #626 (stage 2, shared local benchmark preparation, stacked on #625) are open. Stages 3–6 — multi-arm measurement, pairwise conclusions, thin CI integration, report presentation — follow as their dependencies land. One consequence worth stating plainly for #622 and #623: their base branch |
1 participant
Split out of #583. Stacked on #620.
Generalizes the paired A/B benchmark from a fixed baseline/candidate pair to 2-4 named arms, so a bake-off between competing implementations of the same feature is one measurement on one runner instead of two separate two-arm dispatches whose ratios share no denominator.
perf/runner.py:Arm/CellResultkeyed byarm_idsand an explicit reference, replacing the baseline/candidate pair. The A/A control now runs K copies of the reference (not a cheap pair), since the last arm in a K-arm round carries more within-round drift than an adjacent pair does.perf/stats.py: every non-reference arm gets its own gated contrast against the reference; every pair of non-reference arms gets a symmetric FASTER/SLOWER/TIED head-to-head that never gates the build. Three BH-corrected families (gated / symmetric / ungated) so a bake-off costs the regression gate no power.perf/report.py,fixtures/bench.py,conftest.py:--bench-refs(with--bench-baseline/--bench-candidatekept as two-arm shorthand), arm resolution and comparability checks generalized to K arms, and bake-off results rendered in the job summary and terminal output.xtest.yml:bench-refs/bench-budget-seconds/bench-max-roundsinputs, a validate-and-normalize step, and a per-arm resolution step in the bench job. Budget defaults scale by arm-count/2 since a K-arm round costs K invocations, and the job timeout goes to 240m to cover it.Depends on #620 (otdf-sdk-mgr slash-flattening): named-ref arms can be arbitrary branches, and an unflattened tag breaks build discovery.