Skip to content

fix: discover Criterion samples at their real depth, under new/ (#95) - #130

Merged
Xof merged 1 commit into
fix/94-file-size-stridefrom
fix/95-criterion-cell-discovery
Jul 31, 2026
Merged

fix: discover Criterion samples at their real depth, under new/ (#95)#130
Xof merged 1 commit into
fix/94-file-size-stridefrom
fix/95-criterion-cell-discovery

Conversation

@Xof

@Xof Xof commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Closes #95 (BENCH-1, BUG/correctness).

Stacked on #129 (fix/94-file-size-stride). Base retargets to main when that merges.

The defect

discover_cells bounded its walk to exactly depth 4:

// Each leaf is at depth 4 (criterion_dir/row/mode/size/sample.json).
for entry in walkdir::WalkDir::new(criterion_dir).min_depth(4).max_depth(4)

Criterion 0.5 writes <row>/<mode>/<size>/new/sample.json — depth 5. On this
repo's real tree:

$ find target/criterion -maxdepth 4 -name sample.json   # 0 hits
$ find target/criterion -maxdepth 5 -name sample.json   # 2 hits

So the walk matched nothing, and it failed silently: NoCellsFound never
fires because aux_metrics.jsonl still supplies cells via the Step-3 leftover
path. Every micro-grid cell reached the renderers with timing: None, and
every p50/p95/p99 in summary.md rendered as an em-dash.

The one test covering this walked a fixture tree that omits the new/ level,
so the suite passed against a layout Criterion never produces.

The fix

Walk depth 5, and require the leaf's parent directory to be named new.

The new check is the load-bearing half, not the depth change: Criterion
retains the previous run under a sibling base/, which shares the
(row, mode, size) key and would otherwise emit a duplicate Cell.

Tests

The fixture tree gains the new/ level (so it matches what Criterion actually
writes) plus a base/ sibling holding times 10x larger. Two things now fail if
base/ leaks in: an explicit exactly-one-cell count, and the pre-existing
p50 == 3000 assertions (base/ would give 30000).

Counterfactual — reverting the walk to depth 4, all three discover tests fail:

test summary::discover::tests::discover_cells_joins_criterion_with_aux ... FAILED
test summary::discover::tests::discover_cells_handles_missing_aux_gracefully ... FAILED
test summary::discover::tests::discover_cells_handles_malformed_sample_json ... FAILED

copy_raw_archive was always unbounded-depth and already archived the new/
path correctly — only summarize_smoke's expected archive path needed updating
to match the fixture move.

End-to-end verification

Against the real target/criterion, summarize now reports:

smoke/chisel-strict/256B -> p50 84481021.0 p95 89508941.85

Before the fix that cell was emitted with no timing at all.

Verification

cargo test all green, cargo clippy --all-targets -- -D warnings clean,
cargo fmt --check clean.

`discover_cells` bounded its walk to exactly depth 4 with the comment
"Each leaf is at depth 4 (criterion_dir/row/mode/size/sample.json)".
Criterion 0.5 writes `<row>/<mode>/<size>/new/sample.json` — depth 5.
The walk therefore matched nothing: on this repo's real Criterion tree,
`find target/criterion -maxdepth 4 -name sample.json` returns zero hits
and `-maxdepth 5` returns two.

The failure was silent rather than loud. `NoCellsFound` never fired
because aux_metrics.jsonl still supplied cells through the Step-3
leftover path, so every micro-grid cell reached the renderers with
`timing: None` and every p50/p95/p99 in summary.md rendered as an
em-dash. The only test covering this walked a fixture tree that omits
the `new/` level, so the suite passed against a layout Criterion never
produces.

Walks depth 5 and requires the leaf's parent directory to be named
`new`. The `new` check is the load-bearing half: Criterion retains the
previous run under a sibling `base/`, which shares the (row, mode, size)
key and would otherwise emit a duplicate Cell.

The fixture tree gains the `new/` level and a `base/` sibling holding
times 10x larger, so the tests now cover the real layout and the suite
fails if `base/` is ever picked up (asserted both by an exactly-one-cell
count and by the existing p50 == 3000 assertions). `copy_raw_archive`
was always unbounded-depth and already archived the `new/` path
correctly; the smoke test's expected archive path is updated to match.

Verified end to end: against the real `target/criterion`, summarize now
reports `smoke/chisel-strict/256B` with p50 84481021 / p95 89508941,
where before it emitted the cell with no timing at all.

Closes #95.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] discover_cells walks the wrong depth and never finds a real Criterion sample.json — every micro-grid timing cell renders as an em-dash

1 participant