diff --git a/benchmarks/gc_ratchet/README.md b/benchmarks/gc_ratchet/README.md index fa0a231aa1..909fb2130d 100644 --- a/benchmarks/gc_ratchet/README.md +++ b/benchmarks/gc_ratchet/README.md @@ -71,13 +71,14 @@ sessions x 7 repeats (21 runs per probe), plus 5 traced runs per probe: The GC accounting family is parsed from `PERRY_GC_DIAG=1` output in a separate, untimed pass; enabling the trace was verified not to change `heap_used_bytes`, so the traced pass observes the same collector the untimed pass measures. The -harness takes two traced runs on every invocation and fails if they disagree — -that is the harness proving, each time it runs, that the counters it is about to -gate on really are deterministic. +harness records two traced runs on every invocation. `check` fails if a gated +counter disagrees, even within its tolerance band. Documented probe overrides +apply here too; the measurement keeps both samples and runs every later probe. -Retention and GC accounting are semantic: they are a function of the allocation -sequence and collector policy, not of CPU speed, core count, or machine load. -That is why they can be gated on a shared CI runner and memory and time cannot. +Retention and GC accounting usually transfer across machine classes because +they describe allocation and collector work rather than CPU speed. They must +still demonstrate repeatability: block placement can change the collection +point and therefore the live cohort, as the #9790 investigation below shows. ## A probe may declare the collector it is a probe *of* (the large-Eden arm) @@ -332,10 +333,63 @@ carrying a non-deterministic gating cell cannot be *pinned*. Before #7554 the rule existed only in `tests/test_gc_ratchet.py`, which is why a bad pin could be committed and only wedge CI afterwards. -The section is currently **empty**, which is the goal state and not an -oversight. Its one entry — `12_large_live_set.heap_used_bytes` — was deleted by -#7558, which removed the *cause* rather than the cell. That is rule 4 working -as designed. +The former `12_large_live_set.heap_used_bytes` entry was deleted by #7558, +which removed its cause. The two current entries exclude only +`07_array_grow_evacuate.copied_bytes` and `.freed_bytes`, with the following +evidence. Removing the placement dependency would allow deleting these entries. + +### Array growth: placement changes the collection point (#9790) + +Twenty-one executions of one unchanged binary produced two counter tuples. +Every stdout matched Node 26.5.1. Retained heap, arena capacity, minor/step +counts, copied/promoted object counts, and promoted bytes were identical. +The [receipt](evidence/9790-array-growth-pacing.json) records raw byte-counter +samples, stable metrics, and compiler/runtime/probe hashes. + +Temporary logging in `move_young` cross-checked the counters against the actual +headers moved. Only the fifth minor's live cohort differed: an 8,208-byte array +with length 640, capacity 1,024, and seed 5,207 was reached through the ring's +remembered edge in one run. The other run instead copied a 144-byte array with +length 1, capacity 16, and seed 5,240 from the native stack. The difference is +exactly **8,064 bytes**, while both runs copy one object. Summing the other +copied headers gives the same result in both runs. + +The freed-byte difference also balances against the actual from-space usage: + +| Fifth minor | Higher copied bytes | Lower copied bytes | +|---|---:|---:| +| Eden bytes | 16,775,936 | 16,776,080 | +| Active survivor bytes | 517,248 | 517,248 | +| Copied bytes | 525,312 | 517,248 | +| Promoted bytes | 131,328 | 131,328 | +| Freed bytes | 16,636,544 | 16,644,752 | + +In each column, freed = Eden + active survivor - copied - promoted; malloc +reclamation is zero. There is no unexplained accounting remainder. + +Block-boundary logging located the pacing cause. `arena_cell_alloc` checks GC +pressure when its current block cannot satisfy an allocation. Promotion walks +address-keyed root tables, so equal total promoted bytes can fill individual +old blocks differently. In the higher-copy run, an old block overflowed on a +4,112-byte growth request after young occupancy had crossed the cap. In the +lower-copy run, that old-block rollover occurred earlier, below the cap; the +next nursery block overflow armed collection while starting the next array. +`js_array_grow` can fall back to old allocation when a growth cannot fit the +current nursery block, connecting this workload to that old-block geometry. +The earlier dirty-page statistics also vary with placement; they alone would +not have established the cause. + +The two byte counters therefore describe real, placement-dependent work on +this workload. They remain measured and displayed, with their existing bands; +only their ability to fail the gate is excluded. The probe's correctness, +retention, cycle counts, copied/promoted object counts, and promoted bytes +remain gated, as do these byte counters on every other probe. Runtime pacing +and the probe's allocation sequence are unchanged. + +The determinism check now runs in `check`, where the baseline's reviewed +overrides are available. An unlisted disagreement still fails even if its +median equals the baseline, and the full measurement artifact survives for +inspection. `assemble` still refuses to pin any nondeterministic gated cell. ### What that probe's non-determinism was, and where it went (#7558) diff --git a/benchmarks/gc_ratchet/baseline/gc-ratchet-v1.json b/benchmarks/gc_ratchet/baseline/gc-ratchet-v1.json index eb8998a45a..3d6db44e37 100644 --- a/benchmarks/gc_ratchet/baseline/gc-ratchet-v1.json +++ b/benchmarks/gc_ratchet/baseline/gc-ratchet-v1.json @@ -103,12 +103,12 @@ "job red. Every entry carries evidence that is checked, not merely stored --", "at least 21 runs (the same number every band above is justified by) and a", "spread that is actually non-zero, so a cell cannot be excluded on a hunch.", - "The section is EMPTY, and that is the goal state. Its one entry --", - "12_large_live_set.heap_used_bytes, added by #7554 -- was deleted by #7558,", - "which removed the cause rather than the cell: explicit gc() no longer forces", - "the conservative native-stack scan, so that reading is bit-identical again", - "and gates again. An empty section is not a disarmed rule; the evidence", - "checks and the never-gate-nothing rule still fail any entry added back.", + "The former 12_large_live_set.heap_used_bytes exclusion was removed by #7558.", + "#9790 excludes only 07_array_grow_evacuate.copied_bytes and freed_bytes:", + "promotion order changes block packing and when nursery pressure is checked.", + "The receipt records 21 runs and an allocation-level accounting cross-check.", + "Both cells remain measured and reported. All other cells retain their bands.", + "Deleting the placement dependency means deleting these exclusions.", "", "#7559 -- A heap_used_bytes band used to NOT be a statement about how much", "the collector retained. The reading is taken after the probe's own gc(),", @@ -295,7 +295,30 @@ "rationale": "GATED HERE ONLY. Worst cross-session spread of medians-of-7 was 0.751% on an idle box (load 1.7-2.0); worst raw within-session spread was 5.3%, which the median-of-7 damps out. 10% is ~13x the cross-session figure and ~2x the worst raw spread, so it will not fire on scheduler jitter but will catch the tens-of-percent slowdown a whole-stack conservative scan would introduce. The 15 ms floor covers the fastest probe (126 ms)." } }, - "probe_overrides": {} + "probe_overrides": { + "07_array_grow_evacuate": { + "copied_bytes": { + "gating": false, + "rationale": "NOT GATED ON THIS PROBE (#9790). Address-dependent promotion order changes old-block packing and the allocation boundary where nursery pressure is checked. The fifth minor observes a different live cohort: a completed 8,208-byte array or a new 144-byte array. Header-size sums and from-space reclamation account for the byte deltas; this is placement-dependent pacing, not an accounting discrepancy. Keep this cell visible; correctness, retention, cycle counts, object counts and promoted bytes remain gated. See evidence/9790-array-growth-pacing.json and the README investigation.", + "evidence": { + "observed_runs": 21, + "observed_spread": 8064, + "measured_on": "2026-09-05, macOS arm64, Perry 0.5.1520; 21 executions of one unchanged binary. Compiler/runtime hashes and raw counters: benchmarks/gc_ratchet/evidence/9790-array-growth-pacing.json.", + "issue": "https://github.com/PerryTS/perry/issues/9790" + } + }, + "freed_bytes": { + "gating": false, + "rationale": "NOT GATED ON THIS PROBE (#9790). Address-dependent promotion order changes old-block packing and the allocation boundary where nursery pressure is checked. The fifth minor observes a different live cohort: a completed 8,208-byte array or a new 144-byte array. Header-size sums and from-space reclamation account for the byte deltas; this is placement-dependent pacing, not an accounting discrepancy. Keep this cell visible; correctness, retention, cycle counts, object counts and promoted bytes remain gated. See evidence/9790-array-growth-pacing.json and the README investigation.", + "evidence": { + "observed_runs": 21, + "observed_spread": 8208, + "measured_on": "2026-09-05, macOS arm64, Perry 0.5.1520; 21 executions of one unchanged binary. Compiler/runtime hashes and raw counters: benchmarks/gc_ratchet/evidence/9790-array-growth-pacing.json.", + "issue": "https://github.com/PerryTS/perry/issues/9790" + } + } + } + } }, "notes": "Re-pinned for #8122-recover: the allocation census before minor #0 object-denominates the first nursery cap (first cycle fires earlier on small-object workloads), one descriptor lookup per traced object, untraced-promotion threshold 990 -> 980. Every GC-accounting fingerprint shifts; retention improves on 12_large_live_set (-75%) and 13_large_eden_survivors moves +85 KB because its cycle 0 now holds up an in-place promotion at 581 permille (main at cap 49 retains 651 KB the same way).", "probes": { diff --git a/benchmarks/gc_ratchet/evidence/9790-array-growth-pacing.json b/benchmarks/gc_ratchet/evidence/9790-array-growth-pacing.json new file mode 100644 index 0000000000..49b48be8d8 --- /dev/null +++ b/benchmarks/gc_ratchet/evidence/9790-array-growth-pacing.json @@ -0,0 +1,177 @@ +{ + "issue": 9790, + "measured_at": "2026-09-05", + "platform": "darwin-arm64", + "runtime_version": "0.5.1520", + "source": "benchmarks/gc_ratchet/probes/07_array_grow_evacuate.ts (unchanged workload)", + "run_env": { + "PERRY_GC_DIAG": "1", + "PERRY_GC_TRACE": "1" + }, + "oracle": { + "version": "v26.5.1", + "matching_runs": 21, + "stdout": "probe:07_array_grow_evacuate\nchecksum:21891160\nsurvivors:94\n" + }, + "binaries": { + "perry": { + "bytes": 152440456, + "sha256": "0e482d4099c396832d57acd1ed9a974107735901cafeeac609ae04ec7a3beb69" + }, + "libperry_runtime.a": { + "bytes": 87787184, + "sha256": "f979e631e6e1d468db1b40bc85882660afe7c79d75c9fc11c18a71fe58c5d591" + }, + "probe": { + "bytes": 16811144, + "sha256": "1f5e3b44052edf6c63fdb1cb51e3f62ec819db50435f264c6ef218da23f230d5" + } + }, + "constant_counters": { + "minor_cycles": 5, + "step_cycles": 5, + "copied_objects": 6745, + "promoted_objects": 6197, + "promoted_bytes": 844600 + }, + "constant_retention": { + "heap_used_bytes": 1214784, + "heap_total_bytes": 25165824 + }, + "samples": [ + { + "copied_bytes": 2648008, + "freed_bytes": 83558840 + }, + { + "copied_bytes": 2639944, + "freed_bytes": 83567048 + }, + { + "copied_bytes": 2648008, + "freed_bytes": 83558840 + }, + { + "copied_bytes": 2639944, + "freed_bytes": 83567048 + }, + { + "copied_bytes": 2648008, + "freed_bytes": 83558840 + }, + { + "copied_bytes": 2639944, + "freed_bytes": 83567048 + }, + { + "copied_bytes": 2648008, + "freed_bytes": 83558840 + }, + { + "copied_bytes": 2639944, + "freed_bytes": 83567048 + }, + { + "copied_bytes": 2648008, + "freed_bytes": 83558840 + }, + { + "copied_bytes": 2648008, + "freed_bytes": 83558840 + }, + { + "copied_bytes": 2648008, + "freed_bytes": 83558840 + }, + { + "copied_bytes": 2639944, + "freed_bytes": 83567048 + }, + { + "copied_bytes": 2648008, + "freed_bytes": 83558840 + }, + { + "copied_bytes": 2648008, + "freed_bytes": 83558840 + }, + { + "copied_bytes": 2648008, + "freed_bytes": 83558840 + }, + { + "copied_bytes": 2648008, + "freed_bytes": 83558840 + }, + { + "copied_bytes": 2648008, + "freed_bytes": 83558840 + }, + { + "copied_bytes": 2639944, + "freed_bytes": 83567048 + }, + { + "copied_bytes": 2648008, + "freed_bytes": 83558840 + }, + { + "copied_bytes": 2648008, + "freed_bytes": 83558840 + }, + { + "copied_bytes": 2648008, + "freed_bytes": 83558840 + } + ], + "diagnostic": { + "method": "Temporary move/header and block-boundary logging in runtime source at c7361c87c73738fe97cf352c75c9f00ce0a9b346; manually relinked the same generated object. Both original counter tuples and Node stdout reproduced. Diagnostic edits are not part of the runtime fix.", + "high": { + "copied_bytes": 2648008, + "freed_bytes": 83558840, + "fifth_minor_copied_bytes": 525312, + "fifth_minor_promoted_bytes": 131328, + "fifth_minor_freed_bytes": 16636544, + "fifth_minor_eden_bytes": 16775936, + "fifth_minor_survivor_bytes": 517248, + "extra_array": { + "seed": 5207, + "length": 640, + "capacity": 1024, + "header_size": 8208, + "source": "remembered_set" + }, + "last_block_trigger": { + "generation": "Old", + "request": 4112, + "offset": 1045352, + "block_size": 1048576, + "young_bytes": 17293184 + } + }, + "low": { + "copied_bytes": 2639944, + "freed_bytes": 83567048, + "fifth_minor_copied_bytes": 517248, + "fifth_minor_promoted_bytes": 131328, + "fifth_minor_freed_bytes": 16644752, + "fifth_minor_eden_bytes": 16776080, + "fifth_minor_survivor_bytes": 517248, + "extra_array": { + "seed": 5240, + "length": 1, + "capacity": 16, + "header_size": 144, + "source": "mutable_root_slots/native_stack" + }, + "last_block_trigger": { + "generation": "Nursery", + "request": 144, + "offset": 1048496, + "block_size": 1048576, + "young_bytes": 17293184 + } + }, + "conservation": "fifth_minor_freed_bytes = fifth_minor_eden_bytes + fifth_minor_survivor_bytes - fifth_minor_copied_bytes - fifth_minor_promoted_bytes in both arms; malloc freed bytes are zero. Total copied-byte delta = 8208 - 144 = 8064." + } +} diff --git a/benchmarks/gc_ratchet/gc_ratchet.py b/benchmarks/gc_ratchet/gc_ratchet.py index 87eb80b83c..52cd1a41df 100644 --- a/benchmarks/gc_ratchet/gc_ratchet.py +++ b/benchmarks/gc_ratchet/gc_ratchet.py @@ -721,9 +721,11 @@ def measure( # Separate traced pass. PERRY_GC_DIAG writes one line per collection # phase, which perturbs wall time, so it must not share a pass with - # the timing samples. Two traced runs are taken and required to - # agree: that is the harness proving, every time it runs, that the - # counters it is about to gate on are actually deterministic. + # the timing samples. Keep both traced runs, including disagreement. + # `check` owns the gating policy (and documented probe overrides), + # so it rejects disagreement in every counter that remains gated. + # Aborting here would bypass those overrides and discard all later + # probes before the gate could report their results (#9790). traced = [ parse_gc_diag( run_once( @@ -733,12 +735,6 @@ def measure( ) for _ in range(2) ] - if traced[0] != traced[1]: - differing = sorted(k for k in traced[0] if traced[0][k] != traced[1][k]) - raise RatchetError( - f"{name}: GC counters are not deterministic across traced runs " - f"({', '.join(differing)}); they cannot be gated" - ) # Deliberately NOT rejecting minor_cycles == 0 here. A collector that # has stopped running copying minors at all is the single largest # regression this ratchet exists to catch, and it must surface as a @@ -1862,6 +1858,19 @@ def evaluate( for metric in ALL_METRICS: tolerance = resolve_tolerance(tolerances, overrides, name, metric) + # The two traced samples are an independent premise of the band: + # even a one-byte disagreement inside the allowance invalidates a + # gated counter. Read the samples themselves, not a cached spread. + # Only an explicit probe override (or the profile) can exclude it. + current_samples = cur_entry["metrics"][metric].get("samples", []) + unstable_current = metric in GC_METRICS and tolerance.gating and ( + len(current_samples) < 2 or len(set(current_samples)) != 1 + ) + if unstable_current: + failures.append( + f"{name}: {metric} is not deterministic across traced runs; " + "it cannot be gated" + ) # A cell the pinned artifact cannot support is demoted rather than # trusted: comparing against a number whose own premise failed would # dress a defect up as a verdict. The defect is already in @@ -1882,7 +1891,9 @@ def evaluate( else: breach = False - if breach and quarantined: + if unstable_current: + status = "UNFIT (traced samples disagree)" + elif breach and quarantined: status = "UNFIT (pinned cell unusable)" elif breach: status = "REGRESSION" if tolerance.gating else "drift (informational)" diff --git a/benchmarks/gc_ratchet/probes/07_array_grow_evacuate.ts b/benchmarks/gc_ratchet/probes/07_array_grow_evacuate.ts index cd9a463cba..8fdb35c13f 100644 --- a/benchmarks/gc_ratchet/probes/07_array_grow_evacuate.ts +++ b/benchmarks/gc_ratchet/probes/07_array_grow_evacuate.ts @@ -1,11 +1,13 @@ // GC ratchet probe: array element storage growth and reallocation. // -// A growing array repeatedly abandons its previous element storage, which is a -// separate allocation from the array header. Evacuation has to rewrite the -// header's pointer to the moved storage; per-object pinning would leave the old -// storage in place and fragment the region. The probe grows many arrays past +// A growing array replaces its inline header-plus-elements allocation and +// leaves a forwarding stub at the old address. Evacuation must repair holders +// of the current allocation. The probe grows many arrays past // several reallocation boundaries, keeps a small live sample, cycles the rest // through a ring so they are genuinely heap-allocated, then drops them. +// #9790: copied_bytes/freed_bytes remain informational on this probe because +// block placement changes when nursery pressure is checked. See the ratchet +// README and evidence/9790-array-growth-pacing.json; all other gates remain. declare function gc(): void; diff --git a/benchmarks/gc_ratchet/tolerances.json b/benchmarks/gc_ratchet/tolerances.json index 77d213cadd..4122e3f927 100644 --- a/benchmarks/gc_ratchet/tolerances.json +++ b/benchmarks/gc_ratchet/tolerances.json @@ -27,12 +27,12 @@ "job red. Every entry carries evidence that is checked, not merely stored --", "at least 21 runs (the same number every band above is justified by) and a", "spread that is actually non-zero, so a cell cannot be excluded on a hunch.", - "The section is EMPTY, and that is the goal state. Its one entry --", - "12_large_live_set.heap_used_bytes, added by #7554 -- was deleted by #7558,", - "which removed the cause rather than the cell: explicit gc() no longer forces", - "the conservative native-stack scan, so that reading is bit-identical again", - "and gates again. An empty section is not a disarmed rule; the evidence", - "checks and the never-gate-nothing rule still fail any entry added back.", + "The former 12_large_live_set.heap_used_bytes exclusion was removed by #7558.", + "#9790 excludes only 07_array_grow_evacuate.copied_bytes and freed_bytes:", + "promotion order changes block packing and when nursery pressure is checked.", + "The receipt records 21 runs and an allocation-level accounting cross-check.", + "Both cells remain measured and reported. All other cells retain their bands.", + "Deleting the placement dependency means deleting these exclusions.", "", "#7559 -- A heap_used_bytes band used to NOT be a statement about how much", "the collector retained. The reading is taken after the probe's own gc(),", @@ -222,5 +222,28 @@ } }, - "probe_overrides": {} + "probe_overrides": { + "07_array_grow_evacuate": { + "copied_bytes": { + "gating": false, + "rationale": "NOT GATED ON THIS PROBE (#9790). Address-dependent promotion order changes old-block packing and the allocation boundary where nursery pressure is checked. The fifth minor observes a different live cohort: a completed 8,208-byte array or a new 144-byte array. Header-size sums and from-space reclamation account for the byte deltas; this is placement-dependent pacing, not an accounting discrepancy. Keep this cell visible; correctness, retention, cycle counts, object counts and promoted bytes remain gated. See evidence/9790-array-growth-pacing.json and the README investigation.", + "evidence": { + "observed_runs": 21, + "observed_spread": 8064, + "measured_on": "2026-09-05, macOS arm64, Perry 0.5.1520; 21 executions of one unchanged binary. Compiler/runtime hashes and raw counters: benchmarks/gc_ratchet/evidence/9790-array-growth-pacing.json.", + "issue": "https://github.com/PerryTS/perry/issues/9790" + } + }, + "freed_bytes": { + "gating": false, + "rationale": "NOT GATED ON THIS PROBE (#9790). Address-dependent promotion order changes old-block packing and the allocation boundary where nursery pressure is checked. The fifth minor observes a different live cohort: a completed 8,208-byte array or a new 144-byte array. Header-size sums and from-space reclamation account for the byte deltas; this is placement-dependent pacing, not an accounting discrepancy. Keep this cell visible; correctness, retention, cycle counts, object counts and promoted bytes remain gated. See evidence/9790-array-growth-pacing.json and the README investigation.", + "evidence": { + "observed_runs": 21, + "observed_spread": 8208, + "measured_on": "2026-09-05, macOS arm64, Perry 0.5.1520; 21 executions of one unchanged binary. Compiler/runtime hashes and raw counters: benchmarks/gc_ratchet/evidence/9790-array-growth-pacing.json.", + "issue": "https://github.com/PerryTS/perry/issues/9790" + } + } + } + } } diff --git a/changelog.d/9790-gc-ratchet-array-growth.md b/changelog.d/9790-gc-ratchet-array-growth.md new file mode 100644 index 0000000000..f3abc2b10d --- /dev/null +++ b/changelog.d/9790-gc-ratchet-array-growth.md @@ -0,0 +1,4 @@ +GC Ratchet now applies traced-counter determinism checks where its documented +probe overrides are available, preserving the full measurement report. Record +why array-growth block placement makes two byte counters informational while +keeping the probe's correctness, retention, cycle and object counts gated. diff --git a/tests/test_gc_ratchet.py b/tests/test_gc_ratchet.py index 76bcce51e9..3fed203e45 100644 --- a/tests/test_gc_ratchet.py +++ b/tests/test_gc_ratchet.py @@ -237,6 +237,95 @@ def _hard(failures): return [failure for failure in failures if not failure.startswith("NOTE")] +class CurrentCounterDeterminismTests(unittest.TestCase): + def test_measure_keeps_disagreement_and_runs_the_remaining_probes(self): + stderr = ( + "#gcmetric heap_used_bytes=1000000\n" + "#gcmetric heap_total_bytes=20971520\n" + "#gcmetric rss_bytes=30000000\n" + ) + + def run(copied_bytes=1500000): + return { + "returncode": 0, "stdout": "ok\n", "wall_ms": 200, + "peak_rss_bytes": 31000000, + "stderr": stderr + "[gc-copy-minor] ran copied_objects=20000 " + f"copied_bytes={copied_bytes} promoted_objects=4000 " + "promoted_bytes=500000 freed_bytes=100000000\n[gc-step]\n", + } + + with tempfile.TemporaryDirectory() as tmp: + probes_dir = Path(tmp) + for name in ("01_probe", "02_other"): + (probes_dir / f"{name}.ts").write_text("// stub\n") + with mock.patch( + "benchmarks.gc_ratchet.gc_ratchet.compile_probe", return_value=Path("stub") + ), mock.patch( + "benchmarks.gc_ratchet.gc_ratchet.run_once", + side_effect=[run()] * 4 + [run(1500002)] + [run()] * 5, + ) as runner: + result = measure( + perry=Path("stub"), probes_dir=probes_dir, repeats=3, node=None, warmup=0 + ) + self.assertEqual(runner.call_count, 10) + self.assertEqual(set(result["probes"]), {"01_probe", "02_other"}) + metric = result["probes"]["01_probe"]["metrics"]["copied_bytes"] + self.assertEqual(metric["samples"], [1500000, 1500002]) + self.assertEqual(metric["spread"], 2) + + def test_every_gated_counter_rejects_disagreement_inside_its_band(self): + for profile in PROFILES: + for metric in GC_METRICS: + with self.subTest(profile=profile, metric=metric): + current = _measurement(_pair()) + value = BASE_VALUES[metric] + dist = distribution([value - 1, value + 1]) + # A stale cached spread must not hide the disagreeing samples. + dist["spread"] = 0 + current["probes"]["01_probe"]["metrics"][metric] = dist + rows, failures = evaluate(_baseline(_pair()), current, profile=profile) + self.assertTrue(any( + metric in failure and "not deterministic" in failure + for failure in _hard(failures) + )) + self.assertEqual(len(rows), 2 * len(ALL_METRICS)) + row = next(r for r in rows if r.probe == "01_probe" and r.metric == metric) + self.assertEqual(row.status, "UNFIT (traced samples disagree)") + + def test_only_the_documented_cells_may_vary(self): + payload = _with_override(metric="copied_bytes") + payload["probe_overrides"]["01_probe"]["freed_bytes"] = _override_entry() + baseline = _baseline(_pair(), payload) + current = _measurement(_pair()) + for metric in ("copied_bytes", "freed_bytes"): + value = BASE_VALUES[metric] + current["probes"]["01_probe"]["metrics"][metric] = distribution([value, value + 1]) + for profile in PROFILES: + rows, failures = evaluate(baseline, current, profile=profile) + self.assertEqual(_hard(failures), []) + excluded = [r for r in rows if not r.gating and r.probe == "01_probe"] + self.assertTrue({"copied_bytes", "freed_bytes"} <= {r.metric for r in excluded}) + for probe, metric in (("01_probe", "copied_objects"), ("02_other", "copied_bytes")): + with self.subTest(probe=probe, metric=metric): + perturbed = copy.deepcopy(current) + value = BASE_VALUES[metric] + perturbed["probes"][probe]["metrics"][metric] = distribution([value, value + 1]) + _, failures = evaluate(baseline, perturbed, profile="shared_ci") + self.assertTrue(any( + probe in failure and metric in failure and "not deterministic" in failure + for failure in _hard(failures) + )) + + def test_array_growth_exclusions_match_the_recorded_samples(self): + receipt = json.loads((REPO_ROOT / "benchmarks/gc_ratchet/evidence/9790-array-growth-pacing.json").read_text()) + entries = _shipped_tolerances()["probe_overrides"]["07_array_grow_evacuate"] + self.assertEqual(set(entries), {"copied_bytes", "freed_bytes"}) + for metric, entry in entries.items(): + values = [sample[metric] for sample in receipt["samples"]] + self.assertEqual(entry["evidence"]["observed_runs"], len(values)) + self.assertEqual(entry["evidence"]["observed_spread"], max(values) - min(values)) + + class ParsingTests(unittest.TestCase): def test_measurement_refuses_a_host_without_wait4_before_launching(self): with mock.patch.object(os, "wait4", None, create=True):