Repository navigation
Set the default pass level from the A/A Experiment #1268
Description
This was generated by AI from a grill-with-docs session.
Parent
What to build
Set the Bar's default pass level from the A/A Experiment's numbers posted on #1252 (grill Q3.1).
- The rule: the default is the highest pass level at which an A/A Experiment at the measured disagreement rate, on each suite at its size, passes at least 80% of the time.
- The floor: the default never goes below 70%. If 70% isn't enough for some suite, keep the default at 70%, and say in docs/guide.md and on Spec: a Bar a release that changes nothing can pass #1252 which suite falls short and how many Eval tasks it would need.
- docs/guide.md gives the default, the measured rates, and the interval each suite gets at its size, where it describes the standing Bar.
Acceptance criteria
- The default pass level follows the rule, from the posted numbers, and the calculation is recorded on Spec: a Bar a release that changes nothing can pass #1252.
- A Verdict test shows that an A/A-like Experiment at the measured rate and each suite's size passes under the new default.
- The pinned smallest-passing-suite test is updated to the new default.
- docs/guide.md records the default, the rates and the intervals.
Activity
JacobStephens2 commentedon Oct 26, 2026
Blocked by #1370: take the default pass level from that rerun's numbers, not from #1267's. #1267's suites had a broken task, a task targeting the wrong PR, six code-review tasks that can't be passed by design, and six Spec-review tasks without reference findings.
JacobStephens2 commentedon Oct 27, 2026
Unblocked: #1370 has finished. Take the measured rates from its results (#1370 (comment)), not from #1267's.
Paired task differences, from the Verdict's
difference_counts:code-review[0, 1, 1, 8, 0, 0, 0]: 2 of 10 tasks disagree[0, 0, 0, 10, 0, 0, 0]: none disagreespec-review[2, 25, 3]: 5 of 30 tasks disagree[1, 28, 1]: 2 of 30 tasks disagreeThe scope is unchanged. Expect
code-reviewto fall short of the floor: only 3 of its tasks ever pass, so the floor clause applies. Name it in docs/guide.md and on #1252, with the number of Eval tasks it would need. #1434 tracks repairing and growing that suite, and whether a suite that falls short should be advisory in a release Experiment. That isn't part of this issue.JacobStephens2 commentedon Oct 27, 2026
Done in #1444, merged into the Spec branch
issue-1252as 6d06a1f. The default pass level is the 70% floor; the calculation is on #1252 (#1252 (comment)). The pass rule's upper-bound clause is followed up in #1443.