Repository navigation
attribution audit: grow the golden set past min_n_per_class 10 for verbatim and near-verbatim #5355
Description
Refs #3465
Agent Brief
Type: Task
Summary: Grow the attribution audit's golden set past min_n_per_class 10 for the verbatim and near-verbatim classes.
Current behavior:
The golden set has 10 cases across four groupings, so no class reaches min_n_per_class 10 and every class is report-only. The earlier growth item (#3458) is closed while the set is still at 10. The first report-only sweep harvested adjudication candidates (rejected findings included) that are not yet converted into cases.
Desired behavior:
The set carries at least min_n_per_class cases, and more than 10, for each of verbatim and near-verbatim, including rejected findings and negative cases, so precision at the 0.95 bar is measurable. Each new case records its source, its expected verdict and the rubric version it was scored against.
Key interfaces:
- The golden-set case format and its scoring procedure.
- The
min_n_per_classand precision-bar settings, which this item does not change.
Acceptance criteria:
-
verbatimandnear-verbatimeach have more thanmin_n_per_classcases, or the item records exactly why not. - Each new case cites a real source and an adjudicated verdict; none is authored only to pass.
- The set is scored against the current rubric and per-class precision is recorded with its n.
- No case is fabricated. A case needing an adjudication only the owner can make is listed as blocked instead.
Out of scope:
- Applying dispositions, or declaring any class fix-eligible (an owner decision after the numbers exist).
- The vendored-snapshot tier row and the 10-case re-score (a sibling item; this depends on it).
- Editing provenance sweep sub-topic: repo-wide sweep, upstream-drift convention engagement, and the cleaned-state gate #3465's body or labels.
Activity
kyle-sexton commentedon Sep 30, 2026
Draft PR #5620 adds 14 adjudicated synthetic cases (c25-c38) but does not close this issue.
Why it stays open:
No class is declared fix-eligible.
fix(attribution): reword golden cases c28 and c32 so neither reproduc…
docs(attribution): fold the real-source measurement into the 0.8.1 Ad…
docs(attribution): record the whole-set golden re-score in the 0.8.1 …
docs(attribution): state how the re-score table charges a case to a c…
test(attribution): grow the golden set for verbatim and near-verbatim (…
kyle-sexton commentedon Sep 30, 2026
Update: PR #5620 merged (attribution 0.8.2). This issue stays open.
Owner decision needed on whether the verbatim target should change, and on the blocked and contradicting candidates.
kyle-sexton commentedon Oct 1, 2026
Owner decision (2026-10-01): approved the recommendation ("go with recommended", the owner's reply to the batch decision page that listed this question and its recommendation).
Question: #5355 asked for more than 10 golden cases each for verbatim and near-verbatim. Near-verbatim has 13; verbatim has 3.
Chosen option A: close #5355 as completed. Acceptance criterion 1 allows "or the item records exactly why not", and the 0.8.2 CHANGELOG and the 2026-09-30 comment record why verbatim stays at 3. The open items (10 blocked panels, the contradicting verbatim candidate, the c23/c26/c36 disagreements, whether the verbatim target should change) move to one owner-gated follow-up.
Next: follow-up filed as #5654 (Refs #3465); closing this issue as completed.