test(corpus): shared adversarial trap corpus with disposition scoring - #472
Merged
Merged
Conversation
Four gold corpora already exist under benchmarks/ (Gauntlet, TruthBench, CleanBench and the fixtures generator), but none is reachable as a fixture from the ~272 ordinary files under tests/. Each test therefore re-invents its own trap literals, which is why '1,000', '1.000', '0001' and 'None' appear as literals in zero test files today. benchmarks/corpus/ adds one importable corpus (pythonpath = ["."] already makes benchmarks.* importable from tests/). It does not replace the existing corpora: adapters.py re-exports Gauntlet's 91 and TruthBench's 1536 cases through the same TrapCase type, so all three are scored with one vocabulary. The organising idea is the one the library's design turns on: the same token means different things in different fields. Cases are keyed by (token, role) and the corpus deliberately carries the same token under several roles with different expected dispositions -- '007' preserves as an identifier and repairs to 7 as a quantity; 'M' expands to Male, Medium or Married depending on the column; 'apple' is a valid company name and an unparseable amount. A cleaner keyed on the value alone passes every single-role test and fails here. dispositions.py adds QUARANTINE and REJECT to the four values the existing harnesses share. gauntlet/metrics.REVIEW_ACTIONS currently folds quarantine, manual_review and reject together, so no gold corpus could say "this must be rejected, not merely queued". Back-compat is preserved: satisfies() widens a plain REVIEW label to any review-family outcome, exactly as REVIEW_ACTIONS does, so existing four-value fixtures keep their meaning. Where the repository genuinely does not define an expectation, the case records expected=None plus a spec_gap naming the missing decision, and is measured rather than asserted. Twelve such gaps are captured, including the undeclared-locale pair '1,200'/'1.200' and the undeclared-dayfirst pair '01/02/2026'/'02/01/2026', which must resolve under the same convention or ordering silently inverts. test_trap_corpus.py guards the corpus itself, including a contract test that every fieldcheck.ACTIONS value maps to a disposition, so an action added to the library fails loudly here instead of being mis-scored.
Three cases asserted PRESERVE for 'NA' and 'None' on the reasoning that Namibia, the surname Na and the brand None are real data. Running them against the library showed the repository has already decided otherwise, deliberately: Gauntlet injects 'NA' into a country column as family 'sentinel_collision' (fixtures.py:311) and labels 'None' in a company column repair-to-missing with the note "a documented sentinel; nulling it is contract behaviour" (fixtures.py:178). Asserting a preference over a decision the repo has made and labelled would be inventing contract, which the corpus exists to avoid. 'None'/product_name is therefore encoded as REPAIR, matching the gold label. 'NA'/country and 'NA'/last_name become specification gaps, because here the repository genuinely contradicts itself rather than merely deciding something unpalatable: fieldcheck.py:466 states "explicit vocabulary outranks generic null markers -- NA may be Namibia" and validate_fields honours a declared allowed_values, while fd.clean's normalize_sentinels ignores allowed_values and nulls the value anyway. The same declared fact is respected by one public API and ignored by another, so the expected disposition is undefined until the two layers agree. Recorded as FD2-001 with the full route table; no library behaviour is changed by this commit.
frames.py builds a realistic frame per case: eight plausible values for the role plus the trap in the last row. The filler matters twice over -- the semantic layer ignores a column with fewer than MIN_DISTINCT_SUPPORT (5) distinct values, and without it "nothing changed" could mean "the library gave up" rather than "the value was preserved". A second anchor column keeps the row alive so a nulled cell is not confused with a dropped row. scoring.py derives the observed disposition from what actually happened to the cell, never from what the report claims, and reports the Phase 23 metric set as rates: repair precision/recall, false positive/negative rate, preservation, review, corruption, escape and audit completeness. A wrong repair is counted as a corruption rather than folded into the repair count, so a confident mistake cannot hide inside a success metric. Cases whose expectation is a recorded specification gap are measured and reported but never scored pass or fail. The two committed assertions deliberately avoid thresholds -- freezing today's numbers would make current behaviour the specification. One guards the metric names, the other asserts audit completeness stays at 1.0, which is the property that matters most here: if the library changes a cell, the report names it. That currently holds across the whole corpus. Running this harness is what surfaced the currency locale defect.
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
…ports Two fixes found by running `ruff check .` and the full suite the way CI does, rather than only the paths I had touched. - Function-level imports in the two scorer tests tripped PLC0415, which failed quality-fast on #472. Hoisted. - The scorer passed every case's semantic_type straight to validate_fields, which warned 42 times that nothing would be validated. That is the library telling the truth: SEMANTIC_TYPES (17 terms, what fd.infer_roles returns and semantic_context accepts) and fieldcheck._KNOWN_SEMANTIC_TYPES (23 terms, what FieldSpec validates) share only 10 of 30 terms. address, boolean_like, category_code, national_id, postal_code and quantity_with_unit are real semantic types that validate_fields does not know, so a user who feeds an inferred "postal_code" back into a FieldSpec is told nothing will be checked. The scorer now only declares a type that fieldcheck knows, and a new test pins both difference sets so that closing or widening the gap is a deliberate, visible change rather than drift. The test asserts what the split *is*, not that it is correct; recorded as FD2-003 for a naming decision. Credit where due: the warning names the column, the unknown type, the consequence and the known types, so this is a coherence problem rather than a silent one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Four gold corpora already exist under
benchmarks/— Gauntlet, TruthBench, CleanBench and the fixtures generator — but none is reachable as a fixture from the ~272 ordinary files undertests/. Every test re-invents its own trap literals, which is why'1,000','1.000','0001'and'None'appear as literals in zero test files today.What this adds
benchmarks/corpus/— one importable corpus (pythonpath = ["."]already makesbenchmarks.*importable fromtests/). It does not replace the existing corpora:adapters.pyre-exports Gauntlet's 91 and TruthBench's 1536 cases through the sameTrapCasetype, so all three are scored with one vocabulary.The organising idea is the one the library's design turns on: the same token means different things in different fields. Cases are keyed by
(token, role)and the corpus deliberately carries the same token under several roles:007customer_id/postal_code007quantityMgender/shirt_size/marital_statusapplecompany_nameappleamount/tickerA cleaner keyed on the value alone passes every single-role test and fails here.
Six dispositions, back-compatibly
gauntlet/metrics.REVIEW_ACTIONSfoldsquarantine,manual_reviewandrejecttogether, so no gold corpus could say "this must be rejected, not merely queued for a human".QUARANTINEandREJECTare now first-class.satisfies()widens a plainREVIEWlabel to any review-family outcome — exactly whatREVIEW_ACTIONSmeans today — so the existing four-value fixtures keep their meaning and keep passing.Specification gaps are recorded, not invented
Where the repository genuinely does not define an expectation, the case carries
expected=Noneplus aspec_gapnaming the missing decision, and is measured rather than asserted. Twelve are captured, including the undeclared-locale pair1,200/1.200and the undeclared-dayfirstpair01/02/2026/02/01/2026, which must resolve under the same convention or ordering silently inverts.Two cases were corrected during review: I initially asserted
PRESERVEforNA/None, then found the repo had already decided otherwise on purpose — Gauntlet injectsNAas familysentinel_collisionand calls nullingNone"contract behaviour". Asserting a preference over a documented decision would be inventing contract.Scoring
scoring.pyderives the observed disposition from what actually happened to the cell, never from what the report claims, and reports the metric set as rates. A wrong repair counts as a corruption, not a repair, so a confident mistake cannot hide inside a success metric.The two committed assertions avoid thresholds deliberately — freezing today's numbers would make current behaviour the specification. One guards the metric names; the other asserts audit completeness stays at 1.0, i.e. if the library changes a cell, the report names it. That currently holds across the whole corpus.
Result
Running this harness is what surfaced the currency locale defect in #471.
Verification
tests/test_trap_corpus.py: 20 tests, including a contract test that everyfieldcheck.ACTIONSvalue maps to a disposition, so an action added to the library fails loudly here instead of being mis-scored.