fix(strings): a declared vocabulary outranks a generic null marker

[kevincostner17]

Summary

fieldcheck has applied this rule since the TestAllowedValuesBeatNullMarkers regression — "'NA' may be Namibia: when the schema literally allows a value, it is a value, not a missing marker" (fieldcheck.py:466). fd.clean did not. normalize_sentinels applied DEFAULT_SENTINELS unconditionally, so a caller who had explicitly declared NA as permitted still lost it.

fd.validate_fields(df, {"country": FieldSpec(allowed_values=[..., "NA"])})
#   -> no issue; NA is a value

fd.clean(df, context="Allowed country values are US, GB, FR, DE, JP, CA, AU, BR, NA.")
#   policy compiles cleanly: [('country','allowed_values','soft')], issues: []
#   -> country is NaN

The same declared fact was respected by one public API and ignored by another. NA is Namibia in an ISO-3166 column and None is a real brand; both were destroyed.

Every route tested before this change

Route	NA survives
defaults	no
semantic_context semantic_type="country"	no
semantic_context allowed_values=[…NA…]	no
context="Allowed country values are …, NA." (compiles clean, 0 issues)	no
preserve_columns	no — documented as "never dropped", so correct
context="Never modify country values." → protected	yes, but disables every other repair
normalize_sentinels=False	yes, but global, every column

So the only escapes were protecting the column outright or turning sentinel handling off for the whole frame — which would also stop nulling genuine "N/A" markers elsewhere.

The fix

A column's declared allowed_values — via semantic_context or compiled from a context= policy, which lower to the same place — now removes those tokens from that column's sentinel set, matched casefolded and trimmed for consistency with fieldcheck and extra_sentinels.

Deliberately narrow:

• No declaration means no change. "NA" with no vocabulary is still missing — the gauntlet sentinel_collision gold label. That fixture spells its countries out ("Namibia"), so its labels are untouched and the gate still passes.
• A vocabulary that excludes NA still nulls it, mirroring fieldcheck's test_na_outside_vocabulary_is_still_a_null_marker.
• Scoped to the declaring column — declaring NA for country does not rescue it in note.
• A malformed hint is ignored, not fatal. A bare string is rejected deliberately, so "NA" cannot be iterated into the letters N and A.

The gauntlet fixture states the intended behaviour outright: "without a vocabulary containing 'NA', the null-marker reading wins; with allowed_values that includes 'NA' the value survives".

Verification

• tests/test_sentinel_allowed_values.py: 14 cases. 7 fail on main; the other 7 are the must-not-change cases and pass on both, showing the default is untouched.
• Full suite py3.12: 6588 passed, 22 skipped, 0 failed, coverage 93.91%.
• python -m benchmarks.gauntlet run --check: all gates passed, 0 corruption and 0 false-positive rate across all five fixtures.
• ruff check / ruff format clean. (Local mypy is blocked by the numpy 2.5 stubs issue; CI pins numpy<2.5.)

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 5d9b7435-75d2-4b9e-b014-b9c2138d43ca

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)