fix(privacy): classify from every value, inline rules first, rule keys first, non-string labels, quarantine nullable columns by kevincostner17 · Pull Request #401 · FreshCode-Org/freshdata

kevincostner17 · 2026-09-15T10:59:48Z

Summary

One commit per fix. Builds on #399 (merged).

#246: classify policy columns from every distinct value

classify_columns built its sample from the first 200 non-null cells of each column, so PII after row 200 was never classified and no action reached it.

Classification now reads every distinct non-null value, de-duplicated in first-seen order.
Entity detection runs detect_in_text over newline-joined chunks of at most 64,000 characters and takes the union. It stops once every reportable entity type has been found.
The value-regex classifier still reads each value; the Luhn and context checks read each chunk.
The detectors, patterns and detect_in_text are unchanged.
The report metadata gains classification_values_scanned (distinct values read per column).

#284: inline rules take priority over pack rules

Rules were ranked by classifier specificity across inline and pack rules alike, so an inline rule matching by entity, regex or context lost to a pack column-name rule.

Matches are now ranked by (is_inline, specificity): any in-scope inline match beats every pack rule.
Specificity still decides within each group, and ties still go to the earlier rule.
The PrivacyPolicy docstring and the specificity comment describe this.

#285: a rule's own key before policy defaults

_resolve_key checked policy.key_env before rule.key, so setting an organisation-wide variable silently changed the tokens of a rule with its own key.

The precedence is now rule.key_env > rule.key > policy.key_env > policy.key.
Rules without a key resolve exactly as before, and tokenize with no key anywhere still raises.
PrivacyRule and PrivacyPolicy document the order.

#232 (part 5): non-string column labels

apply_privacy_policy indexed the frame with str(label), so integer labels raised KeyError.

Each report key is now mapped back to its original label for read, write and drop. Report keys stay strings.
classify_columns and apply_privacy_policy raise ValueError on duplicate labels, or on labels that collide once stringified (1 and "1").

Quarantine on nullable integer, boolean and categorical columns

The quarantine action substituted its string placeholder with Series.where, which raises TypeError on nullable Int64/boolean and categorical columns, failing the whole policy run.

When the column's dtype rejects the placeholder, the column is cast to object first, matching object columns.
Missing cells stay missing, and every other dtype is unchanged.
The two quarantine cases excluded in tests/test_privacy_missing_and_labels.py (fix(privacy): missing values stay missing, categorical k-anonymity, duplicate labels, fpe audit metadata, NER status #399) are re-enabled.

Behaviour changes

Columns longer than 200 rows may now be classified where they were not before. Classification cost grows with the number of distinct values.
An inline rule beats a more specific pack rule.
A rule's key beats policy.key_env.
Duplicate or colliding labels raise ValueError.
Quarantined Int64/boolean/category columns come back as object dtype.

Tests

New tests/test_privacy_policy_classification.py:

apply_privacy_policy classifies from the first 200 values only; later PII stays unmasked #246:
- the issue repro at 10, 250 and 10,000 rows
- late value-regex, context and Luhn matches
- a detect_in_text call counter: calls equal ceil(chars / limit), repeated values cost one call, and detection stops early after one chunk
- oversized-value chunking
- missing and unhashable cells
PrivacyPolicy inline rules do not take priority over pack rules as documented #284:
- the repro
- an inline context rule beats a pack column-name rule
- specificity within the inline group
- an out-of-scope inline rule doesn't override
- a pack-only policy is unchanged
apply_privacy_policy: policy-level key_env overrides a rule's explicit key #285:
- the repro
- rule.key_env beats rule.key
- fallback to the policy key
- resolution for a rule without a key is pinned
- the no-key ValueError is pinned
Non-string column labels crash or silently misbehave in six APIs #232 part 5:
- the repro gives ['<EMAIL>', '<EMAIL>']
- string report keys
- drop, quarantine and tokenize with integer labels
- 1/"1" collision and duplicate-label errors for both functions
Quarantine: Int64, boolean, string, category and object columns with pd.NA.

Verification

ruff check .: passed
mypy src/freshdata: no issues
pytest -m "not online and not large" on main 78b7790 + this branch:
- Python 3.12 / pandas 2.3.3: 5573 passed, 13 skipped
- Python 3.9 / pandas 1.5.3: 5569 passed, 17 skipped
pytest tests/truthbench: 250 passed

Closes #246
Closes #284
Closes #285
Refs #232

Root cause: classify_columns built its value sample from the first 200 non-null cells of each column (_SAMPLE_ROWS). Entity, value-regex, Luhn and context classification all read only that sample, so a column whose first 200 values held nothing sensitive stayed unclassified and no action reached later cells. The report gave no sign that a sample was used. Fix: classification now reads every distinct non-null value of a column, de-duplicated in first-seen order (falling back to str() de-duplication for unhashable cells). Entity detection runs detect_in_text over newline-joined chunks of at most 64,000 characters and takes the union, stopping once every entity type the config can report has been found. The value-regex classifier still reads each value; the Luhn and context checks read each chunk. Only the values fed to classification change; the detectors and patterns are untouched. The report metadata gains classification_values_scanned with the distinct value count per column. Closes #246

Root cause: classify_columns ranked every matching rule by classifier specificity alone (column-name > entity > regex > context), comparing inline and pack rules on the same scale; rule order only broke ties. An inline rule matching by entity type, value regex or context therefore lost to any pack rule matching the column name, which contradicts the PrivacyPolicy docstring and is the usual way to override a pack. Fix: rank each match by (is_inline, specificity), where an inline rule is one listed in policy.rules. Any in-scope inline match beats every pack rule; specificity still decides within each group, and ties still go to the earlier rule. Out-of-scope inline rules are filtered out before ranking, as before, and pack-only policies classify exactly as they did. The PrivacyPolicy docstring and the specificity comment now describe this. Closes #284

Root cause: _resolve_key checked rule.key_env, then policy.key_env, then rule.key, then policy.key. A rule with its own literal key was switched to the organisation-wide key as soon as the policy-level environment variable was set, so tokens changed between environments and joins against previously tokenized data broke. Fix: resolve rule settings first and policy defaults second: rule.key_env (when set to a non-empty value), rule.key, policy.key_env, policy.key. A rule that sets no key resolves exactly as before (policy.key_env, then policy.key), and tokenize with no key anywhere still raises. PrivacyRule and PrivacyPolicy docstrings state the order. Closes #285

Root cause: classify_columns keys its results by str(label), and apply_privacy_policy then indexed the frame with that string. Any frame with integer (or other non-str) column labels that had a classified column raised KeyError, although fd.clean accepts such frames (headerless CSVs, DataFrame(ndarray)). Duplicate labels, or distinct labels with the same string form such as 1 and "1", could not be addressed by a single report key either. Fix: apply_privacy_policy maps each report key back to its original label and uses that label to read, write and drop the column; report keys stay strings. classify_columns and apply_privacy_policy raise ValueError for duplicated labels and for labels that collide once stringified. Only the label lookup changes; masking, hashing and tokenize defaults are untouched. Refs #232

… a TypeError Root cause: the quarantine action in apply_privacy_policy substitutes its string placeholder with Series.where. Nullable Int64 and boolean arrays (and categoricals) cannot hold a string, so pandas raised TypeError and the whole policy run failed, while object, string, numpy numeric and datetime columns were quarantined normally. Fix: when Series.where rejects the placeholder for the column's dtype, cast the column to object and substitute there, as an object column behaves. Missing cells stay missing and every dtype that already worked is unchanged. Only the quarantine substitution changes. The Int64 and boolean quarantine cases that tests/test_privacy_missing_and_labels.py had excluded are re-enabled.

coderabbitai · 2026-09-15T10:59:55Z

Important

🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: de3e67b0-e65c-49d7-9255-64cb2ce7d6d6

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-09-15T11:09:36Z

FreshData benchmark report — `performance`

freshdata: ?
python: ?
platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)

Covers #373, #380, #381, #382, #383, #384, #386, #387, #388, #389, #390, #391, #393, #394, #396, #397, #398, #399, #400 and #401.

kevincostner17 added 5 commits September 15, 2026 16:25

kevincostner17 mentioned this pull request Sep 15, 2026

docs: changelog for the round-6 bug-hunt fixes #402

Merged

kevincostner17 merged commit c96d169 into main Sep 15, 2026
21 checks passed

kevincostner17 mentioned this pull request Sep 15, 2026

Non-string column labels crash or silently misbehave in six APIs #232

Closed

6 tasks

kevincostner17 added a commit that referenced this pull request Sep 15, 2026

docs: changelog for the round-6 bug-hunt fixes (#402)

cbd1967

Covers #373, #380, #381, #382, #383, #384, #386, #387, #388, #389, #390, #391, #393, #394, #396, #397, #398, #399, #400 and #401.

Navigation Menu

fix(privacy): classify from every value, inline rules first, rule keys first, non-string labels, quarantine nullable columns - #401

fix(privacy): classify from every value, inline rules first, rule keys first, non-string labels, quarantine nullable columns#401
kevincostner17 merged 5 commits into
mainfrom
fix/privacy-policy-classification

kevincostner17 commented Sep 15, 2026

Uh oh!

coderabbitai Bot commented Sep 15, 2026

Uh oh!

github-actions Bot commented Sep 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

kevincostner17 commented Sep 15, 2026

Summary

#246: classify policy columns from every distinct value

#284: inline rules take priority over pack rules

#285: a rule's own key before policy defaults

#232 (part 5): non-string column labels

Quarantine on nullable integer, boolean and categorical columns

Behaviour changes

Tests

Verification

Uh oh!

coderabbitai Bot commented Sep 15, 2026

Uh oh!

github-actions Bot commented Sep 15, 2026

FreshData benchmark report — performance

Authored-code reduction (Metric 6)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

FreshData benchmark report — `performance`