fix(compliance): evidence-based HIPAA coverage, token hint matching, GDPR measures

[kevincostner17]

Summary

Compliance reports now rest on evidence from the run instead of optimistic defaults.

• HIPAA Safe Harbor report passes without dataframe= even when identifier columns exist #245 — HIPAA Safe Harbor passed without dataframe=.
  • Cause: without a source frame, the adapter only saw columns that cleaning touched. Untouched identifier columns (e.g. ssn, patient_email) were invisible, so every identifier came back not_detected.
  • Fix:
    • CleanReport now records input_columns, the post-rename column list, in run_pipeline and apply_plan.
    • The compliance context uses it when dataframe= is absent.
    • When no full column list exists, the HIPAA report sets coverage_verifiable: False, adds a warning, and does not pass.
  • input_columns is not part of to_dict(), so the JSON schema and golden snapshots are unchanged.
• HIPAA identifier hints match substrings: description, last_updated, business_unit flagged #283 — HIPAA hints matched substrings (ip in description, date in last_updated, sin in business_unit).
  • Column names are now split into tokens on separators, camelCase and letter/digit boundaries.
  • Every hint matches as a whole token or token sequence (DateOfBirth matches date_of_birth), and hints of four characters or fewer match only that way.
  • Longer hints may also match inside run-together names (patientemail), but never starting part-way through a token (ship_address no longer matches ip_address).
  • Added surname and zipcode hints.
• GDPR Article 30 record always lists 'Hash-salt PII masking' as a security measure #287 — GDPR Article 30 always listed "Hash-salt PII masking (SHA-256 + salt)".
  • security_measures is now built from evidence.
  • A masking entry appears only when columns were masked; it names the recorded strategies and makes no cryptographic claims.
  • The Data Trust Score entry appears only when a trust score was available.

Behaviour changes

• Untouched identifiers now count: HIPAA reports from fd.clean / fd.apply_plan without dataframe= detect them, so reports that used to pass can fail.
• No column evidence now fails HIPAA: those reports fail with a warning. This covers synthetic reports, and reports from native backends, streaming and multi-file runs, which don't record input_columns yet.
• Short hints are stricter: they no longer match inside other words. All-lowercase run-together names that relied on one (e.g. visitdate, patientssn) are no longer flagged; separated or camelCase forms still are.
• GDPR is accurate: security_measures no longer claims masking or a trust score that wasn't applied.

Tests

• tests/test_compliance/test_hipaa.py:
  • input_columns is recorded after rename and kept out of to_dict().
  • HIPAA Safe Harbor report passes without dataframe= even when identifier columns exist #245 repro: untouched identifiers fail without dataframe=, with the same gaps as with it; masked identifiers still pass; no column evidence gives unverifiable and not passed.
  • HIPAA identifier hints match substrings: description, last_updated, business_unit flagged #283 repro, plus false-positive cases (hotel, cancelled, province, fluid, surface, backlinks, filename) and true-positive cases (IPv4Address, DateOfBirth, dateofbirth, patientemail, zip5, surname). ship_address is not an IP.
  • apply_plan records input_columns.
• tests/test_compliance/test_gdpr.py:
  • GDPR Article 30 record always lists 'Hash-salt PII masking' as a security measure #287 repro: no masking measure and no SHA-256/salt text when nothing was masked.
  • The masking entry comes from masked_columns.
  • Recorded strategy names and the trust-score entry come from an enterprise result.

Verification

• ruff check .: all checks passed
• mypy src/freshdata: no issues in 202 source files
• pytest -m "not online and not large" on py3.12: 4279 passed, 6 skipped
• pytest -m "not online and not large" on py3.9 / pandas 1.5: 4275 passed, 10 skipped

Closes #245
Closes #283
Closes #287

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: ae1e114e-ac15-4803-bbcf-76f7e83f4301

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)