fix(guard,explain): survive duplicate index labels and unhashable cells

[kevincostner17]

Summary

Two crashes on frames that fd.clean and fd.profile otherwise accept, both found by fuzzing.

• Closes Protected-column guard false positive: fd.clean raises ProtectedColumnError on frames with dupl #449 — the protected-column guard raises ProtectedColumnError on a frame with duplicate index labels, on a column it did not touch
• Closes fd.explain_clean raises TypeError: unhashable type 'list' on an object column containing a list #450 — fd.explain_clean raises TypeError: unhashable type: 'list'

Root cause

• Protected-column guard false positive: fd.clean raises ProtectedColumnError on frames with dupl #449: _series_identical aligned its snapshot with before.loc[after.index]. With repeated labels that reindex multiplies rows instead of selecting them, so as soon as a row-level step dropped any row (here drop_empty_rows), the lengths could never line up and the guard reported a violation. The column was in fact returned unchanged, and the error told the user to report a freshdata executor bug. Net effect: context policies and mutable=False were unusable on any frame with a non-unique index.
• fd.explain_clean raises TypeError: unhashable type 'list' on an object column containing a list #450: _column_stats counts distinct values with nunique(), which hashes every value, so one list or dict cell aborted the whole report. profile.py already guards the same pattern.

Behaviour change

• Guard, duplicate labels: when either index has repeated labels, the surviving values are checked positionally instead of by label. Row-level steps only ever drop rows, so every surviving value must still appear, in order, in the original column.
  • Still reported: a rewritten cell, reordered rows, gained rows, a dtype change and a dropped column.
  • Two missing values count as equal, as before.
  • A rewrite that happens to leave the values a valid in-order subsequence is the one case this check cannot distinguish. It is strictly better than the previous behaviour, which could not verify these frames at all.
• Guard, unique labels: unchanged, still aligned by index label.
• explain_clean: unhashable cells are counted by their text form, so [1], [1], {"k": 2} counts as 2 distinct values. Hashable columns are untouched.

Default-output changes

None for frames that already worked.

• Frames with a duplicate index and a protected column now return the cleaned frame where they used to raise ProtectedColumnError. The guard action in the report is recorded as verified rather than violated.
• explain_clean on a frame with list or dict cells now returns a report with a nunique count where it used to raise.
• Unique-index frames and hashable columns produce byte-identical output.

Tests

• tests/test_guard_protected.py:
  • the repro frame (duplicate labels, a protected column, drop_empty_rows) cleans and matches the unprotected result
  • _series_identical on a duplicate index still reports a rewritten cell, reordered rows and gained rows
  • two missing values compare equal, and a missing-to-value change is still reported
• tests/test_explain.py: explain_clean on a frame with list and dict cells returns a report, counts 2 distinct payloads and still counts the null.

Verification

• ruff check and mypy src/freshdata: clean.
• Campaign repros for Protected-column guard false positive: fd.clean raises ProtectedColumnError on frames with dupl #449 and fd.explain_clean raises TypeError: unhashable type 'list' on an object column containing a list #450 exit 0 on py3.12 / pandas 2.3.3 and py3.9 / pandas 1.5.3; both fail on main.
• Guard, explain and context suites: 357 passed on both Pythons.
• Full not online and not large lanes: py3.12 6507 passed, 17 skipped; py3.9 6478 passed, 21 skipped.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: a20f3f88-c74f-4c75-ae87-0efa9bbef542

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)