fix(labels): accept missing labels, keep label identity, reject duplicates alike

[kevincostner17]

Summary

Four column-label defects found by fuzzing. Stacked on #466 — review that first; this PR targets its branch and I'll retarget to main once it merges.

• Closes fd.suggest_plan / fd.plan raise TypeError on duplicate column labels #459 — suggest_plan/plan raise TypeError on duplicate labels
• Closes KeyError(Index([nan])) in clean/profile/infer_roles/explain_clean when column labels are numeric-or-None #461 — numeric-or-None labels raise KeyError(Index([nan])) in clean, profile, infer_roles and explain_clean
• Closes fd.infer_roles returns coerced column labels (0 -> 0.0, None -> NaN), breaking the documented round-trip #462 — infer_roles returns coerced labels, breaking the documented round-trip
• Closes Duplicate column labels raise AttributeError ('DataFrame' has no attribute 'str'/'dtype')  #437 — clean_text/lint_text_encoding raise AttributeError on duplicate labels (the validate_fields side shipped in fix(fieldcheck): report bytes, container, boolean and duplicate-label inputs instead of raising #445)

Root cause

• KeyError(Index([nan])) in clean/profile/infer_roles/explain_clean when column labels are numeric-or-None #461: pandas coerces Index([0, None]) to float64, so the second label is NaN. DataFrame.duplicated() then compares its subset against the frame's own columns and cannot match NaN against itself, raising KeyError(Index([nan])). Three call sites hit it: the duplicates step (detection and both removal paths), the profile summary, and the role-inference duplicate mask. A None label beside a non-numeric label keeps the index object-dtype, which is why earlier sweeps missed it.
• fd.infer_roles returns coerced column labels (0 -> 0.0, None -> NaN), breaking the documented round-trip #462: infer_roles built its output frame from a list of row dicts, so pandas collected the labels into one Series and coerced them — 0 became 0.0, None became NaN, -2 became -2.0.
• fd.suggest_plan / fd.plan raise TypeError on duplicate column labels #459: with the semantic layer active, build_context does int(s.isna().sum()) on df[label], which is a DataFrame when the label repeats. The preview API was therefore stricter than the executor: fd.clean handled the frame, fd.plan(df, semantic_mode="auto") did not.
• Duplicate column labels raise AttributeError ('DataFrame' has no attribute 'str'/'dtype')  #437: clean_text and lint_text_encoding call .str/.dtype on df[col], which is a DataFrame for a repeated label.

Behaviour change

• Missing labels: duplicate detection addresses columns by position when any label is missing, so detection, drop_duplicates, duplicate_subset and the keep-modes all work on such frames. Frames without missing labels take exactly the previous code path.
• Label identity: infer_roles(...)["column"] now holds the frame's own labels with object dtype, so frame[row["column"]] round-trips. Rows are still ordered by label text, as before.
• Duplicate labels: suggest_plan, plan, clean_text and lint_text_encoding raise ValueError("<func> requires unique column labels; duplicated: [...]"), matching fd.validate, fd.infer_roles, explain_clean, detect_pii and build_baseline. The guard moved to _util.require_unique_labels; api._require_unique_labels stays as an alias.

Default-output changes

• Frames with a missing column label ({0: ..., None: ...}) now return a result from clean, profile, infer_roles and explain_clean instead of raising, and their duplicate rows are detected and removed like any other frame's.
• infer_roles: the column column is object dtype rather than float64 when labels are mixed numeric/None or mixed int/float. Values are the frame's real labels. For all-string labels, output is unchanged.
• suggest_plan/plan/clean_text/lint_text_encoding on duplicate labels: a clear ValueError instead of TypeError/AttributeError. Unique-label frames are unaffected.

One correction to #462 as filed

The issue's repro expects [0, None] back from pd.DataFrame({0: ..., None: ...}), but pandas coerces those labels to [0.0, nan] at construction, before any freshdata code runs — df.columns.dtype is already float64, and df[nan] cannot round-trip in pandas either. That part is not recoverable. What this PR fixes is the coercion freshdata itself introduced: when a frame's columns are genuinely object dtype (pd.Index([0, None], dtype=object), [-2, 0.78], ["a", 1]), the labels now come back exactly as they went in and round-trip. The repro therefore still exits 1; the regression test covers the recoverable case.

Tests

tests/test_label_robustness.py (new):

• KeyError(Index([nan])) in clean/profile/infer_roles/explain_clean when column labels are numeric-or-None #461: clean, profile, infer_roles and explain_clean all accept a NaN-label frame; duplicate rows are still detected and removed; duplicate_subset still narrows the comparison.
• fd.infer_roles returns coerced column labels (0 -> 0.0, None -> NaN), breaking the documented round-trip #462: [0, None], [-2, 0.78] and ["a", 1] object-dtype label sets come back unchanged and round-trip through df[label].
• fd.suggest_plan / fd.plan raise TypeError on duplicate column labels #459 / Duplicate column labels raise AttributeError ('DataFrame' has no attribute 'str'/'dtype')  #437: suggest_plan, plan, clean_text, lint_text_encoding and infer_roles all raise ValueError on duplicate labels without modifying the frame, and all still work on unique labels.

Verification

• ruff check and mypy src/freshdata: clean.
• Campaign repros for fd.suggest_plan / fd.plan raise TypeError on duplicate column labels #459 and KeyError(Index([nan])) in clean/profile/infer_roles/explain_clean when column labels are numeric-or-None #461 exit 0 on py3.12 / pandas 2.3.3 and py3.9 / pandas 1.5.3; both fail on main. The Duplicate column labels raise AttributeError ('DataFrame' has no attribute 'str'/'dtype')  #437 repro passes its text sections. The fd.infer_roles returns coerced column labels (0 -> 0.0, None -> NaN), breaking the documented round-trip #462 repro still exits 1 for the reason above.
• Label, profile, explain and plan suites: 65 passed (py3.12), 64 passed / 1 skipped (py3.9).
• Full not online and not large lanes: py3.12 6582 passed, 17 skipped; py3.9 6553 passed, 21 skipped.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: f66e8ad2-6aa1-4dac-9a0f-74fdcf11f4e0

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)