Non-string column labels crash or silently misbehave in six APIs

[kevincostner17]

Summary

fd.clean accepts DataFrames whose column labels are not strings (for example integer labels from pd.read_csv(header=None) or df.T), but several other entry points assume str labels. Some raise KeyError/TypeError; others silently return wrong results because one code path keys by str(col) and another by the raw label. #209 fixed this class for StreamingCleaner/clean_timeseries (PR #221); the paths below are separate.

Affected functions

• fd.clean(semantic_mode=...) raises KeyError on non-string column labels (src/freshdata/semantic/context.py:84)
• FreshCore engine silently drops a column when two labels stringify alike (1 and '1') (src/freshdata/execution/backends/_freshcore.py:158)
• explain_clean reports 0 changed cells and blank dtypes for non-string column labels (src/freshdata/explain.py:36)
• enforce_contract and diff_schema mishandle non-string column labels (src/freshdata/enterprise/contracts.py:1674)
• apply_privacy_policy raises KeyError on non-string column labels (src/freshdata/enterprise/privacy_policy.py:474)
• explain_clean and infer_roles crash with TypeError on mixed int/str column labels (src/freshdata/explain.py:100)

Environment

freshdata 2.0.0; found at 55a8044 and re-checked on main c87efbd (2026-09-15). Reproduces on Python 3.9.6 / pandas 1.5.3 / numpy 1.26.4 and Python 3.12.14 / pandas 2.3.3 / numpy 2.5.3. The FreshCore item was run against a stand-in freshdata_freshcore module (no Rust toolchain available); see its details.

Suggested fix

• Normalise label handling in one shared helper (key every per-column mapping by the original label and only stringify for display), and use it in each function listed.
• Add one parametrised test that runs each API on a frame with integer labels and with mixed 1/"1" labels.

Details

1. fd.clean(semantic_mode=...) raises KeyError on non-string column labels

fd.clean works with integer column labels, but enabling the semantic layer crashes. build_semantic_context stores name = str(col), proposals carry that string, and run_semantic indexes the frame with it (out['2']). This is the same pattern as #209, which covers only StreamingCleaner and clean_timeseries; this is a separate code path.

Reproduction

import sys, warnings; warnings.simplefilter("ignore")
import pandas as pd
import freshdata as fd
df = pd.DataFrame({1: ["yes", "no", "yes", "no"], 2: ["twenty", "5", "6", "7"]})
fd.clean(df, column_names=False, verbose=False)  # control: core clean works
try:
    fd.clean(df, semantic_mode="auto", column_names=False, verbose=False); err = None
except Exception as e:
    err = f"{type(e).__name__}: {e}"
print("EXPECTED: semantic clean succeeds on integer labels (core clean does)")
print("ACTUAL:", err or "ok")

Expected

The semantic clean completes, as the core clean does.

Actual

EXPECTED: semantic clean succeeds on integer labels (core clean does)
ACTUAL: KeyError: '2'

Where

• src/freshdata/semantic/context.py:84: name = str(col)
• src/freshdata/semantic/apply.py:280: out[col] = _apply_column(out[col], mapping)

2. FreshCore engine silently drops a column when two labels stringify alike (1 and '1')

The FreshCore adapter sends column names as str(label) and rebuilds the output with a dict keyed by name. When two distinct labels have the same str(), such as the int 1 and the string "1", the second column overwrites the first. The report still says cols_after == 2, and columns_dropped is empty. The duplicate-label guard doesn't catch this because it checks raw labels, which differ here.

Reproduction

Needs the built freshdata_freshcore extension.

import pandas as pd, freshdata as fd

df = pd.DataFrame({1: [10.0, 20.0], "1": ["a", "b"]})
kw = dict(strategy="conservative", fix_dtypes=False, return_report=True, verbose=False)
print(list(fd.clean(df, engine="pandas", **kw)[0].columns))
out, rep = fd.clean(df, engine="freshcore", **kw)
print(list(out.columns), out.shape, rep.cols_after, rep.columns_dropped)

Expected

Two columns come back ([1, '1']), as with the pandas engine, or the run falls back.

Actual

['1'] (2, 1) 2 []: the numeric column is gone without any record.

Where

• src/freshdata/execution/backends/_freshcore.py:158: self._column_payload(str(name), ...).
• _freshcore.py:127: frame.columns.duplicated() checks raw labels only.
• _freshcore.py:204-213: data[name] = values, where a later column with the same name overwrites the earlier one.

3. explain_clean reports 0 changed cells and blank dtypes for non-string column labels

ExplainReport keys before_stats/after_stats by str(col) but keys cell_changes by the raw label. For integer column labels:

• to_frame() looks up cell_changes with the string key, so every column shows changed_cells = 0.
• The HTML view looks up the stats with the raw key, so both dtype cells are blank.

With MultiIndex columns, the tuple keys in cell_changes also make to_html() raise TypeError (from json.dumps), and _repr_html_() returns None, so the notebook shows nothing.

Reproduction

import pandas as pd, freshdata as fd

df = pd.DataFrame({0: [" a", "b ", "c", "d"], 1: [1.0, 2.0, 3.0, 4.0]})
rep = fd.explain_clean(df, verbose=False)
print(rep.cell_changes)          # {0: 2, 1: 0}
print(rep.to_frame())            # changed_cells 0 for column 0

mi = pd.DataFrame([[1, " a", None], [2, "b", 3.0]],
                  columns=pd.MultiIndex.from_tuples([("a", "x"), ("a", "y"), ("b", "z")]))
r2 = fd.explain_clean(mi, verbose=False)
print(r2._repr_html_())          # None
r2.to_html()                     # TypeError

Expected

Column 0 shows changed_cells = 2 with dtype object in both to_frame() and HTML, and the MultiIndex report renders.

Actual

cell_changes: {0: 2, 1: 0} | before_stats keys: ['0', '1']
column before_dtype after_dtype  changed_cells
     0       object      object              0
     1      float64     float64              0
HTML row for column 0: blank dtypes

_repr_html_() -> None | to_html(): TypeError: keys must be str, int, float, bool or None, not tuple

Where

• src/freshdata/explain.py:36: stats[str(col)] = entry
• src/freshdata/explain.py:80: changes[col] = changed (raw label)
• src/freshdata/explain.py:183: "changed_cells": self.cell_changes.get(col, 0) with a string col
• src/freshdata/render/renderers.py (render_explain): rep.before_stats.get(col, {}) with a raw col, then H.json_download("explain.json", rep.to_dict(), ...)

4. enforce_contract and diff_schema mishandle non-string column labels

Frames with integer column labels, for example from pd.read_csv(..., header=None), cannot be checked with contracts:

• Declaring the column as 0 reports it as missing.
• Declaring it as "0" raises KeyError.
• diff_schema raises KeyError as well.

Column profiles are keyed by str(col), but the frame is later indexed with that string.

Reproduction

import pandas as pd
import freshdata as fd

df = pd.DataFrame({0: [1, 2, 3], 1: ["a", "b", "c"]})
r = fd.enforce_contract(df, fd.DataContract("c", (fd.ColumnContract(0, dtype="int", min_value=0),)))
print([f.message for f in r.findings if f.status != "passed"])
fd.enforce_contract(df, fd.DataContract("c", (fd.ColumnContract("0", dtype="int", min_value=0),)))
fd.diff_schema(df, contract=fd.DataContract("c", (fd.ColumnContract("x"),)))

Expected

Column 0 is found and checked with no failures, and diff_schema returns a report saying x is missing.

Actual

{'0': ['required column 0 is missing'], "'0'": "KeyError: '0'", 'diff_schema': "KeyError: '0'"}

Where

Profiles are keyed by the string label, and the frame is then indexed with it:

• src/freshdata/enterprise/contracts.py:1674 builds str(col): _profile_column(frame[col], ...).
• contracts.py:1323 does passes &= _contract_values(findings, cc, frame[col]), which raises KeyError: '0'.
• contracts.py:1902 does new_family = _normalize_dtype(str(df[new].dtype)), which raises KeyError: '0' in diff_schema.

5. apply_privacy_policy raises KeyError on non-string column labels

classify_columns keys its results by str(col), and apply_privacy_policy then indexes the frame with that string. Any frame with integer (or other non-str) column labels that has a classified column crashes. Headerless CSVs and DataFrame(ndarray) are common sources of such labels. fd.clean accepts these frames.

Reproduction

import pandas as pd
from freshdata.enterprise import PrivacyPolicy, apply_privacy_policy, load_compliance_pack

pol = PrivacyPolicy(packs=(load_compliance_pack("hipaa"),), jurisdiction="US")
df = pd.DataFrame({0: ["a@b.com", "c@d.com"], 1: [1, 2]})
apply_privacy_policy(df, pol)

Expected

The policy is applied to column 0, giving ['<EMAIL>', '<EMAIL>'] as it does with str labels.

Actual

KeyError: '0'

Where

• src/freshdata/enterprise/privacy_policy.py:474 and :481: result[str(col)] = _ColumnClassification(...)
• src/freshdata/enterprise/privacy_policy.py:628: series = frame[col], using the stringified name.

6. explain_clean and infer_roles crash with TypeError on mixed int/str column labels

fd.clean accepts a frame whose column labels mix integers and strings. fd.explain_clean and fd.infer_roles crash on the same frame, because both sort the per-column context dict by label.

Reproduction

import pandas as pd, freshdata as fd

df = pd.DataFrame({0: [1.0, None, 3.0, 4.0], "name": ["a", "b", None, "d"]})
fd.clean(df, verbose=False)          # ok
fd.explain_clean(df, verbose=False)  # TypeError
fd.infer_roles(df)                   # TypeError

Expected

Both functions work on the frame fd.clean accepts.

Actual

ACTUAL: {'explain_clean': "TypeError: '<' not supported between instances of 'str' and 'int'", 'infer_roles': "TypeError: '<' not supported between instances of 'str' and 'int'"}

Where

• src/freshdata/explain.py:100: for col, ctx in sorted(contexts.items()):
• src/freshdata/api.py:1141: for col, ctx in sorted(contexts.items()):

[kevincostner17]

All six parts are fixed on main: part 1 (semantic repair) in #363, part 2 (FreshCore colliding labels) in #389, parts 3 and 6 (explain_clean stats keys, mixed int/str sort) in #381, part 4 (contracts) in #388, and part 5 (apply_privacy_policy) in #401.