Duplicate row index labels crash or overwrite rows in four APIs

[kevincostner17]

Summary

Frames with a non-unique row index (common after pd.concat without ignore_index=True) break label-based alignment in these functions. Two of them crash, one silently copies one row's value over another, and one finds no candidates at all.

Affected functions

• Semantic date-order consistency check crashes fd.clean on a duplicate row index (src/freshdata/semantic/consistency.py:127)
• clean_text overwrites rows that share an index label with one row's cleaned value (src/freshdata/textclean.py:335)
• validate_fields crashes on frames with duplicate index labels (src/freshdata/fieldcheck.py:726)
• suggest_join_keys finds no candidates when a frame has duplicate index labels (src/freshdata/enterprise/join_assist.py:217-238)

Environment

freshdata 2.0.0; found at 55a8044 and re-checked on main c87efbd (2026-09-15). Reproduces on Python 3.9.6 / pandas 1.5.3 / numpy 1.26.4 and Python 3.12.14 / pandas 2.3.3 / numpy 2.5.3.

Suggested fix

• Operate on positional arrays (.to_numpy() / reset_index(drop=True) internally, restoring the original index on output) instead of index-aligned assignment.
• Add a duplicate-index case to each function's tests.

Details

1. Semantic date-order consistency check crashes fd.clean on a duplicate row index

With the semantic layer on, _check_date_pair_ordering reads cells with df.at[row, col]. On a duplicated index label (for example after pd.concat) that returns a Series, and if end_deviates and not start_deviates raises. _check_fahrenheit_in_celsius has the same pattern (float(numeric.at[row])). The same frame with a unique index cleans fine.

Reproduction

import sys, warnings; warnings.simplefilter("ignore")
import pandas as pd
import freshdata as fd
n = 10
df = pd.DataFrame({"start_date": pd.date_range("2024-01-01", periods=n).astype(str),
                   "end_date": pd.date_range("2024-02-01", periods=n).astype(str)},
                  index=[0, 0, 1, 1, 2, 2, 3, 3, 4, 4])
df.iloc[3, 1] = "2023-01-01"
fd.clean(df.reset_index(drop=True), semantic_mode="auto", verbose=False)  # control
try:
    fd.clean(df, semantic_mode="auto", verbose=False); err = None
except Exception as e:
    err = f"{type(e).__name__}: {e}"
print("EXPECTED: clean completes (same frame with a unique index does)")
print("ACTUAL:", err or "ok")

Expected

The clean completes and the ordering break is reported as a warning.

Actual

EXPECTED: clean completes (same frame with a unique index does)
ACTUAL: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Where

• src/freshdata/semantic/consistency.py:127: start_deviates = df.at[row, start_col] != start_modal
• src/freshdata/semantic/consistency.py:261: value = float(numeric.at[row])

2. clean_text overwrites rows that share an index label with one row's cleaned value

fd.clean_text collects changes in a dict keyed by index label, so a later row with the same label overwrites an earlier one. out.loc[list(cleaned_values), col] = pd.Series(cleaned_values) then writes the surviving value into every row with that label. ' alice'/'bob ' at index [7, 7] become ['bob', 'bob'], while the report still lists alice and bob as the cleaned values.

Reproduction

import sys, warnings; warnings.simplefilter("ignore")
import pandas as pd
import freshdata as fd
df = pd.DataFrame({"t": [" alice", "bob "]}, index=[7, 7])
out, rep = fd.clean_text(df)
print("EXPECTED: ['alice', 'bob']")
print("ACTUAL:", out["t"].tolist(), "| report changes:", [(c["original"], c["cleaned"]) for c in rep.changes])

Expected

['alice', 'bob'].

Actual

EXPECTED: ['alice', 'bob']
ACTUAL: ['bob', 'bob'] | report changes: [(' alice', 'alice'), ('bob ', 'bob')]

Where

• src/freshdata/textclean.py:335: out.loc[list(cleaned_values), col] = pd.Series(cleaned_values)

3. validate_fields crashes on frames with duplicate index labels

validate_fields looks up suspect cells with series.loc[idx], which returns a Series when the label is duplicated. _check_value then evaluates pd.isna(raw) in a boolean context, which raises ValueError. The same frame with a unique index validates normally.

Reproduction

import sys, warnings; warnings.simplefilter("ignore")
import pandas as pd
import freshdata as fd
df = pd.DataFrame({"x": ["ok", "bad id!", "ok2"]}, index=[0, 0, 1])
fd.validate_fields(df.reset_index(drop=True), {"x": "identifier"})  # control
try:
    fd.validate_fields(df, {"x": "identifier"}); err = None
except Exception as e:
    err = f"{type(e).__name__}: {e}"
print("EXPECTED: a report flagging 'bad id!'")
print("ACTUAL:", err or "ok")

Expected

A report flagging bad id!.

Actual

EXPECTED: a report flagging 'bad id!'
ACTUAL: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Where

• src/freshdata/fieldcheck.py:726: found = _check_value(col, idx, series.loc[idx], cleaned,
• src/freshdata/fieldcheck.py:720: raw = series.loc[idx]

4. suggest_join_keys finds no candidates when a frame has duplicate index labels

fd.suggest_join_keys reads cells with frame.at[index_label, column]. When a frame has duplicate index labels, which is common after pd.concat, that returns a Series instead of a scalar. The string of the Series is then compared, so exact matches like "Acme Corp" and "Acme Corp" produce no candidates.

Reproduction

import pandas as pd
import freshdata as fd

left = pd.DataFrame({"company": ["Acme Corp", "Zenith Ltd"]}, index=[0, 0])
right = pd.DataFrame({"company": ["Acme Corp"]})
rep = fd.suggest_join_keys(left, right, on=["company"])
print([(c.left_index, c.right_index, c.score) for c in rep.candidates], len(rep.matches))

Expected

Acme Corp in left is a confident match for Acme Corp in right.

Actual

candidates=[] matches=0

Where

src/freshdata/enterprise/join_assist.py:217-238:

for idx, row in left.iterrows():
    left_blocks.setdefault(block_key(row), []).append(idx)
...
    c: _similarity(str(left.at[li, c]), str(right.at[ri, c])) for c in on

[kevincostner17]

All four parts are fixed on main: part 1 (semantic consistency) in #363, and parts 2–4 (clean_text, validate_fields, suggest_join_keys) in #383.