fix(streaming): real recent-window dedup, dtype-stable row hashes, drift for constant columns

[kevincostner17]

Summary

Fixes three streaming bugs in StreamingCleaner.

• global_duplicates keeps the first window_size rows forever instead of a recent window #292: global_duplicates kept the first window_size rows forever.
  • Cause: the cross-batch hash set only accepted new hashes while it had room and never evicted anything. After window_size distinct rows, duplicates of recent rows passed through for the rest of the stream.
  • Fix: the window is now an OrderedDict. A repeated row moves to the most-recent end, and the least recently seen rows are evicted, so it is the bounded recent window the docs describe.
  • Duplicates within a batch are handled as before.
• global_duplicates misses cross-batch duplicates when a column flips int64/float64 #293: cross-batch duplicates were missed when a column flipped between int64 and float64.
  • Cause: hash_pandas_object hashes dtype-specific bytes, so 1 and 1.0 hashed differently (for example, an int column promoted to float by a missing value).
  • Fix: rows are now hashed with numeric non-bool columns (nullable ones included) converted to float64, missing values as NaN, and -0.0 as 0.0.
  • Integer columns holding values beyond ±2**53 keep their dtype, so distinct large integers are never merged.
• Streaming distribution drift never fires for a column that was constant in earlier batches #294: distribution drift never fired for a column that had been constant.
  • Cause: the z-score check returned early when the running std was 0.
  • Fix: for a constant column with at least two prior values, a batch mean that moves away from the constant by more than 1e-9 * max(1, |mean|) is now reported as high-risk distribution drift.

Docstrings for global_duplicates and drift_zscore, and docs/streaming.md, are updated to match.

Behaviour changes:

• Duplicates of recent rows are now removed for the whole stream, and rows older than the window can reappear.
• Int/float dtype flips no longer hide duplicates.
• A constant column that changes now raises a drift action, a warning and a drift_log entry.
• No serialized format changes: the seen-hash window is in memory only, not part of state_, StreamingState.to_dict() or any checkpoint.

Tests

New module tests/test_streaming_dedup_drift.py (15 tests):

• global_duplicates keeps the first window_size rows forever instead of a recent window #292:
  • the issue repro
  • duplicates after more than window_size distinct rows
  • eviction of the oldest rows, with the window staying bounded
  • a repeat refreshes recency
  • a batch larger than the window
• global_duplicates misses cross-batch duplicates when a column flips int64/float64 #293:
  • the issue repro (a float batch promoted by NaN)
  • float then int
  • nullable Int64 vs float
  • -0.0 vs 0
  • distinct values not merged
  • integers beyond 2**53 stay distinct
• Streaming distribution drift never fires for a column that was constant in earlier batches #294:
  • the issue repro (a constant 5 jumps to 1,000,000)
  • an unchanged constant reports nothing
  • a change within float tolerance reports nothing
  • a single prior value reports nothing

10 of the 15 fail on main; the other 5 guard against false positives.

Verification

• ruff check .: all checks passed
• mypy src/freshdata: no issues in 202 source files
• pytest -m "not online and not large", py3.12 / pandas 2.3: 4254 passed, 6 skipped
• pytest -m "not online and not large", py3.9 / pandas 1.5: 4250 passed, 10 skipped
• All three issue repros were confirmed on both lanes before the fix and pass after.

Closes #292
Closes #293
Closes #294

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 55756c33-96e4-4901-8aee-fdbd68cc1b1b

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)