fix(er): strict pandas blocking parser, NaT-safe comparisons, quoted link keys

[kevincostner17]

Summary

Fixes five entity-resolution bugs in src/freshdata/enterprise/entity_resolution.py.

• resolve_entities(backend='duckdb') rejects valid DuckDB blocking SQL #236 — DuckDB backend rejected valid blocking SQL. The pandas parser runs on every backend to attribute blocking_rule_ids, and _parse_blocking_rules didn't catch its EntityResolutionError. Non-equality DuckDB rules such as jaro_winkler_similarity(l.name, r.name) > 0.8 therefore aborted the run. The error is now caught; rules the parser can't read are simply not attributed, as documented. The pandas backend still raises.
• pandas entity-resolution blocking silently returns zero pairs for <=, != and OR rules #237 — pandas blocking silently returned zero pairs. _parse_blocking split on the first =, so l.id <= r.id became the column id <, which is missing for every record. The parser is now strict:
  • It splits on AND and on =/== outside quoted identifiers and string literals.
  • Each side must be a column reference (l.col, r."quoted col", col) or lower/upper/trim/left/right/substr over one.
  • OR, <, <=, >, >=, !=, <>, literals and arithmetic raise EntityResolutionError.
  • Every existing equality rule (templates, benchmarks, examples, tests) still parses.
  • Quoted identifiers unescape "".
  • A rule that references a column absent from the frame now logs a warning.
• Entity resolution scores two missing datetimes (NaT) as full agreement and merges records #238 — two NaT values scored as agreement. _is_missing only recognised None and float NaN. It now treats every scalar pandas missing value (NaN, NaT, pd.NA, numpy NaT) as missing, in both scoring and blocking keys.
• fd.link(backend='duckdb') fails on key columns containing spaces or hyphens #266 — fd.link(backend="duckdb") failed on keys containing spaces or hyphens. The default blocking SQL now double-quotes identifiers, escaping " as "".
• link_entities reports omit thresholds, so review queues sort around a hardcoded midpoint #271 — link_entities reports omitted thresholds. runtime_metadata now includes match_threshold, clerical_review_threshold and scoring, as resolve_entities does. The external fd.link report includes its thresholds too, so build_review_queue sorts around the configured midpoint instead of 0.75.

Behaviour changes

• Stricter pandas blocking: rules that are not pure column equalities now raise instead of returning zero or wrong pairs. That covers OR, comparison operators, literals, arithmetic, BETWEEN, parenthesised predicates, and unquoted names containing spaces or hyphens. Quote such names, e.g. l."first name" = r."first name".
• Missing values: NaT and pd.NA fields no longer count as agreement, so records previously merged on missing datetimes can now split.

Tests

• tests/test_enterprise_entity_resolution.py:
  • DuckDB non-equality blocking with correct attribution of the equality rules
  • rejected pandas rules (parametrised)
  • pandas vs DuckDB pair parity for valid equality rules
  • quoted identifiers containing spaces, "", = and and
  • the unknown-column warning
  • _is_missing unit cases
  • NaT and pd.NA in comparisons and blocking keys
  • link_entities metadata and review-queue order
• tests/test_link.py:
  • exact links on the keys first name, e-mail, say "hi" and select, on both backends
  • multi-key exact and fuzzy links with spaced keys, on both backends
  • thresholds recorded for fuzzy and external links
• DuckDB tests use pytest.importorskip("duckdb").

Verification

• ruff check .: all checks passed
• mypy src/freshdata: no issues in 202 source files
• pytest -m "not online and not large", Python 3.12: 4298 passed, 6 skipped
• pytest -m "not online and not large", Python 3.9 / pandas 1.5.3: 4294 passed, 10 skipped
• Each issue's reproduction fails on main and passes on this branch, on both interpreters.

Closes #236
Closes #237
Closes #238
Closes #266
Closes #271

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: be1cd457-1b30-449a-a953-937973897343

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)