fix(engines): disclose ingestion fallbacks; validate engine/output_format pairs

[kevincostner17]

Summary

#206: inputs native engines can't ingest faithfully.

• A pandas object column holding mixed value types (e.g. [1, "a", None]) crashed the Polars engine inside pl.from_pandas. DuckDB silently cast it to text (1 became '1').
• Duplicate column labels also crashed Polars, and DuckDB returned them wrongly renamed (['x_2', 'x_1']).
• The new execution/_ingest.py::pandas_ingest_fallback_reason flags both cases before ingestion. The engines then take the recorded pandas fallback, so output matches the pandas engine and report.fallback_events names the column.
• fallback_policy="error" still raises before any pandas work.
• The Polars engine now decides fallbacks before converting the source.
• Single-type object columns stay native.

#205: a mismatched native handle silently returned the wrong type, even under fallback_policy="error". For example, engine="duckdb", output_format="polars-lazy" returned a DuckDBPyRelation.

• EngineConfig now rejects an explicit engine paired with another engine's handle format, raising ValueError (the chosen option for Mismatched engine/output_format silently returns the wrong type, even with fallback_policy='error' #205). The message names the right engine.
• engine="auto", or the default engine, picks the engine that owns the format: fd.clean(df, output_format="duckdb") returns a relation.
• _convert_output only returns a materialized frame in place of a handle when the report records the pandas fallback. Anything else raises instead of substituting silently.

Docs:

• docs/fallback-matrix.md lists the two input-driven fallbacks and points at _ingest.py.
• docs/backends.md says a handle format needs its own engine.

Tests

tests/test_execution/test_ingest_and_output_format.py:

• A mixed object column falls back with pandas-equal values; duplicate labels match pandas (['x', 'x_2']) on Polars and DuckDB.
• The error policy blocks the fallback, and single-type object columns stay native.
• EngineConfig rejects every foreign engine/handle pair, including under fallback_policy="error".
• auto and the default engine select the owning engine, and a handle request that falls back returns disclosed pandas output.

Verification

• Full lane pytest -m "not online and not large" on Python 3.12 (numpy 2.5) and 3.9 (pandas 1.5.3).
  • Only local-only failures remain: the sandbox tests that need an installed package.
  • Two mixed-column cases failed in those runs because the test compared a CleanResult wrapper with a plain DataFrame. That test is fixed and passes 22/22 on both Python versions.
• Targeted engine and streaming tests re-run after rebasing onto fix(streaming): exact large integers and non-string column labels #221.
• ruff check ., mypy src/freshdata and mkdocs build --strict pass.

This touches backends/_duckdb.py and _polars.py in different hunks from #223 and #224. Whichever merges later gets rebased.

Closes #205
Closes #206

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 753fa39d-a1d0-4474-9887-9569078949bb

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)