Skip to content

fix(impute): reject unknown impute_strategy columns; single fallback record for predictor-less MissForest columns - #373

Merged
kevincostner17 merged 1 commit into
mainfrom
fix/impute-column-validation
Sep 15, 2026
Merged

kevincostner17 merged 1 commit into
mainfrom
fix/impute-column-validation

Conversation

@kevincostner17

Copy link
Copy Markdown
Contributor

Summary

  • Pipeline.impute(columns=) and impute_strategy silently ignore unknown column names #310impute_strategy keys (including those produced by Pipeline.impute(columns=)) that match no column were silently ignored: a pre-rename name like "Age" after normalize_columns(), or a typo like "agee", imputed nothing with no error, warning or report action. run_pipeline now validates the keys right after column renaming and raises a ValueError in the same style as duplicate_subset: it lists the unknown keys and available columns and, when a key is a pre-normalization name, suggests the normalized one ('Age' -> 'age'). Validation runs before any drop step, so keys for columns a later step removes are still accepted, and before any mutation of the input.
  • MissForest on a single-column frame logs the column as imputed twice and claims a model #324 — MissForest on a frame with no predictor columns (a one-column frame) filled the column via the fallback, then also logged it as a random-forest regressor/classifier imputation and appended it to columns_imputed twice. Such columns are now filled once by the simple fallback before scikit-learn is loaded and removed from the model loop, so the report shows a single missforest_fallback action and scikit-learn is not required when every column falls back.

Behaviour change

Configs with impute_strategy keys that name no column (after renaming) now raise instead of being silently ignored. This includes a reused config across frames where the column is missing from some of them.

Tests

New tests/test_impute_column_validation.py:

Verification

  • ruff check . — clean
  • mypy src/freshdata — no issues (202 files)
  • pytest -m "not online and not large" — py3.12: 4258 passed, 6 skipped; py3.9: 4254 passed, 10 skipped

Closes #310
Closes #324

@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: e3b4ef50-4f6c-409c-a193-b7e6b7ccc09d


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

FreshData benchmark report — performance

  • freshdata: ?
  • python: ?
  • platform: ?
fixture n_rows n_cols p50 s p95 s peak MB repair % false-repair % preserve % trust monotonic export %

Authored-code reduction (Metric 6)

…t fallback for predictor-less columns

impute_strategy keys (also produced by Pipeline.impute(columns=)) were
matched against str(column) inside the missing-value step, and keys that
matched nothing were silently ignored. A pre-rename name such as "Age"
after column normalization, or a typo such as "agee", imputed nothing
with no error, warning or report action. run_pipeline now validates the
keys right after column renaming (and before any drop step, so keys for
columns an earlier step later drops remain valid) and raises a ValueError
in the same style as duplicate_subset: it lists the unknown keys and the
available columns, and when a key is a pre-normalization name it
suggests the normalized one.

MissForestImputer kept a column with no predictor columns (a one-column
frame) in the eligible set. The model loop fell back to simple
imputation, then _assign_success recorded the column again as a
random-forest regressor/classifier imputation and appended it to
columns_imputed twice. Such columns are now filled once by
_fallback_fill before scikit-learn is loaded and removed from the
eligible set, so the report shows a single missforest_fallback action and
scikit-learn is not required when every column falls back.

Closes #310
Closes #324
@kevincostner17
kevincostner17 force-pushed the fix/impute-column-validation branch from c33a011 to 7c91790 Compare September 15, 2026 09:18
@kevincostner17
kevincostner17 merged commit e38f65b into main Sep 15, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant