Skip to content

fix(missforest): keep integer dtypes by rounding imputed values - #384

Merged
kevincostner17 merged 1 commit into
mainfrom
fix/missforest-nullable-int
Sep 15, 2026
Merged

kevincostner17 merged 1 commit into
mainfrom
fix/missforest-nullable-int

Conversation

@kevincostner17

Copy link
Copy Markdown
Contributor

Summary

impute="missforest" turned nullable integer columns (Int64, Int32, ...) into object dtype holding a mix of Python ints and fractional floats, with no note in the report. MissForest fills from a float64 working frame. Series.where refuses float predictions on a masked integer array, so _assign_success fell back to astype(object). _fallback_fill did the same when the median was fractional.

For numpy and nullable integer columns, both paths now:

  • round imputed values half-to-even to Python ints and keep the column's own dtype;
  • use exact_int_stat for the fallback median when the column holds values beyond 2**53, so no digits are lost;
  • note the rounding in the action rationale and set a new rounded_to_integer metadata flag. The flag is True only when regressor predictions or a median were actually rounded.

Numeric columns that still cannot hold the fill are cast to float64 with a rationale note, never to object. Float, bool, categorical and string targets are unchanged.

Tests

New tests/test_missforest_nullable_int.py (skipped without scikit-learn):

  • Int64/Int32/Int16 columns keep their dtype, imputed values are integers, observed values are untouched, and the report note and flag are present
  • integer predictions equal the rounded float predictions for the same frame
  • float columns keep fractional predictions and have no rounding note
  • numpy int64 columns come back unchanged
  • the small-frame fallback keeps Int64, rounds a 3.5 median to 4, and records the note
  • values beyond 2**53 stay exact in both the fallback and regressor paths

Verification

Closes #263

MissForest fills from a float64 working frame, so regressor predictions
for a nullable Int64 column are fractional floats. Series.where refuses
them on the masked integer array and _assign_success fell back to
astype(object), leaving a column that mixed Python ints with fractional
floats and had no report note. _fallback_fill had the same object
fallback for a fractional median.

For numpy and nullable integer columns both paths now round the imputed
values half-to-even to Python ints and store them in the column's own
dtype. The fallback median uses exact_int_stat for columns holding values
beyond 2**53, so no digits are lost. The action rationale notes the
rounding and the metadata gains a rounded_to_integer flag, set only when
regressor predictions or a median were actually rounded.

Numeric columns that still cannot hold the fill are cast to float64 with
a rationale note instead of object. Float, bool, categorical and string
targets keep their existing behaviour.

Closes #263
@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 5b4cefb6-4fdf-48f4-8ea7-8e4387f59fed


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

FreshData benchmark report — performance

  • freshdata: ?
  • python: ?
  • platform: ?
fixture n_rows n_cols p50 s p95 s peak MB repair % false-repair % preserve % trust monotonic export %

Authored-code reduction (Metric 6)

@kevincostner17
kevincostner17 merged commit e3927a4 into main Sep 15, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

impute='missforest' turns nullable Int64 columns into object dtype with fractional floats

1 participant