fix(missforest): keep integer dtypes by rounding imputed values - #384
Merged
Merged
Conversation
MissForest fills from a float64 working frame, so regressor predictions for a nullable Int64 column are fractional floats. Series.where refuses them on the masked integer array and _assign_success fell back to astype(object), leaving a column that mixed Python ints with fractional floats and had no report note. _fallback_fill had the same object fallback for a fractional median. For numpy and nullable integer columns both paths now round the imputed values half-to-even to Python ints and store them in the column's own dtype. The fallback median uses exact_int_stat for columns holding values beyond 2**53, so no digits are lost. The action rationale notes the rounding and the metadata gains a rounded_to_integer flag, set only when regressor predictions or a median were actually rounded. Numeric columns that still cannot hold the fill are cast to float64 with a rationale note instead of object. Float, bool, categorical and string targets keep their existing behaviour. Closes #263
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
impute="missforest"turned nullable integer columns (Int64,Int32, ...) intoobjectdtype holding a mix of Python ints and fractional floats, with no note in the report. MissForest fills from a float64 working frame.Series.whererefuses float predictions on a masked integer array, so_assign_successfell back toastype(object)._fallback_filldid the same when the median was fractional.For numpy and nullable integer columns, both paths now:
exact_int_statfor the fallback median when the column holds values beyond 2**53, so no digits are lost;rounded_to_integermetadata flag. The flag is True only when regressor predictions or a median were actually rounded.Numeric columns that still cannot hold the fill are cast to float64 with a rationale note, never to
object. Float, bool, categorical and string targets are unchanged.Tests
New
tests/test_missforest_nullable_int.py(skipped without scikit-learn):Int64/Int32/Int16columns keep their dtype, imputed values are integers, observed values are untouched, and the report note and flag are presentint64columns come back unchangedInt64, rounds a 3.5 median to 4, and records the noteVerification
ruff check .clean;mypy src/freshdatacleanpytest -m "not online and not large": Python 3.12 4446 passed / 6 skipped; Python 3.9 4442 passed / 10 skipped (before rebasing onto current main; MissForest suites re-run after the rebase on both)missforest.pyCloses #263