Round to_f64/to_f32 once, to the subnormal-aware precision#91
Merged
Conversation
The conversions rounded the source to a fixed 53/24-bit intermediate, then the bit encoding rounded again into the exponent-dependent subnormal grid. A value just past a subnormal halfway lands on the halfway after the first rounding and the second rounds to even -- one ULP low. Round the source once, straight to the target's precision at its magnitude (fewer than 53/24 bits for subnormals) through a round-to-odd base conversion, so the single rounding is mode-correct and never double-rounds. Also round an over-wide quotient down to the requested precision in the base-changing division instead of requiring the caller to pre-bound the dividend: high-precision decimals (e.g. "123456789012345678.9012345678901") previously panicked in debug and double-rounded in release when converted.
- Cover negative inputs in the to_f{32,64} subnormal-halfway and
high-precision tests (sign bit flip); the new over-wide-quotient path
and subnormal rounding are sign-aware but were only tested positive.
- Reword the convert_base_odd doc: rounding down to width-2 reproduces
the correctly-rounded value for every rounding mode (not just nearest).
- Fold f64/f32_significand_bits into a shared significand_bits helper
parameterized by max_bits and the subnormal exponent.
Co-Authored-By: Claude <noreply@anthropic.com>
cmpute
force-pushed
the
fix-subnormal-double-rounding
branch
from
July 22, 2026 15:14
8416569 to
c8b79e1
Compare
repr_div carries a documented precondition — lhs.digits() <= precision + rhs.digits() — that callers must uphold (Context::div already pre-shrinks the dividend to exactly this bound). The previous fix instead relaxed the contract: it dropped the debug_assert and taught repr_div itself to round an over-wide quotient down. That worked but departed from the layering — repr_div is a lightweight hot kernel whose complexity belongs in the informed caller, and the assert loss demoted a real invariant to an implicit convention. Restore repr_div's contract and assert, and fix the actual offender: the to_f64/to_f32 path reaches repr_div via convert_base's small-exponent division, which fed an oversized significand straight in. convert_base now pre-shrinks the dividend to den.digits() + precision before dividing, mirroring Context::div. Behavior is unchanged (the high-precision and subnormal tests still pass, now through the caller path, with the debug_assert active). Co-Authored-By: Claude <noreply@anthropic.com>
Owner
|
Thanks! Merged with some personal modifications |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FBig/DBig→ float conversion rounds the source to a fixed 53/24-bit intermediate and then letsf{32,64}::encoderound again into the format's exponent-dependent subnormal grid. That second rounding is wrong in two ways. This is the subnormal counterpart of the normal-range double-rounding that was recently addressed.Subnormal double-rounding (1 ULP). A subnormal has fewer than 53/24 significant bits — its spacing is fixed at
2^-1074/2^-149— so the fixed intermediate keeps bits below the subnormal grid. For a value just past a subnormal halfway, the intermediate rounds to the halfway (dropping the excess that lifts it above) andencodethen rounds half-to-even down, one ULP below the correctly-rounded value:Debug panic / release double-round on high-precision inputs.
to_f64/to_f32feed the unrounded source significand into the base-changing division, whosedebug_assert!(lhs.digits() <= self.precision + rhs.digits())fires whenever the significand is wider than the target precision:In release the assert is compiled out and the oversized quotient is re-rounded downstream — a silent double rounding.
Fix. Round the source once, straight to the target's precision at its own magnitude (
min(53, e + 1075)bits for f64 at binary exponente, analogously for f32), soencodere-rounds nothing. To keep that single rounding correct without an exact arbitrary-precision conversion, the source is first taken to a fixed, generous width with round-to-odd; rounding that down to the final width then reproduces the correctly-rounded result for every rounding mode. The division now rounds an over-wide quotient down to the requested precision instead of assuming the caller pre-bounded the dividend, so the assertion is removed and high-precision inputs convert without panicking (and without the release double-round).Correctness holds to within the base conversion's working precision; a value closer to a boundary than the conversion resolves stays a pre-existing limitation of finite-precision base changing (shared with
with_base). The base-2 /Reprpaths are exact for all inputs.Tests cover subnormal halfways nudged just past the boundary across the exponent range, in both base 2 and base 10, for f64 (oracle:
float()) and f32 (oracle: exactFraction→ nearest2^-149, notnumpy.float32, which itself double-rounds), plus the high-precision decimals that previously panicked.cargo test --workspace --exclude dashu-python,cargo clippy --all-features --all-targets --workspace --exclude dashu-python -- -D warnings, andcargo fmt --all -- --checkall pass.