Skip to content

Round to_f64/to_f32 once, to the subnormal-aware precision#91

Merged
cmpute merged 3 commits into
cmpute:masterfrom
gaoflow:fix-subnormal-double-rounding
Jul 22, 2026
Merged

Round to_f64/to_f32 once, to the subnormal-aware precision#91
cmpute merged 3 commits into
cmpute:masterfrom
gaoflow:fix-subnormal-double-rounding

Conversation

@gaoflow

@gaoflow gaoflow commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

FBig/DBig → float conversion rounds the source to a fixed 53/24-bit intermediate and then lets f{32,64}::encode round again into the format's exponent-dependent subnormal grid. That second rounding is wrong in two ways. This is the subnormal counterpart of the normal-range double-rounding that was recently addressed.

Subnormal double-rounding (1 ULP). A subnormal has fewer than 53/24 significant bits — its spacing is fixed at 2^-1074 / 2^-149 — so the fixed intermediate keeps bits below the subnormal grid. For a value just past a subnormal halfway, the intermediate rounds to the halfway (dropping the excess that lifts it above) and encode then rounds half-to-even down, one ULP below the correctly-rounded value:

// value = (2m+1)·2^-1075 + 2^-1135, just above the halfway between subnormals m and m+1
let sig = (IBig::from(2 * m + 1) << 60) + IBig::ONE;
let v = FBig::<HalfEven, 2>::from_parts(sig, -1135);
v.to_f64(); // returned m·2^-1074 (rounded to even); correct is (m+1)·2^-1074

Debug panic / release double-round on high-precision inputs. to_f64/to_f32 feed the unrounded source significand into the base-changing division, whose debug_assert!(lhs.digits() <= self.precision + rhs.digits()) fires whenever the significand is wider than the target precision:

DBig::from_str("123456789012345678.9012345678901")?.to_f64(); // panics in debug builds

In release the assert is compiled out and the oversized quotient is re-rounded downstream — a silent double rounding.

Fix. Round the source once, straight to the target's precision at its own magnitude (min(53, e + 1075) bits for f64 at binary exponent e, analogously for f32), so encode re-rounds nothing. To keep that single rounding correct without an exact arbitrary-precision conversion, the source is first taken to a fixed, generous width with round-to-odd; rounding that down to the final width then reproduces the correctly-rounded result for every rounding mode. The division now rounds an over-wide quotient down to the requested precision instead of assuming the caller pre-bounded the dividend, so the assertion is removed and high-precision inputs convert without panicking (and without the release double-round).

Correctness holds to within the base conversion's working precision; a value closer to a boundary than the conversion resolves stays a pre-existing limitation of finite-precision base changing (shared with with_base). The base-2 / Repr paths are exact for all inputs.

Tests cover subnormal halfways nudged just past the boundary across the exponent range, in both base 2 and base 10, for f64 (oracle: float()) and f32 (oracle: exact Fraction → nearest 2^-149, not numpy.float32, which itself double-rounds), plus the high-precision decimals that previously panicked. cargo test --workspace --exclude dashu-python, cargo clippy --all-features --all-targets --workspace --exclude dashu-python -- -D warnings, and cargo fmt --all -- --check all pass.

The conversions rounded the source to a fixed 53/24-bit intermediate, then
the bit encoding rounded again into the exponent-dependent subnormal grid.
A value just past a subnormal halfway lands on the halfway after the first
rounding and the second rounds to even -- one ULP low. Round the source
once, straight to the target's precision at its magnitude (fewer than 53/24
bits for subnormals) through a round-to-odd base conversion, so the single
rounding is mode-correct and never double-rounds.

Also round an over-wide quotient down to the requested precision in the
base-changing division instead of requiring the caller to pre-bound the
dividend: high-precision decimals (e.g. "123456789012345678.9012345678901")
previously panicked in debug and double-rounded in release when converted.
- Cover negative inputs in the to_f{32,64} subnormal-halfway and
  high-precision tests (sign bit flip); the new over-wide-quotient path
  and subnormal rounding are sign-aware but were only tested positive.
- Reword the convert_base_odd doc: rounding down to width-2 reproduces
  the correctly-rounded value for every rounding mode (not just nearest).
- Fold f64/f32_significand_bits into a shared significand_bits helper
  parameterized by max_bits and the subnormal exponent.

Co-Authored-By: Claude <noreply@anthropic.com>
@cmpute
cmpute force-pushed the fix-subnormal-double-rounding branch from 8416569 to c8b79e1 Compare July 22, 2026 15:14
repr_div carries a documented precondition — lhs.digits() <= precision +
rhs.digits() — that callers must uphold (Context::div already pre-shrinks
the dividend to exactly this bound). The previous fix instead relaxed the
contract: it dropped the debug_assert and taught repr_div itself to round
an over-wide quotient down. That worked but departed from the layering —
repr_div is a lightweight hot kernel whose complexity belongs in the
informed caller, and the assert loss demoted a real invariant to an
implicit convention.

Restore repr_div's contract and assert, and fix the actual offender: the
to_f64/to_f32 path reaches repr_div via convert_base's small-exponent
division, which fed an oversized significand straight in. convert_base now
pre-shrinks the dividend to den.digits() + precision before dividing,
mirroring Context::div. Behavior is unchanged (the high-precision and
subnormal tests still pass, now through the caller path, with the
debug_assert active).

Co-Authored-By: Claude <noreply@anthropic.com>
@cmpute
cmpute merged commit 4b0bd88 into cmpute:master Jul 22, 2026
@cmpute

cmpute commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Thanks! Merged with some personal modifications

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants