Fix disk offload crash on FP8 tensors - #4151
Open
shoemoney wants to merge 1 commit into
Open
Conversation
offload_weight sizes FP8 tensors fine via dtype_byte_size, but crashes on write with 'Got unsupported ScalarType Float8_e4m3fn' since NumPy has no FP8 representation. Mirror the existing bfloat16 workaround: reinterpret the 1 byte of FP8 data as int8 on write and restore the original dtype on read, keyed off the torch.float8_ prefix rather than an explicit list so it stays in sync with the five variants dtype_byte_size already knows about.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Disk-offloading a model with FP8 weights crashes in
offload_weight:Same error for
float8_e5m2. NumPy has no FP8 representation, soweight.cpu().numpy()fails outright.
This is the identical problem
bfloat16already has, andoffload_weightalready has abranch for it: view the tensor as
int16, write that, and record the real dtype in theindex so
load_offloaded_weightcan view it back. FP8 hits the same NumPy gap but has noequivalent branch.
The reason this is easy to miss:
dtype_byte_size(src/accelerate/utils/modeling.py)already understands FP8 (correctly returns 1 byte for
float8_e4m3fn,float8_e5m2,float8_e4m3fnuz,float8_e5m2fnuz,float8_e8m0fnu— the last three added in #4063). Sodevice-map planning happily sizes an FP8 model, decides some layers go to disk, and then
the write itself blows up. The planning path knows about FP8; the write path doesn't.
Fix
Mirror the bfloat16 idiom on both sides of
offload.py:offload_weight: if the dtype string starts withtorch.float8_, view the tensor asint8before handing it to NumPy, and record the real dtype name in the index (matchingwhat's already done for bfloat16 -> int16).
load_offloaded_weight: if the recorded dtype starts withfloat8_, load the memmap asint8and view it back to the original FP8 dtype viagetattr(torch, dtype).Judgment call worth flagging: I matched on the
float8_prefix rather than listingthe five variants explicitly.
dtype_byte_sizealready tracks five FP8 dtypes (two of themadded by #4063 after the original three), and a hardcoded pair/list here would just be
another place for that list to drift out of sync as new FP8 variants land. Happy to switch
this to an explicit list (mirroring the
dtype_byte_sizeset) if maintainers would ratherbe conservative about it.
Precedent
The int-view-and-restore trick is the established pattern for this exact class of bug —
it's what closed issue #454 ("Disk offload fails with bfloat16 weights") produced for
bfloat16. This PR applies the same fix to FP8.
Test plan
Extended the existing dtype loop in
test_offload_weight(tests/test_offload.py) toinclude
float8_e4m3fn/float8_e5m2, guarded withis_torch_version(">=", "2.1.0")+hasattr(torch, name)— the same guard style used by #4063.torch.randn(..., dtype=float8_*)raises
NotImplementedError, so the FP8 tensors are produced the way they actually show upin practice: cast down from a float32 tensor via
.to(dtype).Round-trip correctness is checked on raw bits (
.view(int8/int16/int32)), not floatequality, since float equality can't be trusted to catch a byte-level corruption for these
dtypes anyway.
Manually verified (beyond what's in the test) with NaN/inf-adjacent bit patterns injected
directly, including a full sweep of all 256 possible FP8 byte values:
Confirmed the test actually discriminates: reverted only the source change (kept the new
test), and it fails with the original crash:
Full suite results on this branch:
tests/test_offload.py: 4 passedtests/test_modeling_utils.py: 42 passed, 2 skipped (GPU-only)ruff check/ruff format --check(pinned 0.13.1): cleanWho can review?
@SunMarc @muellerzr