Skip to content

fix(glm4_moe): load compressed-tensors NVFP4 expert checkpoints - #430

Open
gberasmus87 wants to merge 1 commit into
FlashML-org:mainfrom
gberasmus87:glm4-moe-compressed-tensors
Open

fix(glm4_moe): load compressed-tensors NVFP4 expert checkpoints#430
gberasmus87 wants to merge 1 commit into
FlashML-org:mainfrom
gberasmus87:glm4-moe-compressed-tensors

Conversation

@gberasmus87

Copy link
Copy Markdown

fix(glm4_moe): load compressed-tensors NVFP4 expert checkpoints

glm4_moe is the only NVFP4 family that never learned the llm-compressor spelling.
Its expert key pattern matches weight | weight_scale | weight_scale_2 and its
source spec sets neither kind_map nor global_reciprocal, so on a
compressed-tensors export -- gesong2077/GLM-4.5-Air-NVFP4, whose experts are
weight_packed + weight_scale + weight_global_scale -- the only tensors that
match are the weight_scale ones, whose name the two dialects happen to share.
The packed weights and the global scales are missed entirely, so the offload
banks fill with scales and no weights.

The machinery already exists; glm4_moe just is not wired to it.
Nvfp4ExpertSourceSpec.kind_map folds weight_packed -> weight and
weight_global_scale -> weight_scale_2, and global_reciprocal inverts the
quant-side global scale at ingest -- without which the expert dequant is inverted
and the model emits a single repeated token. glm5_next carries exactly this pair
for its own compressed-tensors release; this mirrors it, including selecting the
spec from quant_method.

input_global_scale deliberately does not match: the routed-expert path is W4A16
and never quantizes activations.

detect_expert_quant already recognises these exports, so parse_config needs
no change -- expert_quant is nvfp4 either way. The reader was the only gap.

Verified on gesong2077/GLM-4.5-Air-NVFP4 (compressed-tensors, config_groups
num_bits=4 type=float group_size=16 strategy=tensor_group), against its real
weight index: 69,120 routed-expert tensors, of which 17,280 are
input_global_scale. Before, the spec matched 17,280 -- every one a
weight_scale, no weights at all. After, it matches all 51,840 non-activation
tensors, folding 17,280 weight_packed -> weight and 17,280
weight_global_scale -> weight_scale_2, and still rejects every
input_global_scale. The modelopt path is untouched: an export without
quant_method == "compressed-tensors" still gets the original spec.

glm4_moe is the only NVFP4 family that never learned the llm-compressor spelling.
Its expert key pattern matches `weight | weight_scale | weight_scale_2` and its
source spec sets neither `kind_map` nor `global_reciprocal`, so on a
compressed-tensors export -- `gesong2077/GLM-4.5-Air-NVFP4`, whose experts are
`weight_packed` + `weight_scale` + `weight_global_scale` -- the only tensors that
match are the `weight_scale` ones, whose name the two dialects happen to share.
The packed weights and the global scales are missed entirely, so the offload
banks fill with scales and no weights.

The machinery already exists; glm4_moe just is not wired to it.
`Nvfp4ExpertSourceSpec.kind_map` folds `weight_packed` -> `weight` and
`weight_global_scale` -> `weight_scale_2`, and `global_reciprocal` inverts the
quant-side global scale at ingest -- without which the expert dequant is inverted
and the model emits a single repeated token. glm5_next carries exactly this pair
for its own compressed-tensors release; this mirrors it, including selecting the
spec from `quant_method`.

`input_global_scale` deliberately does not match: the routed-expert path is W4A16
and never quantizes activations.

`detect_expert_quant` already recognises these exports, so `parse_config` needs
no change -- `expert_quant` is `nvfp4` either way. The reader was the only gap.

Verified on `gesong2077/GLM-4.5-Air-NVFP4` (compressed-tensors, config_groups
num_bits=4 type=float group_size=16 strategy=tensor_group), against its real
weight index: 69,120 routed-expert tensors, of which 17,280 are
`input_global_scale`. Before, the spec matched 17,280 -- every one a
`weight_scale`, no weights at all. After, it matches all 51,840 non-activation
tensors, folding 17,280 `weight_packed` -> `weight` and 17,280
`weight_global_scale` -> `weight_scale_2`, and still rejects every
`input_global_scale`. The modelopt path is untouched: an export without
`quant_method == "compressed-tensors"` still gets the original spec.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant