How are the quantization parameters of block_ap passed to the e2e-qp stage?

[tayton42]

Thank you for your work. I noticed that when loading the model between the two stages, load_checkpoint_in_model(model, checkpoint=model_path, device_map=device_map, offload_state_dict=True), all the scales and zero points were indicated as "were not used." Is this normal?

[ChenMnZ]

This is normal.

"were not used." is reported by

EfficientQAT/quantize/int_linear_real.py

Line 181 in 3917549

model = AutoModelForCausalLM.from_config(config=config,torch_dtype=torch.float16, trust_remote_code=True)

We manually load the model parameters at later code:

EfficientQAT/quantize/int_linear_real.py

Line 195 in 3917549

load_checkpoint_in_model(model,checkpoint=model_path,device_map=device_map,offload_state_dict=True)

[tayton42]

This is normal.

"were not used." is reported by

EfficientQAT/quantize/int_linear_real.py

Line 181 in 3917549

model = AutoModelForCausalLM.from_config(config=config,torch_dtype=torch.float16, trust_remote_code=True)
We manually load the model parameters at later code:

EfficientQAT/quantize/int_linear_real.py

Line 195 in 3917549

load_checkpoint_in_model(model,checkpoint=model_path,device_map=device_map,offload_state_dict=True)

Actually, my warning is like this,
2025-05-26 17:57:31,707 - accelerate.utils.modeling - WARNING - Some weights of the model checkpoint at tmp/qwen2505b-prefill/effi-qat/ were not used when initializing Qwen2ForCausalLM: {'model.layers.1.mlp.up_proj.weight_quantizer.zero_point', 'model.layers.6.self_attn.q_proj.weight_quantizer.scale', 'model.layers.20.mlp.down_proj.weight_quantizer.zero_point', 'model.layers.7.mlp.up_proj.weight_quantizer.scale', 'model.layers.18.mlp.up_proj.weight_quantizer.scale', 'model.layers.5.mlp.up_proj.weight_quantizer.scale', 'model.layers.21.self_attn.o_proj.weight_quantizer.scale', 'model.layers.17.self_attn.v_proj.weight', 'model.layers.14.mlp.down_proj.weight_quantizer.zero_point'...
now I understand this is normal, but how are these unloaded scale and zero_point parameters passed to the e2e-qp stage?

[ChenMnZ]

@tayton42
All scales and zero point are loaded after

EfficientQAT/quantize/int_linear_real.py

Line 195 in 3917549

load_checkpoint_in_model(model,checkpoint=model_path,device_map=device_map,offload_state_dict=True)

You can have a check about this.

[tayton42]

@tayton42 All scales and zero point are loaded after

EfficientQAT/quantize/int_linear_real.py

Line 195 in 3917549

load_checkpoint_in_model(model,checkpoint=model_path,device_map=device_map,offload_state_dict=True)
You can have a check about this.

So do I need to enable real_quant to load the quantization parameters? The model reloaded from AutoModelForCausalLM does not have the corresponding QuantLinear layer.