fix(qwen3_moe): export W4A16 NVFP4 checkpoints via NVFP4 MoE path - #176
fix(qwen3_moe): export W4A16 NVFP4 checkpoints via NVFP4 MoE path#176potoior wants to merge 1 commit into
Conversation
QUANT_NVFP4_A16 (nvfp4_a16) was introduced in 0.10.0 to distinguish W4A16 weight-only NVFP4 from W4A4 NVFP4, but Qwen3SparseMoeBlock._prepare_moe_weights only gated on QUANT_NVFP4. Mixed-precision W4A16_NVFP4 checkpoints such as Qwen3.6-35B-A3B-NVFP4 therefore fell through to the INT4 GPTQ Marlin repack and crashed with: AttributeError: 'NVFP4A16MarlinLinear' object has no attribute 'qweight' Two changes: - Treat QUANT_NVFP4_A16 as NVFP4 when gating the NVFP4 MoE plugin path. - Accept NVFP4A16MarlinLinear experts in repack_nvfp4_gated_moe_experts (their raw weight/weight_scale/weight_scale_2 buffers are the same ModelOpt NVFP4 format, so decode_modelopt_nvfp4 reuses them as-is). Verified: tensorrt-edgellm-export completes for Qwen3.6-35B-A3B-NVFP4 (llm).
|
Thanks for the contribution! We will formalize W4A4 vs W4A16 support in following releases. |
|
@nvluxiaoz Friendly ping — could you verify this fix directly? Once verified, this PR should be ready to merge. Thanks! |
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
Could we add a focused regression test for the QUANT_NVFP4_A16 MoE path?
This fix depends on two separate conditions staying aligned: Qwen3 must select the NVFP4 MoE path for QUANT_NVFP4_A16, and repack_nvfp4_gated_moe_experts must accept NVFP4A16MarlinLinear. A small test exercising those together would make it much harder for a future quant-type refactor to send W4A16 experts back through the INT4 path.
The successful 35B export is useful validation, but it would be good to have this behaviour pinned in CI as well.
Summary
QUANT_NVFP4_A16was introduced in 0.10.0 to distinguish W4A16 weight-onlyNVFP4 from W4A4 NVFP4, but
Qwen3SparseMoeBlock._prepare_moe_weightsonlygated on
QUANT_NVFP4. Mixed-precisionW4A16_NVFP4checkpoints such asQwen3.6-35B-A3B-NVFP4therefore fell through to the INT4 GPTQ Marlinrepack
and crashed with:
Changes
modeling_qwen3_moe.py: treatQUANT_NVFP4_A16as NVFP4 when gating theNVFP4 MoE plugin path.
repacking.py: acceptNVFP4A16MarlinLinearexperts inrepack_nvfp4_gated_moe_experts(their rawweight/weight_scale/weight_scale_2buffers are the same ModelOpt NVFP4 format, sodecode_modelopt_nvfp4reuses them as-is).Type of change
Bug fix
Verification
tensorrt-edgellm-export Qwen3.6-35B-A3B-NVFP4 --skip-audio --skip-visualcompletes successfully after the fix (40-layer MoE repacked into
Nvfp4MoePlugin).Compatibility
Backward compatible
Checklist
pre-commit run --all-filesandfixed any reported issues.