Skip to content

fix(qwen3_moe): export W4A16 NVFP4 checkpoints via NVFP4 MoE path - #176

Open
potoior wants to merge 1 commit into
NVIDIA:mainfrom
potoior:fix/qwen3moe-nvfp4-a16
Open

fix(qwen3_moe): export W4A16 NVFP4 checkpoints via NVFP4 MoE path#176
potoior wants to merge 1 commit into
NVIDIA:mainfrom
potoior:fix/qwen3moe-nvfp4-a16

Conversation

@potoior

@potoior potoior commented Aug 13, 2026

Copy link
Copy Markdown

Summary

QUANT_NVFP4_A16 was introduced in 0.10.0 to distinguish W4A16 weight-only
NVFP4 from W4A4 NVFP4, but Qwen3SparseMoeBlock._prepare_moe_weights only
gated on QUANT_NVFP4. Mixed-precision W4A16_NVFP4 checkpoints such as
Qwen3.6-35B-A3B-NVFP4 therefore fell through to the INT4 GPTQ Marlin
repack
and crashed with:

  AttributeError: 'NVFP4A16MarlinLinear' object has no attribute
  'qweight'

Changes

  • modeling_qwen3_moe.py: treat QUANT_NVFP4_A16 as NVFP4 when gating the
    NVFP4 MoE plugin path.
  • repacking.py: accept NVFP4A16MarlinLinear experts in
    repack_nvfp4_gated_moe_experts (their raw weight/weight_scale/
    weight_scale_2 buffers are the same ModelOpt NVFP4 format, so
    decode_modelopt_nvfp4 reuses them as-is).

Type of change

Bug fix

Verification

tensorrt-edgellm-export Qwen3.6-35B-A3B-NVFP4 --skip-audio --skip-visual
completes successfully after the fix (40-layer MoE repacked into
Nvfp4MoePlugin).

Compatibility

Backward compatible

Checklist

  • I have run the hooks manually with pre-commit run --all-files and
    fixed any reported issues.
  • All tests are passing.
  • The change is backward compatible

QUANT_NVFP4_A16 (nvfp4_a16) was introduced in 0.10.0 to distinguish W4A16
weight-only NVFP4 from W4A4 NVFP4, but Qwen3SparseMoeBlock._prepare_moe_weights
only gated on QUANT_NVFP4. Mixed-precision W4A16_NVFP4 checkpoints such as
Qwen3.6-35B-A3B-NVFP4 therefore fell through to the INT4 GPTQ Marlin repack and
crashed with:

  AttributeError: 'NVFP4A16MarlinLinear' object has no attribute 'qweight'

Two changes:
- Treat QUANT_NVFP4_A16 as NVFP4 when gating the NVFP4 MoE plugin path.
- Accept NVFP4A16MarlinLinear experts in repack_nvfp4_gated_moe_experts (their
  raw weight/weight_scale/weight_scale_2 buffers are the same ModelOpt NVFP4
  format, so decode_modelopt_nvfp4 reuses them as-is).

Verified: tensorrt-edgellm-export completes for Qwen3.6-35B-A3B-NVFP4 (llm).
@potoior
potoior requested a review from a team August 13, 2026 10:46
@nvluxiaoz

Copy link
Copy Markdown
Collaborator

Thanks for the contribution! We will formalize W4A4 vs W4A16 support in following releases.

@potoior

potoior commented Sep 1, 2026

Copy link
Copy Markdown
Author

@nvluxiaoz Friendly ping — could you verify this fix directly? Once verified, this PR should be ready to merge. Thanks!

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a focused regression test for the QUANT_NVFP4_A16 MoE path?

This fix depends on two separate conditions staying aligned: Qwen3 must select the NVFP4 MoE path for QUANT_NVFP4_A16, and repack_nvfp4_gated_moe_experts must accept NVFP4A16MarlinLinear. A small test exercising those together would make it much harder for a future quant-type refactor to send W4A16 experts back through the INT4 path.

The successful 35B export is useful validation, but it would be good to have this behaviour pinned in CI as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants