Views
No views yet
fp8_cast — amax set to FP8 range without data-driven calibration of K/V tensors). Matches the convention used by nvidia/Qwen3-32B-NVFP4, nvidia/Qwen3.5-397B-A17B-NVFP4, nvidia/Gemma-4-31B-IT-NVFP4.| Component | Format |
|---|---|
| Language model linear weights/activations (full + linear-attention transformer blocks) | NVFP4 (W4A4, group size 16, FP8 E4M3 scales) |
| KV cache | FP8 (E4M3) |
Vision encoder (model.visual.*, 27 SigLIP-style blocks) | BF16 (untouched) |
| Vision-to-LM projector + image/video embeddings | BF16 |
lm_head | BF16 |
Linear-attention conv1d (48 of 64 layers in this hybrid model) | BF16 |
MTP head (mtp, mtp.layers.0) | BF16 |
Routers / mlp.gate.* | BF16 |
hf_quant_config.json (exclude_modules).1vllm serve berkerdooo/Qwen3.6-27B-NVFP4 \
2 --quantization modelopt \
3 --tensor-parallel-size 2 \
4 --trust-remote-codeqwen3_5 model type (architecture id is Qwen3_5ForConditionalGeneration).algorithm: max takes max(amax_text, amax_image) per block, so order does not matter):nvidia/Nemotron-VLM-Dataset-v2, subsets: sparsetables, plotqa_cot, wiki_en). Real images are processed by AutoProcessor, pixel_values are passed through full_model.forward(...), exercising the vision encoder, the vision-to-LM projector, and the LM with real projected vision tokens at the head of context.abisee/cnn_dailymail). Plain tokenized articles flow through the extracted language_model.forward(...), giving clean amax estimates for channels that vision tokens do not stress.nvidia/Qwen3.5-397B-A17B-NVFP4) with the multimodal coverage that an image-aware deployment actually exercises at inference time.1# Patched hf_ptq.py to chain --calib_with_images + --dataset.
2# When both flags are present, _ChainedCalibLoaders runs the VLM forward
3# through full_model and the text forward through language_model.
4
5CUDA_LAUNCH_BLOCKING=1 CUDA_VISIBLE_DEVICES=0,1 \
6python examples/llm_ptq/hf_ptq.py \
7 --pyt_ckpt_path Qwen/Qwen3.6-27B \
8 --qformat nvfp4 \
9 --calib_with_images \
10 --dataset cnn_dailymail \
11 --calib_size 128 \
12 --batch_size 1 \
13 --gpu_max_mem_percentage 0.70 \
14 --trust_remote_code \
15 --export_path ./qwen3.6-27b-nvfp4hf_ptq.py calls extract_and_prepare_language_model_from_vl(), which walks the model.model.language_model lineage and attaches a disabled quantizer config to every non-LM submodule (vision tower + projector + embeddings) so the export keeps them in BF16. The _default_disabled_quantizer_cfg (in modelopt/torch/quantization/config.py) already covers *linear_attn.conv1d*, *mixer.conv1d*, *lm_head*, *router*, *output_layer*, BatchNorm, LeakyReLU. MTP layers (mtp, mtp.layers.0) are detected at export time and added to the exclusion list automatically.uint8weight_scale)weight_scale_2)qwen3_5 model type is only recognized by transformers ≥5.0)Model-Optimizer main, with a small local patch to chain VLM and text calibration loaders)