Trusted, reproducible NVFP4 quantization of Qwen/Qwen3.5-4B (Apache-2.0) produced with NVIDIA ModelOpt 0.45 for Blackwell inference (vLLM quantization=modelopt_fp4).

This is a text-chat oriented build of the official VLM wrapper (Qwen3_5ForConditionalGeneration). The vision tower is left unquantized / unused at serve time — send text only.

Why this exists

This release exists to provide a known base + auditable calibration for Qwen3.5-4B in ModelOpt NVFP4 form. It publishes the exact preset, public calibration mix, seed, hashes, and measured speed/quality so the recipe can be re-run independently.

Recipe

Item	Value
Base	`Qwen/Qwen3.5-4B`
Tool	NVIDIA ModelOpt `0.45.0`
Preset	`NVFP4_DEFAULT_CFG` (max calib; group size 16)
KV cache quant	none (`kv_cache_quant_algo: null`)
Excludes	`lm_head`, vision / multimodal modules, `linear_attn.conv1d` / `in_proj_*` (ModelOpt defaults + export)
Calibration	512 public samples, seed 20260730
Mix	~70% `HuggingFaceH4/ultrachat_200k` `train_sft` + ~30% `HuggingFaceFW/fineweb-edu` `sample-10BT`, chat-templated, max seq 4096
Calib SHA-256	`762d54c48ad951318dc641aeb0f2091729bb6f8008ef89126c74ac41eea591b2` (`calib.jsonl`)
Quant host	RTX 5060 Ti (SM120), CUDA 12.8, driver 610.43, torch `2.11.0+cu128`

Scripts live with the producer workspace (build_calib.py, quantize_modelopt.py). AWQ_LITE was attempted but export hit a Qwen fuse/pre_quant_scale shape error; DEFAULT (max) is the published path.

Serve (vLLM ≥ 0.25 recommended; tested 0.26.0)

bash

python -m vllm.entrypoints.openai.api_server \
  --model LostGentoo/Qwen3.5-4B-NVFP4 \
  --served-model-name LostGentoo/Qwen3.5-4B-NVFP4 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code

vLLM auto-detects ModelOpt NVFP4 (quantization=modelopt_fp4). On Blackwell you should see:

Using CutlassNvFp4LinearKernel for NVFP4 GEMM

Hardware: native FP4 GEMM requires NVIDIA Blackwell. Non-Blackwell will not get the intended speedup.

Tip: Qwen3.5 may emit a “Thinking Process” preamble. Prefer higher max_tokens for evals, or disable thinking in the chat template if your stack supports enable_thinking=false / /no_think.

Measured — gentoo 1× RTX 5060 Ti (SM120), vLLM 0.26.0

Server-side deltas via vllm_probe.py delta + ctxsweep / prefill_probe (unique prompts).

Decode aggregate @6144 ctx

Conc	Agg tok/s
1	87
4	218
8	292
16	341
24	355

Whole-matrix server delta decode ≈ 334 tok/s. Prefill peak @6144 ≈ 17.1k tok/s (conc1 fit ≈ 16.2k). 0 preemptions / sweep cell errors. Kernel: CutlassNvFp4LinearKernel.

Quality (thinking disabled where noted)

Check	Result
50 synthetic product-support spot prompts	50/50 coherent replies
IFEval (`chat_smoke`, n=541, thinking on)	prompt_level_strict 38.3%, inst_level_strict 49.5%
IFEval (`chat_smoke`, n=541, `enable_thinking=false`)	prompt_level_strict 72.1%, inst_level_strict 79.9%
Needle-in-haystack	Pass @4096 with long completion budget; thinking-on was the main source of flakiness
MMLU-Pro (`max_samples=1000`, `enable_thinking=false`)	NVFP4 31.2%, bf16 28.7%, AWQ 28.5%
MMLU-Pro (`max_samples=3000`, `enable_thinking=false`)	NVFP4 31.27%, bf16 29.23% (Δ +2.03 pp)
MMLU-Pro gate vs bf16	Fails the plan threshold: NVFP4 stays ~+2 pp vs bf16 on both 1k and 3k subsets (target `<=1.5 pp`)

Cloud Run RTX PRO 6000 spot

Service qwen35-4b-axion-nvfp4-b6000, vLLM v0.24.0 image + stdlib proxy, max-num-seqs=128. Native fp4_gemm.

Decode @6144	Agg tok/s
C1	162
C16	1006
C64	1486

Prefill peak @6144: 69.7k tok/s. 0 errors / preemptions. Scaled to minScale=0 after the spot.

5060 Ti remains a consumer SM120 signal; B6000 is the G4-class capacity number.

Files

model.safetensors — NVFP4 weights (+ unquantized excluded modules)
hf_quant_config.json — ModelOpt producer metadata
config.json — architectures: [Qwen3_5ForConditionalGeneration], quantization_config.quant_algo: NVFP4
Tokenizer / processor / chat template from the base model

Limitations

VLM architecture quirk: not a pure ForCausalLM checkpoint (vLLM text-only path still works).
Public UltraChat/FineWeb calib — not domain-tuned for any private corpus.
Default thinking mode can burn completion budget; configure your client accordingly.
Consumer Blackwell ≠ datacenter B6000 bandwidth; re-bench on target SKU before capacity planning.

Acknowledgements

Base model: Qwen team (Qwen/Qwen3.5-4B)
Quantization: NVIDIA ModelOpt
Structural recipe validated against vLLM loading behavior and the exported hf_quant_config.json layout

Citation

bibtex

@misc{lostgentoo_qwen35_4b_nvfp4,
  title  = {Qwen3.5-4B NVFP4 (ModelOpt, public calib)},
  author = {LostGentoo},
  year   = {2026},
  url    = {https://huggingface.co/LostGentoo/Qwen3.5-4B-NVFP4}
}