This repository contains a quantized version of Qwopus3.6-27B-v2 (based on the Qwen3.5 architecture). The model has been quantized from BF16 to NVFP4 format using the NVIDIA TensorRT Model Optimizer (modelopt).

To maintain high performance and accuracy, a hybrid precision strategy was applied: the core LLM weights are quantized to 4-bit, while the Vision Encoder and Multi-Token Prediction (MTP) modules are strictly preserved in their native BF16 precision.

The resulting checkpoint is fully compatible with vLLM for high-throughput, low-latency inference.

Hardware Recommendation

Target Hardware: Highly recommended for NVIDIA Blackwell architecture GPUs (e.g., NVIDIA RTX 50 series, RTX PRO series, and GB100/GB10 chips) which feature native hardware support for FP4.

Model Specifications

Original Size: ~54 GB (BF16)
Quantized Size: ~18 GB (~0.33x compression ratio)
Format: Hugging Face safetensors (compatible with vLLM via --quantization modelopt)
Preserved Modules: Vision tower and MTP heads remain in BF16 to prevent degradation in image understanding and speculative decoding efficiency.

Key Design Decisions & Rationale

Decision	Rationale
Vision encoder unquantized	Visual features are highly precision-sensitive. Quantizing the vision tower severely degrades image understanding and multimodal alignment.
MTP modules unquantized	Multi-Token Prediction (speculative decoding) heads are small in parameter count. Quantizing them yields minimal memory savings but causes significant accuracy loss.
MTP written as separate shard	Isolated into `model_mtp.safetensors`. Safetensors round-tripping during standard saving can corrupt existing float8 scale binary representations.
256 samples / 2048 length	Calibrated using the `neuralmagic/calibration` dataset (LLM split). This setup balances calibration quality perfectly against GPU time.

Quantization Pipeline (7 Steps)

The quantization process followed a strict 7-step engineering pipeline:

Load Model: Loaded the full Qwen3_5ForConditionalGeneration model in BF16 onto the GPU.
Prepare Calibration Data: Sampled 256 conversation sequences from neuralmagic/calibration, tokenized with the official chat template (Max length: 2,048 tokens).
Configure Quantization Rules: Extended NVFP4_DEFAULT_CFG with explicit disable rules for *visual* (vision encoder) and *mtp* (multi-token prediction heads).
Quantize & Calibrate: Executed mtq.quantize() with a forward loop across all 256 calibration samples, collecting per-layer activation statistics (amax values) to compute static FP4 scaling factors.
Export Checkpoint: Serialized quantized weights using export_hf_checkpoint(), outputting FP4 weights and float8 input_scales.
Graft MTP Weights: Extracted all mtp.* tensors from the original BF16 model and wrote them into a separate shard (model_mtp.safetensors). Updated model.safetensors.index.json accordingly to prevent loading/saving artifacts.
Patch Config: Registered all MTP module names under quantization_config.ignore in config.json, instructing vLLM to automatically bypass quantization and load these layers in BF16.

Deployment via vLLM

You can serve this model using vLLM with the following command:

bash

python3 -m vllm.entrypoints.openai.api_server \
    --model /path/to/Qwopus3.6-27B-v2-NVFP4 \
    --served-model-name Qwopus3.6-27B-v2-NVFP4 \
    --quantization modelopt \
    --trust-remote-code \
    --dtype auto \
    --gpu-memory-utilization 0.93 \
    --max-model-len 112000 \
    --max-num-seqs 4 \
    --max-num-batched-tokens 8192 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --default-chat-template-kwargs '{"enable_thinking": true}' \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
    --reasoning-parser qwen3 \
    --host 0.0.0.0 \
    --port 8000

crushleorey/Qwopus3.6-27B-v2-NVFP4

Qwopus3.6-27B-v2-NVFP4

Overview

Hardware Recommendation

Model Specifications

Key Design Decisions & Rationale

Quantization Pipeline (7 Steps)

Deployment via vLLM

crushleorey/
Qwopus3.6-27B-v2-NVFP4