NVFP4 GGUF quantization of Qwen3.5-9B, Alibaba Cloud's efficient 9B multimodal foundation model with 262K context, 201 languages, and hybrid Gated DeltaNet + Gated Attention architecture.
Optimized for NVIDIA Blackwell GPUs with native FP4 tensor core acceleration.
About NVFP4
What is NVFP4?
NVFP4 is NVIDIA's native 4-bit floating-point quantization format introduced with the Blackwell architecture (SM120+). Unlike traditional integer quantization (Q4_0, Q4_K_M, etc.), NVFP4 stores weights in FP4 (E4M3) format — a 4-bit floating-point representation with 1 sign bit, 4 exponent bits, and 3 mantissa bits.
The key difference from INT4 formats:
Property
NVFP4 (FP4)
INT4 (Q4_X)
Representation
Floating-point
Integer
Dynamic range
~±240
~±7
Block size
16
32
Scale format
FP16 (E4M3)
FP16
Hardware support
Blackwell SM120+ (native tensor cores)
All GPUs (software)
Zero-shot perplexity
Near-identical to FP16
Slight degradation
Why use NVFP4?
Blackwell-native acceleration: NVFP4 is processed natively on Blackwell FP4 tensor cores, delivering up to 2x throughput vs INT4 software kernels on the same hardware.
Better dynamic range: Floating-point 4-bit preserves more information for outlier weights compared to integer quantization, resulting in lower perplexity degradation.
Memory efficiency: At ~4.74 bits per weight (BPW), a 9B model fits in ~5 GB — well within 16 GB VRAM with room for 262K context.
No dequantization overhead: Unlike INT4 formats that require runtime dequantization, FP4 operates directly on tensor cores for both compute and memory bandwidth.
When to use NVFP4 vs other formats
NVFP4: Best choice if you have a Blackwell GPU (RTX 5060 Ti, RTX 5090, B200, etc.)
Q4_K_M / Q4_0: Better for pre-Blackwell GPUs (Ampere, Ada Lovelace) or CPU inference
Q8_0 / F16: Use when maximum quality is needed and memory is not a constraint
Hybrid architecture: Alternating linear attention (Gated DeltaNet) and full attention layers for efficient long-context processing
262K context window: Native support for 262,144 token sequences
Vision capabilities: Built-in vision encoder with 27-layer ViT for image understanding
Multi-Token Prediction: MTP head enabling speculative decoding for faster generation
Multilingual: Strong performance across 201 languages
Usage
llama.cpp (CLI)
bash
1# Text + Image2llama-cli \3 -m qwen3.5-9b-nvfp4.gguf \4 --mmproj mmproj-qwen3.5-9b-nvfp4-f16.gguf \5 --image photo.jpg \6 -p "Describe this image in detail"78# Text only9llama-cli \10 -m qwen3.5-9b-nvfp4.gguf \11 -p "Explain quantum computing in simple terms"\12 -n 5121314# OpenAI-compatible server15llama-server \16 -m qwen3.5-9b-nvfp4.gguf \17 --mmproj mmproj-qwen3.5-9b-nvfp4-f16.gguf \18 --port 8080
llama-cpp-python
python
1from llama_cpp import Llama
23llm = Llama.from_pretrained(4 repo_id="FreedomAISVR/Qwen3.5-9B-NVFP4-GGUF",5 filename="qwen3.5-9b-nvfp4.gguf",6 n_gpu_layers=-1,7)89response = llm.create_chat_completion([10{"role":"user","content":"What is the capital of France?"}11])12print(response["choices"][0]["message"]["content"])
