You can deploy this model and start earning money today!

lhoang8500/
Qwen3-VL-8B-Instruct-NVFP4

Name: Qwen3-VL-8B-Instruct-NVFP4
Availability: InStock

safetensors

qwen3_vl

neuralmagic/calibration

Qwen/Qwen3-VL-8B-Instruct

Views

No views yet

Qwen3-VL-8B-Instruct-NVFP4

Intro

Original model: Qwen/Qwen3-VL-8B-Instruct
Quantization: NVFP4 using LLMCompressor
Activation precision: W4A4
GPU support: Blackwell GPUs only (e.g., RTX 50xx series)

This model is fully compatible with vLLM and optimized to run on a single GPU with 16GB VRAM, achieving ~1.5× faster performance compared to the INT4 version, with potentially better accuracy.

Calibration Data

python

DATASET_ID = "neuralmagic/calibration"
NUM_CALIBRATION_SAMPLES = 256
MAX_SEQUENCE_LENGTH = 8192

Tested on:

cuda: 13.0
vllm: 0.11.0
torch: 2.8.0
flashinfer-cubin:0.5.1
flashinfer-jit-cache:0.5.1+cu130
flashinfer-python:0.5.1
transformers:4.57.1

vllm serving command

VLLM_SLEEP_WHEN_IDLE=1 vllm serve lhoang8500/Qwen3-VL-8B-Instruct-NVFP4 --max-model-len 32768 -tp 1 --limit_mm_per_prompt '{"image":1, "video":0}' --kv-cache-dtype fp8 --gpu-memory-utilization 0.9 --max-num-seqs 64 --tool-call-parser hermes --enable-auto-tool-choice

Generation hyper parameteres

greedy='false'
seed=3407
top_p=0.8
top_k=20
temperature=0.7
repetition_penalty=1.0
presence_penalty=1.5
out_seq_length=32768

Quantization code

see file quantize.py, credit to llmcompressor