This repository contains a locally quantized Qwen3-4B-Thinking-2507 model using NVFP4A16 / compressed-tensors for vLLM inference on NVIDIA Blackwell GPUs.

All Linear layers (except lm_head) were quantized to NVFP4A16 using llmcompressor + compressed-tensors. Use --reasoning-parser qwen3 for thinking mode.

This model card documents the local quantization and test performed on RTX 5060 Ti 16GB.

Quantization Summary

Item	Value
Base model	Qwen/Qwen3-4B-Thinking-2507
Architecture	`Qwen3ForCausalLM`
Hidden size	2560
Layers	36
Attention heads	32 (KV: 8)
Context length	262,144
Vocab size	151,936
Quantization format	NVFP4A16
Compressed size	~3.4 GB
Quantization config	`compressed-tensors`

Tested Hardware

Component	Configuration
GPU	NVIDIA GeForce RTX 5060 Ti 16 GB
CPU	Intel Xeon E5-2680 v4
System RAM	64 GB
Runtime	Docker + NVIDIA Container Runtime
Container image	`vllm/vllm-openai:v0.22.0-ubuntu2404`

Suggested vLLM Command

bash

vllm serve /models/Qwen3-4B-Thinking-2507-NFVP4 \
  --trust-remote-code \
  --served-model-name Qwen3-4B-Thinking \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.93 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 4 \
  --tensor-parallel-size 1 \
  --reasoning-parser qwen3 \
  --enforce-eager \
  --port 8000

Status

Quantized to NVFP4A16 (compressed-tensors)
Tested on RTX 5060 Ti 16 GB
Tested with vLLM v0.22.0 Docker image
Loads successfully in vLLM

murilonwt/Qwen3-4B-Thinking-2507-NFVP4

Qwen3-4B-Thinking-2507 NVFP4A16 — vLLM / RTX 5060 Ti

Quantization Summary

Tested Hardware

Suggested vLLM Command

Status

murilonwt/
Qwen3-4B-Thinking-2507-NFVP4