Views
No views yet
lm_head) were quantized to NVFP4A16 using llmcompressor + compressed-tensors.This model card documents the local quantization and test performed on RTX 5060 Ti 16GB.
| Item | Value |
|---|---|
| Base model | Qwen/Qwen3-8B |
| Architecture | Qwen3ForCausalLM |
| Hidden size | 4096 |
| Layers | 36 |
| Attention heads | 32 (KV: 8) |
| Context length | 40,960 |
| Vocab size | 151,936 |
| Quantization format | NVFP4A16 |
| Compressed size | ~6.0 GB |
| Quantization config | compressed-tensors |
| Component | Configuration |
|---|---|
| GPU | NVIDIA GeForce RTX 5060 Ti 16 GB |
| CPU | Intel Xeon E5-2680 v4 |
| System RAM | 64 GB |
| Runtime | Docker + NVIDIA Container Runtime |
| Container image | vllm/vllm-openai:v0.22.0-ubuntu2404 |
1vllm serve /models/Qwen3-8B-NVFP4 \
2 --trust-remote-code \
3 --served-model-name Qwen3-8B \
4 --max-model-len 32768 \
5 --gpu-memory-utilization 0.93 \
6 --max-num-batched-tokens 8192 \
7 --max-num-seqs 4 \
8 --tensor-parallel-size 1 \
9 --enforce-eager \
10 --port 8000