Huihui Mistral Medium 3.5 128B Abliterated NVFP4
Experimental ModelOpt NVFP4 quantization of a Mistral3/Pixtral multimodal model.
Status
Text generation: working
Vision/image input: working after repairing vision-tower Q/K permutation
vLLM serving: working with TP=2 and TP=4
Quantization format: ModelOpt NVFP4
Tested backend: vLLM 0.19.1rc1.dev310+g8ad6ff003
Tested Hardware
2x/4x NVIDIA RTX PRO 6000 Blackwell Workstation Edition
96 GiB VRAM per GPU
Tensor parallel size: 2, 4
CUDA Graph enabled
Quantization: --quantization modelopt
Recommended vLLM Command
1 CUDA_VISIBLE_DEVICES = 4,5 \
2 NCCL_P2P_DISABLE = 1 \
3 NCCL_NVLS_ENABLE = 0 \
4 NCCL_IB_DISABLE = 1 \
5 NCCL_CUMEM_ENABLE = 0 \
6 vllm serve /path/to/Huihui-Mistral-Medium-3.5-128B-abliterated-NVFP4 \
7 --served-model-name huihui-mistral-medium-3.5-128b-nvfp4 \
8 --trust-remote-code \
9 --tensor-parallel-size 2 \
10 --quantization modelopt \
11 --gpu-memory-utilization 0.90 \
12 --max-model-len 8192 \
13 --max-num-seqs 8 \
14 --disable-custom-all-reduce \
15 --host 127.0 .0.1 \
16 --port 8000
The NCCL environment variables above were required on the tested Blackwell workstation to avoid TP=2 NCCL initialization hangs.
For TP=4, use four GPUs and set --tensor-parallel-size 4:
1 CUDA_VISIBLE_DEVICES = 0,1 ,4,5 \
2 NCCL_P2P_DISABLE = 1 \
3 NCCL_NVLS_ENABLE = 0 \
4 NCCL_IB_DISABLE = 1 \
5 NCCL_CUMEM_ENABLE = 0 \
6 vllm serve /path/to/Huihui-Mistral-Medium-3.5-128B-abliterated-NVFP4 \
7 --served-model-name huihui-mistral-medium-3.5-128b-nvfp4 \
8 --trust-remote-code \
9 --tensor-parallel-size 4 \
10 --quantization modelopt \
11 --gpu-memory-utilization 0.90 \
12 --max-model-len 8192 \
13 --max-num-seqs 8 \
14 --disable-custom-all-reduce \
15 --host 127.0 .0.1 \
16 --port 8000
Context Length
Measured with gpu_memory_utilization=0.90, CUDA Graph enabled.
TP --max-model-lenResult Notes 2 8,192 OK Good interactive/default serving setting 2 131,072 OK Long-context serving works, max concurrency about 1.19x 2 160,000 Failed vLLM reported insufficient KV cache memory 4 262,144 OK Official config limit; max concurrency about 2.50x 4 524,288 Blocked Above model config max_position_embeddings=262144
vLLM reported for TP=2:
1 available KV cache memory: 26.25 GiB
2 estimated maximum model length: 156384
vLLM reported for TP=4 at --max-model-len 262144:
1 model loading took 28.47 GiB per GPU
2 available KV cache memory: 54.92 GiB
3 GPU KV cache size: 654,384 tokens
4 maximum concurrency for 262,144 tokens per request: 2.50x
Practical recommendation:
Use 8192 for interactive WebUI and concurrent serving.
Use 131072 for long-context tests.
Treat 156384 as the measured theoretical ceiling for this TP=2/96GiBx2 setup.
Use TP=4 for the full 262144 context length. Longer contexts require overriding the model config and were not validated.
Throughput TP=2
vLLM online serving benchmark, OpenAI chat endpoint:
Dataset: random
Input length: 1024 tokens
Output length: 256 tokens
Prompts: 16
--ignore-eos--temperature 0Server: --max-model-len 8192 --max-num-seqs 8
Max concurrency Output TPS Peak output TPS Total token TPS Mean TTFT Mean TPOT 1 19.83 tok/s 21.00 tok/s 127.29 tok/s 538.81 ms 48.50 ms 2 40.45 tok/s 42.00 tok/s 259.63 tok/s 135.81 ms 49.08 ms 4 78.62 tok/s 80.00 tok/s 504.55 tok/s 152.20 ms 50.46 ms 8 151.44 tok/s 160.00 tok/s 971.92 tok/s 125.18 ms 52.37 ms
Raw benchmark files:
1 /media/tonoken/CT4000/benchmarks/huihui_nvfp4_tp2/serve_c1.json
2 /media/tonoken/CT4000/benchmarks/huihui_nvfp4_tp2/serve_c2.json
3 /media/tonoken/CT4000/benchmarks/huihui_nvfp4_tp2/serve_c4.json
4 /media/tonoken/CT4000/benchmarks/huihui_nvfp4_tp2/serve_c8.json
Throughput TP=4
vLLM online serving benchmark, OpenAI chat endpoint:
Dataset: random
Input length: 1024 tokens
Output length: 256 tokens
Prompts: 16
--ignore-eos--temperature 0Server: --max-model-len 8192 --max-num-seqs 8
Max concurrency Output TPS Peak output TPS Total token TPS Mean TTFT Mean TPOT 1 34.16 tok/s 38.00 tok/s 219.23 tok/s 506.51 ms 27.40 ms 2 69.67 tok/s 72.00 tok/s 447.13 tok/s 97.66 ms 28.42 ms 4 134.91 tok/s 140.00 tok/s 865.82 tok/s 105.67 ms 29.34 ms 8 243.33 tok/s 248.00 tok/s 1561.69 tok/s 127.23 ms 32.49 ms
Multimodal Notes
This model includes Pixtral vision components:
model.vision_towermodel.multi_modal_projectorPixtralProcessorimage tokens: [IMG], [IMG_BREAK], [IMG_END]
The vision tower must also have GGUF Q/K row permutation repaired. Text-only repair of model.language_model.layers.*.self_attn.{q,k}_proj is not enough. Without repairing:
model.vision_tower.transformer.layers.*.attention.{q,k}_projthe model may identify coarse colors but fail badly on UI/OCR-like images.
After repairing the vision tower Q/K tensors, the model correctly read a KIMI Code console screenshot including:
KIMI CodeConsoleWeekly usageRate limit detailsModeratoK2.6API Keys
Known Caveats
ModelOpt NVFP4 checkpoint support in vLLM is experimental.
--reasoning-parser mistral did not work with this HF-converted directory because vLLM expected a MistralTokenizer-compatible tokenizer file.Reasoning High can be passed through chat_template_kwargs, but reasoning text may appear in normal assistant content rather than a separate API field.
TP=2 required NCCL P2P/NVLS/IB/CUMEM workarounds on the tested system.
Support the Base Model Author
Ko-fi: https://ko-fi.com/huihuiai Bitcoin: bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge 
