vllm serve RedHatAI/Kimi-K3-NVFP4 \
  --tensor-parallel-size 8 \
  --trust_remote_code \
  --load-format instanttensor \
  --reasoning-parser kimi_k3 \
  --language-model-only  # optional

May require https://github.com/vllm-project/vllm/pull/50500 to run

Creation Process

This model was compressed using LLM Compressor. For more information, please contact Kyle Sayers via the vLLM Developers Slack or ksayers@redhat.com.

Evaluation

Benchmark	`moonshotai/Kimi-K3`	`RedHatAI/Kimi-K3-NVFP4`
GPQA	93.5	91.0

bash

inspect eval hf/Idavidrein/gpqa/diamond \
  --model vllm/RedHatAI/Kimi-K3-NVFP4 \
  --reasoning-effort high \
  --model-base-url http://localhost:8000/v1

Benchmarking

631451072-ab73c651-6f7e-4029-bfa8-9a73fecc1f7a