Views
No views yet
win10/K1-31B-v5
Scheme: FP8_DYNAMIC — weights quantized to FP8 statically, activations scaled dynamically at runtime (W8A8).
How it was made: One-shot datafree quantization with LLM Compressor on a DGX Spark (GB10 Grace Blackwell, 128GB unified memory). No calibration data required.Multimodal projection layers and MoE routers kept in bf16. Blackwell (GB10/B100/B200) has native FP8 hardware support. Hopper (H100/H200) also supports FP8 natively.
1sudo docker run \
2 --gpus all \
3 --network host \
4 --ipc host \
5 --restart unless-stopped \
6 --ulimit memlock=-1 \
7 --ulimit stack=67108864 \
8 -e NCCL_IGNORE_CPU_AFFINITY=1 \
9 vllm/vllm-openai:gemma4-0505-cu130 \
10 Firworks/K1-31B-v5-fp8 \
11 --gpu-memory-utilization 0.90 \
12 --max-num-seqs 4 \
13 --max-num-batched-tokens 4096 \
14 --enable-chunked-prefill \
15 --enable-prefix-caching \
16 --trust-remote-code \
17 --reasoning-parser gemma4 \
18 --tool-call-parser gemma4 \
19 --enable-auto-tool-choice \
20 --default-chat-template-kwargs '{"enable_thinking": true}' \
21 --mm-processor-kwargs '{"max_soft_tokens": 1120}' \
22 --max-model-len 131072 \
23 --port 8000 \
24 --host 0.0.0.0