Property	Value
Base Model	deepseek-ai/deepseek-moe-16b-chat
Architecture	Mixture of Experts (64 experts, 6 active + 2 shared)
Total Parameters	~16B
Active Parameters	~2.8B
Quantization Method	GPTQ
Format	`float-quantized` (FP8 E4M3)
Strategy	`tensor`
Weight Bits	8 (FP8)
Activation Bits	8 (FP8)
Group Size	N/A (per-tensor)
Calibration Data	512 samples from `HuggingFaceH4/ultrachat_200k` (max seq len 2048)

Quantization Recipe

Quantized with llm-compressor v0.11.0 on an NVIDIA H200 GPU.

python

from llmcompressor.modifiers.gptq import GPTQModifier

recipe = GPTQModifier(
    targets="Linear",
    scheme="FP8",
    ignore=["lm_head", "re:.*mlp\.gate$", "model.layers.0.mlp.down_proj"],
)

Ignored layers:

lm_head — output projection, sensitive to quantization
re:.*mlp\.gate$ — MoE router weights (64-expert routing decisions)
model.layers.0.mlp.down_proj — dense first layer, column count (10944) not divisible by group size 128

Note: llm-compressor v0.11.0 has a known bug where actorder: "static" is written for channel/tensor-strategy quantization. vLLM v0.22.1+ rejects this. The config in this repo has been patched (actorder: null).

Serving with vLLM

bash

pip install vllm
vllm serve soyrsoyr/deepseek-moe-16b-chat-FP8-GPTQ --max-model-len 2048 --trust-remote-code

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
    model="soyrsoyr/deepseek-moe-16b-chat-FP8-GPTQ",
    messages=[{"role": "user", "content": "Explain quantization in one sentence."}],
    max_tokens=128,
)
print(response.choices[0].message.content)

Collection

Part of DeepSeek-MoE-16B-Chat GPTQ Quantized — all four schemes (W4A16, W8A8, FP8, NVFP4) compared side by side.