Property	Value
Base Model	deepseek-ai/deepseek-moe-16b-chat
Architecture	Mixture of Experts (64 experts, 6 active + 2 shared)
Total Parameters	~16B
Active Parameters	~2.8B
Quantization Method	GPTQ
Format	`int-quantized` (INT4)
Strategy	`group`
Weight Bits	4
Activation Bits	16 (unquantized)
Group Size	128
Calibration Data	512 samples from `HuggingFaceH4/ultrachat_200k` (max seq len 2048)

Quantization Recipe

Quantized with llm-compressor v0.11.0 on an NVIDIA H200 GPU.

python

from llmcompressor.modifiers.gptq import GPTQModifier

recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=["lm_head", "re:.*mlp\.gate$", "model.layers.0.mlp.down_proj"],
)

Ignored layers:

lm_head — output projection, sensitive to quantization
re:.*mlp\.gate$ — MoE router weights (64-expert routing decisions)
model.layers.0.mlp.down_proj — dense first layer, column count (10944) not divisible by group size 128

Serving with vLLM

bash

pip install vllm
vllm serve soyrsoyr/deepseek-moe-16b-chat-W4A16-GPTQ --max-model-len 2048 --trust-remote-code

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
    model="soyrsoyr/deepseek-moe-16b-chat-W4A16-GPTQ",
    messages=[{"role": "user", "content": "Explain quantization in one sentence."}],
    max_tokens=128,
)
print(response.choices[0].message.content)

Collection

Part of DeepSeek-MoE-16B-Chat GPTQ Quantized — all four schemes (W4A16, W8A8, FP8, NVFP4) compared side by side.