You can deploy this model and start earning money today!

glenic/
DeepSeek-R1-Distill-Qwen-7B-AWQ

Name: DeepSeek-R1-Distill-Qwen-7B-AWQ
Availability: InStock

deepseek-ai/DeepSeek-R1-Distill-Qwen-7B

quantized

compressed-tensors

Views

No views yet

DeepSeek-R1-Distill-Qwen-7B — AWQ 4-bit

Auto-quantized from deepseek-ai/DeepSeek-R1-Distill-Qwen-7B by fastserve's self-quantization pipeline (publish/) — so it can be trusted the way a random community AWQ requant sometimes can't. Two community quants fastserve auto-detected during its own benchmark run looped garbage tokens instead of answering; every checkpoint published here passed an accuracy gate against its bf16 baseline first.

Quantization

Method: AWQ (W4A16_ASYM), group size 128, via llm-compressor
Calibration: 256 samples from ultrachat-200k

Validation (accuracy gate — this is why you can trust it)

	bf16 baseline	this AWQ checkpoint
GSM8K accuracy (n=30)	0.7667	0.8667

Within 0.1 (absolute) of the bf16 baseline, <30% degenerate (repeated-token loops). A checkpoint that failed this gate would not have been uploaded.

Original vs vLLM vs fastserve

Same GSM8K prompts (n=8), single-stream (batch-1) greedy decode, one A100-80GB. "fastserve" = this AWQ checkpoint + speculative decoding (eagle3:thoughtworks/DeepSeek-R1-Distill-Qwen-7B-Eagle3) on vLLM. Memory = weights only (bf16 vs AWQ); vLLM's KV-cache budget is a separate knob.

	Original (HF bf16)	vLLM (bf16)	fastserve (AWQ+spec)
GSM8K acc	0.75	0.75	0.875
Decode speed	59.3 tok/s	98.2 tok/s	343.7 tok/s
Weights (VRAM)	14.19 GiB	14.19 GiB	5.19 GiB

Serve it with fastserve

fastserve auto-detects this checkpoint — point it at the original model id and it finds this AWQ + wires up speculative decoding:

bash

git clone https://github.com/jireh-father/fastserve && cd fastserve && ./install.sh

# serve an OpenAI-compatible API (auto-picks this AWQ checkpoint)
./fastserve serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B

# or benchmark the speedup vs the naive baseline
./fastserve bench deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --compare-baseline

Then query it like any OpenAI endpoint:

bash

curl localhost:8000/v1/completions \
  -d '{"model": "glenic/DeepSeek-R1-Distill-Qwen-7B-AWQ", "prompt": "Q: What is 17*4?\nA:", "max_tokens": 64}'

Or serve directly with vLLM

bash

pip install vllm
python -m vllm.entrypoints.openai.api_server --model glenic/DeepSeek-R1-Distill-Qwen-7B-AWQ

License

Inherits the base model's license — see deepseek-ai/DeepSeek-R1-Distill-Qwen-7B for terms.