You can deploy this model and start earning money today!

amanwalksdownthestreet/
DeepSeek-R1-Distill-Llama-70B-MXFP8

Name: DeepSeek-R1-Distill-Llama-70B-MXFP8
Availability: InStock

Views

No views yet

DeepSeek-R1-Distill-Llama-70B-MXFP8

MXFP8 quantization of deepseek-ai/DeepSeek-R1-Distill-Llama-70B. Weights only — prompt format, chat template, <think> reasoning behavior, and license (MIT) are the base model's.

Requires Blackwell (compute capability 12.0). MXFP8 needs Blackwell tensor-core support in vLLM; does not run on Ada/Hopper.

Weights: MXFP8 (group size 32), compressed-tensors format (llm-compressor 0.12.1)
Not quantized: lm_head, embeddings. KV cache: BF16
Size: 70 GB, 11 shards (base BF16 ≈ 141 GB)
Calibration: 1024 samples, neuralmagic/calibration (LLM), max seq 1024

Serving

bash

vllm serve <path-or-repo> --quantization compressed-tensors \
  --max-model-len 49152 --gpu-memory-utilization 0.95

Reasoning model: expect <think>…</think> traces per the base model's chat template (--reasoning-parser deepseek_r1 in vLLM separates them).

Evals (lm-evaluation-harness 0.4.11 on vLLM)

Benchmark	DeepSeek paper (BF16)	MXFP8 (this repo)
MMLU-Pro (5-shot, non-thinking, /v1/completions)	not published	67.50 ±0.41
GPQA-Diamond	65.2	pending*
AIME24	70.0 (multi-sample avg)	pending*

* The first GPQA/AIME24 run was invalidated by eval-infrastructure contention (195/198 GPQA responses returned empty — two sweeps fought over one endpoint); numbers will land after a clean re-run. The MMLU-Pro row is a deliberate non-thinking self-baseline (raw completions, no chat template, no <think> budget) — do not compare it against thinking-mode numbers.

All credit for the model to DeepSeek; this repo is a Blackwell-native repack.

amanwalksdownthestreet/DeepSeek-R1-Distill-Llama-70B-MXFP8

DeepSeek-R1-Distill-Llama-70B-MXFP8

Serving

Evals (lm-evaluation-harness 0.4.11 on vLLM)

amanwalksdownthestreet/
DeepSeek-R1-Distill-Llama-70B-MXFP8