This repository delivers an Int4 + selectively-Int8 GPTQ DeepSeek-R1-0528 model: only layers that are highly sensitive to quantization remain in Int8, while the rest stay Int4—preserving generation quality with minimal file-size overhead.
Preliminary trials show that converting the entire model to pure Int4 (AWQ/GPTQ) under the quantization layout used in vLLM’s current DeepSeek-R1 implementation degrades inference accuracy and can produce faulty outputs. Layer-wise fine-grained quantization substantially mitigates this issue.
Temporary patch:
vLLM == 0.9.0 does not yet natively support per-layer quantization for MoE modules.
We added get_moe_quant_method to gptq_marlin.py as an interim fix.
Until the upstream PR is merged, please replace the original file with the one provided in this repo.
Variant Overview
Variant
Characteristics
File Size
Recommended Scenario
Lite
Only the most critical layers upgraded to Int8; size close to pure Int4
355 GB
Resource-constrained, lightweight server deployments
Compact
More Int8 layers, relatively higher output quality
414 GB
VRAM-sufficient deployments focused on answer quality (e.g., 8 × A100)
Medium
Compact plus fully-Int8 attention layers; high quality with reduced long-context loss
445 GB
VRAM-rich deployments needing both top answer quality and high concurrency (e.g., 8 × H20)
Choose the variant that best matches your hardware and quality requirements.
【Model Update Date】
2025-05-31
1. fast commit
【Dependencies】
vllm==0.9.0
transformers==4.52.3
### 【💡Notes on New VLLM Versions💡】
1. Recommend Using V0 Inference Mode
Before launching vLLM, set the environment variable
export VLLM_USE_V1=0
### 【💡 Patch for gptq_marlin.py💡】
At present, vllm==0.9.0 lacks support for per-layer quantization configurations for the moe module, which will lead to errors when loading the model.
I have implemented a simple fix by adding the get_moe_quant_method function to the gptq_marlin.py file.
Until the PR is merged, please replace the gptq_marlin.py file in your installation with the attached version, placing it at:
The DeepSeek R1 model has undergone a minor version upgrade, with the current version being DeepSeek-R1-0528. In the latest update, DeepSeek R1 has significantly improved its depth of reasoning and inference capabilities by leveraging increased computational resources and introducing algorithmic optimization mechanisms during post-training. The model has demonstrated outstanding performance across various benchmark evaluations, including mathematics, programming, and general logic. Its overall performance is now approaching that of leading models, such as O3 and Gemini 2.5 Pro.
Compared to the previous version, the upgraded model shows significant improvements in handling complex reasoning tasks. For instance, in the AIME 2025 test, the model’s accuracy has increased from 70% in the previous version to 87.5% in the current version. This advancement stems from enhanced thinking depth during the reasoning process: in the AIME test set, the previous model used an average of 12K tokens per question, whereas the new version averages 23K tokens per question.
Beyond its improved reasoning capabilities, this version also offers a reduced hallucination rate, enhanced support for function calling, and better experience for vibe coding.
2. Evaluation Results
DeepSeek-R1-0528
For all our models, the maximum generation length is set to 64K tokens. For benchmarks requiring sampling, we use a temperature of $0.6$, a top-p value of $0.95$, and generate 16 responses per query to estimate pass@1.
Category
Benchmark (Metric)
DeepSeek R1
DeepSeek R1 0528
General
MMLU-Redux (EM)
92.9
93.4
MMLU-Pro (EM)
84.0
85.0
GPQA-Diamond (Pass@1)
71.5
81.0
SimpleQA (Correct)
30.1
27.8
FRAMES (Acc.)
82.5
83.0
Humanity's Last Exam (Pass@1)
8.5
17.7
Code
LiveCodeBench (2408-2505) (Pass@1)
63.5
73.3
Codeforces-Div1 (Rating)
1530
1930
SWE Verified (Resolved)
49.2
57.6
Aider-Polyglot (Acc.)
53.3
71.6
Math
AIME 2024 (Pass@1)
79.8
91.4
AIME 2025 (Pass@1)
70.0
87.5
HMMT 2025 (Pass@1)
41.7
79.4
CNMO 2024 (Pass@1)
78.8
86.9
Tools
BFCL_v3_MultiTurn (Acc)
-
37.0
Tau-Bench (Pass@1)
-
53.5(Airline)/63.9(Retail)
Note: We use Agentless framework to evaluate model performance on SWE-Verified. We only evaluate text-only prompts in HLE testsets. GPT-4.1 is employed to act user role in Tau-bench evaluation.
DeepSeek-R1-0528-Qwen3-8B
Meanwhile, we distilled the chain-of-thought from DeepSeek-R1-0528 to post-train Qwen3 8B Base, obtaining DeepSeek-R1-0528-Qwen3-8B. This model achieves state-of-the-art (SOTA) performance among open-source models on the AIME 2024, surpassing Qwen3 8B by +10.0% and matching the performance of Qwen3-235B-thinking. We believe that the chain-of-thought from DeepSeek-R1-0528 will hold significant importance for both academic research on reasoning models and industrial development focused on small-scale models.
AIME 24
AIME 25
HMMT Feb 25
GPQA Diamond
LiveCodeBench (2408-2505)
Qwen3-235B-A22B
85.7
81.5
62.5
71.1
66.5
Qwen3-32B
81.4
72.9
-
68.4
-
Qwen3-8B
76.0
67.3
-
62.0
-
Phi-4-Reasoning-Plus-14B
81.3
78.0
53.6
69.3
-
Gemini-2.5-Flash-Thinking-0520
82.3
72.0
64.2
82.8
62.3
o3-mini (medium)
79.6
76.7
53.3
76.8
65.9
DeepSeek-R1-0528-Qwen3-8B
86.0
76.3
61.5
61.1
60.5
3. Chat Website & API Platform
You can chat with DeepSeek-R1 on DeepSeek's official website: chat.deepseek.com, and switch on the button "DeepThink"
Please visit DeepSeek-R1 repository for more information about running DeepSeek-R1-0528 locally.
Compared to previous versions of DeepSeek-R1, the usage recommendations for DeepSeek-R1-0528 have the following changes:
System prompt is supported now.
It is not required to add "<think>\n" at the beginning of the output to force the model into thinking pattern.
The model architecture of DeepSeek-R1-0528-Qwen3-8B is identical to that of Qwen3-8B, but it shares the same tokenizer configuration as DeepSeek-R1-0528. This model can be run in the same manner as Qwen3-8B.
System Prompt
In the official DeepSeek web/app, we use the same system prompt with a specific date.
该助手为DeepSeek-R1，由深度求索公司创造。
今天是{current date}。
For example,
该助手为DeepSeek-R1，由深度求索公司创造。
今天是2025年5月28日，星期一。
Temperature
In our web and application environments, the temperature parameter $T_{model}$ is set to 0.6.
Prompts for File Uploading and Web Search
For file uploading, please follow the template to create prompts, where {file_name}, {file_content} and {question} are arguments.
search_answer_en_template = \
'''# The following contents are the search results related to the user's message:
{search_results}
In the search results I provide to you, each result is formatted as [webpage X begin]...[webpage X end], where X represents the numerical index of each article. Please cite the context at the end of the relevant sentence when appropriate. Use the citation format [citation:X] in the corresponding part of your answer. If a sentence is derived from multiple contexts, list all relevant citation numbers, such as [citation:3][citation:5]. Be sure not to cluster all citations at the end; instead, include them in the corresponding parts of the answer.
When responding, please keep the following points in mind:
- Today is {cur_date}.
- Not all content in the search results is closely related to the user's question. You need to evaluate and filter the search results based on the question.
- For listing-type questions (e.g., listing all flight information), try to limit the answer to 10 key points and inform the user that they can refer to the search sources for complete information. Prioritize providing the most complete and relevant items in the list. Avoid mentioning content not provided in the search results unless necessary.
- For creative tasks (e.g., writing an essay), ensure that references are cited within the body of the text, such as [citation:3][citation:5], rather than only at the end of the text. You need to interpret and summarize the user's requirements, choose an appropriate format, fully utilize the search results, extract key information, and generate an answer that is insightful, creative, and professional. Extend the length of your response as much as possible, addressing each point in detail and from multiple perspectives, ensuring the content is rich and thorough.
- If the response is lengthy, structure it well and summarize it in paragraphs. If a point-by-point format is needed, try to limit it to 5 points and merge related content.
- For objective Q&A, if the answer is very brief, you may add one or two related sentences to enrich the content.
- Choose an appropriate and visually appealing format for your response based on the user's requirements and the content of the answer, ensuring strong readability.
- Your answer should synthesize information from multiple relevant webpages and avoid repeatedly citing the same webpage.
- Unless the user requests otherwise, your response should be in the same language as the user's question.
# The user's message is:
{question}'''
5. License
This code repository is licensed under MIT License. The use of DeepSeek-R1 models is also subject to MIT License. DeepSeek-R1 series (including Base and Chat) supports commercial use and distillation.
6. Citation
@misc{deepseekai2025deepseekr1incentivizingreasoningcapability,
title={DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning},
author={DeepSeek-AI},
year={2025},
eprint={2501.12948},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2501.12948},
}
7. Contact
If you have any questions, please raise an issue or contact us at service@deepseek.com.
