DeepSeek-R1-Distill-Qwen-14B GGUF Models

Choosing the Right Model Format

BF16 (Brain Float 16) – Use if BF16 acceleration is available

• A 16-bit floating-point format designed for faster computation while retaining good precision.
• Provides similar dynamic range as FP32 but with lower memory usage.
• Recommended if your hardware supports BF16 acceleration (check your device’s specs).
• Ideal for high-performance inference with reduced memory footprint compared to FP32.

F16 (Float 16) – More widely supported than BF16

• A 16-bit floating-point high precision but with less of range of values than BF16.
• Works on most devices with FP16 acceleration support (including many GPUs and some CPUs).
• Slightly lower numerical precision than BF16 but generally sufficient for inference.

Quantized Models (Q4_K, Q6_K, Q8, etc.) – For CPU & Low-VRAM Inference

• Lower-bit models (Q4_K) → Best for minimal memory usage, may have lower precision.
• Higher-bit models (Q6_K, Q8_0) → Better accuracy, requires more memory.

Very Low-Bit Quantization (IQ3_XS, IQ3_S, IQ3_M, Q4_K, Q4_0)

• IQ3_XS: Ultra-low-bit quantization (3-bit) with extreme memory efficiency.
  • Use case: Best for ultra-low-memory devices where even Q4_K is too large.
  • Trade-off: Lower accuracy compared to higher-bit quantizations.
• IQ3_S: Small block size for maximum memory efficiency.
  • Use case: Best for low-memory devices where IQ3_XS is too aggressive.
• IQ3_M: Medium block size for better accuracy than IQ3_S.
  • Use case: Suitable for low-memory devices where IQ3_S is too limiting.
• Q4_K: 4-bit quantization with block-wise optimization for better accuracy.
  • Use case: Best for low-memory devices where Q6_K is too large.
• Q4_0: Pure 4-bit quantization, optimized for ARM devices.
  • Use case: Best for ARM-based devices or low-memory environments.

Summary Table: Model Format Selection

Included Files & Details

DeepSeek-R1-Distill-Qwen-14B-bf16.gguf

• Model weights preserved in BF16.
• Use this if you want to requantize the model into a different format.
• Best if your device supports BF16 acceleration.

DeepSeek-R1-Distill-Qwen-14B-f16.gguf

• Model weights stored in F16.
• Use if your device supports FP16, especially if BF16 is not available.

DeepSeek-R1-Distill-Qwen-14B-bf16-q8_0.gguf

• Output & embeddings remain in BF16.
• All other layers quantized to Q8_0.
• Use if your device supports BF16 and you want a quantized version.

DeepSeek-R1-Distill-Qwen-14B-f16-q8_0.gguf

• Output & embeddings remain in F16.
• All other layers quantized to Q8_0.

DeepSeek-R1-Distill-Qwen-14B-q4_k.gguf

• Output & embeddings quantized to Q8_0.
• All other layers quantized to Q4_K.
• Good for CPU inference with limited memory.

DeepSeek-R1-Distill-Qwen-14B-q4_k_s.gguf

• Smallest Q4_K variant, using less memory at the cost of accuracy.
• Best for very low-memory setups.

DeepSeek-R1-Distill-Qwen-14B-q6_k.gguf

• Output & embeddings quantized to Q8_0.
• All other layers quantized to Q6_K .

DeepSeek-R1-Distill-Qwen-14B-q8_0.gguf

• Fully Q8 quantized model for better accuracy.
• Requires more memory but offers higher precision.

DeepSeek-R1-Distill-Qwen-14B-iq3_xs.gguf

• IQ3_XS quantization, optimized for extreme memory efficiency.
• Best for ultra-low-memory devices.

DeepSeek-R1-Distill-Qwen-14B-iq3_m.gguf

• IQ3_M quantization, offering a medium block size for better accuracy.
• Suitable for low-memory devices.

DeepSeek-R1-Distill-Qwen-14B-q4_0.gguf

• Pure Q4_0 quantization, optimized for ARM devices.
• Best for low-memory environments.
• Prefer IQ4_NL for better accuracy.

🚀 If you find these models useful

What I'm Testing

The other Available AI Assistants

Final Word

DeepSeek-R1

1. Introduction

2. Model Summary

• We directly apply reinforcement learning (RL) to the base model without relying on supervised fine-tuning (SFT) as a preliminary step. This approach allows the model to explore chain-of-thought (CoT) for solving complex problems, resulting in the development of DeepSeek-R1-Zero. DeepSeek-R1-Zero demonstrates capabilities such as self-verification, reflection, and generating long CoTs, marking a significant milestone for the research community. Notably, it is the first open research to validate that reasoning capabilities of LLMs can be incentivized purely through RL, without the need for SFT. This breakthrough paves the way for future advancements in this area.
• We introduce our pipeline to develop DeepSeek-R1. The pipeline incorporates two RL stages aimed at discovering improved reasoning patterns and aligning with human preferences, as well as two SFT stages that serve as the seed for the model's reasoning and non-reasoning capabilities. We believe the pipeline will benefit the industry by creating better models.

• We demonstrate that the reasoning patterns of larger models can be distilled into smaller models, resulting in better performance compared to the reasoning patterns discovered through RL on small models. The open source DeepSeek-R1, as well as its API, will benefit the research community to distill better smaller models in the future.
• Using the reasoning data generated by DeepSeek-R1, we fine-tuned several dense models that are widely used in the research community. The evaluation results demonstrate that the distilled smaller dense models perform exceptionally well on benchmarks. We open-source distilled 1.5B, 7B, 8B, 14B, 32B, and 70B checkpoints based on Qwen2.5 and Llama3 series to the community.

3. Model Downloads

DeepSeek-R1 Models

DeepSeek-R1-Distill Models

4. Evaluation Results

DeepSeek-R1-Evaluation

Distilled Model Evaluation

5. Chat Website & API Platform

6. How to Run Locally

DeepSeek-R1 Models

DeepSeek-R1-Distill Models

Usage Recommendations

1. Set the temperature within the range of 0.5-0.7 (0.6 is recommended) to prevent endless repetitions or incoherent outputs.
2. Avoid adding a system prompt; all instructions should be contained within the user prompt.
3. For mathematical problems, it is advisable to include a directive in your prompt such as: "Please reason step by step, and put your final answer within \boxed{}."
4. When evaluating model performance, it is recommended to conduct multiple tests and average the results.

7. License

• DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Qwen-14B and DeepSeek-R1-Distill-Qwen-32B are derived from Qwen-2.5 series, which are originally licensed under Apache 2.0 License, and now finetuned with 800k samples curated with DeepSeek-R1.
• DeepSeek-R1-Distill-Llama-8B is derived from Llama3.1-8B-Base and is originally licensed under llama3.1 license.
• DeepSeek-R1-Distill-Llama-70B is derived from Llama3.3-70B-Instruct and is originally licensed under llama3.3 license.

8. Citation

@misc{deepseekai2025deepseekr1incentivizingreasoningcapability,
      title={DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning}, 
      author={DeepSeek-AI},
      year={2025},
      eprint={2501.12948},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2501.12948}, 
}

9. Contact