Views
No views yet
deepseek-ai/DeepSeek-R1-Distill-Llama-70B with INT4 weight compression.deepseek-ai/DeepSeek-R1-Distill-Llama-70Bdeepseek-ai/DeepSeek-R1-Distill-Llama-70B model card states that the code repository and model weights are licensed under the MIT License. It also states that the DeepSeek-R1 series supports commercial use, modifications, and derivative works.DeepSeek-R1-Distill-Llama-70B is derived from Llama3.3-70B-Instruct, which was originally licensed under the Llama 3.3 license.LICENSE file is included in this repository for attribution and compliance.1optimum-cli export openvino \
2 --model deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
3 --weight-format int4 \
4 --group-size 128 \
5 --ratio 1.0 \
6 ov_DeepSeek-R1-Distill-Llama-70B_int4
7Quantization
8Weight format: INT4
9Group size: 128
10Ratio: 1.0
11Export tool: Optimum Intel
12Compression backend: NNCF through Optimum Intel
13Runtime format: OpenVINO IR
14
15INT4 compression is intended to reduce model size and memory usage compared with higher precision weights. As with any converted and quantized model, output quality and numerical behavior may differ from the original model and should be validated for your use case.
16Test with Optimum Intel first
17
18Run from inside the model directory:
19
20python examples/test_deepseek70b_llama_ov_optimum.py \
21 --model-dir . \
22 --device CPU \
23 --max-new-tokens 128 \
24 --prompt "Explain OpenVINO in one short paragraph."
25Test with OpenVINO GenAI
26
27OpenVINO GenAI provides a clean runtime path for generative inference with OpenVINO-converted models.
28
29Run from inside the model directory:
30
31python examples/test_deepseek70b_llama_ov_genai.py \
32 --model-dir . \
33 --device CPU \
34 --max-new-tokens 64 \
35 --prompt "Explain OpenVINO in one short paragraph."
36For GPU
37
38First confirm that OpenVINO detects GPU devices. The included OpenVINO GenAI script prints all available OpenVINO devices.
39
40Then run:
41
42python examples/test_deepseek70b_llama_ov_genai.py \
43 --model-dir . \
44 --device GPU.0 \
45 --max-new-tokens 64 \
46 --prompt "Explain OpenVINO in one short paragraph."
47Here is my full setup and export flow:
48
49cd ~
50
51python3.11 -m venv deepseek70b_llama_ov_env
52source ~/deepseek70b_llama_ov_env/bin/activate
53
54python -m pip install --upgrade pip setuptools wheel
55
56pip install -U \
57 "openvino>=2025.1.0" \
58 "optimum-intel[openvino]>=1.22.0" \
59 "nncf>=2.14.0" \
60 "transformers>=4.48.0" \
61 "accelerate" \
62 "safetensors" \
63 "huggingface_hub" \
64 "sentencepiece" \
65 "protobuf"
66
67cd ~/ov_models
68
69MODEL_ID="deepseek-ai/DeepSeek-R1-Distill-Llama-70B"
70OUT_DIR="ov_DeepSeek-R1-Distill-Llama-70B_int4"
71
72mkdir -p export_logs
73
74optimum-cli export openvino \
75 --model "$MODEL_ID" \
76 --weight-format int4 \
77 --group-size 128 \
78 --ratio 1.0 \
79 "$OUT_DIR" \
80 2>&1 | tee export_logs/deepseek_r1_distill_llama_70b_int4_export.log
81Notes
82This model is text-only.
83This repository uses both Optimum Intel and OpenVINO GenAI examples.
84The Optimum Intel path is useful for validating the exported model with Transformers-style APIs.
85The OpenVINO GenAI path is recommended for generative inference with OpenVINO-converted models.
86OpenVINO Model Server compatibility is not claimed unless separately validated.
87Limitations
88
89This repository inherits the limitations of the original deepseek-ai/DeepSeek-R1-Distill-Llama-70B model. Additional differences may arise from OpenVINO conversion, INT4 compression, runtime package versions, and generation configuration.
90
91Attribution
92
93This is an unofficial OpenVINO conversion of the original DeepSeek model. All rights to the original model, training, and licensing remain with the original authors.