This is
DeepSeek-R1-Distill-Qwen-1.5B model converted to the
OpenVINO™ IR (Intermediate Representation) format with weights compressed to INT8 by
NNCF .
For more information on quantization, check the
OpenVINO model optimization guide from transformers import AutoTokenizer
from optimum.intel.openvino import OVModelForCausalLM
model_id = "OpenVINO/DeepSeek-R1-Distill-Qwen-1.5B-int8-ov"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = OVModelForCausalLM.from_pretrained(model_id)
inputs = tokenizer("What is OpenVINO?", return_tensors="pt")
outputs = model.generate(**inputs, max_length=200)
text = tokenizer.batch_decode(outputs)[0]
print(text)For more examples and possible optimizations, refer to the
OpenVINO Large Language Model Inference Guide .
pip install -U --pre openvino openvino-tokenizers openvino-genai --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/pre-release
pip install huggingface_hubimport huggingface_hub as hf_hub
model_id = "OpenVINO/DeepSeek-R1-Distill-Qwen-1.5B-int8-ov"
model_path = "DeepSeek-R1-Distill-Qwen-1.5B-int8-ov"
hf_hub.snapshot_download(model_id, local_dir=model_path)
import openvino_genai as ov_genai
device = "CPU"
pipe = ov_genai.LLMPipeline(model_path, device)
print(pipe.generate("What is OpenVINO?", max_length=200))More GenAI usage examples can be found in OpenVINO GenAI library
docs and
samples Check the original model card for
original model card for limitations.
The original model is distributed under
mit license. More details can be found in
original model card .
Intel is committed to respecting human rights and avoiding causing or contributing to adverse impacts on human rights. See
Intel’s Global Human Rights Principles . Intel’s products and software are intended only to be used in applications that do not cause or contribute to adverse impacts on human rights.
