This repository contains Q4_KM and Q5_KM quantized versions of the
DeepSeek-R1-Distill-Llama-8B model, optimized for efficient deployment while maintaining strong performance.
Discover our full range of quantized language models by visiting our
SandLogic Lexicon HuggingFace. To learn more about our company and services, check out our website at
SandLogic.
These models are quantized versions of DeepSeek-R1-Distill-Llama-8B, which is a distilled 8B parameter model based on the Llama architecture. The original model demonstrates that reasoning patterns from larger models can be effectively distilled into smaller architectures.
Please refer to the llama-cpp-python
documentation to install with GPU support.
Here's an example demonstrating how to use the high-level API for basic text completion:
1from llama_cpp import Llama
2
3llm = Llama(
4 model_path="model/path/",
5 verbose=False,
6 # n_gpu_layers=-1, # Uncomment to use GPU acceleration
7 # n_ctx=2048, # Uncomment to increase the context window
8)
9
10output = llm(
11 "Q: Name the planets in the solar system? A: ", # Prompt
12 max_tokens=32, # Generate up to 32 tokens
13 stop=["Q:", "\n"], # Stop generating just before a new question
14 echo=False # Don't echo the prompt in the output
15)
16
17print(output["choices"][0]["text"])
This model inherits the license of the original DeepSeek-R1-Distill-Llama-8B model. Please refer to the original model's license for usage terms and conditions.
We thank the DeepSeek AI team for open-sourcing their distilled models and demonstrating that smaller models can achieve impressive performance through effective distillation techniques.
