from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "Deepak7376/DeepSeek-R1-Distill-Qwen-1.5B-bnb-4bit"

bnb_config_4bit = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config_4bit)
tokenizer = AutoTokenizer.from_pretrained(model_id)

pipe = pipeline(
        'text-generation',
        model=model,
        tokenizer=tokenizer,
        max_length=1024,
        truncation=True,
        do_sample=True,
        temperature=0.6,
        top_p=0.95,
    )

messages = [
    {"role": "user", "content": "suggest me top movies in 2021? <think>\n"},
]
pipe(messages)

python

from transformers import pipeline

pipe = pipeline("text-generation", model="Deepak7376/DeepSeek-R1-Distill-Qwen-1.5B-bnb-4bit")

messages = [
    {"role": "user", "content": "suggest me top movies in 2021? <think>\n"},
]
pipe(messages)

Model Performance

Base Model: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
Quantization: 4-bit ( NF4 )
Library: <a href="https://github.com/TimDettmers/bitsandbytes" target="_blank" rel="noopener noreferrer" class="text-blue-400 hover:text-blue-300 underline underline-offset-2 transition-colors">bitsandbytes
Framework: transformers
Use Case: Text generation, chatbot applications, and other NLP tasks.

Quantizing the model significantly reduces memory usage while maintaining good performance. Here are the memory footprints:

Model Version	Memory Usage
Base Model	~3.5GB
4-bit Quantized	~1.5GB

License

This model follows the apache-2.0 license.

Acknowledgments

DeepSeek-AI for the original model.
BitsAndBytes for quantization support.