Deepseek-R1-0528-Distill-Qwen2.5-1.5B-QA
Model Overview
This repository presents Deepseek-R1-0528-Distill-Qwen2.5-1.5B-QA, a specialized Large Language Model (LLM) tailored for Question Answering (QA) tasks. This model is the result of a knowledge distillation process, where the advanced capabilities of the larger DeepSeek-R1 0528 model were transferred to the more efficient Qwen 2.5 1.5B architecture.
The primary motivation behind this project is to create a highly performant yet resource-efficient QA model. By distilling knowledge from a state-of-the-art teacher model, we aim to make sophisticated QA inference more accessible for deployment in environments with limited computational resources, such as edge devices or applications requiring low latency.
Distillation Process
The model's development involved the following key steps in a knowledge distillation pipeline:
Teacher Model Selection: The powerful DeepSeek-R1 0528, known for its strong reasoning and language understanding, was chosen as the teacher model.
Student Model Selection: The Qwen 2.5 1.5B model, a compact and efficient base LLM, served as the student model.
Distillation Dataset Creation: A custom multiturn.jsonl dataset was meticulously prepared. This dataset was populated by feeding a diverse set of user prompts (questions) to the DeepSeek-R1 0528 teacher model. The high-quality, detailed responses generated by DeepSeek-R1 0528 then formed the "ground truth" for the student model to learn from. This process effectively trained the Qwen 2.5 1.5B model to mimic the DeepSeek-R1's conversational style, reasoning patterns, and factual accuracy in the context of QA.
Fine-tuning Methodology: The Qwen 2.5 1.5B student model was fine-tuned on this synthetic instruction-following dataset. To optimize for memory efficiency and training speed, 4-bit quantization (QLoRA) was employed, leveraging the Unsloth library for its accelerated training capabilities on consumer-grade GPUs.
Performance & Efficiency
This distilled model represents a significant stride in balancing model performance with operational efficiency:
Efficiency: The model's size is approximately 995MB on disk. This compact footprint is achieved through aggressive 4-bit quantization, making it considerably smaller than its teacher model. This reduction in size translates directly into faster loading times, lower memory consumption, and improved inference speeds, opening up possibilities for deployment in resource-constrained environments.
Performance: While knowledge distillation inherently involves some degree of information compression, the objective was to retain a substantial portion of DeepSeek-R1's robust QA capabilities. Initial observations indicate successful knowledge transfer, positioning this model as a viable and efficient solution for various QA applications where computational resources are a key consideration.
How to Use
To load and perform inference with this fine-tuned and merged model, you will need the Hugging Face transformers library and bitsandbytes for 4-bit loading.
Installation:
pip install transformers bitsandbytes torch
Python Code for Inference:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from transformers import StoppingCriteria, StoppingCriteriaList
--- Configuration for Loading ---
Path to your saved merged model directory (local path or Hugging Face Hub repo ID)
model_path = "./qwen_distill_merged_model" # Example: "your-username/Deepseek-R1-0528-Distill-Qwen2.5-1.5B-QA" if uploaded to Hub
Max sequence length used during training. Crucial for consistent inference.
max_seq_length = 2048
Configure 4-bit quantization (as the model was saved in this format)
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16, # Use torch.float16 if bfloat16 is not supported by your GPU
bnb_4bit_use_double_quant=True,
)
--- Load the Merged Model and Tokenizer ---
print(f"Loading model from: {model_path}...")
model = AutoModelForCausalLM.from_pretrained(
model_path,
quantization_config=bnb_config,
device_map="auto",
torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(model_path)
print("Model and tokenizer loaded successfully!")
--- Set the Chat Template (Crucial for consistent inference) ---
This must match the template used during training.
if tokenizer.chat_template is None:
tokenizer.chat_template = (
"{% for message in messages %}"
"{% if message['role'] == 'system' %}"
"{{ '<|im_start|>system\n' + message['content'] + '<|im_end|>' + '\n' }}"
"{% elif message['role'] == 'user' %}"
"{{ '<|im_start|>user\n' + message['content'] + '<|im_end|>' + '\n' }}"
"{% elif message['role'] == 'assistant' %}"
"{{ '<|im_start|>assistant\n' + message['content'] + '<|im_end|>' + '\n' }}"
"{% endif %}"
"{% endfor %}"
"{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
Define custom StoppingCriteria classes for robust generation control
class EosTokenStoppingCriteria(StoppingCriteria):
def init(self, eos_token_id):
self.eos_token_id = eos_token_id
def call(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
return input_ids[0, -1] == self.eos_token_id
class StopStringsStoppingCriteria(StoppingCriteria):
def init(self, tokenizer, stop_strings):
self.tokenizer = tokenizer
self.stop_strings = stop_strings
def call(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
generated_text = self.tokenizer.decode(input_ids[0])
for stop_string in self.stop_strings:
if stop_string in generated_text:
return True
return False
Combine stopping criteria
stopping_criteria_list = [EosTokenStoppingCriteria(tokenizer.eos_token_id)]
Add any specific strings that should stop generation if observed in output (e.g., from data artifacts)
problematic_stop_strings = ["user", "BitFields", "GuidId", "mPid", "_Statics", "代码", "VariablesVariablesVariables"]
stopping_criteria_list.append(StopStringsStoppingCriteria(tokenizer, problematic_stop_strings))
--- Example Inference ---
prompt = "Can you explain how to calculate the factorial of a number, including the formula and an example?"
messages = [
{"role": "system", "content": "You are a helpful and knowledgeable assistant. Provide clear and concise answers."},
{"role": "user", "content": prompt}
]
text_for_inference = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
model_inputs = tokenizer(
[text_for_inference],
return_tensors="pt",
max_length=max_seq_length,
truncation=True
).to(model.device)
print("\nGenerating response...")
generated_ids = model.generate(
**model_inputs,
max_new_tokens=512,
use_cache=True,
do_sample=True,
temperature=0.7,
top_p=0.9,
repetition_penalty=1.1,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
stopping_criteria=stopping_criteria_list,
)
generated_text = tokenizer.batch_decode(
generated_ids[:, model_inputs.input_ids.shape[1]:],
skip_special_tokens=True
)[0]
print("\n--- Generated Response ---")
print(generated_text)
Limitations & Current Issues
Knowledge Compression: While effective, distillation inherently involves some knowledge compression, meaning the student model may not perfectly replicate the teacher's performance across all highly nuanced or complex reasoning tasks.
Repetitive and Nonsensical Generation (Under Investigation): The model currently exhibits a tendency to generate repetitive patterns, including parts of the input prompt, and unusual strings such as "BitFields", "GuidId", "mPid", "_Statics", "VariablesVariablesVariables", and Chinese characters (e.g., "代码"). This behavior is actively being investigated.
Primary Hypothesis: It is strongly suspected that these problematic patterns were inadvertently included or malformed within the multiturn.jsonl training dataset, causing the model to learn and reproduce them.
Debugging: Custom StoppingCriteria have been implemented in the inference code to help mitigate and diagnose this issue by forcing the model to stop upon generating these specific strings or its end-of-sequence token.
Future Work & Improvements
Critical Data Cleaning: The immediate and most crucial next step is a comprehensive inspection and rigorous cleaning of the multiturn.jsonl distillation dataset to remove any artifacts, metadata, or malformed conversational turns.
Re-fine-tuning: Once the data is thoroughly cleaned, the model will be re-fine-tuned on the corrected dataset to address the learned repetitive patterns.
Quantitative Evaluation: Formal evaluation on standard QA benchmarks will be conducted post-cleaning and re-training to quantify the model's performance and assess the effectiveness of the distillation.
Acknowledgements
DeepSeek AI: For the powerful DeepSeek-R1 0528 teacher model.
Qwen Team (Alibaba Cloud): For the efficient Qwen 2.5 1.5B base model.
Unsloth AI: For the highly optimized fine-tuning framework that accelerated the training process.
Hugging Face: For the transformers library, bitsandbytes, and the Hugging Face Hub, which are indispensable tools for LLM development and sharing.
