LRC-4B-SFT is a Small Language Model (SLM) with approximately 4 billion parameters. It is the Supervised Fine-Tuned (SFT) version of LRC-4B-Base. The LRC method is an efficient knowledge distillation technique used to construct the base model from its teacher, Qwen2.5-7B-Instruct, using 18 billion tokens. This SFT version was then further fine-tuned on an instruction-following dataset ultrachat_200k.
The LRC approach trains a set of low-rank projection matrices that enable soft pruning by compressing teacher weights and an "activation clone" mechanism that aligns student activations (including FFN signals) with those of the teacher. The base model, LRC-4B-Base, was trained on 18 billion tokens.
Uses
Direct Use
LRC-4B-SFT is an instruction-tuned model and is intended for tasks requiring instruction following, question answering, and general chat capabilities.
Biases, Risks, and Limitations
SFT Dataset Limitations: LRC-4B-SFT was fine-tuned solely on the UltraChat dataset (using 0.2B tokens). While UltraChat enhances general instruction-following, it may not be sufficiently diverse or targeted to instill robust safety alignment or complex instruction adherence compared to models trained with more extensive or specialized alignment techniques (e.g., RLHF, or SFT on broader safety/instruction datasets). Consequently, the model might exhibit deficiencies in safety and its ability to follow highly complex or nuanced instructions. This is particularly evident in the IFeval and TruthfulQA benchmarks where performance decreased after SFT compared to its base model.
Inherited Biases: The model may reflect biases present in its pre-training data (a mix of Fineweb-Edu, OpenHermes 2.5, DCLM, and CosmopiediaV2) and the teacher model (Qwen2.5-7B-Instruct).
Hallucination: Like all LLMs, LRC-4B-SFT can generate factually incorrect or nonsensical information (hallucinations).
Limited Scope of Evaluation: The paper's primary evaluation focuses on pre-training efficiency and general downstream tasks. Extensive testing on safety benchmarks or complex reasoning tasks beyond the reported MMLU, ARC, etc., was not detailed.
How to Get Started with the Model
❗ Critical: For vLLM serving, please specify model-impl==transformers when using qwen series model. This is because, in the current implementation of vLLM, the qwen model does not support setting a custom head_dim through the config. Fortunately, vLLM allows using transformers as the backend.
Tested versions that can serve properly: vllm==0.8.5.post1 and transformers==4.51.3.
1from transformers import AutoTokenizer, AutoModelForCausalLM
23tokenizer = AutoTokenizer.from_pretrained('JitaiHao/LRC-4B')# Assuming this will be the HF repo name4model = AutoModelForCausalLM.from_pretrained('JitaiHao/LRC-4B')# Assuming this will be the HF repo name56# Prepare a multi-turn chat history7messages =[8{"role":"user","content":"Hello, who are you?"},9{"role":"assistant","content":"Hello, I am an AI assistant."}10]1112# Use apply_chat_template to create a prompt for the model13input_text = tokenizer.apply_chat_template(14 messages,15 tokenize=False,# Only generate the string prompt, do not tokenize yet16 add_generation_prompt=True# Add a generation prompt for the assistant17)1819print(input_text)# View the generated prompt string2021# If you want to generate a response with the model22inputs = tokenizer(input_text, return_tensors="pt")23outputs = model.generate(**inputs)24print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Training Details
Training Data
Pre-training (for LRC-4B-Base):18 billion tokens from the "Mixed-2.0" dataset, which includes Fineweb-Edu, OpenHermes 2.5, DCLM, and CosmopiediaV2.
Supervised Fine-Tuning (SFT):0.2 billion tokens from the UltraChat dataset.
Training Procedure
Pre-training (LRC-4B-Base): Trained using the Low-Rank Clone (LRC) method.
Teacher Model: Qwen2.5-7B-Instruct
Supervised Fine-Tuning (SFT):
Dataset: UltraChat (0.2B tokens)
Learning Rate (SFT): 1.0 x 10⁻⁵
Evaluation
Zero-Shot Comparison with other publicly available SFT models (from Table 2 of the paper):
Model
# Tokens
ARC-E
ARC-C
LogiQA
CSQA
PIQA
WinoG
BoolQ
SciQ
MMLU
Avg.
Gemma3-4B
4T
82.53
57.08
33.03
69.37
76.44
69.38
83.94
95.50
57.58
69.43
Minitron-4B
94B
79.59
54.35
30.26
71.09
77.64
65.93
82.60
96.60
56.77
68.31
Qwen3-4B
36T
80.47
53.58
33.64
75.76
75.08
65.27
84.95
95.50
68.38
70.29
LRC-4B-SFT
18B
78.37
52.47
34.10
79.28
76.82
67.72
84.50
95.00
64.41
70.30
Performance on safety and instruction-following tasks (LRC-4B refers to the SFT version, LRC-4B-B refers to the base version):
Benchmark
Metric
Score (LRC-4B-SFT)
Score (LRC-4B-Base)
ToxiGen
Accuracy Norm
43.72
43.83
IFeval
Instance-Level Loose Acc
13.67
36.09
TruthfulQA
MC2
50.71
55.89
The significant decrease in IFeval and TruthfulQA scores after SFT compared to the base model highlights the impact of the SFT dataset's specificity and potential lack of coverage for these nuanced capabilities.
