This is the instruction-tuned version of BUVN-2.0, fine-tuned on conversational data from Alpaca (52K instructions) and OpenAssistant (13.9K English conversation pairs).
The model learns to follow the chat template format and generate responses in a conversational style.
Step 0: val loss 3.19 (ppl 24.4) ← loaded pre-trained weights
Step 200: val loss 1.98 (ppl 7.2) ← rapidly learning chat format
Step 600: val loss 1.87 (ppl 6.5)
Step 1000: val loss 1.85 (ppl 6.4)
Step 2000: val loss 1.84 (ppl 6.3)
Step 3000: val loss 1.83 (ppl 6.3) ← converged
Chat Template
The model was trained with this conversation format:
<|user|>
What is the capital of France?
<|end|>
<|assistant|>
The capital of France is Paris.
<|end|>
Honest Assessment
At 109.5M parameters, the model successfully learns the chat format but has limited instruction-following accuracy. This is a known limitation of small models:
Model Size
Instruction Quality
109.5M (this model)
Learns format; answers often tangential to the question
350M+
Basic Q&A works for simple questions
1B+
Solid instruction following
3B+ (Phi-2, TinyLlama)
Good quality, genuinely useful
7B+ (LLaMA, Mistral)
Strong instruction following
The model needs to be scaled to 350M+ parameters for meaningful instruction following. The SFT pipeline and infrastructure are validated and ready for scaling.
-blue?style=for-the-badge&labelColor=0d1117){kind=icon}
