| Base model | HuggingFaceTB/SmolLM2-135M-Instruct |
| Model type | Causal Language Model |
| Fine-tuning method | SFT + LoRA |
| Language | English |
| License | Apache 2.0 |
all| Parameter | Value |
|---|---|
| Learning rate | 2e-4 |
| Training steps | 200 |
| Batch size | 2 |
| Packing | True |
| Optimizer | AdamW |
| LR scheduler | Linear |
| Precision | fp16 |
| Parameter | Value |
|---|---|
Rank (r) | 16 |
Alpha (lora_alpha) | 16 |
| Dropout | 0.05 |
| Target modules | all-linear |
| Bias | none |
| Trainable parameters | ~1.7M / 135M (1.26%) |
Machine learning is a field of AI where systems learn from data to make predictions or decisions without being explicitly programmed for each task.
A Transformer is a neural network architecture that uses attention mechanisms to understand relationships between all words in a sentence simultaneously, rather than reading them one by one like older models did.
Full fine-tuning updates all parameters of a model which requires a lot of memory. LoRA instead freezes the base model and adds small trainable matrices to each layer, training only ~1% of parameters while achieving similar results.