You can deploy this model and start earning money today!

AndreasBis/
TinyLlamaTalos-1B

Name: TinyLlamaTalos-1B
Availability: InStock

safetensors

Views

No views yet

TinyLlama UltraChat Fine-Tune

This repository contains a full fine-tuning run of TinyLlama/TinyLlama-1.1B-Chat-v1.0 on HuggingFaceH4/ultrachat_200k, trained with tinyllama-training.ipynb and saved under tinyllama-ultrachat-rslora/.

Notebook Highlights

Base model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
Dataset: HuggingFaceH4/ultrachat_200k
Full run: quick_test = False
Conversations are expanded into assistant-turn training rows
Context length: 512
LoRA variant: Rank-Stabilized LoRA
LoRA rank: 128
LoRA alpha: 256
LoRA dropout: 0.04
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Precision: bf16
Gradient checkpointing: enabled
Per-device train batch size: 256
Gradient accumulation steps: 4
Effective batch size: 1024
Learning rate: 1e-4
Epochs: 1.0
Eval subset during training: 4096
Output directory: ./tinyllama-ultrachat-rslora

Run Summary

Item	Value
Processed training rows	`657,771`
Processed evaluation rows	`73,150`
Optimizer steps	`643`
Epochs	`1.0`
Effective batch size	`1024`
Train runtime	`6:32:07`
Eval runtime	`0:00:38.21`
Train samples / second	`27.958`
Train steps / second	`0.027`
Eval samples / second	`107.196`
Eval steps / second	`0.419`
Final train loss	`1.0671`
Final eval loss	`1.0621`
Final perplexity	`2.8925`
Adapter output	`./tinyllama-ultrachat-rslora/adapter`
Merged output	`./tinyllama-ultrachat-rslora/merged`

Loss Table

Step	Training Loss	Validation Loss
32	`1.088547`	`1.108700`
64	`1.098695`	`1.115909`
96	`1.099704`	`1.114013`
128	`1.093774`	`1.107312`
160	`1.087225`	`1.102051`
192	`1.082110`	`1.098298`
224	`1.080011`	`1.094395`
256	`1.075894`	`1.090381`
288	`1.070766`	`1.085714`
320	`1.064308`	`1.081495`
352	`1.064451`	`1.078283`
384	`1.058922`	`1.074668`
416	`1.055673`	`1.071584`
448	`1.050642`	`1.068871`
480	`1.049796`	`1.066419`
512	`1.047866`	`1.064614`
544	`1.046279`	`1.063277`
576	`1.040401`	`1.062529`
608	`1.043520`	`1.062152`
640	`1.045642`	`1.062093`
643	`1.045642`	`1.062105`

Output Tree

text

tinyllama-ultrachat-rslora/
|-- adapter/
|   |-- README.md
|   |-- adapter_config.json
|   |-- adapter_model.safetensors
|   |-- chat_template.jinja
|   |-- tokenizer.json
|   |-- tokenizer_config.json
|   `-- training_args.bin
|-- checkpoint-608/
|   |-- README.md
|   |-- adapter_config.json
|   |-- adapter_model.safetensors
|   |-- chat_template.jinja
|   |-- optimizer.pt
|   |-- rng_state.pth
|   |-- scheduler.pt
|   |-- tokenizer.json
|   |-- tokenizer_config.json
|   |-- trainer_state.json
|   `-- training_args.bin
|-- checkpoint-640/
|   |-- README.md
|   |-- adapter_config.json
|   |-- adapter_model.safetensors
|   |-- chat_template.jinja
|   |-- optimizer.pt
|   |-- rng_state.pth
|   |-- scheduler.pt
|   |-- tokenizer.json
|   |-- tokenizer_config.json
|   |-- trainer_state.json
|   `-- training_args.bin
|-- checkpoint-643/
|   |-- README.md
|   |-- adapter_config.json
|   |-- adapter_model.safetensors
|   |-- chat_template.jinja
|   |-- optimizer.pt
|   |-- rng_state.pth
|   |-- scheduler.pt
|   |-- tokenizer.json
|   |-- tokenizer_config.json
|   |-- trainer_state.json
|   `-- training_args.bin
|-- merged/
|   |-- chat_template.jinja
|   |-- config.json
|   |-- generation_config.json
|   |-- model.safetensors
|   |-- tokenizer.json
|   `-- tokenizer_config.json
|-- all_results.json
|-- eval_results.json
|-- train_results.json
`-- trainer_state.json

Artifact Highlights

adapter/ contains the saved PEFT adapter weights and tokenizer files.
merged/ contains the fully merged Hugging Face model, including model.safetensors.
checkpoint-608/, checkpoint-640/, and checkpoint-643/ are the saved training checkpoints retained by Trainer.
Root-level JSON files contain the final train metrics, eval metrics, aggregate results, and trainer state.

AndreasBis/TinyLlamaTalos-1B