Views
No views yet
mini-math-teacher-v1-30K is the 30,000-step teacher-model training run for this project: a 146.8M-parameter causal language model built with a GQA-based transformer and trained on a mixed web-and-math corpus.base model: model.safetensorscheckpoint model: checkpoints_model_30000.safetensors (includes optimizer/scaler state for resuming training)| Field | Value |
|---|---|
| Model type | Causal LM (GQATransformer) |
| Attention | Grouped Query Attention |
| Parameters | 146,790,432 (100% trainable) |
| Layers | 18 |
| Hidden size | 768 |
| Attention heads | 16 |
| Context length | 1024 |
| Precision during training | bfloat16 AMP |
| Optimizer | AdamW |
| Learning rate | 1.2e-4 |
| Weight decay | 0.1 |
| Epsilon | 1e-8 |
| Gradient clipping | 1.0 |
| Gradient accumulation | 12 |
| Effective global batch size | 144 |
| Gradient checkpointing | enabled |
| Field | Value |
|---|---|
| Total steps | 30,000 |
| World size | 2 GPUs |
| Training time | 139,166.93 seconds (~38.66 hours) |
| Tokens trained (local) | 2,209,680,000 |
| Tokens trained (global estimate) | 4,419,360,000 |
| Best train loss | 2.7606 |
| Final train loss | 3.5598 |
| Worst train loss | 10.5604 |
| Best val loss | 3.2709 (step 30,000) |
| Final val loss | 3.2709 |
| Dataset | Split | Text column | Rows | Share |
|---|---|---|---|---|
HuggingFaceFW/fineweb (sample-10BT) | train | text | 8,363,014 | 56.75% |
open-web-math/open-web-math | train | text | 6,315,233 | 42.85% |
incredible45/Gutenberg-BookCorpus-Cleaned-Data-English | train | context | 58,653 | 0.40% |
success=true indicates inference success, not answer correctness.