nanochat-d34-finetuned

Model Description

• Base Model: karpathy/nanochat-d34 (2.2B parameters, pre-trained on 88B tokens)
• Architecture: GPT-style transformer with depth=34
• Training Pipeline: Mid-training → SFT → RL (optional)
• Hardware: 8x A100-80GB GPUs

Training Details

Base Model (Pre-trained by Karpathy)

• Parameters: 2,217,082,880
• Training tokens: 88,683,315,200 (40x param:token ratio)
• Max sequence length: 2048
• Base CORE score: 0.3382

Dataset

Eval Reports

Fine-tuning Pipeline

1. Mid-training: General instruction tuning on SmolTalk, MMLU, GSM8K, Spelling tasks
2. SFT (Supervised Fine-Tuning): Chat-specific training on ARC, GSM8K, SmolTalk
3. RL (Reinforcement Learning): Optional GRPO-style training on GSM8K (if included)

WandB Reports

Repository Structure

├── tokenizer/
│   ├── tokenizer.pkl          # Tokenizer
│   └── token_bytes.pt         # Token byte mappings
├── mid_checkpoints/d34/       # Mid-training checkpoint
│   ├── model_*.pt
│   └── meta_*.json
├── chatsft_checkpoints/d34/   # SFT checkpoint
│   ├── model_*.pt
│   └── meta_*.json
├── chatsft_checkpoints_int8/d34/   # SFT checkpoint
│   ├── model_*.pt
│   └── meta_*.json
├── chatrl_checkpoints/d34/    # RL checkpoint (if available)
│   ├── model_*.pt
│   └── meta_*.json
├── report/                    # Evaluation reports
│   └── report.md
└── logs/                      # Training logs

License

Acknowledgments

• Andrej Karpathy for the nanochat framework and pre-trained base model

• The nanochat community