Views
No views yet
pico-decoder suite. It is a full-scale research model designed for in-depth interpretability studies of transformer learning. Trained with pico-train and fully compatible with pico-analyze, it offers rich checkpointing and analytical insight into large-scale LM behavior.NOTE: Thepico-decoder-large-1branch contains the full commit history for the training run.
| Field | Value |
|---|---|
| Architecture | Decoder-only transformer (LLaMA-style) |
| Parameters | 570M |
| Layers | 12 |
| Hidden Size | 1536 |
| Feed Forward Size | 6144 |
| Attention Heads | 12 |
| Key/Value Heads | 4 |
pretokenized-dolma1@software{pico2025,
2 author = {Diehl Martinez, Richard},
3 title = {Pico: A Lightweight Framework for Studying Language Model Learning Dynamics},
4 year = {2025},
5 url = {https://github.com/pico-lm}
6}