Views
No views yet
| architecture | looped |
| d_model | 640 |
| effective layers | 32 |
| compute budget | 1e18 FLOPs |
| training steps | 45,632 |
| parameters (stored) | 143,627,520 |
| peak LR | 0.005 |
| batch size | 16 |
| muP width_ratio | 2.5 (d_base=256) |
| final val loss | n/a |
1from transformers import AutoModelForCausalLM, AutoTokenizer
2m = AutoModelForCausalLM.from_pretrained(
3 "ml-ryanlee/looped-32L-d640-1e18", trust_remote_code=True)
4tok = AutoTokenizer.from_pretrained("gpt2")max_length=1024 when evaluating — the RoPE buffer is sized to the training context.