Views
No views yet
avg_8m_k4resultsloss_log.csvcheckpoints/final.pt1import torch
2from huggingface_hub import hf_hub_download
3
4path = hf_hub_download('FAIRC/token-averaging-avg_8m_k4', 'checkpoints/final.pt')
5state = torch.load(path, map_location='cpu', weights_only=False)
6model.load_state_dict(state['model']) # your OLMAveraged / OLMTransformerBody
7print(state['step'], state['tokens_seen'], state['cumulative_flops'])transformers weights. Rebuild the
architecture from config.json → model_config (or from
experiments/chinchilla/model_configs.py in the source repo) and load
the raw state_dict.1{
2 "d_model": 128,
3 "n_heads": 4,
4 "n_layers": 6,
5 "context_len": 512,
6 "averaging_k": 4,
7 "tie_embeddings": true,
8 "lr": 0.0004,
9 "warmup_steps": 500,
10 "target_tokens": 800000000,
11 "n_params_approx": 7612544
12}