Views
No views yet
avg_500m_k2resultsloss_log.csvloss_log_0.1.csvcheckpoints/final.ptcheckpoints/step_00050000.ptcheckpoints/step_00100000.ptcheckpoints/step_00150000.ptcheckpoints/step_00200000.ptcheckpoints/step_00250000.ptcheckpoints/step_00300000.ptcheckpoints/step_00350000.ptcheckpoints/step_00400000.ptcheckpoints/step_00450000.ptcheckpoints/step_00500000.ptcheckpoints/step_00550000.ptcheckpoints/step_00600000.pt1import torch
2from huggingface_hub import hf_hub_download
3
4path = hf_hub_download('FAIRC/token-averaging-avg_500m_k2', 'checkpoints/final.pt')
5state = torch.load(path, map_location='cpu', weights_only=False)
6model.load_state_dict(state['model']) # your OLMAveraged / OLMTransformerBody
7print(state['step'], state['tokens_seen'], state['cumulative_flops'])transformers weights. Rebuild the
architecture from config.json → model_config (or from
experiments/chinchilla/model_configs.py in the source repo) and load
the raw state_dict.1{
2 "d_model": 1280,
3 "n_heads": 20,
4 "n_layers": 22,
5 "context_len": 1024,
6 "averaging_k": 2,
7 "tie_embeddings": true,
8 "lr": 0.00012,
9 "warmup_steps": 2000,
10 "target_tokens": 20000000000,
11 "n_params_approx": 496866560
12}