Views
No views yet
avg_50m_k8_learnable_posresultsloss_log.csvcheckpoints/final.ptcheckpoints/step_00350000.ptcheckpoints/step_00400000.ptcheckpoints/step_00450000.pt1import torch
2from huggingface_hub import hf_hub_download
3
4path = hf_hub_download('FAIRC/token-averaging-avg_50m_k8_learnable_pos', 'checkpoints/final.pt')
5state = torch.load(path, map_location='cpu', weights_only=False)
6model.load_state_dict(state['model']) # your OLMAveraged / OLMTransformerBody
7print(state['step'], state['tokens_seen'], state['cumulative_flops'])transformers weights. Rebuild the
architecture from config.json → model_config (or from
experiments/chinchilla/model_configs.py in the source repo) and load
the raw state_dict.1{
2 "d_model": 512,
3 "n_heads": 8,
4 "n_layers": 8,
5 "context_len": 1024,
6 "averaging_k": 8,
7 "tie_embeddings": true,
8 "method_name": "learnable_pos_k8",
9 "lr": 0.0002,
10 "warmup_steps": 2000,
11 "target_tokens": 8144000000,
12 "n_params_approx": 50897408
13}