Views
No views yet
avg_50m_k4_learnableresultsloss_log.csvcheckpoints/final.ptcheckpoints/step_00050000.ptcheckpoints/step_00100000.ptcheckpoints/step_00150000.ptcheckpoints/step_00200000.pt1import torch
2from huggingface_hub import hf_hub_download
3
4path = hf_hub_download('FAIRC/token-averaging-avg_50m_k4_learnable', 'checkpoints/final.pt')
5state = torch.load(path, map_location='cpu', weights_only=False)
6model.load_state_dict(state['model']) # your OLMAveraged / OLMTransformerBody
7print(state['step'], state['tokens_seen'], state['cumulative_flops'])transformers weights. Rebuild the
architecture from config.json → model_config (or from
experiments/chinchilla/model_configs.py in the source repo) and load
the raw state_dict.1{
2 "d_model": 512,
3 "n_heads": 8,
4 "n_layers": 8,
5 "context_len": 1024,
6 "averaging_k": 4,
7 "tie_embeddings": true,
8 "method_name": "learnable_k4",
9 "lr": 0.0002,
10 "warmup_steps": 2000,
11 "target_tokens": 4072000000,
12 "n_params_approx": 50897408
13}