You can deploy this model and start earning money today!

FAIRC/
token-averaging-model1_50m_tied_2ctx

Name: token-averaging-model1_50m_tied_2ctx
Availability: InStock

Views

No views yet

FAIRC/token-averaging-model1_50m_tied_2ctx

Checkpoint dump from the token averaging research project.

run name: model1_50m_tied_2ctx
results tree: results

Loss logs

loss_log.csv

Checkpoints

checkpoints/final.pt

Loading a checkpoint

python

import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download('FAIRC/token-averaging-model1_50m_tied_2ctx', 'checkpoints/final.pt')
state = torch.load(path, map_location='cpu', weights_only=False)
model.load_state_dict(state['model'])  # your OLMAveraged / OLMTransformerBody
print(state['step'], state['tokens_seen'], state['cumulative_flops'])

These are not Hugging Face transformers weights. Rebuild the architecture from config.json → model_config (or from experiments/chinchilla/model_configs.py in the source repo) and load the raw state_dict.

FAIRC/token-averaging-model1_50m_tied_2ctx

FAIRC/token-averaging-model1_50m_tied_2ctx

Contents

Loss logs

Checkpoints

Loading a checkpoint

FAIRC/
token-averaging-model1_50m_tied_2ctx