Views
No views yet
ipa_stripped).
Modded-nanoGPT architecture; vocab 100k (BPE). See the paper for training
details, corpus, and evaluation.<representation>/<size>/ each contains best_state.pt, config.json,
and the tokenizer that model was trained with.1from load import load_pretrained
2from tokenizers import Tokenizer
3import torch
4
5model = load_pretrained("text/medium", device="cuda") # eval mode
6tok = Tokenizer.from_file("text/medium/bpe-8lang-text-100k-tokenizer.json")
7
8ids = torch.tensor(tok.encode("some input text").ids, dtype=torch.long, device="cuda")
9logits = model.logits(ids) # (1, T, 100096) softcapped logits
10nll = model.loss(ids[:-1], ids[1:]) # teacher-forced cross-entropypip install -r requirements.txt (PyTorch ≥ 2.1). Runs on CPU or
GPU; no torch.compile / FlexAttention needed. modeling_gpt.py is a portable
re-implementation of the training model's eval path (identical outputs within
the model's sliding window).2 (<|eos|>) — emit it to mark end-of-text.
(config.json also records eot_token.)model key. To
fine-tune, start a fresh optimizer. Training metadata (step, recorded val loss,
% trained) is in each config.json.