You can deploy this model and start earning money today!

nanochat-students/
base-d20

Name: base-d20
Availability: InStock

Views

No views yet

Pretrained base-d20

This model is trained with the nanochat recipe by Andrej Karpathy.

It was trained with a depth of 20 on 2 billion tokens and corresponds to this tokenizer. I will combine this repo with the tokenizer.

Usage

python

from transformers import AutoConfig, AutoModel, AutoTokenizer
import torch

model_dir = "nanochat-students/base-d20"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = AutoModel.from_pretrained(model_dir, trust_remote_code=True)
model = model.to(device)
model.eval()

tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)

prompt = "The capital of Belgium is "
input_ids = tokenizer.encode(prompt, prepend=tokenizer.get_bos_token_id())
ids = torch.tensor([input_ids], dtype=torch.long, device=device)

max_new_tokens = 50
with torch.inference_mode():
    for _ in range(max_new_tokens):
        outputs = model(input_ids=ids)
        logits = outputs["logits"] if isinstance(outputs, dict) else outputs.logits
        next_token = torch.argmax(logits[:, -1, :], dim=-1, keepdim=True)
        ids = torch.cat([ids, next_token], dim=1)

decoded = tokenizer.decode(ids[0].tolist())
print(decoded)

Base model evaluation

timestamp: 2025-10-14 16:16:53

Model: base_model (step 21400)
CORE metric: 0.1963
hellaswag_zeroshot: 0.2634
jeopardy: 0.0959
bigbench_qa_wikidata: 0.4993
arc_easy: 0.5269
arc_challenge: 0.1251
copa: 0.4400
commonsense_qa: 0.0653
piqa: 0.3743
openbook_qa: 0.1440
lambada_openai: 0.3683
hellaswag: 0.2630
winograd: 0.2674
winogrande: 0.0923
bigbench_dyck_languages: 0.1050
agi_eval_lsat_ar: 0.0326
bigbench_cs_algorithms: 0.3674
bigbench_operators: 0.1524
bigbench_repeat_copy_logic: 0.0000
squad: 0.2222
coqa: 0.1957
boolq: -0.4615
bigbench_language_identification: 0.1801

Base model loss

timestamp: 2025-10-14 16:11:41

train bpb: 0.8147
val bpb: 0.8121
sample 0: <|bos|>The capital of France is Paris. It is the largest city in France and the capital of the country.
sample 1: <|bos|>The chemical symbol of gold is Au and the atomic number is 79. Gold is a soft, malleable,
sample 2: <|bos|>If yesterday was Friday, then tomorrow will be Saturday. If today is Monday, then tomorrow will be Tuesday. If today is
sample 3: <|bos|>The opposite of hot is cold. The opposite of hot is cold. The opposite of hot is cold.
sample 4: <|bos|>The planets of the solar system are: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune,
sample 5: <|bos|>My favorite color is blue. I love the color blue because it is a color that is so versatile
sample 6: <|bos|>If 5x + 3 = 13, then x is a factor of 5. If 5x + 3 =

Base model training

timestamp: 2025-10-14 14:28:31

run: dummy
depth: 20
max_seq_len: 2048
num_iterations: -1
target_flops: -1.0000
target_param_data_ratio: 20
device_batch_size: 32
total_batch_size: 524,288
embedding_lr: 0.2000
unembedding_lr: 0.0040
weight_decay: 0.0000
matrix_lr: 0.0200
grad_clip: 1.0000
eval_every: 250
eval_tokens: 10,485,760
core_metric_every: 2000
core_metric_max_per_task: 500
sample_every: 2000
model_tag:
Number of parameters: 560,988,160
Number of FLOPs per token: 3.491758e+09
Calculated number of iterations: 21,400
Number of training tokens: 11,219,763,200
Tokens : Params ratio: 20.0000
DDP world size: 8
warmup_ratio: 0.0000
warmdown_ratio: 0.2000
final_lr_frac: 0.0000
Minimum validation bpb: 0.8120
Final validation bpb: 0.8120
CORE metric estimate: 0.2059
MFU %: 48.36%
Total training flops: 3.917670e+19
Total training time: 172.18m
Peak memory usage: 75422.02MiB

Training Logs

Logs are available on the trackio space here