You can deploy this model and start earning money today!

pbarcelos1/
nanochat-pt-latn-d12

Name: nanochat-pt-latn-d12
Availability: InStock

Views

No views yet

nanochat PT-Latn d12 — base language model

Pretrained Portuguese (PT-Latn) language model, forked from karpathy/nanochat and trained on HuggingFaceFW/fineweb-2 subset por_Latn.

Model details


Architecture	Decoder-only transformer (relu² FFN, RoPE, GQA, RMSNorm)
Layers	12
d_model	768
Attention heads	6 (KV heads: 6)
Sequence length	2048
Vocabulary	32,768 (byte-level BPE, PT-trained)
Trained steps	4,830
Training tokens	~2.53 B
Final val bpb	0.9487 (PT-Latn validation shard)
Training time	12.2 h on 1× NVIDIA RTX A6000

Usage

This is a base model (not instruction-tuned). Load with nanochat:

python

import os
import torch
from huggingface_hub import snapshot_download
from nanochat.checkpoint_manager import build_model, find_last_step
from nanochat.engine import Engine

# Download repo to local cache (~760 MB, cached after first run)
local_dir = snapshot_download(repo_id="pbarcelos1/nanochat-pt-latn-d12")
os.environ["NANOCHAT_BASE_DIR"] = local_dir  # needed for tokenizer lookup

device = torch.device("cuda:0")
step = find_last_step(local_dir)
model, tokenizer, meta = build_model(local_dir, step, device, phase="eval")

engine = Engine(model, tokenizer)
tokens = tokenizer("Era uma vez,", prepend="<|bos|>")
samples, _ = engine.generate_batch(tokens, num_samples=1, max_tokens=200, temperature=0.8)
print(tokenizer.decode(samples[0]))

Limitations

Base model only — raw continuations, no chat/instruction following
Portuguese text only — no cross-lingual transfer evaluated
CORE benchmark (English ICL) is not applicable; results are near-random