You can deploy this model and start earning money today!

pbarcelos1/
nanochat-pt-latn-d12-midtrain

Name: nanochat-pt-latn-d12-midtrain
Availability: InStock

pbarcelos1/nanochat-pt-latn-d12

finetune

mit

Views

No views yet

nanochat PT-Latn d12 — mid-trained on QA-PT (no loss mask)

This is a mid-trained model: the base pre-trained checkpoint continued on Portuguese QA data (no loss mask). It bridges raw web-text pre-training and supervised fine-tuning.

Forked from karpathy/nanochat and trained on ju-resplande/qa-pt (full sequence, no mask).

Model details


Architecture	Decoder-only transformer (relu² FFN, RoPE, GQA, RMSNorm)
Layers	12
d_model	768
Attention heads	6 (KV heads: 6)
Sequence length	2048
Vocabulary	32,768 (byte-level BPE, PT-trained)
Trained steps	572
Final val bpb	0.6753
Training time	0.9 h on 1× NVIDIA RTX A6000

Usage

Load with nanochat:

python

import os
import torch
from huggingface_hub import snapshot_download
from nanochat.checkpoint_manager import build_model, find_last_step
from nanochat.engine import Engine

local_dir = snapshot_download(repo_id="pbarcelos1/nanochat-pt-latn-d12-midtrain")
os.environ["NANOCHAT_BASE_DIR"] = local_dir

device = torch.device("cuda:0")
step = find_last_step(local_dir)
model, tokenizer, meta = build_model(local_dir, step, device, phase="eval")

engine = Engine(model, tokenizer)
tokens = tokenizer("Era uma vez,", prepend="<|bos|>")
samples, _ = engine.generate_batch(tokens, num_samples=1, max_tokens=200, temperature=0.8)
print(tokenizer.decode(samples[0]))