You can deploy this model and start earning money today!

yasuiniko/
nanochat-d8-base-39000

Name: nanochat-d8-base-39000
Rating: 2.5 (8 reviews)

Views

No views yet

Thanks to Andrej Karpathy and Harley Cooper for doing all the work.

nanochat

This repository contains a nanochat checkpoint exported for Hugging Face. The architecture mirrors GPT-style decoders with rotary embeddings, RMSNorm, multi-query attention, relu² MLPs, and untied embeddings.

Model Summary

Parameters: ~8.4M
Layers: 8
Hidden size: 512
Attention heads: 4 (KV heads: 4)
Context length: 2048
Tokenizer: nanochat rustbpe (BPE, 65k vocab)

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("yasuiniko/nanochat-d8-base-39000", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("yasuiniko/nanochat-d8-base-39000", trust_remote_code=True)

inputs = tokenizer("Hello nanochat!", return_tensors="pt")
output_ids = model.generate(**inputs, max_length=128)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Remember to pass trust_remote_code=True so Transformers picks up the custom model and tokenizer implementations bundled in this repository.