This is a ChatGPT-like model trained using the nanochat pipeline.
Model Description
nanochat is a minimalist but full-featured GPT-style language model trained from scratch, following the complete pipeline from tokenizer training to chat fine-tuning. This model demonstrates that high-quality conversational AI can be trained efficiently with modern techniques.
Training Details
Architecture
Model Type: GPT decoder-only transformer
Parameters: ~561M (d20 configuration)
Context Length: 2048 tokens
Depth: 20 layers
Vocabulary Size: 65,536 tokens
Training Infrastructure
Hardware: 8× NVIDIA A100-SXM4-80GB (634GB total GPU memory)
This model requires the nanochat codebase to run. See the nanochat repository for inference instructions.
Quick Start
bash
1# Clone the repository2git clone https://github.com/karpathy/nanochat.git
3cd nanochat
45# Download model from HuggingFace67# Run inference8python generate.py --checkpoint path/to/model_000650.pt
Model Files
model_000650.pt: Chat SFT model weights (final checkpoint)
meta_000650.json: Model metadata and configuration
tokenizer.pkl: Custom BPE tokenizer
token_bytes.pt: Token to byte mapping
Sample Outputs
From the base model (before chat tuning):
The capital of France is Paris. It is the largest city in France and the capital of the country.
The chemical symbol of gold is Au. It is a soft, silvery-white metal that is malleable and ductile.
The planets of the solar system are: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune,
