Sana 24M Base — Model Card
Model Summary
Sana 24M Base is a small decoder-only language model trained from scratch using a custom PyTorch implementation. The model is designed as an educational and experimental base model for studying language model pretraining, tokenizer construction, fine-tuning, and inference without relying on high-level model libraries such as Hugging Face Transformers or SentencePiece.
The model has approximately 24 million parameters and uses a hand-built 16,000-token BPE tokenizer trained in pure Python. The full training pipeline includes data preparation, tokenizer training, pretraining, supervised fine-tuning, testing, and interactive inference.
Architecture
Sana 24M Base is a GPT-style Transformer language model with:
- Approximately 24M parameters
- 10 Transformer blocks
- 384 hidden dimension
- 6 attention heads
- 64-dimensional attention heads
- 1,024-dimensional SwiGLU feed-forward layer
- RMSNorm
- RoPE positional embeddings
- Tied input/output embeddings
- 512-token context length
- KV cache support during generation
- Custom 16k BPE tokenizer
The model and tokenizer are implemented from scratch using PyTorch and standard Python utilities.
Training Data
The base model was pretrained on educational web text from FineWeb-Edu. The pretraining objective is standard causal language modelling: predicting the next token from previous tokens.
The model is intended to learn general language patterns, basic factual associations, reasoning patterns, and text continuation ability from pretraining.
Intended Use
This model is intended for:
- Educational language model research
- Understanding Transformer internals
- Studying tokenizer training
- Experimenting with pretraining and fine-tuning
- Small-scale inference demos
- Portfolio and learning projects
It is not intended for production use, high-stakes decision-making, medical advice, legal advice, financial advice, or safety-critical applications.
Limitations
Sana 24M Base is a very small language model compared with modern production LLMs. It may:
- Generate incorrect or hallucinated information
- Struggle with long-context reasoning
- Produce repetitive or low-quality text
- Fail at complex maths or coding tasks
- Lack reliable factual accuracy
- Reflect biases or errors from its training data
The model does not have built-in retrieval, browsing, or external knowledge access unless connected to a separate RAG or tool system.
Safety and Risks
Because this is a base model, it is not fully instruction-following or safety-aligned. Outputs should be treated as experimental. Users should verify important information independently and avoid using the model for sensitive or high-impact decisions.
Implementation Notes
This project intentionally avoids high-level model frameworks. The goal is to show the full language model pipeline end-to-end:
- Custom Transformer implementation
- Custom BPE tokenizer
- FineWeb-Edu data preparation
- Binary shard tokenization
- Pretraining loop
- Supervised fine-tuning loop
- Inference with KV cache
- Testing scripts for pretrained and fine-tuned checkpoints
License
license: apache-2.0
