Sana 24M Instruct — Model Card
Model Summary
Sana 24M Instruct is a small decoder-only instruction-tuned language model trained using a custom PyTorch implementation. The model is designed as an educational and experimental instruct model for studying language model pretraining, tokenizer construction, supervised fine-tuning, and inference without relying on high-level model libraries such as Hugging Face Transformers or SentencePiece.
The model has approximately 24 million parameters and uses a hand-built 16,000-token BPE tokenizer trained in pure Python. The full training pipeline includes data preparation, tokenizer training, base-model pretraining, supervised instruction fine-tuning, testing, and interactive inference.
Unlike a base model, Sana 24M Instruct has been fine-tuned to respond more directly to user prompts and simple instructions.
Architecture
Sana 24M Instruct is a compact GPT-style Transformer language model with:
- Approximately 24M parameters
- Decoder-only Transformer architecture
- RMSNorm
- RoPE positional embeddings
- SwiGLU feed-forward layers
- Multi-head self-attention
- Tied input/output embeddings
- 512-token context length
- KV cache support during generation
- Custom 16k BPE tokenizer
The model and tokenizer are implemented from scratch using PyTorch and standard Python utilities.
Training Data
The base model was pretrained on educational web text from FineWeb-Edu using the standard causal language modelling objective: predicting the next token from previous tokens.
After pretraining, the model was supervised fine-tuned on instruction-style conversation data so that it can better follow prompts, answer simple questions, and produce assistant-like responses.
The instruction tuning stage is intended to improve usability, but the model remains small and experimental.
Intended Use
This model is intended for:
- Educational language model research
- Understanding Transformer internals
- Studying tokenizer training
- Experimenting with pretraining and instruction fine-tuning
- Small-scale chatbot and inference demos
- Portfolio and learning projects
- Testing custom inference pipelines and KV cache generation
It is not intended for production use, high-stakes decision-making, medical advice, legal advice, financial advice, or safety-critical applications.
Limitations
Sana 24M Instruct is a very small language model compared with modern production LLMs. It may:
- Generate incorrect or hallucinated information
- Misunderstand user instructions
- Struggle with multi-step reasoning
- Produce repetitive or low-quality responses
- Fail at complex maths or coding tasks
- Lack reliable factual accuracy
- Give overly short, vague, or inconsistent answers
- Reflect biases or errors from its training and fine-tuning data
The model does not have built-in retrieval, browsing, or external knowledge access unless connected to a separate RAG or tool system.
Safety and Risks
Although Sana 24M Instruct has been fine-tuned to follow instructions, it is not fully safety-aligned. Outputs should be treated as experimental and should not be trusted for sensitive or high-impact decisions.
Users should verify important information independently. The model may produce unsafe, biased, misleading, or factually incorrect responses, especially when asked about complex or sensitive topics.
Implementation Notes
This project intentionally avoids high-level model frameworks. The goal is to show the full language model pipeline end-to-end:
- Custom Transformer implementation
- Custom BPE tokenizer
- FineWeb-Edu data preparation
- Binary shard tokenization
- Base-model pretraining loop
- Supervised instruction fine-tuning loop
- Inference with KV cache
- Testing scripts for pretrained and instruction-tuned checkpoints
- Interactive chat-style inference
License
license: apache-2.0
