Anexa is a minimal, educational, yet production-grade implementation of a GPT (Generative Pre-trained Transformer) language model.
It began as a "from scratch" Pure Python project to understand the math behind AI, and has evolved into a high-performance PyTorch model capable of training on GPUs and generating coherent text.
🚀 Why Anexa?
In a world of closed-source giants like ChatGPT and Gemini, Anexa stands for:
Understanding: Built line-by-line to teach you how Transformers work.
Speed: Runs locally on your machine. No queues, no latency, no internet required.
Privacy: Your data never leaves your laptop.
Control: You own the weights. You own the code.
📚 Documentation
We have detailed documentation for both versions of the architecture:
Old Architecture (Pure Python): The original educational version. Learn how backpropagation and matrix multiplication work under the hood without any libraries.
New Architecture (PyTorch): The modern version of Anexa. Features GPU acceleration, SafeTensors persistence, and standard PyTorch layers.
⚡ Quick Start
Anexa uses PyTorch and SafeTensors for maximum performance.
1. Installation
Create a virtual environment and install dependencies (requires NVIDIA GPU for CUDA support):
Run the model script. It will automatically download "The Adventures of Sherlock Holmes", train the model for 5000 steps, and save it to model.safetensors.
python anexagpt.py
3. Interactive Mode
After training, the script enters an interactive mode where you can chat with the model:
Training finished...
Model saved to model.safetensors
--- Interactive Generation ---
Enter a prompt to continue (or 'quit' to exit)
Prompt: Sherlock
Response: Sherlock Holmes, who was usually very late in the mornings...
✨ New: Thinking Mode (Experimental)
Anexa now supports Reasoning Tokens (<think>...</think>), similar to DeepSeek-R1.
How it works: The model outputs a "Chain of Thought" before the final answer.
Visualization: In chat.py, thoughts are displayed in Gray to distinguish them from the answer.
Note: At 0.8M parameters, the reasoning is currently structural only (it mimics the format but makes arithmetic errors).
⚔️ Anexa vs The Giants
Here is how your custom 0.8M parameter model compares to state-of-the-art LLMs. While Anexa is tiny, it beats them in Privacy and Wait Time.
Model
Parameters
Training Data
Hardware
Capability
Speed
Anexa
0.8 Million
1 Book (Sherlock)
1 Laptop GPU
Basic Completion
Instant (Local)
GPT-2 (Small)
124 Million
WebText (8M Docs)
Clusters
Paragraphs
Fast
DeepSeek-V3
671 Billion
Trillions of Tokens
H100 Clusters
Reasoning/Math
Network Lag
Kimi-k2.5
hundreds of Bills
Long-context experts
Massive Clusters
2M+ Context
Network Lag
Gemini 1.5
~Trillions
Google's Index
TPU Pods
Multimodal
Network Lag
GPT-4
~1.8 Trillions
Entire Internet
Data Centers
Advanced Reasoning
Network Lag
Detailed Comparison
Anexa (0.8M):
Best For: Learning how AI works, running locally on old hardware, 100% privacy, zero latency.
Weakness: Can only write simple sentences. Cannot code or solve math.
Analogy: A bicycle. You built it, you understand it, it's free, and reliable for short trips.
ChatGPT / DeepSeek / Kimi:
Best For: Coding, complex math, summarizing long documents, creative writing.
Weakness: Requires internet, privacy concerns (data sent to servers), can be slow (queues).
Analogy: A commercial jet. Powerful, fast, gets you across the world, but you don't fly it and you adhere to their schedule.
Analogy: A commercial jet. Powerful, fast, gets you across the world, but you don't fly it and you adhere to their schedule.
🏆 Benchmark Comparison (Hypothetical)
Here is where the giants stand (and where Anexa is, humbly):
