Views
No views yet
[!Note]
The i3 architecture is present within the model for highly efficient pretraining. It is designed to reduce memory usage, speed up training, and allow pretraining from scratch on tiny hardware. Internal details are abstracted for simplicity.
1from transformers import pipeline
2
3pipe = pipeline("text-generation", model="FlameF0X/i3-12m")
4messages = [
5 {"role": "user", "content": "Who are you?"},
6]
7pipe(messages)Input Tokens
│
+-----------------+
| Embedding Layer |
+-----------------+
│
+-----------------+
| i3 Architecture |
+-----------------+
│
+------------------------+
| KQV Low-Rank Attention |
+------------------------+
│
+-----------------------+
| LayerNorm + Residuals |
+-----------------------+
│
+-------------------+
| Output Projection |
+-------------------+
│
Predicted TokensKey idea: Every component is optimized for memory efficiency and pretraining speed on small hardware, while preserving essential transformer dynamics.
1@software{lorpt2025,
2 title={LoRPt: Low-Rank Pretraining for Resource-Efficient Language Models},
3 author={[FlameF0X]},
4 year={2025},
5 url={https://github.com/FlameF0X/Low-Rank-Pretraining}
6}