Vexion-GPT is a classic dense language model engine built from scratch on PyTorch. The project was created to deeply understand the architecture of Transformers, optimize memory, and pretraining processes without the use of heavy third-party frameworks.
⚙️ Key Features of the Engine (Under the Hood)
Unlike many "training" models, Vexion-GPT is designed for real-world big data work and maximum GPU utilization:
Pure Dense Architecture: Classic, mathematically pure GPT architecture without additives (no MoE, no RoPE). Only the proven Causal Attention and GELU/SiLU activation functions.
Flash Attention Integrated: Full support for Fused Kernels for on-the-fly attention computation. VRAM consumption has been dramatically reduced (the model can easily train on consumer GPUs with batch sizes that previously caused OOMs).
Ultra-fast Custom DataLoader: The dataloader has been rewritten to stream binary data (.bin), bypassing Python's garbage collector and Windows system caching. The token feed rate is static and does not degrade over long distances.
HF-Compatible Config: The architecture is completely decoupled from hardcoded data. Model configuration is implemented via config.json according to Hugging Face standards (full support for hidden_size, num_hidden_layers, etc.).
Adaw: Standard Adaw allows the model to store all its values in 32-bit format, which helps achieve lower LOSS and higher-quality models.
📊 Vexion Model Family
The project's development is divided into several stages, from compact test versions to fully-fledged billion-dollar vehicles.
Model
Parameters
Context
Status
Train_Loss
Val_Loss
Vocab_Size
Total_Steps
Vexion-gpt
117M
1024
Base model
3.21
3.1
40960
220000
Vexion-gpt medium
345M
1024
Base model
2.58
2.53
40960
382500
Vexion-gpt large
645M
2048
Base model
3.76
3.72
40960
71000
Vexion-gpt XL
1B
1024
train
-
-
40960
-
Vexion-gpt XXL
2.5B
1024
-
-
-
40960
-
Vexion-gpt-3b
3B
1024
-
-
-
40960
-
📊 Model training dynamics
Снимок экрана (435)
Снимок экрана (438)
Technical details:
Hardware: 1x RTX 3060 Ti 8GB or later on 1x RTX 4060 Ti 16GB.
Settings: batch_size - 12, accumulate_steps - 8 at ctx 256. Batch_size - 8, accumulate_steps - 32 at ctx 1024.
Number of tokens in pre-train - 39 BILLION
Total training time: 570 hours
Pre-train dataset: CulturaX
📊 News (temporarily)
Okay, I didn’t expect myself that it would be possible to fit 3 billion parameters into 16 GB of memory — it’s hard to even imagine, let alone do. Yes, the speed is almost a minute per step, but in theory, in theory it’s possible to create it, and it even seems like it could be done. But is it worth spending a little over a year on this? I don't know. Will it replace XXL, or what will happen to it? It will remain in the table and in the plans. Simply because 2.5 BILLION parameters still fit adequately into the memory, and you can set the speed a bit faster than training “to the brim” in memory with 3 billion. I can’t guarantee that it will definitely be released, simply because the training would be too unbearably long. It might be released, but first XL and XXL will be released, and then we’ll take a look at XXXL. But without hardware improvements, it will be a bit difficult.
⚠️ Important compatibility warning
This model is built on a completely custom architecture written in pure PyTorch.
It DOES NOT support the Hugging Face transformers library (Transformer API). You will not be able to load it through standard classes like AutoModelForCausalLM. For inference and retraining, use only the scripts provided in this repository (model.py, generate.py, generation.py).
🚀 Code Usage
Running/Retraining the Model:
Before training the model, launch the command prompt (CMD) as administrator and enter the following command: cd C:\Users\Username\Desktop\model folder
