🧠 GT-730 MoE Reasoning Transformer (GGUF)
DemonKing1234/gt730-moe-transformer is a lightweight, edge-optimized Mixture-of-Experts (MoE) Language Model designed for efficient token generation, algorithmic problem-solving, and agentic reasoning on low-resource hardware.
🚀 What This Model Can Do & What It's Used For
1. 🐍 Python Code & Algorithm Generation
Trained on Python data structures, algorithms (binary search trees, graphs, sorting, async operations), and common developer patterns. It can generate small code snippets, logic blocks, and algorithmic skeletons.
2. 🤖 Agentic Problem Solving & Tool-Use
Employs step-by-step reasoning tokens ( chains) and action simulation (ACTION: write_file,
un_command), enabling it to simulate autonomous assistant and tool-use behaviors.
3. 🧩 Multi-Step Logic & Math Reasoning
Leverages MoE routing to direct logic and mathematical steps through specialized feed-forward expert networks for structured, multi-step problem solving.
4. ⚡ Ultra-Efficient Edge Inference
Because the model uses a sparse Mixture-of-Experts architecture with top-k gating, only a fraction of the network parameters are activated per token, making it fast and lightweight for edge devices, CPUs, and low-end GPUs.
📊 Model Architecture & Specifications
| Parameter | Value |
|---|
| Architecture | Causal Mixture-of-Experts (MoE) Transformer |
| Layers | 5 Transformer Blocks |
| Routing / Experts | 16 Feed-Forward Experts (Top-2 Activated per token) |
| Attention Heads | 4 Heads with Causal Masking |
| Hidden Size | 64 |
| Feed-Forward Dimension | 128 |
| Vocabulary Size | 2,386 Tokens |
| Precision | FP32 (GGUF Format) |
| Weight File | moe_reasoning_transformer_f32.gguf (8.86 MB) |
📈 Training Details & Token Count
| Metric | Details |
|---|
| Total Cumulative Tokens Trained | ~175,000,000+ Tokens (~175M token exposures across pretraining passes) |
| Base Corpus Size | ~2.1 Million Characters (~400,000 base tokens) |
| Training Passes / Epochs | 300 to 500 iterative optimization epochs |
| Hardware | NVIDIA GeForce GT 730 (CUDA) + Multi-Core CPU |
| Frameworks | JAX/Flax & CuPy |
📚 Training Dataset (Mega Hybrid Corpus)
Pretrained on a curated dataset combining:
- High-performance Python codebases, data structures, and algorithms
- Agentic scratchpads and structured chain-of-thought demonstrations ()
- Step-by-step mathematical & logical reasoning samples (GSM8K/MATH style)
- Literary knowledge & structured conversational dialogues
⚙️ Intended Use & Limitations
- Intended For: Research into sparse MoE scaling on ultra-constrained edge devices, offline embedded AI experiments, and fast local prototyping.
- Limitations: As a micro-scale model (8.86 MB), it is optimized for compactness and structural reasoning rather than expansive general world knowledge.
