Views
No views yet
transformers. It incorporates state-of-the-art LLM architectural choices—Grouped-Query Attention (GQA), SwiGLU MLPs, RMSNorm, and Rotary Position Embeddings (RoPE)—and natively saves weights in the zero-copy Safetensors format.ModernLLMForCausalLM)vocab_size: 151,936)FP16).safetensorsdevoppro/FastLLM| Parameter | Configuration |
|---|---|
| Hidden Size ($d_{\text{model}}$) | 768 |
| Intermediate Size (SwiGLU) | 2048 |
| Hidden Layers | 12 |
| Query Heads | 12 |
| Key/Value Heads (GQA) | 4 (3:1 Query-to-KV ratio) |
| Max Context Length | 2048 tokens |
| Normalization | RMSNorm ($\epsilon = 10^{-6}$) |
| Positional Embedding | Rotary Embeddings (RoPE, $\theta = 1000000.0$) |