ULTRON-Core-512M Foundation Model (Generation 3 - CPT Knowledge Enhanced)

Specifications

• Total Parameters: 562,382,848 (~512.05M Non-Embedding)
• Architecture: Multi-Head Latent Attention (MLA: $d_c=512, d_R=64$) + QK-LayerNorm + SwiGLU Gating + RoPE ($\theta=500k$)
• Hidden Size ($d_{model}$): 1024 (36 Layers)
• Vocabulary Size ($V$): 24,576 (Pure-Python Byte-Level BPE, NFC Standard)
• Total Tokens Trained: 81,920,000 tokens
• Final CPT Loss: 3.1075 (Perplexity: 22.36)