Quasar Foundation Models (RoPE Base)

⚠️ Important Note

• A base model
• Pretrained for only a few billion tokens
• Designed for distillation and scaling, not benchmarking

Model Overview

• Model Name: Quasar 3B (RoPE Base)
• Organization: SILX AI
• Architecture: Quasar-RoPE Hybrid Transformer
• Total Parameters: 3B
• Active Parameters: ~1B (Mixture-of-Experts)
• Training Stage: Stage 1 (Base Pretraining)
• Sequence Length: 16K tokens (RoPE phase)

Training Strategy

Stage 1 — RoPE Pretraining

• Train using Rotary Positional Embeddings (RoPE)
• Context length: 16K tokens
• Objective: stabilize training and build core reasoning

Stage 2 — Distillation (SN24)

• Distributed training on Bittensor subnet (SN24)
• Miners distill knowledge from:
  • Qwen
  • GLM
• Target: transfer reasoning + capabilities into Quasar

Stage 3 — DroPE Long-Context Training

• Remove positional embeddings entirely (DroPE phase)
• Transition to position-free reasoning
• Train on ultra-long context (up to 5M tokens)

• Stable early training
• Efficient knowledge transfer
• Extreme context scaling without positional bottlenecks

Quasar-RoPE Hybrid Architecture

• Looped computation
• Persistent latent memory
• Hybrid attention mechanisms
• Stable Mixture-of-Experts routing

1. Looped Transformer Logic

• A fixed set of layers is reused multiple times (num_loops)
• This multiplies effective depth without increasing VRAM

Key Mechanism:

• Anchor P (Input Injection):
  • Embedding output is stored as P
  • Injected into the hidden state at every loop
• Gradient Stabilization:
  • Injection gradients scaled by 1 / num_loops
  • Prevents instability during recirculation

2. Hybrid Layer Composition

Quasar Layers

• Use Latent Memory Module
• Handle long-range dependencies
• Read/write persistent state

GLA Layers (Gated Linear Attention)

• Fast, RNN-like recurrence
• Efficient local sequence modeling

3. Persistent Latent Memory

• Memory Slots:
  • Fixed parameter banks (e.g., 128–256 slots)
• Segment Compression:
  • Tokens grouped into segments (default: 64 tokens)
  • Reduced noise during updates
• Saliency Gating:
  • Learns which information is important
  • Writes only high-value signals to memory

4. SMEBU (Stability-Maximized Expert Balancing Unit)

• Global Bias Buffers
  • Stored outside optimizer
  • Prevent routing collapse
• Zero-Loop Updates
  • Expert balancing done in vectorized pass
  • No recursive instability
• Sparse Activation
  • ~1B active parameters per forward pass

5. Technical Specifications

• Normalization: RMSNorm (Pre-Norm)
• Positional Encoding: RoPE (theta = 1,000,000)
• Initialization: Depth-scaled 1/sqrt(2L)
• Architecture Type: Hybrid Transformer + Memory + MoE

Architecture Overview

Core Data Flow

Token IDs
  ↓
Embedding Layer
  ↓
Anchor P Snapshot
  ↓
┌──────────────────────────────────────────────┐
│ Loop (i < num_loops)                         │
│                                              │
│   Quasar Block                               │
│        ↓                                     │
│   GLA Block                                  │
│        ↓                                     │
│   SMEBU MoE                                  │
│        ↓                                     │
│   Inject Anchor P (Residual Conditioning)    │
└──────────────────────────────────────────────┘
  ↓
Next Loop Iteration (state updated)

Final Loop Output
  ↓
RMSNorm
  ↓
LM Head
  ↓
Logits

Latent Memory Update Path

Hidden States
  ↓
Layer Normalization (RMSNorm)
  ↓
Segment Compressor
  ↓
Segment Representation (Z)
  ↓
  ├──────────────→ Saliency Gate (importance scoring)
  │                        ↓
  │                     Write Signal
  │                        ↓
  └──────────────→ Memory Write Operation
                           ↓
              Persistent Memory Bank (M)
                           ↓
                  Updated Memory (M')
                           ↓
                  Memory Read Module
                           ↓
              Memory-Augmented Hidden State
                           ↓
                         Output

SMEBU MoE Stability Flow

Router Network
  ↓
Token Routing Scores
  ↓
  * Global Bias Buffer (non-trainable stability path)
  ↓
Top-K Expert Selection
  ↓
Selected Experts
  ↓
Expert Output Aggregation
  ↓
Final MoE Output
  ↓
Post-Loop Bias Update (vectorized, stabilized)

Intended Use

• Bittensor SN24 miners participating in distributed training and knowledge distillation
• Distillation pipelines transferring capabilities from frontier models (e.g., Qwen, GLM)
• Research on long-context architectures, especially beyond traditional positional encoding limits
• Agentic system development, where persistent memory and long-horizon reasoning are required

Next Steps

• Training on SN24 in the coming days
• Miners distill knowledge into this model
• Then we go for Run 2 — DroPE training at 5M tokens