It injects 14 Domain Anchors and 28 Sub-shelves (totaling 392 specialized experts per MoE layer) into 21 transformer layers, activated via a zero-synchronization Turbo V2 Dense Weight Mask engine.

🏛️ Architecture Overview

Domain Anchor Gate: 14 broad domains (Coding, CoT, Math, Agent, Medical/Science, Multilingual Turkish, General).
Sub-Shelf Dynamic Routing: Top-2 experts activated out of 28 sub-shelves per domain.
Turbo V2 Engine: Fully vectorized dense mask execution with zero GPU-CPU synchronization stalls.
Soft Residual Injection: Dynamic RMSNorm + trainable Gamma scaling.

📊 Training Specifications & Convergence

Base Model: Gemma-4B (4-Bit NF4 Base + Float32 MoE Master Weights).
Trainable Parameters: 170.9M parameters (2.09% of base model).
Distilled Dataset: 132,443 high-quality CoT pairs across 7 frontier domains.
Loss Trajectory: Initial loss ~0.91 ➜ Final converged loss 0.3878 - 0.4068 at step 2000!

🚀 Quick Start / Inference Example

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

REPO_ID = "sana0756/hipponet-gemma-398moe-omega2"

tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
model = AutoModelForCausalLM.from_pretrained(
    REPO_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

prompt = """<start_of_turn>user
Aşağıdaki Python fonksiyonunu analiz et ve optimize edilmiş halini açıkla:
def fibonacci(n):
    if n <= 1: return n
    return fibonacci(n-1) + fibonacci(n-2)<end_of_turn>
<start_of_turn>model
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

⚖️ License

Apache 2.0