Beta

Explore
Marketplace
Neural Labs
Chat
Wallet
Docs

Beta

Explore Marketplace Neural Labs Wallet Docs

Build

Manage

My assets Manage wallet Organization

Community

Discord X (Twitter)Telegram

Docs Privacy Policy Terms and Conditions

Proud member of

Proud member of

Docs Cookie Policy

Privacy Policy Terms and Conditions

© 2026 AlphaNeural AI. All rights reserved.

phi-mini-moe-pruned-specialized-v1 – AI Model by maxie-12321 | AlphaNeural AI

You can deploy this model and start earning money today!

maxie-12321/
phi-mini-moe-pruned-specialized-v1

like

0

mixture-of-attention

Views

No views yet

Phi-mini-MoE + MoA + Pruning + Specialization

What's Special

This model adds Mixture of Attention (MoA) routing to Phi-mini-MoE, then:

✂️ Pruned 25% of attention heads (kept only important ones)
🎯 Forced expert specialization (each expert focuses on specific tasks)
⚡ ~3x faster than OLMoE-1B-7B

Stats

Base: Phi-mini-MoE (7.6B total, 2.4B active)
Attention heads: 32 → 24 (pruned 25%)
Training iterations: 10
Expert specialization: 16.7%

Files

moa_router.pt - Trained + pruned MoA router
training_data.json - Self-play examples
expert_stats.json - Expert specialization profiles
pruning_stats.json - Which heads were pruned

By