Mintaka Micro — 173M Hybrid SSM-Transformer Foundation Model
Mintaka Micro is a 173M parameter hybrid foundation model combining Selective State Space Model (Mamba) layers with Transformer attention, trained from scratch on high-quality curated data. It is designed for efficient inference on edge devices, mobile hardware, and resource-constrained environments.
Model Description
Mintaka Micro introduces a novel hybrid architecture that combines the long-context efficiency of Mamba (SSM) layers with the precise reasoning capability of causal self-attention — achieving strong language modeling performance at a fraction of the compute cost of comparable pure-Transformer models.
Architecture
[Mamba → Mamba → Attention → MLP] × 8 blocks
Parameter
Value
Total parameters
173.61M
Hidden dimension
768
Number of blocks
8
Mamba layers per block
2
Attention layers per block
1
Attention heads
12
Head dimension
64
FFN dimension
3072
Max sequence length
2048
Vocabulary size
49,152
Attention type
Causal Self-Attention with RoPE
MLP activation
SwiGLU
Normalization
RMSNorm
Key Design Choices
Hybrid SSM-Transformer: Mamba layers handle long-range sequence modeling efficiently (linear complexity), while sparse attention layers provide precise token recall and reasoning capability
RoPE embeddings: Rotary positional embeddings on attention layers for better length generalization
SwiGLU MLP: Same activation function used in LLaMA/Mistral for improved performance
Weight tying: Embedding and LM head weights are tied to reduce parameter count
BF16 training: Full BF16 mixed precision on A100 80GB
Training Details
Detail
Value
Training tokens
4.2B
Dataset
HuggingFaceTB/smollm-corpus (cosmopedia-v2)
Tokenizer
HuggingFaceTB/cosmo2-tokenizer
Optimizer
AdamW (fused, β1=0.9, β2=0.95)
Learning rate
3e-4 (cosine decay)
Warmup steps
2,000
Batch size
16 × 8 grad accum × 2048 seq = ~262K tokens/step
Weight decay
0.1
Gradient clipping
1.0
Hardware
1× NVIDIA A100 80GB SXM4
Training time
~20 hours
Final loss
~1.65 (cross-entropy)
Perplexity
~5.2
Training Efficiency
Mintaka Micro reached a cross-entropy loss of 1.65 in just 4.2B tokens — significantly more data-efficient than comparable pure-Transformer models of similar size, which typically require 40B+ tokens to reach similar loss values. This efficiency is attributed to:
The hybrid Mamba-Transformer architecture learning representations faster
High-quality synthetic pretraining data (Cosmopedia v2)
Modern training optimizations (fused AdamW, BF16, RoPE)
⚠️ Occasional topic drift after 3-4 sentences (expected at pretraining stage)
⚠️ No instruction following yet (requires SFT)
Sample Outputs
Prompt:"The history of artificial intelligence began"
The history of artificial intelligence began to take shape in the late 20th century, spurred by advances in machine learning and big data analytics. Early examples of AI were used in various applications such as fraud detection in data mining and AI-driven recommendation systems. Today, these technologies have become ubiquitous in many businesses, driving innovation and growth.
Prompt:"Once upon a time, there was a small village"
Once upon a time, there was a small village named Serenity Springs. Despite its challenges, it had been home to everyone who welcomed its residents with open arms. Among them were two best friends, Maya and Jamie, who shared a profound love for their homeland.
Inference Performance
Hardware
Tokens/sec (generation)
A100 80GB
~5,000
RTX 4090
~2,000
RTX 3080
~800
MacBook M2
~200
iPhone 15 Pro
~50-80
Model Size
Format
Size
BF16 (full)
~347MB
INT8 quantized
~173MB
INT4 quantized
~87MB
Roadmap
Mintaka Micro is the foundation (Stage 1 — Language Teaching Phase) of a three-stage training pipeline:
Stage 1 — Language Teaching (this model): Language foundation on 4.2B tokens — grammar, coherence, and world knowledge surface
Stage 2 — Factual Teaching(experimental): Grounding in factual knowledge via supervised fine-tuning and knowledge distillation from frontier models
Stage 3 — Reasoning(experimental): Multi-step reasoning and agentic alignment via GRPO reinforcement
Note: Stages 2 and 3 are purely experimental. Mintaka Micro is a research model — results and training approaches at these stages are exploratory and not production-validated.
The goal is to demonstrate that a 173M hybrid SSM-Transformer model, when trained with high-quality distilled data and modern alignment techniques, can achieve performance comparable to much larger models on practical tasks — particularly agentic and instruction-following benchmarks.
Limitations
Context window: Currently limited to 2,048 tokens due to quadratic attention
No instruction following: This is a base pretrained model — not suitable for chat or instruction tasks without fine-tuning
English only: Trained on English text only
Pretraining only: No RLHF or alignment applied yet
Citation
If you use Mintaka Micro in your research, please cite:
bibtex
1@misc{mintaka-micro-2025,
2 title={Mintaka Micro: A 173M Hybrid SSM-Transformer Foundation Model},
3 author={Rishi Ravikumar},
4 year={2025},
5 publisher={HuggingFace},
6 url={https://huggingface.co/rishi-r-rishi-r/mintaka-micro}
7}
About
Mintaka Micro is developed by Rishi Ravikumar as part of the Mintaka AI project — building efficient, edge-deployable foundation models that bring frontier AI capability to resource-constrained environments.
