BitMamba-2-255M is the ultra-efficient baseline model of the BitMamba-2 family. It integrates 1.58-bit ternary quantization (BitNet) into the Mamba-2 architecture. Despite its small size, it demonstrates stable convergence and surprising reasoning capabilities, serving as the proof-of-concept for scaling ternary State Space Models.
Training Tokens: Trained on high-quality data (FineWeb-Edu, Cosmopedia, Stack-Dedup).
Hardware: Trained on Google Cloud TPU v6e.
📊 Benchmark Results
This model serves as the baseline for our scaling laws analysis.
Benchmark
Metric
BitMamba-2-255M
ARC-Easy
Accuracy
55.51%
PIQA
Accuracy
64.42%
BoolQ
Accuracy
59.30%
HellaSwag
Acc Norm
35.22%
WikiText-2
Perplexity
51.69
As shown in the scaling analysis below, the 255M model (blue line) establishes a stable learning trajectory, which is significantly improved upon by the 1B model (red line).
Scaling Laws
🚀 Usage (Inference)
This model is optimized for extreme edge deployment (IoT, Mobile, Legacy Hardware) using our custom C++ inference engine.
1. Download the Quantized Model
Download the bitmamba_255m.bin file located in the files tab.
