Mamba-130M compressed from 489 MB (FP32) to 128 MB — a pure state-space model proving the codec works beyond transformers. No calibration data. No architecture-specific tuning. Just pip install and from_pretrained().
Install and Run
pip install "helix-substrate[hf]"
python
1import helix_substrate # registers the HXQ quantizer with HuggingFace2from transformers import AutoModelForCausalLM, AutoTokenizer
34model = AutoModelForCausalLM.from_pretrained("EchoLabs33/mamba-130m-helix")5tokenizer = AutoTokenizer.from_pretrained("EchoLabs33/mamba-130m-helix")67inputs = tokenizer("The future of artificial intelligence", return_tensors="pt")8outputs = model.generate(**inputs, max_new_tokens=64)9print(tokenizer.decode(outputs[0], skip_special_tokens=True))
That's it. import helix_substrate registers the quantizer. from_pretrained() handles the rest automatically.
Benchmark
Dense (FP32)
HXQ
Size
489 MB
128 MB
Perplexity (WikiText-2)
20.77
24.60 (+18.4%)
Compression ratio
—
3.8x
Compressed modules
—
96 HelixLinear + 1 nn.Linear (embedding)
Architecture
Mamba (24 layers, pure SSM)
unchanged
Eval: WikiText-2 test split, 2048 tokens, stride 512.
Good to Know
+18.4% PPL delta — higher than transformer models. Expected: Mamba-130M is tiny (24 layers, 768 hidden), so each weight carries more information per parameter. The companion Zamba2-1.2B (which includes Mamba2 layers) compresses at +2.90% — SSM architectures compress well at scale.
GPU and CPU supported — runs on any CUDA GPU or CPU via standard PyTorch. Fused kernels for additional speedup are in progress.
Fine-tunable via LoRA — compressed weights remain frozen, but LoRA adapters attach to each HelixLinear layer via HelixLinearSTE. See helix-substrate for training infrastructure.
Requires helix-substrate — the quantizer is not built into transformers. You need pip install "helix-substrate[hf]".
mamba-ssm recommended — without it, falls back to a slower sequential code path.
Why This Model Exists
This is the architecture proof, not the fidelity champion. HelixCode compresses any nn.Linear — including the in_proj, out_proj, x_proj, and dt_proj layers inside Mamba's selective scan blocks. No architecture-specific tuning was needed.
What is HelixCode?
HelixCode is a universal weight compression codec based on vector quantization:
Each weight matrix is replaced by a 256-entry codebook (float32) + uint8 index matrix + optional sidecar corrections for outlier values
The compressed form is the executable — HelixLinear performs codebook[indices] @ x directly, no decompression step
Works on any nn.Linear regardless of architecture (Transformer, Mamba, MLP, CNN)
No calibration data required — unlike GPTQ/AWQ, codebooks are fit from the weights alone
How It Works
import helix_substrate registers the hxq quantizer with HuggingFace
from_pretrained() reads quantization_config.quant_method = "hxq" from config.json
The quantizer replaces 96 nn.Linear modules with HelixLinear shells before weight loading
Safetensors populates the codebook, indices, and sidecar buffers directly
The model runs in compressed form — no decompression needed
Mamba-Specific Details
Mamba's architecture includes non-weight parameters that are stored at full precision:
A_log — log-space diagonal state matrix (24 layers)
D — skip connection parameter (24 layers)
dt_bias — timestep bias (24 layers)
conv1d — causal convolution (24 layers)
These are not nn.Linear and are not compressed. Only the projection matrices (in_proj, out_proj, x_proj, dt_proj) are VQ-compressed.
Compression Receipt
Compressed tensors: 97
From original model: 145 (A_log, D, dt_bias, conv1d, norms)
Total keys: 573
Output size: 128 MB
HXQ ratio: 3.92x (weight bytes)
HelixLinear ratio: 5.61x (in-memory, includes format overhead reduction)
PPL delta: +18.4% (24.60 vs 20.77 dense)
Eval: WikiText-2 test, 2048 tokens, stride=512
Companion Models
Same codec, same pip install, multiple architectures:
