Zamba2-1.2B-instruct — LiteRT-LM
Zyphra/Zamba2-1.2B-instruct converted to the
LiteRT-LM (
.litertlm) format for on-device inference with Google's
LiteRT-LM runtime.
Requires litert-lm ≥ 0.15.
Zamba2 is Zyphra's shared-attention hybrid: a Mamba2 selective-scan backbone (32 layers) with a single shared transformer block applied at 6 interleaved positions — one set of attention+MLP weights reused at every position, specialized by per-position LoRA adapters, attending over the concatenation of the running hidden state and the original embeddings. To our knowledge this is the first Zamba2 conversion to a mobile runtime.
| File | Recipe | Size |
|---|
Zamba2-1.2B-instruct_int8.litertlm | int8 dynamic on linears + embedding (convs and the scan stay float); fp32 activations declared for GPU | 1.33 GB |
Correctness
- Logits parity vs PyTorch: the float export matches the HF model teacher-forced across 48 decode positions — top-1 and top-5 identical at every position, mean per-position logit correlation 1.0000, mean KL ≈ 0.
- 8-question sanity gate: 8/8 on every lane — GPU and CPU, litert-lm 0.15.0 and 0.16.0. No degeneration, no greedy flips.
- Prompt-length robustness: hermetic prefill-chunk sweep (fresh engine per length) — CPU fills 12–51 and GPU fills 12–31 all clean.
- iPhone 17 Pro (Metal): the 8-item composite quality probe answers 8/8 on CPU; on GPU 6/8, where both misses (8×7 and the rhyme) are questions the HF fp32 reference itself answers incorrectly on this composite ("63"; a broken echo) — the GPU path tracks the reference model's own behavior, and Mac and iPhone GPU produce identical text. The individual 8-question gate is 8/8 on both backends.
Usage
1litert-lm run ./Zamba2-1.2B-instruct_int8.litertlm --prompt "What is the capital of France? Answer in one word."
2
3# GPU
4litert-lm run ./Zamba2-1.2B-instruct_int8.litertlm --backend gpu --cache no --prompt "..."
Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks. The bundle carries the tokenizer and the stock ChatML Zamba2 chat template.
Performance
litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:
| Backend | Prefill (256) | Decode | TTFT |
|---|
| GPU | 1033 tok/s | 74.0 tok/s | 0.26 s |
| CPU | 450 tok/s | 22.7 tok/s | 0.61 s |
On device (cold start, single runs, composite prompt, quality harness):
| Device | Backend | Prefill | Decode | TTFT | Peak memory |
|---|
| iPhone 17 Pro | GPU (Metal) | 96.6 tok/s | 12.8 tok/s | 1.63 s | 5.43 GB |
| iPhone 17 Pro | CPU | 87.3 tok/s | 7.4 tok/s | 1.73 s | 1.82 GB |
Honest notes:
- GPU runs with fp32 activations (declared in the bundle) — expect a corresponding memory multiple over CPU. The 5.43 GB GPU peak includes the six shared-attention positions' wide KV caches (32 KV heads × 128 head dim at 4096 context) held in fp32; a 12 GB phone runs the full 12-signature ladder with no memory pressure.
- On the long composite probe the two backends split two model-edge questions differently (see Correctness) — deterministically, and identically on Mac and iPhone. For maximum fidelity to the HF reference use GPU; for the best composite score use CPU.
Galaxy S26 — GPU backend
The published bundle runs on the Android GPU backend: LiteRT takes the whole graph and the model generates.
| file | GPU backend | delegation | peak |
|---|
Zamba2-1.2B-instruct_int8.litertlm | runs | 54510 / 54510 ops across 12 subgraphs on LiteRT GPU | 3620 MB |
Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.
No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.
GPU wiring, including the Gallery import toggle:
GPU guide.
Conversion notes
Converted with
litert-torch plus a hybrid-cache patch (reproduction script + patch:
hf-to-litertlm zamba2_work/):
- Folded selective scan: the Mamba2 scan is re-expressed as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no
BROADCAST_TO, no int64 index math) — this is what makes the graph fully delegable on GPU.
- Min-only dt clamp handling: Zamba2 clamps
softplus(dt) at time_step_min with no upper clamp; padded prefill positions are forced to exact identity steps AFTER the clamp (without this, every runtime pad token decays the recurrent state).
- Shared block + adapters: the tied transformer block traces once per position with its own LoRA adapter statically selected; tied weights are stored once.
- Composite hybrid cache layer: the 6 shared-attention positions hold KV + conv + recurrent state at ONE layer index (the runtime binds states by tensor name, so co-residency is just packaging).
- Prefill-pad guard: the runtime runs partially-filled prefill chunks; pad positions are made exact identity steps for the SSM and the stored conv window is gathered at the last valid column.
- Streaming detokenization: the tokenizer's
Strip decoder is removed from the bundle — Zamba2's metaspace (SP-BPE) tokenizer otherwise loses every interior space under the runtime's per-token streaming decode; the only behavior change is a sequence-initial space, which the runtime trims.
- Quantization: post-hoc dynamic int8 over linears + embedding only; convs and the scan stay float.
2026-08-28 — start_token fix (weights unchanged)
The LiteRT-LM engine prepends the metadata start_token to every prompt, and this model's chat template already renders <|im_start|> itself — so the model was reading <|im_start|><|im_start|>…, a stream it was never trained on. The start token has been removed.
Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file — only the file's own sha256 differs. What changed is the input: the token stream the model reads for a given conversation can differ from the previous file's, and it now matches this model's own reference chat stream. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file in either direction. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-28, re-download.
Raspberry Pi 5 (CPU)
Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with
litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens,
--cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (
vcgencmd get_throttled stayed
0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.
| File | Prefill (tok/s) | Decode (tok/s) | TTFT | Peak RSS |
|---|
Zamba2-1.2B-instruct_int8.litertlm | 24.3 (24.2–24.3) | 1.8 (1.8–1.8) | 11.1 s | 2.9 GB |
