granite-4.0-h-1b — LiteRT-LM
Update 2026-08-13: re-converted from the same weights with the selective scan re-expressed as batched matmuls (all tensors rank ≤ 4, no giant broadcast intermediates). Two consequences, measured on the same machine and protocol: CPU decode is ~3.5× faster than the previous file (11.4 → 39.2 tok/s) and the graph now delegates fully to the GPU — verified end-to-end on macOS (Metal), iPhone 17 Pro (Metal) and Pixel 8a (Arm Mali, OpenCL). GPU execution requires fp32 activations, which the bundle now declares; see the honest notes below for the memory cost. If you downloaded the file before this date, re-download.
ibm-granite/granite-4.0-h-1b converted to the
LiteRT-LM (
.litertlm) format for on-device inference with Google's
LiteRT-LM runtime.
Requires litert-lm ≥ 0.15 (the published file passes the sanity gate on CPU and GPU on both 0.15.0 and 0.16.0; the speed rows below were measured on 0.16.0). To our knowledge this is the first Mamba2-hybrid LLM served by the released LiteRT-LM runtime — and, since the 2026-08-13 update, the first to run its selective scan on a mobile GPU.
Granite 4.0-h is IBM's hybrid architecture: Mamba2 (selective-scan) blocks interleaved with a few grouped-query attention blocks (this 1B/1.5B-class model has 36 mamba + 4 attention layers). The mamba blocks carry constant-size per-layer conv + SSM recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 4 attention layers keep KV (4096-token budget here).
| File | Recipe | Size |
|---|
granite-4.0-h-1b_int8.litertlm | int8 dynamic on linears + embedding (convs and the selective scan stay float) | 1.68 GB |
Correctness
The rewritten scan is verified equivalent to the reference implementation at export time (a built-in numeric self-check runs on every export), and the float export behind this exact file matches the HF PyTorch model at the logits level (teacher-forced 8-step decode comparison: per-position max|logit diff| ≤ 1.1e-4, correlation 1.000000, top-1 and top-5 identical at every position). The published int8 file scores 8/8 on an 8-question sanity gate on both the CPU and the GPU backend (Mac, litert-lm 0.16.0), 8/8 on iPhone 17 Pro on the GPU, and generates correctly on a Pixel 8a with the whole graph on the OpenCL delegate (zero rejected ops, all subgraphs fully delegated).
Quality — GSM8K
GSM8K, greedy, 0-shot chain-of-thought, max-tokens 512, n=100, same harness, prompt and answer extraction for both rows:
| Configuration | GSM8K |
|---|
| PyTorch bf16 (reference, MPS) | 77% |
| LiteRT int8 (this file) | 76% |
Usage
1litert-lm run ./granite-4.0-h-1b_int8.litertlm --prompt "What is the capital of France? Answer in one word."
2
3# GPU
4litert-lm run ./granite-4.0-h-1b_int8.litertlm --backend gpu --cache no --prompt "..."
The bundle carries the tokenizer and Granite's full chat template (tool-calling and RAG-documents sections included). Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks.
Performance
litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:
| Backend | Prefill (256) | Decode | TTFT |
|---|
| GPU | 1684 tok/s | 134.7 tok/s | 0.16 s |
| CPU | 247 tok/s | 39.2 tok/s | 1.06 s |
For scale: the pre-update file measured 69 tok/s prefill / 11.4 tok/s decode on the same machine and protocol (CPU) — the rewrite is a ~3.5× CPU decode / ~3.6× CPU prefill win before the GPU enters the picture.
On device (cold start, single runs, 148-token prompt, quality-gate harness):
| Device | Backend | Prefill | Decode | TTFT | Peak memory |
|---|
| iPhone 17 Pro | GPU (Metal) | 210 tok/s | 29.8 tok/s | 0.79 s | 2.53 GB |
| iPhone 17 Pro | CPU | 123 tok/s | 15.9 tok/s | 1.31 s | 0.99 GB |
Pixel 8a (Tensor G3, litert_lm_main built from the v0.16.0 tag, 279-token prompt with generation capped at 256, 3 runs per backend):
| Backend | Prefill (279 tok) | Decode | TTFT |
|---|
| GPU (OpenCL) | 49.0–49.4 tok/s | 9.4–9.6 tok/s | 5.8 s |
| CPU | 27.1–30.3 tok/s | 5.0–10.0 tok/s | 9.3–10.4 s |
On this low-end phone the GPU wins prefill (~1.7×) and TTFT and holds a steadier decode (the CPU rows spread because they throttle; the GPU rows repeat within 1%) — full delegation, zero rejected ops, correct output.
Honest notes:
- Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal) and Pixel 8a (Arm Mali, OpenCL). We have not verified GPU execution on Qualcomm Adreno devices — if you are on a Snapdragon phone, use the CPU backend unless you have confirmed the GPU path on your own device.
- GPU inference runs with fp32 activations (the bundle declares
prefer_activation_type = "fp32"; an fp16-activation formulation of the scan is unfinished work). That is where the GPU rows' extra memory goes — 2.53 GB vs 0.99 GB on iPhone above.
- On low-end Android the GPU does not win decode (decode is memory-bandwidth-bound; phone CPUs and GPUs share the same LPDDR, and the CPU path reads int8 weights while the fp32-activation GPU path reads expanded weights). The GPU still wins prefill and TTFT there. On Apple hardware the GPU wins across the board.
- The scan itself stays unquantized (float); int8 covers linears + embedding.
Conversion notes
Converted with
litert-torch plus a hybrid-cache patch (reproduction script + patch:
hf-to-litertlm granite_work/):
- Folded selective scan (the 2026-08-13 change): the HF chunked Mamba2 scan spells its contractions as broadcast-multiply-reduce over rank-5/6 intermediates that scale as
chunks × chunk_len² × heads × d_state. The export re-expresses every contraction as a batched matmul with the chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math). This is why the same weights got ~3.5× faster on CPU and became fully delegable on GPU. The decode path likewise drops per-token expand/materialization in favor of implicit broadcasting.
- Export cache for Mamba2 layers: conv
[B, conv_dim, K] + SSM recurrent [B, heads, head_dim, state] cache layers registered for Granite's layer types, so torch.export traces the model's own state contract.
- State continuation tracing: prefill graphs trace the chunk-continuation branch (previous conv/SSM state consumed, so multi-chunk prefill composes), and the decode graph traces the single-step branch (conv window rolled by one).
- Prefill-pad guard: the runtime's chunk planner runs partially-filled prefill chunks; pad positions are made identity steps for the SSM (dt forced to ~0) and the stored conv window is gathered at the last valid column. One GPU-specific trap here: reductions in the guard must keep their batch dimension (
sum(-1, keepdim=True)) — a rank-0 scalar entering broadcast arithmetic is silently miscomputed by the GPU delegate.
- Runtime state binding: litert-lm ≥ 0.15 binds per-layer states through an
ExecutorMetadata section listing each state tensor; it is appended at package time.
- Quantization: post-hoc dynamic int8 over linears + embedding only. Export-time conv-int8 measurably costs quality on this family, so the convs and scan stay float.
- GPU activation precision: the bundle's TOML declares
prefer_activation_type = "fp32" — required for correct GPU numerics on this family today.
License and changes
Distributed under Apache-2.0 (inherited from the base model). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified into the .litertlm bundle. This repository is a community conversion and is not affiliated with IBM.
