LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

granite-4.0-h-1b — LiteRT-LM

Update 2026-08-28: metadata-only fix — removed the bundle's start_token, which silently prepended <|end_of_text|> (a token the reference chat stream never contains) to every prompt. Weights and tokenizer are unchanged: every non-metadata section is byte-identical, and only the file's sha256 changed. Scores measured on the previous file do not all carry over, because dropping that token shifts greedy decoding. The sanity gate was re-run on the file now published and moved from 8/8 to 7/8; the GSM8K and iPhone rows below were not re-run. Re-download if you fetched before this date.

Update 2026-08-13: re-converted from the same weights with the selective scan re-expressed as batched matmuls (all tensors rank ≤ 4, no giant broadcast intermediates). Two consequences, measured on the same machine and protocol: CPU decode is ~3.5× faster than the previous file (11.4 → 39.2 tok/s) and the graph now delegates fully to the GPU — verified end-to-end on macOS (Metal), iPhone 17 Pro (Metal) and Pixel 8a (Arm Mali, OpenCL). GPU execution requires fp32 activations, which the bundle now declares; see the honest notes below for the memory cost. If you downloaded the file before this date, re-download.

ibm-granite/granite-4.0-h-1b converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 (the published file passes the sanity gate on CPU and GPU on both 0.15.0 and 0.16.0; the speed rows below were measured on 0.16.0). To our knowledge this is the first Mamba2-hybrid LLM served by the released LiteRT-LM runtime — and, since the 2026-08-13 update, the first to run its selective scan on a mobile GPU.

Granite 4.0-h is IBM's hybrid architecture: Mamba2 (selective-scan) blocks interleaved with a few grouped-query attention blocks (this 1B/1.5B-class model has 36 mamba + 4 attention layers). The mamba blocks carry constant-size per-layer conv + SSM recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 4 attention layers keep KV (4096-token budget here).

File	Recipe	Size
`granite-4.0-h-1b_int8.litertlm`	int8 dynamic on linears + embedding (convs and the selective scan stay float)	1.68 GB

Correctness

The rewritten scan is verified equivalent to the reference implementation at export time (a built-in numeric self-check runs on every export), and the float export behind this exact file matches the HF PyTorch model at the logits level (teacher-forced 8-step decode comparison: per-position max|logit diff| ≤ 1.1e-4, correlation 1.000000, top-1 and top-5 identical at every position). The published int8 file scores 7/8 on an 8-question sanity gate on both the CPU and the GPU backend (Mac, re-run on litert-lm 0.15.0 and 0.16.0 after the 2026-08-28 metadata fix — the same 7/8 on all four). The miss is 17 + 25, answered 43; the file before that fix answered 42 and scored 8/8. Removing the prepended token moves greedy decoding on close calls in both directions, and this is one of them: over 50 GSM8K questions the two files agree on 40 and split the other 10 six-to-four in favour of the older one, which at that sample size is noise. It generates correctly on a Pixel 8a with the whole graph on the OpenCL delegate (zero rejected ops, all subgraphs fully delegated). An earlier 8/8 on iPhone 17 Pro on the GPU was measured on the pre-fix file and has not been re-run.

Quality — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max-tokens 512, n=100, same harness, prompt and answer extraction for both rows:

Configuration	GSM8K
PyTorch bf16 (reference, MPS)	77%
LiteRT int8	76%

⚠ The LiteRT row was measured before the 2026-08-28 metadata fix and has not been re-run. See Correctness for what changing the start token does to a greedy score.

Usage

bash

litert-lm run ./granite-4.0-h-1b_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./granite-4.0-h-1b_int8.litertlm --backend gpu --cache no --prompt "..."

The bundle carries the tokenizer and Granite's full chat template (tool-calling and RAG-documents sections included). Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

Backend	Prefill (256)	Decode	TTFT
GPU	1684 tok/s	134.7 tok/s	0.16 s
CPU	247 tok/s	39.2 tok/s	1.06 s

For scale: the pre-update file measured 69 tok/s prefill / 11.4 tok/s decode on the same machine and protocol (CPU) — the rewrite is a ~3.5× CPU decode / ~3.6× CPU prefill win before the GPU enters the picture.

On device (cold start, single runs, 148-token prompt, quality-gate harness):

Device	Backend	Prefill	Decode	TTFT	Peak memory
iPhone 17 Pro	GPU (Metal)	210 tok/s	29.8 tok/s	0.79 s	2.53 GB
iPhone 17 Pro	CPU	123 tok/s	15.9 tok/s	1.31 s	0.99 GB

Pixel 8a (Tensor G3, litert_lm_main built from the v0.16.0 tag, 279-token prompt with generation capped at 256, 3 runs per backend):

Backend	Prefill (279 tok)	Decode	TTFT
GPU (OpenCL)	49.0–49.4 tok/s	9.4–9.6 tok/s	5.8 s
CPU	27.1–30.3 tok/s	5.0–10.0 tok/s	9.3–10.4 s

On this low-end phone the GPU wins prefill (~1.7×) and TTFT and holds a steadier decode (the CPU rows spread because they throttle; the GPU rows repeat within 1%) — full delegation, zero rejected ops, correct output.

Honest notes:

Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal), Pixel 8a (Arm Mali, OpenCL) and Qualcomm Adreno — the Galaxy S26 section below is that measurement.
GPU inference runs with fp32 activations (the bundle declares prefer_activation_type = "fp32"; an fp16-activation formulation of the scan is unfinished work). That is where the GPU rows' extra memory goes — 2.53 GB vs 0.99 GB on iPhone above.
On low-end Android the GPU does not win decode (decode is memory-bandwidth-bound; phone CPUs and GPUs share the same LPDDR, and the CPU path reads int8 weights while the fp32-activation GPU path reads expanded weights). The GPU still wins prefill and TTFT there. On Apple hardware the GPU wins across the board.
The scan itself stays unquantized (float); int8 covers linears + embedding.

Galaxy S26 — GPU backend

The published bundle runs on the Android GPU backend: LiteRT takes the whole graph and the model generates.

file	GPU backend	delegation	peak
`granite-4.0-h-1b_int8.litertlm`	runs	48929 / 48929 ops across 12 subgraphs on `LiteRT GPU`	2944 MB

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm granite_work/):

Folded selective scan (the 2026-08-13 change): the HF chunked Mamba2 scan spells its contractions as broadcast-multiply-reduce over rank-5/6 intermediates that scale as chunks × chunk_len² × heads × d_state. The export re-expresses every contraction as a batched matmul with the chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math). This is why the same weights got ~3.5× faster on CPU and became fully delegable on GPU. The decode path likewise drops per-token expand/materialization in favor of implicit broadcasting.
Export cache for Mamba2 layers: conv [B, conv_dim, K] + SSM recurrent [B, heads, head_dim, state] cache layers registered for Granite's layer types, so torch.export traces the model's own state contract.
State continuation tracing: prefill graphs trace the chunk-continuation branch (previous conv/SSM state consumed, so multi-chunk prefill composes), and the decode graph traces the single-step branch (conv window rolled by one).
Prefill-pad guard: the runtime's chunk planner runs partially-filled prefill chunks; pad positions are made identity steps for the SSM (dt forced to ~0) and the stored conv window is gathered at the last valid column. One GPU-specific trap here: reductions in the guard must keep their batch dimension (sum(-1, keepdim=True)) — a rank-0 scalar entering broadcast arithmetic is silently miscomputed by the GPU delegate.
Runtime state binding: litert-lm ≥ 0.15 binds per-layer states through an ExecutorMetadata section listing each state tensor; it is appended at package time.
Quantization: post-hoc dynamic int8 over linears + embedding only. Export-time conv-int8 measurably costs quality on this family, so the convs and scan stay float.
GPU activation precision: the bundle's TOML declares prefer_activation_type = "fp32" — required for correct GPU numerics on this family today.

2026-08-28 — start_token fix (weights unchanged)

The LiteRT-LM engine prepends the metadata start_token to every prompt, but this model's reference prompt has no leading BOS at all — the bundle was feeding an extra <|end_of_text|> the model was never trained on. The start token has been removed.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file — only the file's own sha256 differs. What changed is the input: the token stream the model reads for a given conversation can differ from the previous file's, and it now matches this model's own reference chat stream. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file in either direction. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-28, re-download.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

File	Prefill (tok/s)	Decode (tok/s)	TTFT	Peak RSS
`granite-4.0-h-1b_int8.litertlm`	42.3 (41.6–42.6)	4.4 (4.2–4.4)	6.3 s	2.5 GB

License and changes

Distributed under Apache-2.0 (inherited from the base model). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified into the .litertlm bundle. This repository is a community conversion and is not affiliated with IBM.

litert-community/granite-4.0-h-1b