granite-4.1-3b — LiteRT-LM
ibm-granite/granite-4.1-3b converted to the
LiteRT-LM (
.litertlm) format for on-device inference with Google's
LiteRT-LM runtime.
Requires litert-lm ≥ 0.16.
Granite 4.1-3B is IBM's 3.4B-parameter dense instruct model (40 layers, hidden 2560, GQA 40:8, tied embeddings, 100k vocab), long-context and tool-calling capable, released April 2026.
| File | Recipe | Size |
|---|
granite-4.1-3b_int4.litertlm | int4 blockwise-32 + OCTAV on linears, int8 embedding | 2.19 GB |
granite-4.1-3b_int8.litertlm | int8 dynamic on linears + embedding | 3.83 GB |
The int4 file is the one to use on a phone: on Apple hardware it is both smaller and faster than int8 (see below). On Android the int8 file is not out of reach either — it runs fully delegated on a Galaxy S26 GPU at a 1476 MB peak against int4's 1394 MB (see below).
Correctness
The int4 file scores 8/8 on an 8-question sanity gate on both CPU and GPU (Mac, litert-lm 0.16.0), 8/8 on iPhone 17 Pro on Metal (and 7/8 on its CPU — the one miss is the rhyme line in a single combined prompt, which the model answers correctly when asked on its own), and generates correctly on a Pixel 8a with every node on the OpenCL delegate — 1784/1784 in the 1024-token prefill graph and 1614/1614 in decode, zero rejected ops. The int8 file scores 8/8 on all the same Mac lanes and also runs fully delegated on the Pixel.
Quality — GSM8K
GSM8K, greedy, 0-shot chain-of-thought, max-tokens 512, n=100, same harness, prompt and answer extraction for every row:
| Configuration | GSM8K |
|---|
| PyTorch bf16 (reference, MPS) | 88.0% |
| LiteRT int8 | 87.0% |
| LiteRT int4 (this file) | 84.0% |
int4 costs 4 points on this model — an ordinary blockwise-int4 result, and worth the 1.7× size reduction for a phone.
Usage
1litert-lm run ./granite-4.1-3b_int4.litertlm --prompt "What is the capital of France? Answer in one word."
2
3# GPU
4litert-lm run ./granite-4.1-3b_int4.litertlm --backend gpu --cache no --prompt "..."
The bundle carries the tokenizer and Granite's role markers (<|start_of_role|>…<|end_of_role|>), with a 4096-token KV budget and six prefill signatures (1024, 256, 64, 16, 4, 1). Six rather than eleven is deliberate: every exported signature is charged engine memory whether or not it is called, and an eleven-signature build of these same weights was killed by iOS during Metal initialisation unless the app capped the context at 1024. This file creates its engine at the full 4096 on an iPhone 17 Pro.
Performance
litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --cache no, quiet machine, serialized:
| File | Backend | Prefill (256) | Decode | TTFT | Init |
|---|
| int4 | GPU (Metal) | 1241 tok/s | 86.3 tok/s | 0.23 s | 5.96 s |
| int4 | CPU | 104 tok/s | 22.0 tok/s | 3.01 s | 12.5 s |
| int8 | GPU (Metal) | 1221 tok/s | 71.8 tok/s | 0.24 s | 5.19 s |
| int8 | CPU | 257 tok/s | 20.5 tok/s | 3.06 s | 43.2 s |
Pixel 8a (Tensor G3, Mali-G715, 7.75 GB RAM; litert_lm_main built from the v0.16.0 tag, single runs):
| File | Backend | Prefill | Decode | TTFT | Init |
|---|
| int4 | GPU (OpenCL) | 20.0 tok/s | 7.07 tok/s | 0.99 s | 32.5 s |
| int4 | CPU | 5.40 tok/s | 6.98 tok/s | 3.29 s | — |
| int8 | GPU (OpenCL) | 15.5 tok/s | 3.86 tok/s | 1.36 s | 65.2 s |
Honest notes:
- On this low-end phone the GPU wins prefill (3.7×) and TTFT (3.3×), not decode (7.07 vs 6.98 tok/s). Decode is memory-bandwidth-bound and a phone's CPU and GPU share the same LPDDR. On Apple hardware the GPU wins across the board.
- int4 beats int8 on the GPU on both platforms — 1.2× decode on Metal, 1.8× on Mali — while int8 wins CPU prefill (2.5× on Mac), where XNNPACK's large matmuls outrun the blockwise dequant.
- A 3.4B model on a 2020s mid-range phone is not a fast experience in absolute terms; the numbers above are what the hardware gives.
Galaxy S26 — GPU backend
Both published bundles run on the Android GPU backend and generate.
| file | GPU backend | delegation | peak |
|---|
granite-4.1-3b_int4.litertlm | runs | 12119 / 12119 ops across 7 subgraphs on LiteRT GPU | 1394 MB |
granite-4.1-3b_int8.litertlm | runs | 21039 / 21039 ops across 12 subgraphs on LiteRT GPU | 1476 MB |
Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24 and 2026-08-25.
The op counts above are the LiteRT GPU partitions. XNNPACK additionally takes 1 of the 4 nodes in decode_embedder and 1 of the 4 nodes in prefill_embedder_1024; the runtime accepts that split.
No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.
GPU wiring, including the Gallery import toggle:
GPU guide.
Conversion notes
Converted with
litert-torch 0.9.3 / litert-converter 0.3.1 / ai-edge-quantizer 0.8.0 / litert-lm-builder 0.16.0, from a pristine released stack — no patched checkout (reproduction script:
hf-to-litertlm granite41_work/).
- No start token in the metadata. Granite's tokenizer sets
add_bos_token: False, and its BOS is its EOS (<|end_of_text|>). The converter writes a start_token from tokenizer.bos_token regardless, and the runtime prepends it on the first turn — which this model reads as a document that already ended, answering by echoing the question back. Removing the field is the whole fix, and the difference is 5/8 vs 8/8 on the sanity gate. The same behaviour reproduces on the bf16 PyTorch model when the token is prepended by hand, so it is a prompt-construction issue rather than anything about the conversion or the quantization.
- Prefill ladder trimmed to six signatures — in the int4 file. Every exported signature is charged engine memory whether or not it is called. With the full eleven, engine creation on iPhone 17 Pro was killed by the OS during Metal initialisation; six (1024, 256, 64, 16, 4, 1) leaves comfortable headroom and costs only padded chunks at intermediate prompt lengths. The int8 file still carries the full eleven-signature ladder — that is the difference behind its twelve GPU subgraphs against int4's seven in the Galaxy S26 table above — so int4 is the file to use on iOS.
- Embedder externalised so the tied 100352×2560 vocab table sits in its own section, clear of the ~2 GiB single-section mmap ceiling on iOS.
- Quantization: int4 is blockwise-32 with OCTAV clipping on the linears and int8 on the embedding; int8 is dynamic per-channel on linears and embedding.
Raspberry Pi 5 (CPU)
Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with
litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens,
--cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (
vcgencmd get_throttled stayed
0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.
| File | Prefill (tok/s) | Decode (tok/s) | TTFT | Peak RSS |
|---|
granite-4.1-3b_int4.litertlm | 11.6 (11.5–11.7) | 2.2 (2.1–2.2) | 25.0 s | 3.4 GB |
granite-4.1-3b_int8.litertlm | 12.6 (12.0–12.8) | 1.7 (1.7–1.7) | 20.9 s | 4.9 GB |
License and changes
Distributed under Apache-2.0 (inherited from the base model). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged into the .litertlm bundle. This repository is a community conversion and is not affiliated with IBM.
