LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).
Nanbeige4.2-3B — LiteRT-LM (.litertlm)
On-device build of Nanbeige/Nanbeige4.2-3B for Google's LiteRT-LM runtime — an Apache-2.0 3B-class looped-transformer reasoning model (its 22 layers run twice per token, num_loops=2).
Note on speed: the loop makes this model ~2× the per-token compute of a normal 3B (that's where its quality comes from) — expect roughly 8B-class decode rates on CPU.
The loop is unrolled at export: 44 layer executions over 22 shared-weight layers, one KV-cache slot per (loop, layer) — 44 slots. Weights are stored once; only compute and KV double.
The chat template auto-opens the model's <think> block (this is how the official template works — the model starts already inside its reasoning).
Performance
litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine.
Device
Backend
Prefill (256)
Decode
TTFT
Load
Peak footprint
Apple M4 Max (macOS)
CPU
56 tok/s
12.5 tok/s
4.74 s
—
—
Apple M4 Max (macOS)
GPU (Metal), model_fp32act.litertlm
409 tok/s
39.7 tok/s
0.67 s
—
—
iPhone 17 Pro
—
see the iPhone note below
Reproducibility: the GPU rows repeat to within about 1% across invocations; the CPU rows are noisier — re-running the 1B control six times spread its CPU decode over 29.0–33.3 tok/s, so treat the CPU column as accurate to roughly ±7%.
The iPhone 17 Pro runs this file.model_fp32act.litertlm answers all eight questions of an on-device sanity gate on both backends — 8/8 on GPU (Metal) and 8/8 on CPU, engine init 6.77 s and 2.53 s, measured 2026-08-28. The columns above are blank rather than filled because that harness measures correctness, not throughput: init is engine initialisation and not TTFT, and no tok/s figure is claimed.
⚠ That gate set no sampler, so it ran the runtime default — not the sampling recipe this card asks you to use. The default is deterministic (a control model's GPU and CPU runs are byte-identical), so the 8/8 is a greedy result. Read it as "the bundle loads and generates correctly on this phone", not as a score for the recommended temperature 0.6 / top_k 20 / top_p 0.95 configuration, and not as evidence against the sampling note below.
⚠ An earlier version of this card said neither file would start on this phone. That was wrong, and the fault was in our harness: it never created a window, so iOS never considered its launch finished and the process-launch watchdog killed it at exactly 20.00 seconds (0x8BADF00D) regardless of the model. On the console that is indistinguishable from an out-of-memory kill. A model passed only if loading and generating fitted inside 20 s, which a 3B model that reasons before answering cannot do. With the harness fixed the same file, the same phone and the same bundle pass.
The published iPhone 17 Pro | CPU | 3.2 tok/s row from before 2026-08-27 stays withdrawn for the separate reason that its run log was never retained. model.litertlm has not been tested on this phone.
Accuracy note (GSM8K, n=50, 0-shot CoT, official sampling)
build
GSM8K
bf16 (transformers 4.51, reference)
94.0% (47/50)
this int4 .litertlm (LiteRT-LM CPU)
90.0% (45/50)
Local gate: 8/8 basic-quality questions, clean <think>…</think> closure, ChatML stop.
Galaxy S26 — GPU backend
file
GPU backend
delegation
peak
model_fp32act.litertlm
runs
4716 / 4716 ops across 2 subgraphs on LiteRT GPU
2691 MB
model.litertlm
does not run
4716 / 4716 ops delegated, then <unk> at generation
1580 MB
Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from
litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu. Peak is the process high-water mark during that run.
Note that delegation is identical for both files — the graph was never the problem, only the activation dtype.
No speed rows for this phone, deliberately. Each benchmark decodes about 3900 tokens because the model
thinks before answering, so a pair of runs holds the phone for over half an hour and the second is measurably
throttled (prefill 46.2 then 27.0 tok/s; engine init 12.4 s then 29.2 s). A same-device CPU control puts
decode at 3.45 tok/s against the GPU's 3.91 and 3.31 — a wash. The GPU's one clear advantage on this phone is
memory: 2590 MB against the CPU path's 4152 MB.
On Apple the picture is the opposite: GPU wins both ways, 409 against 57 tok/s prefill and 39.7 against
12.9 tok/s decode on an M4 Max, all four numbers from the same machine.
⚠ Usage notes
Sampling is required. The model collapses under greedy decoding (official recipe: temperature 0.6, top_k 20, top_p 0.95 — see its generation_config.json). Configure the runtime session accordingly; do not run at temperature 0.
Backend depends on the file.model.litertlm is CPU-only: on a GPU its fp16 activations mis-read the loop boundary of the unrolled graph and it emits nothing but <unk>. model_fp32act.litertlm is the same int4 weights with prefer_activation_type = "fp32" declared, and it generates correctly on Apple Metal and on a Galaxy S26 GPU.
⚠ The GPU row this card used to carry (556 / 52.6 tok/s) was measured on the file that emits <unk>.litert-lm benchmark reports throughput without checking that the backend produced text. The honest Mac GPU figure is the fp32act file's 409 / 39.7 tok/s above; fp32 activations cost about 26% of prefill and 23% of decode, and that is the price of the graph being computed correctly.
Reasoning outputs open inside <think>; the visible answer follows </think>.
Run it
Android: push the file and import it in Google AI Edge Gallery (+ → import, keep CPU).
CLI / other platforms: see the LiteRT-LM repo for litert-lm runtime usage.
The conversion script (scripts/convert_nanbeige42.py) documents the two traps this model hits: transformers-5.x zeroes the modeling's init-computed rotary inv_freq buffer (rope must be recomputed from config), and the looped architecture needs a 44-slot KV cache registered through the export pipeline's cache_implementation hook.
2026-08-28 — start_token fix (weights unchanged)
The LiteRT-LM engine prepends the metadata start_token to every prompt, and this model's chat template already renders <|im_start|> itself — so the model was reading <|im_start|><|im_start|>…, a stream it was never trained on. The start token has been removed. This bundle also has its prompt-template system.prefix corrected to <|im_start|>system\n: the leading marker used to come from the start token, so removing that alone would have broken the system-prompt path.
Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file — only the file's own sha256 differs. What changed is the input: the token stream the model reads for a given conversation can differ from the previous file's, and it now matches this model's own reference chat stream. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file in either direction. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-28, re-download.
2026-08-29 — default system prompt restored (weights unchanged)
The upstream chat template emits a default system turn whenever the caller sends no system message — for this model: 你是南北阁，一款由BOSS直聘自主研发并训练的专业大语言模型。. The converter's template probe renders the template with a system message already present, so that block was never seen and never reached the bundle: with no system message the model was running without the default system turn it was tuned with. The chat template in model.litertlm, model_fp32act.litertlm now emits the block exactly once when no system message is given. In model.litertlm, model_fp32act.litertlm, the block is not emitted when you pass a system message. model_fp32act.litertlm also receives the 2026-08-28 start_token fix that model.litertlm already carries: the duplicate leading <|im_start|> is removed and system.prefix is corrected to <|im_start|>system\n. The on-device rows on this card (iPhone 17 Pro, Galaxy S26) were measured before this change; the on-device gate has not been re-run on this model_fp32act.litertlm. The restored block adds 21 prefill tokens to a conversation that sends no system message, so time-to-first-token grows by that much; per-token speed is unchanged.
Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file per token — only the file's own sha256 differs. What changed is the input: with no system message, the prompt now renders byte-identical to the upstream chat template's output, verified on the LiteRT-LM runtime. A system message you pass yourself renders as before. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-29, re-download.
2026-08-31 — thought channel declared (metadata only, weights unchanged)
model.litertlm, model_fp32act.litertlm now declare the reasoning channel in their metadata (LlmMetadata.channels: channel name thought, markers <think>…</think> exactly as this model emits them). Without the declaration the runtime has no way to tell the reasoning apart from the answer: the raw thinking streamed inline into the visible text, and a thinking_token_budget was silently ignored (the API returns OK and only logs a warning). With the channel declared, LiteRT-LM returns the reasoning separated in channels["thought"] and the thinking budget takes effect.
Metadata-only change: every section of the bundle except LlmMetadata is byte-identical to the previous file (verified per section, tokenizer included), so the weights, the graph, the tokenizer and the chat template are unchanged and the speed and accuracy numbers on this card still describe this file — only the file's own sha256 differs. Verified on the LiteRT-LM runtime (litert-lm-api 0.16.1): the visible answer stays clean, the reasoning lands in channels["thought"], and on one bundle of this batch thinking_token_budget=16 was confirmed to truncate the reasoning at exactly 16 tokens where it was a no-op before. If you downloaded before 2026-08-31, re-download to get the channel-aware file.
Raspberry Pi 5 (CPU)
Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.
