Qwen3.5-2B — LiteRT-LM

Correctness

• Logits-level parity vs PyTorch: the float export matches the HF model teacher-forced across 48 positions — top-1 and top-5 agreement 100%, Pearson 1.0000, KL ≈ 0.
• 8-question sanity gate: 8/8 on CPU and GPU, on both litert-lm 0.15.0 and 0.16.0 (Mac).
• iPhone 17 Pro (Metal GPU): on the composite 8-question probe (all 8 questions in one 138-token prompt), the GPU output matches the HF fp32 reference word-for-word through answer 7 (including reproducing the model's own arithmetic slip on question 1 — faithful conversion includes the model's mistakes). At that point fp32 ends its turn without attempting question 8; the int8 path diverges by that one end-of-turn token and adds the correct 8th answer before stopping.
• Prompt-length robustness: first-token check against the runtime's real prefill chunk plans, fresh engine per length — 40/40 lengths clean on CPU, 20/20 on GPU.
• Multi-turn conversations carry state correctly across turns.

Usage

Performance

• Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal) and Pixel 8a (Arm Mali, OpenCL). We have not verified GPU execution on Qualcomm Adreno devices — if you are on a Snapdragon phone, use the CPU backend unless you have confirmed the GPU path on your own device.
• GPU inference runs with fp32 activations (declared in the bundle; the fp16-activation formulation is unfinished for this family). That is where the GPU memory multiple over CPU comes from.
• Pixel 8a (8 GB-class Android): GPU does not fit. The OpenCL delegate accepts the whole graph (zero rejections, full delegation on every compiled signature), but engine creation ran the phone out of memory before finishing — the GPU path's fp32-expanded weight buffers are several times the 1.97 GB file. A CPU-backend measurement was also blocked on our test phone, by storage rather than RAM: the runtime's XNNPACK weight cache needs roughly another file-size of free disk, which a nearly-full phone doesn't have. Treat this release as Apple-hardware-first; on Android, prefer 12 GB+ devices for GPU.
• On low-end Android GPUs, decode is memory-bandwidth-bound and does not beat the CPU anyway; the GPU win is on Apple hardware (and, generally, prefill/TTFT).
• On quality: individual questions gate 8/8 on every backend and both runtime versions. On the harder composite probe (8 questions in one prompt), GPU reproduces the fp32 reference; the CPU int8 path degrades (answers 3 of the 8, deterministically, identical on Mac and iPhone). For complex multi-part prompts, prefer the GPU backend or ask for a float variant.

Vision build (Qwen3.5-2B-VL_int8.litertlm)

Measured

Honest notes

• Six prefill signatures, not eleven. Every exported signature is charged engine memory whether or not it is called. The text build's full 1–1024 ladder peaks at ~5.3 GB on iPhone GPU and fits; adding the vision tower pushes the same ladder past a 12 GB phone's jetsam ceiling — all three Metal legs were killed after initialising 11 of 12 signatures. This build ships 1024, 256, 64, 16, 4, 1. Prompts longer than a signature are chunked as usual, so the only cost is a slightly coarser chunk plan. Cutting the ladder also tripled CPU decode (3.8 → 14.1 tok/s) and took first-token latency from 23.7 s to 5.1 s, because the runtime's XNNPACK repack is charged per signature too.
• Positions are 1-D. The fast_vlm contract feeds sequential positions, so the checkpoint's M-RoPE collapses to plain RoPE. Against the full M-RoPE reference on nine image/prompt pairs, one generation is identical and the other eight are fluent same-content paraphrases that diverge deep in the answer. Expect that class of difference — counting and dense-layout questions are where it shows.
• The vision graph is faithful; the int8 damage is in the encoder. The exported tower matches the model's own vision path at correlation 0.9999999992 in fp32. Quantizing the encoder to int8 drops that to 0.97–0.98 on real photographs, so this build ships the encoder in fp16 and only the adapter in int8. An int8-encoder variant is 300 MB smaller and still answers correctly, but it is not what ships here.
• Android is not gated for this build. The text build's notes apply (Pixel-class 8 GB GPU does not fit), and on Mali the fp16 vision encoder is known to crash the device on other models of this shape — an int8-vision build would be the Android path, and we have not measured one on a phone.
• Quality on the composite 8-question probe, separated by cause. On Mac this file answers 7 of 8, missing only 17 + 25 — and it misses it identically on GPU and CPU, so that one is the model, not the conversion (the text build's card records the same slip). On iPhone Metal two further answers degrade that do not degrade on Mac. Individual questions asked one at a time gate 8/8 on both Mac backends.

Conversion notes

• Rank-4 chunk kernel (shared with the 0.8B/4B): the reference chunked delta rule spells its contractions as broadcast-multiply-reduce over high-rank intermediates. The vendored kernel re-expresses them as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math), and writes every tail-pad as a concat with a zeros constant rather than a PAD op (the GPU delegate miscomputes rank-3 non-final-axis PAD — LiteRT#9272).
• Export cache for GatedDeltaNet layers: conv [B, conv_dim, K] + recurrent [B, heads, k_dim, v_dim] cache layers registered for layer_types == "linear_attention", so torch.export traces the model's own state contract.
• Vision tower (VL build only): the upstream tower is dynamic-resolution, and its grid_thw preprocessing aborts torch.export outright. It is re-authored for one static 512×512 image — Conv3d patch-embed folded to Conv2d over the duplicated temporal frame, learned position embeddings bilinearly resampled to the static grid ahead of time, 2-D rope precomputed, explicit full attention instead of the variable-length split. Patches stay in raster order through the encoder and the 2×2 merge happens in the adapter as four strided slices and a concat, so no GATHER_ND reaches the mobile GPU delegate. Two numerical guards matter: every activation keeps a leading batch dim (Metal computes rank-2 elementwise against a rank-2 constant wrong, silently), and LayerNorm is pre-scaled by a calibrated power of two because from block 11 on the residual stream carries values large enough to overflow fp16 accumulation.
• Pad guard on the externalised-embedder graph: fast_vlm moves the embedder into its own tflite, which removes token ids from the decoder graph — so the gated-delta prefill-pad guard, which keyed off input_ids != 0, silently stopped arming. It now derives the valid mask from position monotonicity instead, the same fallback upstream's own exportable module uses. The 40/40 prompt-length sweep is what proves it armed.
• Unlike the 4B, the 2B's linear-attention heads are ungrouped (16 key / 16 value), so the 4B's head-interleave rewrite never traces, and the full prefill signature ladder fits on-phone.

Raspberry Pi 5 (CPU)