CLIPSeg (CVPR 2022, Apache-2.0) re-authored for
LiteRT: type what you want to segment ("a cat", "the sky") and get a mask — no fixed class list.
Three graphs — CLIP text and vision encoders on the CompiledModel GPU, the tiny 3-layer
decoder on CPU (its 4-head/head_dim-16 attention fp16-miscomputes on the Mali delegate; the
12-head/head_dim-64 vision encoder survives at 0.998).
CLIPSeg on-device: text-prompted segmentation
Input | prompt "a dog" | prompt "the grass" — the same image, two prompts, masks from the
on-device model. Photo: "Lily the Golden Retriever in the grass" (Wikimedia Commons, Public Domain).
Verified on a Pixel 8a: text 761/761 GPU (~8.7 ms) + vision 613/613 GPU (~8.2 ms) + decoder
CPU (exact); end-to-end device-vs-PyTorch logits corr 0.99998, mask IoU 0.9986.
1// vision + text: Accelerator.GPU; decoder: Accelerator.CPU2val vis = CompiledModel.create(File(dir,"clipseg_vision_fp16.tflite").path, CompiledModel.Options(Accelerator.GPU),null)3val dec = CompiledModel.create(File(dir,"clipseg_decoder.tflite").path, CompiledModel.Options(Accelerator.CPU),null)4// vision: image[1,3,352,352] -> t3,t6,t9 decoder: (t3,t6,t9,cond[512]) -> logits[1,352,352]5// text graph + host BPE/emb-lookup/text_projection produce cond; see ClipSeg.kt in the LiteRT sample.6val logits = decOut[0].readFloat()// sigmoid -> mask
Conversion
Re-authored with litert-torch: qkv-3D-BMM attention, quick-GELU, baked interpolated pos-embed
(14²→22² @352), host-side token-embedding lookup, safe_ln_up (up-scaled LayerNorm keeping the eps
fp16-normal), convT4x4 (exact non-overlapping ConvTranspose as 1×1-conv + 4-D interleave). The
decoder ships on CPU because its small-head-dim attention fp16-miscomputes on the Mali GPU delegate
(re-authoring is exact — desktop fp16 corr 0.999996).
Upstream
CIDAS/clipseg-rd64-refined (Apache-2.0). Please
cite Lüddecke & Ecker, Image Segmentation Using Text and Image Prompts (CVPR 2022).
Performance
Measured on a Pixel 8a (Tensor G3, Android 16) with the standard TFLite benchmark_model tool — 10 warm-up runs then 50 timed runs, reported as the tool's mean.
Any on-device figure recorded when this model shipped came from a different runtime. It was taken through LiteRT's own CompiledModel accelerator (logcat reports it as LITERT_CL), which is the path the Kotlin sample app and the LiteRT API use, and it appears elsewhere on this card. The rows above are the classic TFLite OpenCL delegate, measured with a tool anyone can download and re-run. The two are not comparable, so read the rows above as a reproducible floor rather than as this model's speed on LiteRT.
On this delegate the CPU is the faster choice for clipseg_decoder.tflite (11.7 ms on CPU against 96.2 ms on GPU), clipseg_text_fp16.tflite (53.8 ms on CPU against 154.4 ms on GPU) — worth knowing before you reach for the GPU on a mid-range phone.
Note that the GPU does not take the whole graph here (44 / 196 in clipseg_decoder.tflite, 32 / 761 in clipseg_text_fp16.tflite, 51 / 613 in clipseg_vision_fp16.tflite); the remainder runs on the CPU and the split costs a per-partition round trip.
Snapdragon NPU (Hexagon)
clipseg_decoder.tflite — the NPU is 1.95x faster than the GPU (1.67 ms against 3.25 ms) and loads 7.72x faster (102 ms against 784 ms).
clipseg_text_fp16.tflite — the NPU is 2.95x faster than the GPU (2.10 ms against 6.20 ms) and loads 7.55x faster (138 ms against 1041 ms).
clipseg_vision_fp16.tflite — the NPU is 2.23x faster than the GPU (15.90 ms against 35.51 ms) and loads 8.17x faster (192 ms against 1569 ms).
file
backend
compiled
inference (median / min)
load
clipseg_decoder.tflite
NPU (Hexagon v81)
on-device JIT
1.67 ms / 1.63 ms
102 ms
clipseg_decoder.tflite
GPU (Adreno)
—
3.25 ms / 2.82 ms
784 ms
clipseg_text_fp16.tflite
NPU (Hexagon v81)
on-device JIT
2.10 ms / 2.07 ms
138 ms
clipseg_text_fp16.tflite
GPU (Adreno)
—
6.20 ms / 6.13 ms
1041 ms
clipseg_vision_fp16.tflite
NPU (Hexagon v81)
on-device JIT
15.90 ms / 15.28 ms
192 ms
clipseg_vision_fp16.tflite
GPU (Adreno)
—
35.51 ms / 32.45 ms
1569 ms
Measured on a Samsung Galaxy S26 (Snapdragon 8 Elite Gen 5 / SM8850, Hexagon v81, Android 16) with LiteRT CompiledModel 2.2.0, one accelerator per process, 5 warm-up runs then N=50 timed runs, median reported. Every run held thermal status NONE throughout. Headroom 0.71–0.74, where 1.0 is the throttling threshold.
The NPU rows ran the published file unchanged. LiteRT compiled it for the Hexagon on the device at first load. Those first compiles took 1.0 s to 18 s here. The load column above is the cached load every later run pays. Recipe and the runtime libraries it needs: NPU guide.
Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with the LiteRT benchmark_model tool from litert-cli-nightly 0.2.0.dev20260805: CPU inference (XNNPACK, 4 threads), 3 invocations per file of 10 warm-up plus 50 timed runs (the tool caps a phase at 150 s, so very slow graphs run fewer — the Runs column is the actual timed total). The latency is the median across invocations; the spread is the min–max over all timed runs. No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0).
