D-FINE-S on a Pixel 8a — both transformer graphs on CompiledModel GPU
D-FINE (USTC, 2024 — ustc-community/dfine-small-coco), the SOTA
real-time DETR, converted to LiteRT and running 100% on the CompiledModel GPU (ML Drift) on a
phone, with no CPU/ONNX fallback.
D-FINE is a transformer detector — HGNetV2 backbone + a hybrid AIFI/CCFM encoder + an FDR (Fine-grained
Distribution Refinement) decoder. Off-the-shelf it is GPU-incompatible (deformable grid_sample →
GATHER_ND, two-stage query selection → TOPK/GATHER). Here it is converted with litert-torch and
split into two GPU graphs with a host step between them, so both transformer graphs run on the GPU.
image[1,3,640,640]
→[GPU Graph A]→ enc_class, memory_raw
→[host: top-300 by max class score; per-token tail on the 300 selected (fp32):
target = enc_output(valid·memory_raw) (Linear + LayerNorm)
ref = enc_bbox_head(target) + anchors (3-layer MLP)]
→[GPU Graph B (memory_raw, target, ref)]→ boxes[1,300,4], logits[1,300,80]
→[host: sigmoid + threshold + cxcywh→xyxy + light NMS]→ detections
The on-device gate — a Mali 3D-sequence fan-out bug (NOT the FDR decoder)
A naïve Graph A (emitting enc_class/enc_coord/output_memory/memory_raw together) gave 0 detections
on device, and it first looked like the FDR decoder collapsing in fp16. That was a red herring. The real
cause is a Mali delegate bug: a 3-D token tensor [1,N,256] (from conv.flatten(2).transpose(1,2)) that
is both a graph output and consumed by another node — or that fans out to several consumers — is silently
clobbered on the longer branch (4-D conv-map outputs are fine). Here the raw memory output (Graph B's
cross-attention input) was garbage (device corr −0.02) → the decoder cross-attended to noise → no detections.
Fix: Graph A emits only the two fp16-clean leaves (enc_class + memory_raw×2) and the per-token tail
(enc_output + enc_bbox_head) runs on the host over the 300 selected tokens (exact, since per-token ops
commute with the gather). With clean memory the FDR decoder is perfect — correlation is not the ship
criterion, real-image detection IoU is.
Minimal usage
Android (Kotlin, CompiledModel GPU)
kotlin
1val ga = CompiledModel.create(context.assets,"dfine_graphA_fp16.tflite",2 CompiledModel.Options(Accelerator.GPU),null)3val gb = CompiledModel.create(context.assets,"dfine_graphB_fp16.tflite",4 CompiledModel.Options(Accelerator.GPU),null)5val aIn = ga.createInputBuffers();val aOut = ga.createOutputBuffers()6val bIn = gb.createInputBuffers();val bOut = gb.createOutputBuffers()7aIn[0].writeFloat(chw)// [1,3,640,640] RGB in [0,1], NCHW8ga.run(aIn, aOut)// -> enc_class[1,8400,80], memory_raw*2[1,8400,256]9// host step: /2 -> top-300 -> per-token tail (host_params.bin) -> target[1,300,256], ref[1,300,4]10// (resolve buffer slots by float size; full math in the Python below / litert-samples object_detection)11bIn[0].writeFloat(memory); bIn[1].writeFloat(target); bIn[2].writeFloat(ref)12gb.run(bIn, bOut)13val boxes = bOut[0].readFloat()// [1,300,4] cxcywh in [0,1]14val logits = bOut[1].readFloat()// [1,300,80] -> sigmoid + threshold + light NMS
Python (desktop verification)
python
1import numpy as np
2from PIL import Image
3from ai_edge_litert.interpreter import Interpreter
45NP_, NQ, NC, H =8400,300,80,2566img = Image.open("photo.jpg").convert("RGB").resize((640,640))7x =(np.asarray(img, np.float32)/255.0).transpose(2,0,1)[None]# [1,3,640,640], [0,1] only89# host_params.bin (fp32 LE): enc_output W[256,256],b,gamma,beta · bbox-MLP W0,b0,W1,b1,W2[4,256],b2 · valid[8400] · anchors[8400,4]10p = np.fromfile("host_params.bin", np.float32); o =011deftake(*s):12global o; n =int(np.prod(s)); v = p[o:o+n].reshape(s); o += n;return v
13eoW, eoB, eoG, eoBe = take(H, H), take(H), take(H), take(H)14W0, b0, W1, b1, W2, b2 = take(H, H), take(H), take(H, H), take(H), take(4, H), take(4)15valid, anchors = take(NP_), take(NP_,4)1617defrun(path, feeds):# feed/fetch tensors by shape (converter slot order is arbitrary)18 it = Interpreter(model_path=path); it.allocate_tensors()19for d in it.get_input_details(): it.set_tensor(d["index"], feeds[tuple(d["shape"][1:])])20 it.invoke();return{tuple(d["shape"][1:]): it.get_tensor(d["index"])for d in it.get_output_details()}2122a = run("dfine_graphA_fp16.tflite",{(3,640,640): x})23enc_cls, mem = a[(NP_, NC)][0], a[(NP_, H)][0]/2.0# Graph A emits memory_raw*2 — undo2425top = np.argsort(-enc_cls.max(-1))[:NQ]# top-300 by max class logit26t =(valid[top,None]* mem[top]) @ eoW.T + eoB # per-token tail: enc_output Linear...27t =(t - t.mean(-1, keepdims=True))/ np.sqrt(t.var(-1, keepdims=True)+1e-5)* eoG + eoBe # ...+ LayerNorm28h = np.maximum(t @ W0.T + b0,0); h = np.maximum(h @ W1.T + b1,0)29ref = h @ W2.T + b2 + anchors[top]# enc_bbox_head MLP + anchors3031b = run("dfine_graphB_fp16.tflite",32{(NP_, H): mem[None],(NQ, H): t[None].astype(np.float32),(NQ,4): ref[None].astype(np.float32)})33boxes, logits = b[(NQ,4)][0], b[(NQ, NC)][0]# cxcywh in [0,1] / 80-way logits34labels =open("coco_labels.txt").read().splitlines()35score =1/(1+ np.exp(-logits.max(-1))); cls = logits.argmax(-1)36for q in np.where(score >0.4)[0]:# + light NMS (IoU 0.7) in a real app37 cx, cy, w, hh = boxes[q]38print(f"{labels[cls[q]]:12s}{score[q]:.2f} xyxy=({cx-w/2:.3f},{cy-hh/2:.3f},{cx+w/2:.3f},{cy+hh/2:.3f})")
On-device (Pixel 8a, Tensor G3 — verified)
Both graphs run 100% GPU-resident (LITERT_CL): Graph A 511/511, Graph B 850/850. On a COCO val image
(giraffe + cows) the device chain reproduces the PyTorch detections at IoU 0.99–1.00 with matching class
and score. End-to-end ~450 ms/frame — accurate and fully-GPU but not real-time on this device (the
deformable decoder over the 8400 tokens / 80×80 levels is GPU-compute-bound; the GATHER-free tent-matmul
grid_sample turns an O(points) gather into an O(H·W) matmul). For a real-time camera DETR see
RF-DETR Nano.
Preprocessing / outputs
Input: square resize to 640×640, RGB, [0,1] rescale only (no ImageNet normalization), NCHW.
Output: Graph B boxes are cxcywh normalized to [0,1]; logits are 80-way (contiguous COCO id 0–79). Host applies sigmoid + score threshold + cxcywh→xyxy + light NMS.
Conversion notes
Converted with litert-torch (NCHW preserved — onnx2tf destroys ViT attention). Re-authoring (per-graph
tflite-vs-torch correlation 1.0): deformable grid_sample → a GATHER/CAST-free tent-matmul, multi-level
MSDeformAttn ≤4D, the FDR LQE prob.topk → iterative max-and-mask, distance2bboxstack→cat, baked AIFI
sine pos-embed, a down-scaled fp16-safe LayerNorm, and the 3D-fan-out fix above (emit clean leaves +
host-side per-token tail).
A runnable Android sample (CompiledModel GPU) and the conversion scripts are in the official
ai-edge-litert/litert-samplesobject_detection example.
Performance
Measured on a Pixel 8a (Tensor G3, Android 16) with the standard TFLite benchmark_model tool — 10 warm-up runs then 50 timed runs, reported as the tool's mean.
Any on-device figure recorded when this model shipped came from a different runtime. It was taken through LiteRT's own CompiledModel accelerator (logcat reports it as LITERT_CL), which is the path the Kotlin sample app and the LiteRT API use, and it appears elsewhere on this card. The rows above are the classic TFLite OpenCL delegate, measured with a tool anyone can download and re-run. The two are not comparable, so read the rows above as a reproducible floor rather than as this model's speed on LiteRT.
XNNPACK declines these fp16 graphs — it reports failed to delegate DEPTHWISE_CONV_2D and then fails to allocate tensors — so there is no usable CPU number. Disabling XNNPACK falls back to reference kernels, which measured about 20× slower than the GPU on models of this size and would not represent CPU inference anyone would ship.
Note that the GPU does not take the whole graph here (233 / 511 in dfine_graphA_fp16.tflite, 128 / 850 in dfine_graphB_fp16.tflite); the remainder runs on the CPU and the split costs a per-partition round trip.
Snapdragon NPU (Hexagon)
dfine_graphA_fp16.tflite — the GPU is faster: 19.73 ms against 23.81 ms on the NPU, a factor of 1.21. The NPU still loads 14.28x faster (121 ms against 1730 ms).
dfine_graphB_fp16.tflite — the GPU is faster: 134.0 ms against 172.4 ms on the NPU, a factor of 1.29. The NPU still loads 8.31x faster (184 ms against 1531 ms).
file
backend
compiled
inference (median / min)
load
dfine_graphA_fp16.tflite
NPU (Hexagon v81)
on-device JIT
23.81 ms / 19.45 ms
121 ms
dfine_graphA_fp16.tflite
GPU (Adreno)
—
19.73 ms / 14.55 ms
1730 ms
dfine_graphB_fp16.tflite
NPU (Hexagon v81)
on-device JIT
172.4 ms / 169.8 ms
184 ms
dfine_graphB_fp16.tflite
GPU (Adreno)
—
134.0 ms / 132.9 ms
1531 ms
Measured on a Samsung Galaxy S26 (Snapdragon 8 Elite Gen 5 / SM8850, Hexagon v81, Android 16) with LiteRT CompiledModel 2.2.0, one accelerator per process, 5 warm-up runs then N=50 timed runs, median reported. Every run held thermal status NONE throughout. Headroom 0.76–0.80, where 1.0 is the throttling threshold.
The NPU rows ran the published file unchanged. LiteRT compiled it for the Hexagon on the device at first load. Those first compiles took 5.7 s to 81 s here. The load column above is the cached load every later run pays. Recipe and the runtime libraries it needs: NPU guide.
Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with the LiteRT benchmark_model tool from litert-cli-nightly 0.2.0.dev20260805: CPU inference (XNNPACK, 4 threads), 3 invocations per file of 10 warm-up plus 50 timed runs (the tool caps a phase at 150 s, so very slow graphs run fewer — the Runs column is the actual timed total). The latency is the median across invocations; the spread is the min–max over all timed runs. No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0).
