The mixed INT4 .litertlm artifact was produced with a TorchAO-based quantize-first recipe from the original Hugging Face checkpoint. This is a mixed quantization layout rather than a uniform all-INT4 model: eligible linear projection weights are stored as blockwise INT4 with group size 32 and floating-point scales, token embedding weights use weight-only INT8 quantization, and normalization/reduction paths plus KV cache tensors remain floating point.

The mixed INT4 bundle also uses LiteRT-LM StableHLO composite ops for attention/cache execution, including odml.runtime_bmm and odml.cache_update.

Performance

Desktop benchmark: AMD Radeon AI PRO R9700, LiteRT-LM WebGPU, 256 prefill tokens, 32 decode tokens. Android rows use LiteRT-LM v0.13.1 with GPU OpenCL, 256 prefill tokens, and 64 decode tokens. Values report the warmed iteration from a two-iteration run unless noted.

Hardware benchmark disclosure: Results were measured by us on retail devices purchased through normal channels. These results are not affiliated with, sponsored by, endorsed by, or verified by Samsung, vivo, Qualcomm, MediaTek, Google, MLCommons, or Hugging Face. Results depend on device SKU, OS build, thermal state, battery mode, backend, model quantization, runtime version, and benchmark settings.

Device / Backend	Prefill (tok/s)	Decode (tok/s)	TTFT (s)	Peak Private Footprint
Desktop GPU WebGPU	1324.75	100.75	0.20	1697 MB
Samsung SM-S937U1 GPU OpenCL	375.67	18.51	0.74	1587 MB
vivo V2502A GPU OpenCL	174.12	12.60	1.55	4720 MB
TECNO LJ9 GPU OpenCL	102.57	9.65	2.60	4903 MB

Galaxy S26 — GPU vs CPU (litert-lm 0.16.0, community measurement)

A different runtime release and device generation than the table above (v0.13.1), so compare within one table, not across. Physical Samsung Galaxy S26 (SM-S942Q, Snapdragon 8 Elite Gen 5 / SM8850, Android 16), litert_lm_advanced_main from the litert-lm v0.16.0 release, GPU backend OpenCL (LITERT_CL), one fixed 205-token prompt text (203 tokens under this tokenizer), --benchmark. Two runs per backend taken back-to-back — cells show the range. Peak RSS is the process VmHWM. Before quoting, the file was run on each backend with a real prompt and both produced a correct text answer; the GPU rows are full delegation (1753/1753 and 1680/1680 ops on LITERT_CL, nothing on the CPU fallback).

Backend	Prefill (203 tok)	Decode	Time-to-first-token	Init	Peak RSS
GPU (OpenCL)	305.7–306.2 tok/s	19.87–19.88 tok/s	0.71 s	15.6–21.4 s	1533–1534 MB
CPU (XNNPACK)	46.1–55.6 tok/s	8.4–9.2 tok/s	3.8–4.5 s	2.6–3.5 s	4582–6093 MB

What the table says:

At 4B the GPU leads every runtime column by a wide margin: prefill 5.5–6.6×, decode 2.2–2.4× (19.9 against 8.4–9.2 tok/s), time-to-first-token 0.71 s against ~4 s.
Peak RSS decides deployability: 1.53 GB on the GPU against 4.6–6.1 GB on the CPU — on most phones the CPU path pressures the memory killer, the GPU path does not.
The two GPU runs agree to three digits — this bundle holds its clock across back-to-back runs. GPU engine init is the one cost the table hides: 16–21 s per process, against 2.6–3.5 s on CPU.

Try It

Install uv and run:

bash

uv tool install litert-lm
uvx litert-lm run --from-huggingface-repo=litert-community/Qwen3-4B-Instruct-2507 qwen3_4b_instruct_2507_mixed_int4.litertlm --prompt="What is the capital of France?"

Integration

Ready to integrate this into your product? Get started in the LiteRT-LM documentation.

Citation

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report},
      author={Qwen Team},
      year={2025},
      eprint={2505.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.09388},
}