Falcon-H1-0.5B-Instruct — LiteRT-LM
tiiuae/Falcon-H1-0.5B-Instruct converted to the
LiteRT-LM (
.litertlm) format for on-device inference with Google's
LiteRT-LM runtime.
Requires litert-lm ≥ 0.15. To our knowledge this is the first Falcon-H1 in LiteRT form, and the first
fully-hybrid (parallel attention + Mamba2 SSM in
every layer) architecture served by the released runtime — including on the GPU.
Falcon-H1 is TII's hybrid design: every one of the 36 layers runs a grouped-query attention branch and a Mamba2 selective-scan branch in parallel on the same input and sums them. Each layer therefore carries both a KV cache and constant-size conv + SSM recurrent state.
| File | Recipe | Size |
|---|
Falcon-H1-0.5B-Instruct_int8.litertlm | int8 dynamic on linears + embedding (convs and the scan stay float); fp32 activations declared for GPU | 650 MB |
Correctness
- Logits parity vs PyTorch: the float export matches the HF model teacher-forced across 8 decode positions — max|logit diff| 7.6e-05, correlation 1.000000, top-1 and top-5 identical at every position.
- 8-question sanity gate: int8 = float = GPU — all three answer 6/8 with near-verbatim identical text; the two misses ("merci" and 8×7) are the 0.5B model's own level (the float graph misses them the same way), not conversion or quantization damage.
- Prompt-length robustness: hermetic prefill-chunk sweep (fresh engine per length, 12–60 tokens) — 41/41 clean.
- Devices: iPhone 17 Pro (Metal) runs the composite quality probe at 7/8; Pixel 8a (OpenCL) delegates every subgraph fully (e.g. 5366/5366, zero rejections) with correct output.
Usage
1litert-lm run ./Falcon-H1-0.5B-Instruct_int8.litertlm --prompt "What is the capital of France? Answer in one word."
2
3# GPU
4litert-lm run ./Falcon-H1-0.5B-Instruct_int8.litertlm --backend gpu --cache no --prompt "..."
Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks. The bundle carries the tokenizer and the stock ChatML-style Falcon-H1 chat template.
Performance
litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:
| Backend | Prefill (256) | Decode | TTFT |
|---|
| GPU | 2650 tok/s | 127.5 tok/s | 0.10 s |
| CPU | 473 tok/s | 59.0 tok/s | 0.56 s |
On device (cold start, single runs, 145-token composite prompt, quality harness):
| Device | Backend | Prefill | Decode | TTFT | Peak memory |
|---|
| iPhone 17 Pro | GPU (Metal) | 365.9 tok/s | 52.7 tok/s | 0.46 s | 2.50 GB |
| iPhone 17 Pro | CPU | 332.3 tok/s | 36.2 tok/s | 0.48 s | 0.66 GB |
Pixel 8a (Tensor G3, litert_lm_main built from the v0.16.0 tag, OpenCL, default 19-token prompt): TTFT 0.93 s, prefill 23.6 tok/s, decode 11.8 tok/s, full delegation. The CPU on the same Pixel 8a (litert_lm_advanced_main from litert-lm 0.16.0, 205-token prompt, two cold runs, 2026-09-05) decodes 22.5–25.9 tok/s (prefill 89.6–98.7 tok/s, TTFT 2.19–2.41 s, engine init 4.5 s, peak 1374 MB), about twice the GPU figure. The two runs differ in binary and prompt length, so that ordering is indicated, not measured under one protocol; the manifest recommends cpu on 8 GB-class Android until a same-binary GPU run settles it.
Honest notes:
- Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal), Pixel 8a (Arm Mali, OpenCL) and Qualcomm Adreno — the Galaxy S26 section below is that measurement.
- On low-end Android the GPU buys prefill and time-to-first-token, not decode (decode is memory-bandwidth-bound there; the CPU path reads int8 weights while the fp32-activation GPU path reads expanded ones). Pick the backend for your workload: long prompts favour the GPU, long answers favour the CPU. On Apple hardware the GPU wins across the board.
- GPU runs with fp32 activations (declared in the bundle) — that is the GPU memory multiple above (2.50 GB vs 0.66 GB on iPhone).
- At 0.5B the model itself is weak at arithmetic and non-English trivia; int8 adds borderline greedy flips on exactly those items (e.g. 8×7 differs between backends). Everything stays coherent — there is no degeneracy.
Galaxy S26 — GPU and CPU
The published bundle runs on the Android GPU backend: LiteRT takes the whole graph and the model generates.
| file | GPU backend | delegation | peak |
|---|
Falcon-H1-0.5B-Instruct_int8.litertlm | runs | 59096 / 59096 ops across 12 subgraphs on LiteRT GPU | 2613 MB |
Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.
Speed on the same handset with one binary and one 205-token prompt file; only --backend changes (litert_lm_advanced_main from litert-lm 0.16.0, --benchmark, --sampler_backend=cpu). The GPU row is the gate run of 2026-08-24 (one run); the CPU rows were taken 2026-09-05, two cold runs (compile and weight caches deleted between runs) with the phone cooled below 42 °C before each run.
| Backend | Prefill | Decode | TTFT | Engine init | Peak (VmHWM) |
|---|
GPU (LiteRT GPU, full delegation) | 386.7 tok/s | 28.8 tok/s | 0.58 s | 11.3 s | 2613 MB |
| CPU (XNNPACK) | 217–317 tok/s | 32.1–40.5 tok/s | 0.69–1.01 s | 1.8 s | 1408 MB |
The CPU decodes 1.3× faster, at half the peak memory and without the 11 s engine init; the GPU wins prefill. This repo's litertlm_manifest.json recommends cpu for flagship-class Android on that pair. Pick gpu for long prompts with short answers.
GPU wiring, including the Gallery import toggle:
GPU guide.
Conversion notes
Converted with
litert-torch plus a hybrid-cache patch (reproduction script + patch:
hf-to-litertlm falcon_h1_work/):
- Composite hybrid cache layer: every layer holds KV + conv + recurrent state at ONE layer index — a cache layer class that is full-attention and Mamba2 at the same time (the runtime binds states by tensor name, so co-residency is just packaging).
- Folded selective scan: the Mamba2 scan is re-expressed as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no
BROADCAST_TO, no int64 index math) — this is what makes the graph fully delegable on GPU.
- Falcon-specific wiring: the µP multiplier vector (
mup_vector, a non-persistent model-level buffer) and ssm_in_multiplier are preserved in the traced scan; the exporter's timestamp-index kwargs are re-injected at the attention layer (FalconH1's layer loop drops kwargs).
- Prefill-pad guard: the runtime runs partially-filled prefill chunks; pad positions are made exact identity steps for the SSM and the stored conv window is gathered at the last valid column.
- Quantization: post-hoc dynamic int8 over linears + embedding only; convs and the scan stay float.
Raspberry Pi 5 (CPU)
Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with
litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens,
--cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (
vcgencmd get_throttled stayed
0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.
| File | Prefill (tok/s) | Decode (tok/s) | TTFT | Peak RSS |
|---|
Falcon-H1-0.5B-Instruct_int8.litertlm | 53.9 (53.4–54.7) | 7.0 (7.0–7.2) | 4.9 s | 1.3 GB |
License and changes
Distributed under the Falcon LLM License (inherited from the base model — see the license link). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified. This repository is a community conversion and is not affiliated with TII.
