Falcon-H1-3B-Instruct — LiteRT-LM
tiiuae/Falcon-H1-3B-Instruct converted to the
LiteRT-LM (
.litertlm) format for on-device inference with Google's
LiteRT-LM runtime.
Requires litert-lm ≥ 0.15. Sibling of
litert-community/Falcon-H1-0.5B-Instruct and
litert-community/Falcon-H1-1.5B-Instruct — same conversion, same patch.
Falcon-H1 is TII's fully-hybrid design: every one of the 32 layers runs a grouped-query attention branch and a Mamba2 selective-scan branch in parallel on the same input and sums them. Each layer therefore carries both a KV cache and constant-size conv + SSM recurrent state.
| File | Recipe | Size |
|---|
Falcon-H1-3B-Instruct_int8.litertlm | int8 dynamic on linears + embedding (convs and the scan stay float); fp32 activations declared for GPU | 3.15 GB |
Correctness
- Logits parity vs PyTorch: the float export matches the HF model teacher-forced across 48 decode positions — top-1 and top-5 identical at every position, mean per-position logit correlation 1.0000, mean KL ≈ 0.
- 8-question sanity gate: 8/8 on every lane — GPU and CPU, litert-lm 0.15.0 and 0.16.0. No degeneration, no greedy flips (the first Falcon-H1 size where int8 drops nothing).
- Prompt-length robustness: hermetic prefill-chunk sweep (fresh engine per length) — CPU fills 12–51 and GPU fills 12–31 all clean.
- iPhone 17 Pro (Metal): the 8-item composite quality probe answers 8/8 on GPU and 8/8 on CPU, identical answers on both backends.
Usage
1litert-lm run ./Falcon-H1-3B-Instruct_int8.litertlm --prompt "What is the capital of France? Answer in one word."
2
3# GPU
4litert-lm run ./Falcon-H1-3B-Instruct_int8.litertlm --backend gpu --cache no --prompt "..."
Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks. The bundle carries the tokenizer and the stock ChatML-style Falcon-H1 chat template.
Performance
litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:
| Backend | Prefill (256) | Decode | TTFT |
|---|
| GPU | 979 tok/s | 65.3 tok/s | 0.28 s |
| CPU | 121 tok/s | 20.9 tok/s | 2.17 s |
On device (cold start, single runs, 146-token composite prompt, quality harness):
| Device | Backend | Prefill | Decode | TTFT | Peak memory |
|---|
| iPhone 17 Pro | GPU (Metal) | 111.5 tok/s | 14.0 tok/s | 1.49 s | 3.03 GB |
| iPhone 17 Pro | CPU | 48.7 tok/s | 7.8 tok/s | 3.14 s | 1.46 GB |
Honest notes:
- Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal), Pixel 8a (Arm Mali, OpenCL) and Qualcomm Adreno — the Galaxy S26 section below is that measurement.
- On low-end Android the GPU buys prefill and time-to-first-token, not decode (decode is memory-bandwidth-bound there; the CPU path reads int8 weights while the fp32-activation GPU path reads expanded ones). Pick the backend for your workload: long prompts favour the GPU, long answers favour the CPU. On Apple hardware the GPU wins across the board.
- GPU runs with fp32 activations (declared in the bundle) — expect a corresponding memory multiple over CPU.
Galaxy S26 — GPU backend
The published bundle runs on the Android GPU backend: LiteRT takes the whole graph and the model generates.
| file | GPU backend | delegation | peak |
|---|
Falcon-H1-3B-Instruct_int8.litertlm | runs | 55723 / 55723 ops across 12 subgraphs on LiteRT GPU | 2128 MB |
Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.
No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.
GPU wiring, including the Gallery import toggle:
GPU guide.
Conversion notes
Converted with
litert-torch plus a hybrid-cache patch (reproduction script + patch:
hf-to-litertlm falcon_h1_work/):
- Composite hybrid cache layer: every layer holds KV + conv + recurrent state at ONE layer index — a cache layer class that is full-attention and Mamba2 at the same time (the runtime binds states by tensor name, so co-residency is just packaging).
- Folded selective scan: the Mamba2 scan is re-expressed as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no
BROADCAST_TO, no int64 index math) — this is what makes the graph fully delegable on GPU.
- Falcon-specific wiring: the µP multiplier vector (
mup_vector, a non-persistent model-level buffer) and ssm_in_multiplier are preserved in the traced scan; the exporter's timestamp-index kwargs are re-injected at the attention layer (FalconH1's layer loop drops kwargs).
- Prefill-pad guard: the runtime runs partially-filled prefill chunks; pad positions are made exact identity steps for the SSM and the stored conv window is gathered at the last valid column.
- Quantization: post-hoc dynamic int8 over linears + embedding only; convs and the scan stay float.
Raspberry Pi 5 (CPU)
Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with
litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens,
--cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (
vcgencmd get_throttled stayed
0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.
| File | Prefill (tok/s) | Decode (tok/s) | TTFT | Peak RSS |
|---|
Falcon-H1-3B-Instruct_int8.litertlm | 23.3 (22.6–23.4) | 1.9 (1.9–1.9) | 11.5 s | 4.3 GB |
License and changes
Distributed under the Falcon LLM License (inherited from the base model — see the license link). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified. This repository is a community conversion and is not affiliated with TII.
