Falcon3-3B-Instruct — LiteRT-LM (blockwise int4)
tiiuae/Falcon3-3B-Instruct
converted to the
LiteRT-LM (
.litertlm) format for on-device inference with
Google's
LiteRT-LM runtime (the
engine behind the official
litert-community/* models).
Text-only conversion (the Falcon3 decoder; no vision/audio towers).
| |
|---|
| File | model.litertlm (~1.74 GB) |
| Quantization | int4 weights — blockwise (block 128), symmetric; embeddings INT8 |
| Compute | integer |
| Context (KV cache) | 2048 |
| Base model | tiiuae/Falcon3-3B-Instruct |
| Decode speed | ~27 tok/s (iPhone 17 Pro, Metal GPU) · ~89 tok/s (Mac M4 Max, LiteRT-LM, greedy) |
Usage
Run with the LiteRT-LM runtime:
1# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
2litert_lm_main \
3 --model_path model.litertlm \
4 --backend gpu \
5 --input_prompt "Explain on-device AI in one sentence."
The .litertlm bundle carries the tokenizer and the prompt template (Falcon3's
native <|user|> / <|assistant|> format, stop token <|endoftext|>), so no
separate tokenizer files are needed.
Run on desktop (LiteRT-LM CLI)
The same
.litertlm bundle runs on macOS / Linux / Windows with the official
LiteRT-LM CLI — including as a
local
OpenAI-compatible API server:
1pip install litert-lm
2litert-lm import --from-huggingface-repo mlboydaisuke/Falcon3-3B-Instruct-LiteRT model.litertlm falcon3-3b-instruct-litert
3litert-lm run falcon3-3b-instruct-litert # interactive chat in the terminal
4litert-lm serve # local OpenAI-compatible API server
Quality — GSM8K parity
Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought asking for #### <n>,
identical prompt and answer-extraction for every row). The 4-bit MLX build is the
known-good 4-bit control:
| Configuration | GSM8K |
|---|
| bf16 (reference) | 75% |
| MLX 4-bit (control) | 76% |
| This model — LiteRT int4 | 77% |
LiteRT int4 is fully at parity — it matches or slightly exceeds both the 4-bit
control and bf16 here (the small spread is sampling noise at n=100). This is a
direct-answering instruct model (no <think> block) and terminates cleanly at
<|endoftext|>.
Conversion
Converted with
litert-torch using a
blockwise int4 recipe (INT4 weights, block size 128, symmetric) with embeddings
kept at INT8, KV cache 2048, and Falcon3's native chat template. Falcon3-3B is a
standard
LlamaForCausalLM architecture, so it rides the existing converter and
runtime with no custom code. Blockwise (not channelwise) int4 is what preserves
reasoning accuracy.
Reproduce (official tools only)
Built with stock litert-torch — no custom code, no graph patches. The only
non-default choice is the int4 recipe: the tool's default named int4 is
channelwise (which degrades small models), so this uses blockwise-128 (the
scheme the official models ship), passed as a recipe file to the standard export:
1from litert_torch.generative.export_hf.export import export
2export(
3 model="tiiuae/Falcon3-3B-Instruct",
4 output_dir="out",
5 quantization_recipe="falcon_int4_block128.json", # included in this repo
6 cache_length=2048,
7 trust_remote_code=True,
8)
falcon_int4_block128.json is included in this repo. (If the export errors with a
missing ai_edge_quantizer/recipes/ directory, create it empty — a packaging gap
in some releases that trips the .json-recipe path.)
License
Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.
