Tashkeel-350M-v2 — LiteRT-LM (on-device Arabic diacritization)
On-device conversion of
Etherll/Tashkeel-350M-v2 —
an Arabic diacritization (tashkeel) fine-tune of
ibm-granite/granite-4.0-h-350m trained on
Misraj/Sadeed_Tashkeela — to
.litertlm
bundles for the
LiteRT-LM runtime. All credit for
the model itself goes to its author; this repo only packages it for phones and desktops.
Requires litert-lm ≥ 0.15 to run (the Mamba2 hybrid state binds through the
ExecutorMetadata section). CPU is the backend to use on phones. The int8 file does run on
mobile GPUs — on a Galaxy S26 (Adreno) and a Pixel 8a (Mali) it delegates fully to LiteRT GPU and
diacritizes the probe correctly (litert-lm 0.16.0, 2026-09-05) — but on both phones the CPU decodes
faster (S26: 69 vs 46 tok/s; Pixel 8a: 48 vs 18) at a third to a half of the GPU's peak memory and
without its 22–25 s engine init. An earlier version of this card said the selective-scan ops exceed
the GPU delegate's rank limit; that describes the base model's CPU-lineage exports and did not hold
for this bundle when it was gated. The fp16 file was measured on the CPU only.
| file | quant | size | task-gate parity vs HF fp32 |
|---|
Tashkeel-350M-v2_fp16.litertlm | fp16 weights (convs/SSM fp32) | 769 MB | 10/10 byte-identical |
Tashkeel-350M-v2_int8.litertlm | dynamic int8 (linears + embedding) | 481 MB | 8/10 (see note) |
Which file: fp16 is the exact-parity build — greedy output byte-identical to the HF fp32
reference on all 10 diacritization probes. int8 is 38% of the size; its two misses are
single-diacritic greedy flips (both probes are byte-exact on the float parent, i.e.
quantization cost, not conversion error — the same 350M-scale int8 sensitivity the base
model's card documents). On phones the CPU runtime unpacks fp16 to fp32 in RAM, so int8 is
the practical mobile choice; use fp16 where RAM allows.
Conversion & verification
Converted with
hf-to-litertlm
(family recipe
granite_work/convert_granite4h.py, 2026-08-25):
- The checkpoint's chat template (byte-equal to the granite base's, 6418/6418) is embedded
and applied at runtime.
- The metadata start token is dropped — granite's template has no leading BOS, and at 350M
scale a prepended
<|end_of_text|> flips correct diacritization into garbage (measured on
this checkpoint).
- Task gate: the model card's worked example plus nine undiacritized MSA probes, bundle vs
HF fp32 greedy on identical rendered strings (
granite_work/gate_tashkeel.py).
Performance (Apple M4 Max, litert-lm 0.16.0, -p 256 -d 256 --runs 3 --cache no, CPU)
| file | prefill tok/s | decode tok/s | TTFT |
|---|
| fp16 | 661.6 | 58.2 | 0.40 s |
| int8 | 761.3 | 97.2 | 0.35 s |
Performance (Android, litert-lm 0.16.0 litert_lm_advanced_main, 205-token prompt, --benchmark, 2 cold runs per row)
| phone | file | backend | prefill tok/s | decode tok/s | peak RSS |
|---|
| Galaxy S26 (SM8850, Adreno) | int8 | CPU | 192–212 | 67.2–69.0 | 1.1 GB |
| Galaxy S26 | int8 | GPU (LITERT_CL, full delegation) | 753 | 46.3 | 2.3 GB |
| Galaxy S26 | fp16 | CPU | 66–77 | 23.2–23.7 | 5.1 GB |
| Pixel 8a (Tensor G3, Mali) | int8 | CPU | 112 | 47.3–49.1 | 1.1 GB |
| Pixel 8a | int8 | GPU (LITERT_CL, full delegation) | 251–263 | 18.3 | 3.7 GB |
Each GPU/CPU pair is the same handset, binary and prompt, taken 2026-09-05 with the phone cooled
between runs; the gate prompt was the trained form above and both backends returned the correct
diacritization. The same rows, with conditions, are in this repo's litertlm_manifest.json.
Usage
The model diacritizes the text you send as a user message in its trained prompt form:
1pip install litert-lm
2litert-lm run Tashkeel-350M-v2_fp16.litertlm --backend cpu \
3 --prompt 'قم بتشكيل هذا النص :
4السلام عليكم'
Output: السَّلَامُ عَلَيْكُمْ
License
apache-2.0, inherited from the source model and its granite base.
