ibm-granite/granite-embedding-311m-multilingual-r2 converted to LiteRT (.tflite) for on-device inference. A multilingual ModernBERT bi-encoder for retrieval, search and RAG, producing 768-dimensional L2-normalized vectors — fully offline, on CPU.

CLS pooling and L2 normalization are inside the graph: one call in, one finished embedding out.

File	Recipe	Signatures	Size
`granite-embedding-311m-r2_wi8fc.tflite`	int8 dynamic-range (linears + embedding table)	64, 128, 256, 512	336 MB	mobile + desktop
`granite-embedding-311m-r2_fp16.tflite`	fp16 weights, float compute	64, 128, 256, 512	629 MB	desktop (runs on phone, but see below)

Since litert-lm 0.17.0 the same weights are also published as LiteRT-LM EmbeddingEngine bundles (.litertlm) that the runtime loads without any host-side tokenization or pooling code — see the section below.

Both files are verified bit-exact against desktop on an iPhone 17 Pro (cosine 1.000000, max diff 0.0 across five scripts and both signature lengths). Use int8 on device for CPU/XNNPACK: it peaks at 518 MiB for all four signatures — no special memory entitlement needed — where fp16 peaks at 3700 MiB and is up to 6× slower, because XNNPACK expands fp16 weights to fp32 while packing each signature subgraph. That trade-off is XNNPACK's alone: on a Galaxy S26 the int8 file compiles on neither accelerator, and fp16 is the file that runs — 49.37 ms on the Hexagon NPU after an ahead-of-time compile (see Snapdragon NPU (Hexagon) below).

Signatures

Batch-1, right-padded static shapes: input_ids int32 [1, S], attention_mask int32 [1, S] (1 = real token, 0 = pad). Output output_0 float32 [1, 768] — the CLS token, L2-normalized.

Pad the token ids into the smallest signature that fits and set the mask accordingly. The result is independent of which signature you route through: the same text through embed_64 / 128 / 256 / 512 returns bitwise identical vectors, and pad-region token ids cannot influence the output at all.

Embeddings are L2-normalized, so cosine similarity is a dot product. IBM documents Matryoshka truncation on the base model — slice the first 256 dimensions and re-normalize (not independently verified here).

Prompts

This model takes plain text — no prefix. Its config_sentence_transformers.json ships empty query/document prompts, and that is the contract used for every number below. (If you are coming from an E5-style model like nvidia/Nemotron-3-Embed, note the difference: there the query: /passage: prefix is mandatory, here it is not part of the model's contract.)

Footnote: a prefix helps symmetric similarity but hurts retrieval

Out of curiosity we measured an unofficial query: prefix on both sides: en-en STS17 rises from 0.783 to 0.826. But on retrieval it reverses — nDCG@10 0.836 → 0.832 and recall@5 0.900 → 0.860. If your workload is purely symmetric (clustering, dedup, similarity scoring) a constant prefix may be worth testing on your own data; for retrieval, use bare text.

Usage (Python)

python

import numpy as np
from ai_edge_litert.interpreter import Interpreter
from transformers import AutoTokenizer

PAD_ID = 0
tok = AutoTokenizer.from_pretrained("ibm-granite/granite-embedding-311m-multilingual-r2")
it = Interpreter(model_path="granite-embedding-311m-r2_wi8fc.tflite", num_threads=8)

LENS = sorted(int(n.split("_")[1]) for n in it.get_signature_list())
runners = {s: it.get_signature_runner(f"embed_{s}") for s in LENS}

def embed(text):
    ids = tok(text)["input_ids"][:LENS[-1]]
    S = next(s for s in LENS if len(ids) <= s)
    x = np.full((1, S), PAD_ID, np.int32)
    m = np.zeros((1, S), np.int32)
    x[0, :len(ids)] = ids
    m[0, :len(ids)] = 1
    return list(runners[S](input_ids=x, attention_mask=m).values())[0][0]

q = embed("What is the tallest mountain in Japan?")
d = embed("富士山は、静岡県と山梨県にまたがる活火山で、標高3776.12 mで日本最高峰の独立峰である。")
print("cosine:", float(q @ d))   # cross-lingual match

Texts longer than 512 tokens must be chunked (the upstream model accepts 32768, but a static on-device graph at that length is not practical).

LiteRT-LM EmbeddingEngine bundles (`.litertlm`, litert-lm ≥ 0.17.0)

Since litert-lm 0.17.0 the runtime hosts embedding models directly through EmbeddingEngine (Python, C and Kotlin), so the same weights are also published as bundles that the engine loads without any host-side tokenization or pooling code:

file	contents	size
`granite-embedding-311m-r2_wi8fc.litertlm`	int8 embedding table + int8 (dynamic-range) encoder, signatures for 64/128/256/512 tokens	332 MB
`granite-embedding-311m-r2_fp16.litertlm`	int8 embedding table + fp16 encoder	436 MB

The vectors are identical to the .tflite path (cosine 1.000000 on the check set below; CLS pooling and L2 normalization are inside the graph). Keep insert_special_tokens at its default (True): the bundle declares <bos> as the token the CLS position reads, and the engine inserts it — the runtime's tokenizer does not run the tokenizer's own post-processor, so turning the option off silently returns wrong vectors.

Python (pip install litert-lm>=0.17.0):

python

import litert_lm
from litert_lm.embedding_engine import EmbeddingEngine, EmbeddingOptions

engine = EmbeddingEngine("granite-embedding-311m-r2_wi8fc.litertlm", backend=litert_lm.Backend.CPU())
vec = engine.compute_embedding("What is the tallest mountain in Japan?").embedding   # 768 floats, L2-normalized
batch = engine.compute_embedding_batch(["first text", "second text"])                  # list of responses

Inputs longer than 512 tokens are an error by default; pass EmbeddingOptions(input_overflow_strategy=InputOverflowStrategy.TRUNCATE) or CHUNK_AND_AVERAGE to choose. EmbeddingOptions(output_size=256) keeps the first 256 dimensions.

Kotlin (com.google.ai.edge.litertlm:litertlm-android:0.17.0, needs a Kotlin 2.4 project):

kotlin

val engine = EmbeddingEngine(EmbeddingEngineConfig(modelPath = "/data/local/tmp/granite-embedding-311m-r2_wi8fc.litertlm", backend = Backend.CPU()))
engine.initialize()
val vec = engine.computeEmbedding(listOf(InputData.Text("What is the tallest mountain in Japan?"))).embedding
engine.close()

Measured (CPU, single text, 10-text check set of 6–70 tokens, median):

device	runtime	init	per text
Galaxy S26 (SM-S942Q, Android 16)	litertlm-android 0.17.0, Kotlin `EmbeddingEngine`	1.99 s	12–24 ms
Mac (M4 Max)	litert-lm 0.17.0, Python	0.5 s	15 ms (wi8fc) / 19 ms (fp16)

Quality

Three independent checks, each on every variant.

1. The base card's own cross-lingual matrix. IBM publishes an exact 3×3 cosine matrix (EN/DE/JA queries × JA/EN/DE passages). fp32 and fp16 reproduce it to every published digit (max abs 0.0000 vs both the card and the PyTorch reference); int8 is 0.0039 away. All variants rank the correct cross-lingual passage first, 3/3.

2. STS17 semantic similarity, 11 language pairs × 100 pairs, Spearman:

Variant	mean	en-en	ar-ar	es-es	ko-ko	en-de	en-ar	en-tr	es-en	fr-en	it-en	nl-en
fp32	0.7363	0.783	0.748	0.792	0.836	0.666	0.762	0.601	0.726	0.726	0.752	0.709
int8	0.7327	0.779	0.746	0.791	0.837	0.664	0.753	0.596	0.727	0.720	0.744	0.702
fp16	0.7364	0.783	0.748	0.792	0.836	0.667	0.762	0.601	0.726	0.726	0.752	0.709

int8 costs 0.0036 mean, per-language ≤ 0.01. fp16 is indistinguishable from fp32.

3. Retrieval (SciFact-derived, 50 queries over a 600-document corpus): fp32 nDCG@10 0.8400, int8 0.8360, fp16 0.8400, with recall@5 0.9000 and hit@1 0.7600 identical across all three. The corpus is subsampled, so the absolute number is not comparable to published BEIR scores — read the variant deltas.

Speed

CPU/XNNPACK, median of 12 runs, Apple M4 Max at 12 threads:

Variant	embed_64	embed_128	embed_256	embed_512
int8	28.6 ms	35.2 ms	45.5 ms	71.3 ms (5383 tok/s)
fp16	29.8 ms	37.5 ms	53.5 ms	84.5 ms

A static signature computes all S positions regardless of how many are real, so route to the smallest signature that fits. Latency scales gently with length (2.5× from 64→512) because 14 of the 22 layers attend over a 64-wide window rather than the full sequence.

Snapdragon NPU (Hexagon)

granite-embedding-311m-r2_fp16.tflite — the NPU runs it at 49.37 ms. The GPU does not — LiteRtException: Failed to compile model.
granite-embedding-311m-r2_wi8fc.tflite — neither accelerator produced a usable row on the S26. Both ended the same way: LiteRtException: Failed to compile model.

file	backend	compiled	inference (median / min)	load
`granite-embedding-311m-r2_fp16.tflite`	NPU (Hexagon v81)	AOT (SM8850)	49.37 ms / 48.81 ms	745 ms

Measured on a Samsung Galaxy S26 (Snapdragon 8 Elite Gen 5 / SM8850, Hexagon v81, Android 16) with LiteRT CompiledModel 2.2.0, one accelerator per process, 5 warm-up runs then N=50 timed runs, median reported. The run held thermal status NONE throughout. Headroom 0.76, where 1.0 is the throttling threshold.

The NPU row marked AOT ran an artifact compiled ahead of time for SM8850 (ai-edge-litert 2.2.0 + QAIRT 2.47.0), not the published file. That artifact is not distributed here; the compile is one command in the NPU guide.

GPU wiring: GPU guide.

Conversion

Encoder lane — a direct multi-signature litert_torch trace of the HF model, not an LLM export. Two things worth knowing if you reproduce it:

ModernBERT alternates local and global attention (22 layers, every 3rd global, 64-wide half-window) with a separate rope frequency set per layer type. ModernBertModel.forward accepts attention_mask as a dict of pre-built masks, so both are built explicitly rather than through transformers' mask machinery.
A sliding window plus right padding creates fully-masked query rows — once a pad position is further than the window from every real token, softmax runs over all -inf. Eager PyTorch absorbs it; the exported graph emits NaN (and int8 hides it). The masks here always allow self-attention, which removes the NaN and is provably output-neutral.

Script and full notes: hf-to-litertlm.

License

Apache-2.0, inherited from the base model; LICENSE is included.

Modification notice: these files are converted, not original. The weights were exported to LiteRT and quantized (int8 dynamic-range / fp16); CLS pooling and L2 normalization were folded into the graph. No fine-tuning or weight modification beyond quantization was performed.

litert-community/granite-embedding-311m-multilingual-r2

granite-embedding-311m-multilingual-r2 — LiteRT