You can deploy this model and start earning money today!

metricspace/
gemma4-E2B-it-litert-64k-mtp

Name: gemma4-E2B-it-litert-64k-mtp
Availability: InStock

litert-lm

gemma4

on-device

google/gemma-4-E2B-it

finetune

apache-2.0

Views

No views yet

metricspace/gemma4-E2B-it-litert-64k-mtp

Community-identical Gemma 4 E2B .litertlm bundle, binary-patched to support a 64k context window on both CPU and GPU. Correct text output verified against the community 32k bundle.

How it was built

Starting from litert-community/gemma-4-E2B-it-litert-lm (32k bundle, magic cache dim 32003), tf_lite_prefill_decode and tf_lite_mtp_drafter are binary-patched 32003 → 65003 (prime, avoids an XNNPack slow path that triggers on composite dims like 65536 or 64006). The patch walks four layers of the TFLite flatbuffer:

Tensor shapes — every shape[i] == 32003 rewritten to 65003 (1 354 entries across decode / prefill / verify, 52 more in the drafter).
External buffers backing arange(N) / arange(N)+512 int32 index tables are regenerated for the new length, appended past the end of the file, and Buffer.offset / Buffer.size fields rewritten.
Small RESHAPE / GATHER int32 shape-parameter buffers (≤ 256 B) that contain the baked dim are rewritten in place.
StableHLOCompositeOptions.CompositeAttributes FlexBuffer blobs of every odml.cache_update composite are walked and their kv_cache_batch_size / key_size / value_size / scale_* attributes rewritten from 32003 → 65003. Missing step 4 gave graphs whose tensor shapes were right but whose per-layer composite attributes still referenced 32003 — the runtime then corrupts decode output to " agregado<pad><pad>...". All earlier -patched revisions had this bug; this release (v2) is the first with correct text output.

Everything else — weights, tokenizer, embedder, per-layer embedder, audio / vision towers, metadata — is byte-identical to the community source bundle.

Performance (Apple M5, LiteRT-LM built from `main`)

Median of 3 runs, 1024-token prefill / 256-token decode, synchronous (--async=false), --conv_type=float, --max_num_tokens=2048 (typical short-chat cache — see note below).

Backend	Prefill tok/s	Decode tok/s	TTFT	vs CPU
CPU	~50	~17.6	~20 s	1×
GPU	~1920	~69	~0.55 s	3.9×

Decode cost scales roughly linearly with --max_num_tokens. At --max_num_tokens=65000 (full bundle capacity), decode lands near 33 tok/s with TTFT ≈ 2.6 s for a 1024-token prompt.

Squeezing more out of the WebGPU delegate (optional, macOS)

The LiteRT-LM v0.10.1 prebuilt of libLiteRtTopKWebGpuSampler.dylib on macOS exports only 3 of the 7 C symbols the runtime dynamically loads: _UpdateConfig, _CanHandleInput, _HandlesInput, and _SetInputTensorsAndInferenceFunc are compiled in but marked non-external (was a private external) because of the macos_exported_symbols.lds export list. The runtime then prints GPU sampler unavailable. Falling back to CPU sampling. and every decode step transfers the full logits tensor GPU → CPU for argmax.

You can promote those 4 symbols in the already-shipped dylib by rewriting its LC_DYLD_EXPORTS_TRIE:

text

# ctox repo helper (scripts/models/patch_webgpu_sampler_exports.py) —
# appends a new 7-symbol exports trie at EOF, extends __LINKEDIT by 200 B,
# and ad-hoc re-signs the dylib. No source rebuild required.
python scripts/models/patch_webgpu_sampler_exports.py \
    <LITERT_LM_PREBUILT_DIR>/libLiteRtTopKWebGpuSampler.dylib

On M5 / Gemma 4 E2B this patch makes the WebGPU sampler load cleanly but gave no measurable decode uplift in my runs — the decode bottleneck is WebGPU-delegate kernel-launch overhead, not the sampler. It's still worth applying for correctness (avoids one GPU → CPU transfer per step) and in anticipation of upstream sampler improvements.

Known upstream ceilings (not bundle-level)

Speculative decoding (--enable_speculative_decoding=true) crashes the WebGPU path with ALREADY_EXISTS: wgpu::Instance already set when drafter and main model both try to create a WebGPU instance. On CPU it runs but produces 0 % acceptance rate — the drafter's first token never matches the verifier, so MTP is a strict slowdown on this runtime version. Keep it off until upstream ships a fix.
--gpu_external_tensor_mode=true fails on Gemma 4 bundles with INVALID_ARGUMENT: bytes-per-token exceeds output tensor size — the external-tensor path has a hard-coded token count that doesn't match Gemma 4's PLE layout. Can't enable.
Embedder / per-layer embedder are hardcoded to CPU in runtime/components/embedding_lookup/embedding_lookup_text.cc on non-Android builds. Forcing them to GPU regresses throughput by ≈ 9 × (E2B drops 68 → 9 tok/s) because the WebGPU delegate has no embedding-lookup kernel and silently falls back with bad scheduling. Leaving them CPU-pinned is correct.
No native Metal backend. The gpu_artisan backend referenced in --helpfull is internal-only. WebGPU → Dawn → Metal is the only GPU path on macOS today.

Run

bash

litert-lm run --from-huggingface-repo metricspace/gemma4-E2B-it-litert-64k-mtp \
    model.litertlm -b gpu --prompt "Explain transformers."

For the honest benchmark above use litert_lm_advanced_main:

bash

litert_lm_advanced_main \
    --model_path=model.litertlm \
    --backend=gpu --async=false \
    --max_num_tokens=2048 \
    --conv_type=float \
    --benchmark --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256

Choosing `--max_num_tokens`

This flag sets the magic-number rewrite target — the actual KV-cache size the runtime allocates and every attention step reads over. The bundle's dim 65003 is only a cap, not the working size. Decode cost is roughly linear in this value, so match it to your real use case:

Short chats (≤ 2k tokens): --max_num_tokens=2048 → ~69 tok/s decode.
Medium documents (≤ 8k): --max_num_tokens=8192 → ~60 tok/s decode.
Long context (up to 64k): --max_num_tokens=65000 → ~33 tok/s decode, 2.6 s TTFT for a 1024-token prompt.

Build recipe

Patched with scripts/models/patch_tflite_cache_length.py (composite- attribute walker included, v2) and repacked with scripts/models/repack_litertlm.py from the ctox repository. The optional sampler-dylib trie patcher lives in scripts/models/patch_webgpu_sampler_exports.py.

metricspace/gemma4-E2B-it-litert-64k-mtp

metricspace/gemma4-E2B-it-litert-64k-mtp

How it was built

Performance (Apple M5, LiteRT-LM built from main)

Squeezing more out of the WebGPU delegate (optional, macOS)

Known upstream ceilings (not bundle-level)

Run

Choosing --max_num_tokens

Build recipe

metricspace/
gemma4-E2B-it-litert-64k-mtp

Performance (Apple M5, LiteRT-LM built from `main`)

Choosing `--max_num_tokens`