INT4-quantized, ExecuTorch-lowered .pte of google/gemma-4-e2b-it, packaged for Raspberry Pi 5 (Cortex-A76, 8 GB) deployment via the ExecuTorch 1.2.0 Python runtime with the XNNPACK backend.

This artifact is the deployable output of the full export → quantize → lower → runtime pipeline documented at:

Source code & documentation: https://github.com/bamb00boy/Gemma4_executorch_deployment

Identical 14-token prompt + 9-token decode for "The capital of France is", bit-exact output across all rows.

Pi 5 decode rate varies ±15% across sessions (small per-prompt sample, thermal state, default schedutil cpufreq governor). For stable benchmarking, pin the governor to performance and let the SoC return to <55°C between runs.

Output quality: bit-exact 9/9 token match against the FP32 reference on the canonical prompt.

The Pi 5 decode is approximately 7.7× slower than llama.cpp on identical hardware. The shipped build uses XnnpackPartitioner(per_op_mode=True) to work around an ARM XNNPACK rejection bug in ExecuTorch 1.2.0 — initially believed to be the entire cause of the gap. Three controlled follow-up experiments (PT2E quantization, config_precisions=DYNAMIC_QUANT, ExecuTorch nightly 1.4.0.dev with the default fused partitioner) confirm that the partitioner mode is not the bottleneck — even with 508 fused subgraphs on ARM nightly the decode rate does not improve. The ARM XNNPACK rejection itself IS fixed in nightly; stable 1.3+ should ship the fix. The remaining gap likely lives in the XNNPACK kernel format (vs llama.cpp's GGUF Q4_K_M hand-tuned ARM kernels), the KleidiAI link status of the executorch wheel's XNNPACK build, or the external KV-cache materialization pattern. Full diagnosis with the measured three-way Pi benchmark at KNOWN_ISSUES.md #1 in the source repo. If maximum Pi 5 decode throughput is the priority, llama.cpp is the appropriate tool today.

The Pi setup guide (OS install, performance tuning, SSH) lives in docs/pi5_setup.md in the source repo.

The .pte runs on any host with ExecuTorch 1.2.0 + XNNPACK. It has been validated on:

x86_64 Linux is expected to work (XNNPACK supports it) but is untested by this project.

Disk size: 5.14 GB. Runtime cache footprint: 18.9 MB across 15 layers (12 sliding-window @ head_dim=256, 3 full-attention @ head_dim=512).

The weights in this file are derived from google/gemma-4-e2b-it and are licensed under Apache License 2.0 by Google DeepMind. Use is subject to:

This is a derivative work: INT4 weight-only quantization of nn.Linear weights and INT8 per-row quantization of embed_tokens_per_layer, followed by ExecuTorch program lowering with the XNNPACK backend. No additional fine-tuning has been performed.

The packaging code (pi_runner.py, gemma4_terminal_chat.py, and the export/quantize/lower pipeline) is released under MIT — see the source GitHub repo.

Original Gemma 4 weights © Google DeepMind, released under Apache 2.0.
INT4 quantization + ExecuTorch lowering: derivative work by the
Gemma4_executorch_deployment contributors (https://github.com/bamb00boy/Gemma4_executorch_deployment).

For the upstream Gemma 4 model, see google/gemma-4-e2b-it.

If this artifact saved you time or you'd like to help fund continued work on on-device LLM packaging, you can support the project at ko-fi.com/bamb00boy. All contributions are appreciated and go directly toward more experiments, benchmarks, and open releases.