Views
No views yet
medgemma-1.5-4b-it model converted to ExecuTorch format for on-device inference in the Android application.optimum-executorch that includes critical fixes for:[1, 106] for correct generation termination.1# Setup environment
2uv venv --python 3.12
3source .venv/bin/activate
4
5# Clone and setup the custom optimum-executorch repository
6git clone https://github.com/kamalkraj/optimum-executorch.git
7cd optimum-executorch
8git checkout merge-eos-and-max-seq
9
10# Install dependencies (requires torch 2.9.0 and torchao 0.14.1 for correct tracing)
11uv pip install '.[dev]' torch==2.9.0 torchao==0.14.1optimum-cli with XNNPACK recipe, using 8-bit dynamic activation and 4-bit weight quantization (8da4w).1optimum-cli export executorch \
2 --model "google/medgemma-1.5-4b-it" \
3 --task "multimodal-text-to-text" \
4 --recipe "xnnpack" \
5 --device cpu \
6 --use_custom_sdpa \
7 --use_custom_kv_cache \
8 --qlinear 8da4w \
9 --qlinear_group_size 32 \
10 --qlinear_encoder "8da4w,8da8w" \
11 --qlinear_encoder_group_size 32 \
12 --qembedding "8w" \
13 --qembedding_encoder "8w" \
14 --max_seq_len 131072 \
15 --output_dir="medgemma-1.5-4b-it-8da4w-executorch"Memory Usage Note: The above command uses a maximum context length of 128K tokens, requiring around 10-11GB of RAM on-device. To reduce memory usage, you can decrease--max_seq_len(e.g., to4096or8192) before exporting, which will still allow effective inference while fitting within the constraints of lower-end devices.