You can deploy this model and start earning money today!

mlboydaisuke/
Qwen3.5-2B-ExecuTorch

Name: Qwen3.5-2B-ExecuTorch
Availability: InStock

Views

No views yet

Qwen3.5-2B — ExecuTorch XNNPACK 8da4w + 8-bit embedding

qwen3_5_2b_xnnpack_8da4w_e8.pte (1483.7 MB)

Source: Qwen/Qwen3.5-2B
License: Apache-2.0
Quantization: 8da4w linear + 8-bit embedding (embedding_quantize: "8,0"). The 248320-row embedding is the reason: left in fp32 it is a gigabyte on its own, and at this vocabulary size that is most of what a small model weighs.
Export: executorch 1.4.0 export_llm, static shape (seq_len=1), max_seq_length 2048, XNNPACK extended_ops
Config: llm_params/qwen3_5_2b_xnnpack_8da4w_e8.yaml

Verification (Mac arm64, 2026-08-21)

llm_params/gen_static.py, token-by-token prefill then greedy decode, a fresh process per prompt so no answer is read through the previous one's cache:

prompt	answer	decode
capital of France?	"The capital of France is Paris."	33.25 tok/s
日本の首都は?	「日本の首都は東京都です。」	33.25 tok/s
17 times 4?	opens a `<think>` block and works through it	33.25 tok/s

All three rates are from one run with nothing else on the machine, which matters more than it sounds: the same file measured 16.6 tok/s while an export was running alongside.

Chat template: ChatML, bos 248045, eos [248046, 248044].

Not measured on a phone. The 0.8B was measured at 10.5 tok/s on an iPhone 17 Pro, but on its earlier build, before use_sdpa_with_kv_cache was turned on.

The setting that doubles it

use_sdpa_with_kv_cache: True. Upstream's own qwen3_5 config has it off with no reason given, while the equally hybrid lfm2 config has it on. Measured on the 2B in one run: 8.20 tok/s off, 16.64 on.

That it is only a kernel swap and not a different model was checked three ways: three of four prompts came out token-identical; the first-step logits agree at corr 0.995 with the same top-1; and 338 tokens deep they agree at corr 0.990, same top-1 and top-2. The fourth prompt differs where a thinking model decides how long to think, which is a near-tie either way.

Running it

bash

python llm_params/gen_static.py \
  --pte qwen3_5_2b_xnnpack_8da4w_e8.pte \
  --tokenizer tokenizer.json \
  --prompt $'<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n' \
  --eos_ids "[248046,248044]"

The 8-bit embedding needs from executorch.kernels import quantized before the program is loaded. Without it the method will not even load — kernel 'quantized_decomposed::embedding_byte.dtype_out' not found — which reads like a broken export rather than a runtime missing its kernels.

(conversion scripts: executorch-models · iOS sample: executorch-samples)

mlboydaisuke/Qwen3.5-2B-ExecuTorch

Qwen3.5-2B — ExecuTorch XNNPACK 8da4w + 8-bit embedding

Verification (Mac arm64, 2026-08-21)

The setting that doubles it

Running it

mlboydaisuke/
Qwen3.5-2B-ExecuTorch