Model Card for QwenPaw-Flash-9B MTP (Core ML, Apple Neural Engine)
A compiled Core ML build of QwenPaw-Flash-9B, packaged to run entirely on device
on the Apple Neural Engine (ANE). It targets macOS (Apple silicon), needs no
network at inference time, and streams tokens locally with the weights quantized
to 6 bits.
This repository holds the compiled model set only, not training or original
weights. It is a self-contained artifact that a companion on-device runtime
loads to drive generation. Nothing here calls out to a server.
Model Details
Model Description
QwenPaw-Flash-9B shares the Qwen3.5 hybrid design: of its 32 layers, 24 are
Gated DeltaNet linear-attention (state-space) blocks and 8 are full
softmax-attention blocks, interleaved "three linear, one attention". The linear
blocks carry a small fixed-size recurrent state instead of a growing key/value
cache, so memory stays flat as context grows, while the periodic attention
blocks preserve exact long-range recall. It is the largest of the 0.8B, 2B, and
4B builds, with the same vocabulary and a deeper, wider network. Unlike the
smaller builds it does not tie its input and output embeddings, so it ships a
separate output-projection matrix. This build converts the upstream model to
Core ML and palettizes the weights to 6 bits for on-device use.
- Developed by: AgentScope / QwenPaw authors (base model); Core ML
conversion by leok7v
- Model type: Hybrid Gated DeltaNet + attention causal language model,
converted to Core ML (mlprogram)
- Language(s): English and the languages of the base model
- License: Apache 2.0 (inherited from the base model)
- Converted from model:
agentscope-ai/QwenPaw-Flash-9B
Model Sources
Uses
Direct Use
On-device assistants and chat, drafting, and summarization, in offline or
privacy-sensitive settings where prompts must not leave the device. At 9B
parameters it is the most capable of these builds but also the heaviest: it is
intended for Apple silicon Macs with ample memory, not phones. The set is
driven by an on-device runtime that runs prompt ingestion (prefill) as a batched
pass on the Neural Engine and generates tokens one at a time, carrying the
linear-attention state forward and reading the attention layers from a paged
key/value cache.
Out-of-Scope Use
Not for tasks requiring verified factual accuracy, and not a basis for legal,
medical, financial, or safety decisions. Not intended for high-stakes automated
decisions without human review.
Bias, Risks, and Limitations
A language model predicts plausible text; it does not verify facts and has no
understanding of truth. Output can be confident and still be wrong, incomplete,
or biased, and 6-bit quantization can widen small errors relative to the base
model. The base model's biases and limitations carry over; see the base model
card.
Recommendations
Treat every response critically and check important output against a trusted
source. This model defaults to emitting a "thinking" prefix; preserve that
per-model default when integrating a chat template.
How to Get Started with the Model
The repository contains a set of compiled .mlmodelc programs plus the
tokenizer. It is designed to be loaded by an on-device runtime that implements
paged key/value attention on the host between the Neural Engine trunk programs;
it is not a single .mlpackage you call with one prediction. The included
chat_template.jinja documents the chat format (roles, the "thinking" prefix,
tool-call rendering).
Technical Specifications
Model Architecture and Objective
| Property | Value |
|---|
| Objective | Autoregressive next-token prediction |
| Hidden size | 4096 |
| Layers | 32 (24 Gated DeltaNet + 8 attention) |
| Attention heads | 16 query / 4 key-value (grouped-query) |
| Vocabulary | 248320 |
| Context length | up to 262144 tokens |
| Tied embeddings | no (separate output projection) |
| Weight format | 6-bit palettized (per-grouped-channel k-means) |
| Activations | fp16 |
Files
mf0of9.mlmodelc ... mf8of9.mlmodelc - the transformer trunk, cut into 9
programs around the attention layers; each is a multifunction bundle whose
"prefill" (batched), "decode" (single-token) and "verify" (speculative)
graphs share ONE quantized weight blob, so the weights are resident once
and speculative decode adds no second copy.head.mlmodelc - the vocabulary matrices as one bundle of four functions: the
token-embedding matrix (shared by the single-token and batched embed lookups,
one quantized copy) and a separate output-projection (lm_head) matrix, since
9B does not tie them.mtp_front.mlmodelc, mtp_back.mlmodelc - the multi-token-prediction
drafter (174 MB), a single attention block that proposes the next tokens
for the trunk to verify. Baked from the full-precision mtp.* tensors
that ship in the Qwen/Qwen3.5-9B base.tile.mlmodelc, tile_prefill.mlmodelc, tile_carry.mlmodelc - weightless
flash-attention kernels for the paged key/value cache.tokenizer.json, tokenizer_config.json, chat_template.jinja - the
byte-level BPE tokenizer and chat template.
Speculative decode
The trunk carries a third "verify" graph, so multi-token prediction runs without
a second copy of the weights: the drafter proposes 2 tokens, one verify pass
scores them, the matching prefix is accepted, and the recurrent state is rolled
back on the host. Measured on an Apple M3, same set, back to back:
| tokens/s |
|---|
| Plain decode | 9.9 |
| Speculative | 14.2 (+43%) |
Accepted 2.24 tokens per cycle. Verify shares the trunk's own quantized
weights, so its argmax is the trunk's.
The draft count is fixed at 2 in the runtime; the verify graph is 8 positions
wide, which caps it at 7. See _mtp_doc in generation_config.json.
Compute Infrastructure
- Hardware: Apple silicon Mac with a Neural Engine.
- Software: macOS 15+. The first launch compiles the programs for the Neural
Engine once; the OS caches that build, so later launches are fast.
Citation
Please cite the upstream QwenPaw-Flash-9B model. See the base model card at
agentscope-ai/QwenPaw-Flash-9B
for the canonical reference and license.
Model Card Contact
leok7v on Hugging Face.
