Model Card for Qwen3.8-27B (Core ML, Apple Neural Engine, 4-bit)
A compiled Core ML build of Qwen3.8-27B, packaged to run entirely on device on
the Apple Neural Engine (ANE). It targets macOS (Apple silicon), needs no
network at inference time, and streams tokens locally with the trunk weights
palettized to 4 bits.
This repository holds the compiled model set only, not training or original
weights. It is a self-contained artifact that a companion on-device runtime
loads to drive generation. Nothing here calls out to a server.
This is an experimental build, not a shipped model. It exists as an online
base for further work, because moving 14 GB between machines by hand is
expensive. No released application fetches it, and nothing here is promised to
stay put: the set may be re-emitted, re-quantized or replaced without notice.
Which of the two builds to take
This set is the sibling of
Qwen3.8-27B-coreml-q6.
They are the same conversion at two LUT widths, and the choice is about size:
this one is about 14 GB on disk against the 6-bit set's 20 GB. Take the 6-bit
build where the disk and memory are available, and this one where they are not.
Model Details
Model Description
Qwen3.8-27B is a dense (not mixture-of-experts) hybrid: of its 64 layers, 48
are Gated DeltaNet linear-attention blocks and 16 are full softmax-attention
blocks, interleaved three-linear-one-attention. The linear blocks carry a small
fixed-size recurrent state instead of a growing key/value cache, so memory
stays flat as context grows, while the periodic attention blocks preserve exact
long-range recall. It is multimodal: a 27-layer vision tower projects images
into the language model's embedding space. Input and output embeddings are not
tied, so a separate output-projection matrix ships.
The trunk, the head, the speculative-verify function and the drafter are all
4-bit. The vision tower is fp16.
Everything in this repository derives from the origin checkpoint's own tensors,
including the vision tower and the drafter.
- Developed by: Qwen, Alibaba (base model); Core ML conversion by leok7v
- Model type: Hybrid Gated DeltaNet + attention causal language model,
multimodal (image and text in, text out)
- Language(s): English and the languages of the base model
- License: Apache 2.0 (inherited from the base model)
- Converted from model:
Qwen/Qwen3.8-27B
Model Sources
Uses
Direct Use
On-device chat, summarization, drafting, question answering over supplied text,
and image understanding, on an Apple silicon Mac with no network. The set also
carries a multi-token-prediction drafter for self-speculative decoding, which a
runtime may use to raise decode throughput without changing what is generated.
Out-of-Scope Use
Anything the base model's licence and card exclude. This build adds no
alignment, no safety tuning and no filtering of its own; it is the upstream
model in a different numeric format. It is not a general Core ML package that a
single prediction call drives, and it is not intended for server deployment.
Bias, Risks, and Limitations
The base model's biases and failure modes carry over unchanged.
Quantization to 4 bits is lossy, and more so than to 6. Outputs will not match
the upstream fp16 model token for token, and the gap is largest where the model
was already uncertain. Anyone depending on this build should evaluate it on
their own task rather than assume the 6-bit set's behaviour.
Recommendations
Treat generations as drafts to verify. The sampling parameters in
generation_config.json are the base model card's own recommendations.
How to Get Started with the Model
The repository contains a set of compiled .mlmodelc programs plus the
tokenizer. It is designed to be loaded by an on-device runtime that implements
paged key/value attention on the host between the Neural Engine trunk programs;
it is not a single .mlpackage you call with one prediction. The included
chat_template.jinja documents the chat format (roles, the thinking block,
tool-call rendering).
Two behaviours of that template are worth knowing before wiring a runtime to
it. preserve_thinking defaults to true, so past assistant turns re-render
with their reasoning block rather than stripped; a runtime that does not retain
reasoning should pass preserve_thinking=false. And reasoning_effort
defaults to xhigh, which injects an instruction to reason thoroughly; low
and medium are supported and cost fewer tokens per turn.
Technical Specifications
Model Architecture and Objective
| Property | Value |
|---|
| Objective | Autoregressive next-token prediction |
| Hidden size | 5120 |
| Layers | 64 (48 Gated DeltaNet + 16 attention) |
| Attention heads | 24 query / 4 key-value (grouped-query) |
| Head dimension | 256 (partial rotary, 64) |
| FFN size | 17408 |
| Vocabulary | 248320 |
| Context length | up to 262144 tokens |
| Tied embeddings | no (separate output projection) |
| Vision tower | 27 layers, width 1152, projected to 5120 |
| Weight format | 4-bit palettized (per-grouped-channel k-means, group 16) |
| Activations | fp16 |
Files
mf0of17.mlmodelc ... mf16of17.mlmodelc - the transformer trunk, cut into
17 programs around the attention layers, 600 to 740 MB each. Each is a
multifunction bundle whose "prefill" (batched), "decode" (single-token) and
"verify" (speculative batch) graphs share one quantized weight blob, so the
weights are resident once.head.mlmodelc - the vocabulary matrices as one bundle of four functions:
the token-embedding matrix (shared by the single-token and batched embed
lookups, one quantized copy), and the output projection in single-row and
batched forms, since this model does not tie its embeddings.vision.mlmodelc - the vision tower and its merger, fp16, unchanged from the
6-bit set.mtp_front.mlmodelc, mtp_back.mlmodelc - the multi-token-prediction
drafter for self-speculative decoding.tile.mlmodelc, tile_prefill.mlmodelc, tile_carry.mlmodelc - weightless
flash-attention kernels for the paged key/value cache.tokenizer.json, tokenizer_config.json, vocab.json, merges.txt,
chat_template.jinja - the byte-level BPE tokenizer and chat template.config.json, generation_config.json, preprocessor_config.json,
video_preprocessor_config.json - geometry, the sampling parameters
recommended by the base model card, and the origin's own image and video
preprocessing settings.
Compute Infrastructure
- Hardware: Apple silicon Mac with a Neural Engine. The set is about 14 GB
on disk and the whole trunk is read per token, so decode is bandwidth bound.
- Software: macOS 15+. The first launch compiles the programs for the
Neural Engine, which takes a while for a set this size and longer on a
fanless machine; the OS caches that build, so later launches are fast.
Citation
Please cite the upstream Qwen3.8-27B model. See the base model card at
Qwen/Qwen3.8-27B for the canonical
reference and license.
Model Card Contact
leok7v on Hugging Face.
