Views
No views yet
Package.swift:1.package(url: "https://github.com/john-rocky/CoreML-LLM", branch: "main"),
2
3// In your target:
4.product(name: "CoreMLLLM", package: "CoreML-LLM"),1import CoreMLLLM
2
3// First call pulls the bundle from this repo to Documents/Models/.
4// Subsequent calls reuse the on-disk copy.
5let llm = try await CoreMLLLM.load(repo: "mlboydaisuke/qwen3.5-2B-CoreML")
6
7let stream = try await llm.generate(
8 [CoreMLLLM.Message(role: .user, content: "Hello!")],
9 maxTokens: 256
10)
11for await chunk in stream {
12 print(chunk, terminator: "")
13}[CoreMLLLM.Message] array, append the
user/assistant turns, and pass the whole history to
generate(_:) again. Call llm.reset() to start a new
conversation (clears the KV cache).Qwen/Qwen3.5-2B, split into 4 INT8 chunks + a raw fp16 embedding sidecar so every chunk fits the iPhone ANE single-mlprogram compile envelope.phys_footprint, 0 GB sustained Metal heap, ~91 % ANE op placement across all 4 body chunks. First-load ANE compile ≈ 15 min across chunks (cached after).qwen3_5_2b_decode_chunks/
├── chunk_a.mlpackage # 340 MB — embed + layers 0-5 + their states
├── chunk_b.mlpackage # 340 MB — layers 6-11 + states
├── chunk_c.mlpackage # 340 MB — layers 12-17 + states
├── chunk_d.mlpackage # 850 MB — layers 18-23 + final_norm + lm_head
└── embed_weight.bin # 1.02 GB — raw fp16 embed table (248320 × 2048)mmaps the raw fp16 file so the 1 GB embed table stays in clean virtual pages and only the rows actually touched per prompt page in. Loading the embed as a Core ML weight would dequantize the entire table into the CPU heap and add ~1 GB to phys_footprint.model_config.json — Core ML serializes input/output shapes into each .mlpackage directly. coremltools loads it without external config.1from transformers import AutoTokenizer
2tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-2B")1import coremltools as ct
2import numpy as np
3from huggingface_hub import snapshot_download
4from transformers import AutoTokenizer
5
6local = snapshot_download("mlboydaisuke/qwen3.5-2B-CoreML")
7root = f"{local}/qwen3_5_2b_decode_chunks"
8
9chunks = [
10 ct.models.MLModel(f"{root}/chunk_{x}.mlpackage")
11 for x in ("a", "b", "c", "d")
12]
13embed = np.memmap(f"{root}/embed_weight.bin",
14 dtype=np.float16, mode="r",
15 shape=(248320, 2048))
16tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-2B")embed[token_id] → hidden (1, 1, 2048) fp16hidden + scalar inputs (position, cos, sin) + state slice to chunk_a.predict(...), take its hidden_out and updated states.chunk_b, chunk_c, chunk_d.chunk_d emits logits (1, 1, 248320) fp16; argmax (or sample) it and feed back as input_token for the next step.new_state_* outputs to the next call's state_* inputs.conversion/qwen35_2b_chunks_parity.py.Qwen35Generator.swift handles the chunk chaining + embed mmap. Tap Qwen3.5 2B (ANE) in the model picker.[L L L F] × 6.| linear_attention | full_attention | |
|---|---|---|
| count | 18 | 6 |
| state A | (1, 6144, 4) | (1, 2, 2048, 256) |
| state B | (1, 16, 128, 128) | (1, 2, 2048, 256) |
1python conversion/build_qwen35_2b_decode_chunks.py \
2 --out-dir ./output \
3 --max-seq 2048 --nbits 8