Views
No views yet
Package.swift:1.package(url: "https://github.com/john-rocky/CoreML-LLM", branch: "main"),
2
3// In your target:
4.product(name: "CoreMLLLM", package: "CoreML-LLM"),1import CoreMLLLM
2
3// First call pulls the bundle from this repo to Documents/Models/.
4let llm = try await CoreMLLLM.load(repo: "mlboydaisuke/gemma-4-E4B-multimodal-coreml")
5
6// Text-only
7let stream = try await llm.generate(
8 [CoreMLLLM.Message(role: .user, content: "Hello!")],
9 maxTokens: 256
10)
11for await chunk in stream { print(chunk, terminator: "") }
12
13// Image + text
14let image: CGImage = // ... your image
15let stream2 = try await llm.generate(
16 [CoreMLLLM.Message(role: .user, content: "Describe this picture.")],
17 image: image, maxTokens: 256)
18
19// Audio + text (16 kHz mono PCM Float)
20let pcm: [Float] = // ... your audio samples
21let stream3 = try await llm.generate(
22 [CoreMLLLM.Message(role: .user, content: "What language is this?")],
23 audio: pcm, maxTokens: 256)LLM_VISION_FORCE_ANE=1 to route the vision encoder through the Apple Neural Engine (built ANE-targeted, 256 tokens per image at the LM hidden dim).google/gemma-4-E4B-it with vision (still image), video, and audio (Conformer) encoders. Sliding-window-attention chunks targeting Apple Neural Engine; vision encoder is ANE-targeted; audio runs on GPU + a small Swift/Accelerate projection sidecar.john-rocky/CoreML-LLM; see docs/E4B_MULTIMODAL_BUILD.md for the full reproduction guide and scripts/assemble_gemma4_e4b_multimodal.sh for the assembly script.# Decode chunks (3-chunk Topology II — auto-detected by ChunkedEngine)
chunk1.mlmodelc/ # L0-11 — own KV
chunk2_3way.mlmodelc/ # L12-32 — merged 21 layers (own + KV-shared internal)
chunk3_3way.mlmodelc/ # L33-41 + lm_head + argmax
# Prefill chunks (legacy 4-chunk with prefill_b8 multifunction inside)
chunk2.mlmodelc/ # L12-22 prefill (own KV writes via recurrent shift)
chunk3.mlmodelc/ # L23-32 prefill (KV-shared)
chunk4.mlmodelc/ # L33-41 prefill + lm_head
# Vision encoder (ANE-targeted)
vision.ane.mlmodelc/ # SigLIP, output [1, 256, 2560]
# Audio encoder + Swift projection sidecars
audio.mlmodelc/ # Conformer, output [1, 50, 1024]
audio_config.json
mel_filterbank.bin
output_proj_weight.npy # 1024 -> 1536 (audio_soft_token_size)
output_proj_bias.npy
embed_proj_weight.npy # 1536 -> 2560 (LM hidden) — E4B-specific shape
# Token / per-layer embeddings (mmap'd, dequantised on demand by Swift)
embed_tokens_q8.bin 640 MB — INT8 token embeddings (262144 x 2560)
embed_tokens_scales.bin 512 KB
embed_tokens_per_layer_q8.bin 2.6 GB — INT8 per-layer embeddings (PLE)
embed_tokens_per_layer_scales.bin 512 KB
per_layer_projection.bin 53 MB
per_layer_norm_weight.bin 512 B
# RoPE cos/sin tables (pre-baked, mmap'd)
cos_sliding.npy / sin_sliding.npy
cos_full.npy / sin_full.npy
# Tokenizer + runtime config
hf_model/
tokenizer.json, tokenizer_config.json, config.json, generation_config.json
model_config.json| Stage | Compute | Files used |
|---|---|---|
| Token / PLE embed lookup | Swift CPU (mmap) | embed_tokens*.bin, per_layer_*.bin |
| Decode (T=1) | ANE | chunk1 + chunk2_3way + chunk3_3way |
| Prefill (batched, T=8) | ANE | chunk1 + chunk2 + chunk3 + chunk4 (prefill_b8 multifunction) |
| Vision encoder | ANE | vision.ane.mlmodelc (with LLM_VISION_FORCE_ANE=1) |
| Audio encoder | GPU | audio.mlmodelc |
| Audio projection (1024 → 1536 → 2560) | Swift / Accelerate | output_proj_*.npy, embed_proj_weight.npy |
chunk2_3way + chunk3_3way and routes prefill through the legacy 4-chunk prefill_b8 multifunction (the engine's fillBatchMasksVisionAware keeps bidirectional within-image attention working at T=8 batches).model.mlpackage)?phys_footprint. We mmap the raw INT8 + scale .bin files instead, dequantize the few rows touched per token in pure Swift, and feed the result to the chunks. The chunks themselves are pure transformer bodies and stay ANE-resident..npy RoPE tables are pre-baked at conversion-time so Swift doesn't need to ship a cos/sin builder.output_proj_* / embed_proj_weight) lives outside the ANE because of a Core ML GPU runtime bug with RMSNorm(with_scale=False) that produces all-zero outputs. Sgemm in Accelerate is fast enough on CPU.hf_model/. Three multimodal placeholder token IDs:<|image|> = 258880 — image-pad span (256 per still image)<|audio|> = 258881 — audio-pad span (~188 per 2 sec)<|video|> = 258884 — video-pad span (64 per frame)<|image|>/<|video|> rows during prefill (and per-token at decode for tail spans). Audio output rows replace <|audio|>. per_layer_raw is forced to zero at multimodal positions — the chunks compute per_layer_combined entirely from the spliced hidden state.google/gemma-4-E4B-it. Use is governed by the Gemma Terms of Use. Vision / audio extensions inherit the same license.