Views
No views yet
Package.swift:1.package(url: "https://github.com/john-rocky/CoreML-LLM", branch: "main"),
2
3// In your target:
4.product(name: "CoreMLLLM", package: "CoreML-LLM"),1import CoreMLLLM
2
3let llm = try await CoreMLLLM.load(repo: "mlboydaisuke/gemma-4-E2B-coreml")
4
5let stream = try await llm.generate(
6 [CoreMLLLM.Message(role: .user, content: "Hello!")],
7 maxTokens: 256
8)
9for await chunk in stream { print(chunk, terminator: "") }1// Image
2let stream = try await llm.generate(
3 [CoreMLLLM.Message(role: .user,
4 content: "Describe this image")],
5 image: cgImage)
6
7// Video (frames + audio extracted internally)
8let stream = try await llm.generate(
9 [CoreMLLLM.Message(role: .user,
10 content: "What happens in this clip?")],
11 videoURL: localFileURL)generate(_:) overloads — see the Swift file
CoreMLLLM.swift
for the full surface.google/gemma-4-E2B-it (the 2B-effective Gemma 4 / Gemma 3n decoder), optimized for Apple Neural Engine. Text + image + audio + short video, INT4 weights.Branches:mainis the long-running 4-chunk text+vision+audio bundle. The default ship target forCoreMLLLMChatv1.6+ is then1024branch (3-chunk merged decoder, slightly faster prefill). Both ship the same architecture — only the chunk topology and tokenizer artifacts differ. Pick whichever matches the Swift runtime you're using; if in doubt, usen1024.
n1024 branch — recommended)chunk1.mlmodelc/ # L0–7 — INT4 palettized
chunk2_3way.mlmodelc/ # L8–24 — merged middle (3-chunk decoder)
chunk3_3way.mlmodelc/ # L25–34 + lm_head — multifunction
prefill_chunk{1..4}.mlmodelc/ # T=N prefill bodies (mlmodelc, weights shared
# with decode chunks via hardlink)
vision.mlmodelc/ # SigLIP encoder, 322 MB
vision_video.mlmodelc/ # video frame encoder (64 tok/frame)
audio.mlmodelc/ # 282 MB Whisper-style audio encoder
embed_tokens_q8.bin 402 MB — INT8 token embeddings (262144 × 1536)
embed_tokens_scales.bin 512 KB
embed_tokens_per_layer_q8.bin 2.19 GB — INT8 PLE
embed_tokens_per_layer_scales.bin 512 KB
per_layer_projection.bin 26 MB
per_layer_norm_weight.bin 1 KB
cos_{full,sliding}.npy 8 MB / 4 MB — precomputed RoPE cos
sin_{full,sliding}.npy 8 MB / 4 MB — precomputed RoPE sin
mel_filterbank.bin 129 KB — for audio path
embed_proj_weight.npy 4.5 MB — vision/audio → text embed projection
output_proj_{weight,bias}.npy 3 MB / 3 KB — audio output projection
model_config.json 434 B — runtime config (hidden=1536, layers=35, …)
audio_config.json 402 B — audio path config
hf_model/{tokenizer.json, tokenizer_config.json, config.json}main branch additionally carries the older 4-chunk topology (chunk2.mlmodelc + chunk3.mlmodelc + chunk4.mlmodelc) and several legacy variant directories (sdpa/, sdpa-8k/, swa/, stateless/, stateless-ctx2048/, lite/, lite-chunks/, mf/, w8a8-8k/, model.mlmodelc, model.mlpackage). These are research builds — only the chunk*.mlmodelc (or chunk{1,2_3way,3_3way}.mlmodelc) family is the shipping path.cos/sin .npy are pre-baked so the Swift side doesn't ship a RoPE builder.hf_model/. Or pull from upstream:1from transformers import AutoTokenizer
2tok = AutoTokenizer.from_pretrained("google/gemma-4-E2B-it")1from huggingface_hub import snapshot_download
2import coremltools as ct, json
3
4local = snapshot_download(
5 "mlboydaisuke/gemma-4-E2B-coreml", revision="n1024",
6 allow_patterns=[
7 "chunk1.mlmodelc/*", "chunk2_3way.mlmodelc/*", "chunk3_3way.mlmodelc/*",
8 "prefill_chunk*.mlmodelc/*",
9 "embed_tokens*.bin", "per_layer_*.bin",
10 "cos_*.npy", "sin_*.npy",
11 "model_config.json", "hf_model/*",
12 ],
13)
14cfg = json.load(open(f"{local}/model_config.json"))
15chunks = [
16 ct.models.MLModel(f"{local}/chunk1.mlmodelc"),
17 ct.models.MLModel(f"{local}/chunk2_3way.mlmodelc"),
18 ct.models.MLModel(f"{local}/chunk3_3way.mlmodelc"),
19]Sources/CoreMLLLM/ChunkedEngine.swift — the canonical reference.vision.mlmodelc expects pixel_values (1, 3, 256, 256) fp16, outputs (1, 256, 1536) text-aligned tokens.audio.mlmodelc expects mel-spectrogram features (use mel_filterbank.bin for the front-end), outputs an audio token stream injected into the same text decoder.vision_video.mlmodelc packs 64 tokens per frame for short video.CoreMLLLMChat — it auto-downloads this repo (the picker fetches the n1024 branch by default) and runs it via ChunkedEngine.| value | |
|---|---|
num_hidden_layers | 35 |
hidden_size | 1536 |
num_key_value_heads | 1 |
intermediate_size | 6144 |
num_kv_shared_layers | 20 |
| KV producers (sliding/full) | L13 / L14 |
| sliding window | 512 |
| context length (shipping) | 1024 (n1024) / 2048 (main) |
| vocab | 262144 |