You can deploy this model and start earning money today!

anziank/
grio-llama-3.2-1b-coreml-anyLM-seq512

Name: grio-llama-3.2-1b-coreml-anyLM-seq512
Availability: InStock

Views

No views yet

Llama 3.2 1B Instruct — CoreML (seq=512 fixed, AnyLanguageModel-compatible)

On-device CoreML .mlpackage converted from meta-llama/Llama-3.2-1B-Instruct for use with HuggingFace's AnyLanguageModel Swift framework and swift-transformers ≥ 1.0.

Input/output tensor names match the inputIds / attentionMask / logits convention required by swift-transformers 1.x LanguageModel. Drop-in compatible with CoreMLLanguageModel(url:computeUnits:chatTemplateHandler:).

Variants in this collection

grio-llama-3.2-1b-coreml-anyLM-seq512 (this repo) Fixed [1, 512] — short-context transcript cleanup and post-processing. Fastest Llama profile. Always pays for full 512 positions per step.
grio-llama-3.2-1b-coreml-anyLM-seq1024 Fixed [1, 1024] — medium-context copy-editing, translation.

Model details

Spec	Value
Base	meta-llama/Llama-3.2-1B-Instruct
Precision	Float16 (`mlprogram`)
Context	512 tokens, fixed shape (prompt + completion combined)
Inputs	`inputIds`, `attentionMask`
Input shape	`Int32 [1, 512]`
Output	`logits: Float16 [1, 512, 128256]` (rank-3, per-position)
Min OS	iOS 18 / macOS 15
Compute	`.all` recommended; fixed shape is ANE-friendly on Llama 3.2
Format	`.mlpackage` (compiled on first load)
Toolchain	`coremltools` 9.0 + `torch` 2.7 + `transformers` 5.8.1

Verified output (greedy, deterministic)

Smoke output from this .mlpackage under .cpuAndGPU. Output is byte-identical to the matching seq1024 variant in this collection (same weights, same graph topology; only the static seq dimension differs).

ChatML System: "You are a helpful assistant. Answer concisely." User: "What is the capital of France?" Output: "The capital of France is Paris." (stops at <|eot_id|> after 8 tokens)

Observed performance (M1 Pro, macOS)

Benchmark notes: full-pad 512, 32-token greedy decode.

Compute	Predict	Load	Result
`.cpuAndGPU`	~580	~18s	✅ Clean. Production-recommended for this shape.
`.all`	—	—	✅ Clean — fixed shape is ANE-friendly.

Runtime gotchas (please read before integrating)

Fixed context window — input + output ≤ 512 tokens. Pad shorter prompts with zeros and set attentionMask = 0 on padded positions.
logits is per-position, rank-3 [1, 512, 128256]. Pick the row at the last real prompt token for greedy/sampled decode.
Llama 3.2 uses two stop tokens in generation_config.json: <|end_of_text|> (id 128001) and <|eot_id|> (id 128009). tokenizer.eos_token_id returns <|eot_id|>. If your runtime stops only on the single tokenizer EOS id, it may decode past <|eot_id|> into out-of-distribution territory. Make sure your generation loop checks both.
Pay for full pad dimension: Shorter prompts are padded to the full sequence length, so you pay for the full static shape cost per step. Since this is the seq512 variant, it runs roughly 2× faster per step than the seq1024 variant on equivalent inputs.

Conversion notes (for the CoreML community)

Findings worth flagging for anyone converting Llama 3.2 to CoreML:

Use attn_implementation="sdpa". SDPA produces a much cleaner MIL graph that lowers reliably to both GPU and Apple Neural Engine.
Use torch.export.default_decompositions() to ensure standard graph representation and avoid conversion scaling errors during export.
FP16 / greedy decoding is not byte-deterministic across backends. Outputs are semantically equivalent to PyTorch CPU FP16 reference but may differ slightly on tokens where top-1 candidates are near-tied. This is expected behavior.

Usage (Swift)

swift

import AnyLanguageModel

let modelURL: URL = // path to this .mlpackage on disk
let lm = try await CoreMLLanguageModel(
    url: modelURL,
    computeUnits: .all,
    chatTemplateHandler: { instructions, prompt in
        // Llama 3.2 Instruct uses the tokenizer-owned header/chat template;
        // tokenizer_config.json and tokenizer files must stay alongside the model
        var messages: [Message] = []
        if let system = instructions?.description, !system.isEmpty {
            messages.append(["role": "system", "content": system])
        }
        messages.append(["role": "user", "content": prompt.description])
        return messages
    }
)
let session = LanguageModelSession(model: lm, instructions: "You are a helpful assistant.")
let response = try await session.respond(to: "Improve this text: ...")
print(response.content)

Keep tokenizer.json, tokenizer_config.json, config.json, and generation_config.json (all bundled in this repo) as siblings of the .mlpackage on disk. swift-transformers reads the chat template from tokenizer_config.json at runtime.

Reproducibility

Conversion done with coremltools==9.0, torch==2.7.0, transformers==5.8.1. Approximate single-call recipe:

python

import coremltools as ct, torch, torch.nn as nn
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-1B-Instruct",
    torch_dtype=torch.float16,
    attn_implementation="sdpa",
)
model.eval()

class Wrapper(nn.Module):
    def __init__(self, m): super().__init__(); self.m = m
    def forward(self, inputIds, attentionMask):
        return self.m(input_ids=inputIds, attention_mask=attentionMask, use_cache=False).logits

wrapper = Wrapper(model).eval()
ep = torch.export.export(
    wrapper,
    (torch.randint(0, 128256, (1, 128), dtype=torch.int32),
     torch.ones((1, 128), dtype=torch.int32)),
).run_decompositions(torch.export.default_decompositions())

ct.convert(
    ep,
    inputs=[
        ct.TensorType(name="inputIds",      shape=(1, 512), dtype=int),
        ct.TensorType(name="attentionMask", shape=(1, 512), dtype=int),
    ],
    outputs=[ct.TensorType(name="logits")],
    minimum_deployment_target=ct.target.iOS18,
    compute_precision=ct.precision.FLOAT16,
    convert_to="mlprogram",
)

License

Llama 3.2 Community License. Weights from meta-llama/Llama-3.2-1B-Instruct by Meta.

anziank/grio-llama-3.2-1b-coreml-anyLM-seq512