You can deploy this model and start earning money today!

anziank/
grio-qwen2.5-1.5b-coreml-anyLM-seq2048

Name: grio-qwen2.5-1.5b-coreml-anyLM-seq2048
Availability: InStock

Views

No views yet

Qwen 2.5 1.5B Instruct — CoreML (seq≤2048, AnyLanguageModel-compatible)

On-device CoreML .mlpackage converted from Qwen/Qwen2.5-1.5B-Instruct for use with HuggingFace's AnyLanguageModel Swift framework and swift-transformers ≥ 1.0.

Input/output tensor names match the inputIds / attentionMask / logits convention required by swift-transformers 1.x LanguageModel. Drop-in compatible with CoreMLLanguageModel(url:computeUnits:chatTemplateHandler:).

Variants in this collection

grio-qwen2.5-1.5b-coreml-anyLM-seq512 Fixed [1, 512] — short-form enhancement. Smallest graph. Runs on .all.
grio-qwen2.5-1.5b-coreml-anyLM-seq1024 Fixed [1, 1024] — medium-context enhancement, multi-sentence translation. Runs on .all.
grio-qwen2.5-1.5b-coreml-anyLM-seq2048 (this repo) RangeDim [1, 1..2048] — variable-length translation / long-context; fastest on short prompts since attention scales with actual length. Must use .cpuAndNE or .cpuAndGPU; .all SIGSEGVs (see Runtime gotchas).

Model details

Spec	Value
Base	Qwen/Qwen2.5-1.5B-Instruct
Precision	Float16 (`mlprogram`)
Context	1-2048 tokens, flexible `RangeDim(1, 2048)`
Inputs	`inputIds`, `attentionMask`
Input shape	`Int32 [1, <=2048]`
Output	`logits: Float16 [1, seq_len, 151936]` (rank-3, per-position)
Min OS	iOS 18 / macOS 15
Compute	`.cpuAndNE` fastest, `.cpuAndGPU` safe; do not use `.all`
Format	`.mlpackage` (compiled on first load)
Toolchain	`coremltools` 9.0 + `torch` 2.7 + `transformers` 5.8.1

Verified output (greedy, deterministic)

Smoke outputs from this .mlpackage. Useful as a regression marker if you re-convert or quantize.

Raw completion, cpu_and_gpu Prompt: "The capital of France is" Output: " Paris. The capital of France is also the capital of which country?"
ChatML System: "You are a helpful assistant. Answer concisely." User: "What is 2+2?" Output: "2+2 equals 4." (EOS at 7 tokens)
ChatML translation System: "You are a French translator. Translate the user message to French. Output only the translation." User: "The capital of France is Paris." Output: "La capitale de la France est Paris." (EOS at 9 tokens)
ChatML translation System: "You are a French translator. Translate the user message to French. Output only the translation." User: "The quick brown fox jumps over the lazy dog." Output: "Le renard brun rapide saute par-dessus le chien paresseux." (byte-identical to PyTorch FP16 reference)

Observed performance (M1 Pro, macOS)

Benchmark notes: real prompt about 27 tokens.

Compute	Predict	Load	Result
`.cpuAndNE`	~167	~50s	✅ Fastest. ANE-only path is healthy on this RangeDim graph.
`.cpuAndGPU`	~339	~28–35s	✅ Safe everywhere; `swift-transformers`' default.
`.all`	❌ SIGSEGV at `predict()`	—	RangeDim + ANE+GPU mix crashes CoreML's MPSGraph/E5 path. Do not use.
`.cpuOnly`	(slow)	—	BNNS produces NaN for FP16 transformer attention. Do not use.

iPad device perf not benchmarked in this build run.

Runtime gotchas (please read before integrating)

.all compute units crash this RangeDim model at inference. This is structural to RangeDim + ANE-mixed compute inside Core ML's MPSGraph / E5 program library on iOS 18 / macOS 15. The fixed-shape seq512 / seq1024 variants in this collection do NOT have this issue. If your runtime currently defaults to .all, route this RangeDim variant to .cpuAndGPU or .cpuAndNE explicitly.
logits shape metadata is empty in the .mlpackage description (a coremltools artifact for RangeDim outputs). Verify at runtime with a real predict() call — the actual output is rank-3 [1, seq_len, 151936] and works with swift-transformers' assert(scores.rank == 3).
Qwen 2.5 uses two stop tokens in generation_config.json: <|endoftext|> (id 151643) and <|im_end|> (id 151645). tokenizer.eos_token_id returns only <|im_end|>. If your runtime stops only on the single tokenizer EOS id, it may miss the document-end token in some prompts. Read the full eos_token_id list from generation_config.json (bundled in this repo) for robustness. swift-transformers ≥ 1.3 handles this correctly.

Conversion notes (for the CoreML community)

This model was produced from PyTorch source via torch.export + coremltools.convert. Findings worth flagging for anyone converting Qwen 2.5 (or similar HF causal LMs) to a swift-transformers-compatible CoreML format:

attn_implementation silently corrupts RangeDim graphs. With attn_implementation="eager" (the HF default for Qwen 2.5), coremltools raises RuntimeWarning: overflow encountered in cast in coremltools/converters/mil/mil/passes/defs/optimize_repeat_ops.py:433 at pass ~65 of the MIL default pipeline when the input shape contains RangeDim. Conversion completes silently. ABI verifies as rank-3 [1, S, 151936]. But model.predict() returns degenerate logits — greedy argmax collapses to a single low-index token on every step. Switching to attn_implementation="sdpa" produces a different op graph (3667 MIL ops vs eager's 3443) that skips the buggy pass and produces correct outputs. Fixed-shape (seq512 / seq1024) builds are not affected by the warning, presumably because the overflow is on RangeDim bounds arithmetic. We still use sdpa for the fixed-shape builds for consistency.
run_decompositions({}) SIGSEGVs at 1.5B with RangeDim. Use torch.export.default_decompositions() instead. Same optimize_repeat_ops neighborhood, but as a hard crash rather than a silent corruption.
.mlpackage output description for RangeDim models reports empty logits shape metadata. Verify rank with a real predict() probe at conversion time — do not trust descriptor introspection alone.
FP16 / greedy decoding is not byte-deterministic across backends. Outputs are semantically equivalent to PyTorch CPU FP16 reference but may differ on tokens where the model has near-tied top-1 candidates (e.g., picking "Kaffeetasse" vs "Kaffeekanne" for "a coffee" in German translation). Expected behavior, not a conversion bug.

Usage (Swift)

swift

import AnyLanguageModel

let modelURL: URL = // path to this .mlpackage on disk
let lm = try await CoreMLLanguageModel(
    url: modelURL,
    computeUnits: .cpuAndNE,  // or .cpuAndGPU — NOT .all on this RangeDim variant
    chatTemplateHandler: { instructions, prompt in
        var messages: [Message] = []
        if let system = instructions?.description, !system.isEmpty {
            messages.append(["role": "system", "content": system])
        }
        messages.append(["role": "user", "content": prompt.description])
        return messages
    }
)
let session = LanguageModelSession(model: lm, instructions: "You are a French translator. Output only the translation.")
let response = try await session.respond(to: "The capital of France is Paris.")
print(response.content) // "La capitale de la France est Paris."

Keep tokenizer.json, tokenizer_config.json, config.json, and generation_config.json (all bundled in this repo) as siblings of the .mlpackage on disk. swift-transformers reads the chat template from tokenizer_config.json at runtime.

Reproducibility

Conversion done with coremltools==9.0, torch==2.7.0, transformers==5.8.1. Approximate single-call recipe:

python

import coremltools as ct, torch, torch.nn as nn
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct",
    torch_dtype=torch.float16,
    attn_implementation="sdpa",         # CRITICAL for RangeDim — see Conversion notes
)
model.eval()

class Wrapper(nn.Module):
    def __init__(self, m): super().__init__(); self.m = m
    def forward(self, inputIds, attentionMask):
        return self.m(input_ids=inputIds, attention_mask=attentionMask, use_cache=False).logits

wrapper = Wrapper(model).eval()
seq = torch.export.Dim("sequence_length", min=1, max=2048)
ep = torch.export.export(
    wrapper,
    (torch.randint(0, 151936, (1, 128), dtype=torch.int32),
     torch.ones((1, 128), dtype=torch.int32)),
    dynamic_shapes={"inputIds": {1: seq}, "attentionMask": {1: seq}},
).run_decompositions(torch.export.default_decompositions())  # NOT {}

ct.convert(
    ep,
    inputs=[
        ct.TensorType(name="inputIds",      shape=(1, ct.RangeDim(1, 2048)), dtype=int),
        ct.TensorType(name="attentionMask", shape=(1, ct.RangeDim(1, 2048)), dtype=int),
    ],
    outputs=[ct.TensorType(name="logits")],
    minimum_deployment_target=ct.target.iOS18,
    compute_precision=ct.precision.FLOAT16,
    convert_to="mlprogram",
)

License

Apache-2.0. Weights from Qwen/Qwen2.5-1.5B-Instruct by Qwen Team / Alibaba Cloud. Re-uploaded as a CoreML port; original model card terms apply.

anziank/grio-qwen2.5-1.5b-coreml-anyLM-seq2048