You can deploy this model and start earning money today!

anziank/
grio-qwen2.5-1.5b-coreml-anyLM-seq512

Name: grio-qwen2.5-1.5b-coreml-anyLM-seq512
Availability: InStock

Views

No views yet

Qwen 2.5 1.5B Instruct — CoreML (seq=512 fixed, AnyLanguageModel-compatible)

On-device CoreML .mlpackage converted from Qwen/Qwen2.5-1.5B-Instruct for use with HuggingFace's AnyLanguageModel Swift framework and swift-transformers ≥ 1.0.

Input/output tensor names match the inputIds / attentionMask / logits convention required by swift-transformers 1.x LanguageModel. Drop-in compatible with CoreMLLanguageModel(url:computeUnits:chatTemplateHandler:).

Variants in this collection

grio-qwen2.5-1.5b-coreml-anyLM-seq512 (this repo) Fixed [1, 512] — short-form enhancement / single-sentence translation. Smallest graph. Runs on .all.
grio-qwen2.5-1.5b-coreml-anyLM-seq1024 Fixed [1, 1024] — medium-context enhancement, multi-sentence translation. Runs on .all.
grio-qwen2.5-1.5b-coreml-anyLM-seq2048 RangeDim [1, 1..2048] — variable-length / long-context. Must use .cpuAndNE or .cpuAndGPU; .all SIGSEGVs on the RangeDim variant.

Model details

Spec	Value
Base	Qwen/Qwen2.5-1.5B-Instruct
Precision	Float16 (`mlprogram`)
Context	512 tokens, fixed shape (prompt + completion combined)
Inputs	`inputIds`, `attentionMask`
Input shape	`Int32 [1, 512]`
Output	`logits: Float16 [1, 512, 151936]` (rank-3, per-position)
Min OS	iOS 18 / macOS 15
Compute	`.all` recommended; fixed shape is ANE-friendly on Qwen 2.5
Format	`.mlpackage` (compiled on first load)
Toolchain	`coremltools` 9.0 + `torch` 2.7 + `transformers` 5.8.1

Verified output (greedy, deterministic)

Smoke output from this .mlpackage. Useful as a regression marker if you re-convert or quantize.

Raw completion, cpu_and_gpu Prompt: "The capital of France is" Output: " Paris. The capital of France is also the capital of which country?"
ChatML System: "You are a helpful assistant. Answer concisely." User: "What is the capital of France?" Output: "Paris." (stops at <|im_end|> after 2 tokens)

This output is identical to the matching seq1024 and seq2048 variants in this collection, since the underlying weights and graph topology are the same; only the static seq dimension differs.

Observed performance (M1 Pro, macOS)

Benchmark notes: full-pad 512, 32-token greedy decode.

Compute	Predict	Load	Result
`.cpuAndGPU`	~538	~31s	✅ Clean.
`.all`	(not benchmarked in this rebuild)	—	✅ Clean — fixed shape is ANE-friendly.

Smoke runs pad to the full 512-position graph every step, so the ms/tok number reflects fixed-shape compute over 512 positions — not the real-prompt-length cost you would see with the seq2048 RangeDim variant.

Runtime gotchas (please read before integrating)

Fixed context window — input + output ≤ 512 tokens. Pad shorter prompts with zeros and set attentionMask = 0 on padded positions.
logits is per-position, rank-3 [1, 512, 151936]. Pick the row at the last real prompt token for greedy/sampled decode.
.all works on this fixed-shape variant — no SIGSEGV. The .all crash documented in the seq2048 variant is specific to RangeDim graphs.
Qwen 2.5 uses two stop tokens in generation_config.json: <|endoftext|> (id 151643) and <|im_end|> (id 151645). tokenizer.eos_token_id returns only <|im_end|>. If your runtime stops only on the single tokenizer EOS id, it may miss the document-end token in some prompts. Read the full eos_token_id list from generation_config.json for robustness. swift-transformers ≥ 1.3 handles this correctly.

Conversion notes (for the CoreML community)

This model was produced from PyTorch source via torch.export + coremltools.convert. Findings worth flagging for anyone converting Qwen 2.5 (or similar HF causal LMs) to a swift-transformers-compatible CoreML format:

Use attn_implementation="sdpa". The HF default "eager" for Qwen 2.5 produces an op graph that triggers RuntimeWarning: overflow encountered in cast in coremltools/converters/mil/mil/passes/defs/optimize_repeat_ops.py:433 under RangeDim shapes, which silently corrupts logits. "sdpa" produces a cleaner op graph that lowers reliably. Fixed-shape (seq512 / seq1024) builds of Qwen 2.5 are not affected by the warning, but we still use SDPA here for consistency across the collection.
Use torch.export.default_decompositions() — the all-decompositions mode ({}) can SIGSEGV at 1.5B+ scale under RangeDim.
FP16 / greedy decoding is not byte-deterministic across backends. Outputs are semantically equivalent to PyTorch CPU FP16 reference but may differ on tokens where the model has near-tied top-1 candidates. Expected behavior, not a conversion bug.

Usage (Swift)

swift

import AnyLanguageModel

let modelURL: URL = // path to this .mlpackage on disk
let lm = try await CoreMLLanguageModel(
    url: modelURL,
    computeUnits: .all,  // fixed shape — ANE-friendly
    chatTemplateHandler: { instructions, prompt in
        // Qwen 2.5 uses ChatML format; tokenizer.json's Jinja template applies the special tokens.
        var messages: [Message] = []
        if let system = instructions?.description, !system.isEmpty {
            messages.append(["role": "system", "content": system])
        }
        messages.append(["role": "user", "content": prompt.description])
        return messages
    }
)
let session = LanguageModelSession(model: lm, instructions: "You are a helpful assistant.")
let response = try await session.respond(to: "Improve this text: …")
print(response.content)

Keep tokenizer.json, tokenizer_config.json, config.json, and generation_config.json (all bundled in this repo) as siblings of the .mlpackage on disk. swift-transformers reads the chat template from tokenizer_config.json at runtime.

Reproducibility

Conversion done with coremltools==9.0, torch==2.7.0, transformers==5.8.1. Approximate single-call recipe:

python

import coremltools as ct, torch, torch.nn as nn
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct",
    torch_dtype=torch.float16,
    attn_implementation="sdpa",
)
model.eval()

class Wrapper(nn.Module):
    def __init__(self, m): super().__init__(); self.m = m
    def forward(self, inputIds, attentionMask):
        return self.m(input_ids=inputIds, attention_mask=attentionMask, use_cache=False).logits

wrapper = Wrapper(model).eval()
ep = torch.export.export(
    wrapper,
    (torch.randint(0, 151936, (1, 128), dtype=torch.int32),
     torch.ones((1, 128), dtype=torch.int32)),
).run_decompositions(torch.export.default_decompositions())

ct.convert(
    ep,
    inputs=[
        ct.TensorType(name="inputIds",      shape=(1, 512), dtype=int),
        ct.TensorType(name="attentionMask", shape=(1, 512), dtype=int),
    ],
    outputs=[ct.TensorType(name="logits")],
    minimum_deployment_target=ct.target.iOS18,
    compute_precision=ct.precision.FLOAT16,
    convert_to="mlprogram",
)

License

Apache-2.0. Weights from Qwen/Qwen2.5-1.5B-Instruct by Qwen Team / Alibaba Cloud. Re-uploaded as a CoreML port; original model card terms apply.

anziank/grio-qwen2.5-1.5b-coreml-anyLM-seq512