See CHANGELOG.md for version-to-version changes, docs/stateful-runtime-notes.md for the current stateful runtime contract, and docs/performance.md for the detailed benchmark log.

This repo is organized for Hugging Face Hub delivery, not GitHub Releases or SwiftPM binary targets. The intended upload payload is:

Artifacts/stateless/zenz-stateless-fp16.mlpackage
Artifacts/stateless/zenz-stateless-8bit.mlpackage
Artifacts/stateful/zenz-stateful-fp16.mlpackage
Artifacts/stateful/zenz-stateful-8bit.mlpackage
tokenizer/*
hf_manifest.json

The original model remains the source of truth for tokenizer semantics, weights provenance, and training lineage. This Core ML port should be linked back to the upstream model when published on Hugging Face.

Runtime shape

stateless is the whole-sequence baseline.
stateful is the single-model cached generation path.

The stateful model keeps the same Core ML state layout:

keyCache
valueCache

The current stateful runtime contract is:

prefill incrementally over the prompt
decode incrementally with one token at a time
reuse the same Core ML state
provide an attention_mask that reflects the active sequence length during decode

Compute Units

stateless: .all
stateful: .cpuAndGPU

Benchmark

Summary

Device	Stateful FP16	Stateful 8-bit
iPhone Air	0.436	0.431
iPhone 12	1.124	1.041

Recommendation

iPhone 15 Pro and newer: use Stateful FP16 first.
Older devices than iPhone 15 Pro: use the Stateful 8-bit model first.

Notes

On iPhone Air, both stateful variants produced correct outputs in the recorded run.
On iPhone Air, 8-bit was slightly faster on mean latency than FP16.
On iPhone 12, 8-bit was faster on mean latency than FP16.
On iPhone 12, the recorded FP16 run under .all showed degraded outputs while the recorded 8-bit run remained correct.
The current recommendation is based on stateful running under .cpuAndGPU, not .all.
My current read is that FP16 is the better top-end option when the device can sustain it cleanly, but 8-bit is the safer deployment default once you care about broader keyboard coverage.
In other words: FP16 is the premium path, 8-bit is the compatibility path.

See docs/performance.md for the detailed tables and case-level notes.

Local export

bash

python -m pip install -r requirements.txt
python Scripts/export_all.py

Or run each stage separately: