Live music generation at 25 frames per second, on the phone in your pocket. Ten unbroken minutes of 48 kHz stereo without melting your phone. Zero dropouts. Even on a 2020 iPhone 12 Pro. And the GPU never wakes up.

This is Google DeepMind's Magenta RealTime 2 (mrt2_small, 230M parameters), surgically cut into three Core ML graphs and placed on the silicon each one wants. The temporal transformer holds p99 ≈ 14 ms on the iPhone's Neural Engine against a 40 ms frame budget. Output correlation vs Google's MLX reference: 0.999985904188 — we publish all twelve digits because we measured all twelve. Decoder SNR: 118.85 dB. Depth rollout tokens vs the reference sampler: 0/900 mismatched in FP32. App-attributed GPU time in a 60-second Instruments capture of the all-ANE pipeline: zero — the only process with GPU intervals is iOS's screen compositor.

Exporters, validation harness, full method docs: github.com/mattmireles/magenta-realtime-2-iphone Pre-converted models (this repo): you are here.

Corrected artifact generation (paper Surgical Inference, §6.3–6.5)

This repo hosts the corrected artifact generation — the binaries produced by the corrected exporters in the GitHub repo, which reproduce the paper's three headline findings. Three earlier conclusions are superseded, and the supersession is itself a reported result; the earlier binaries are retained in superseded/ as negative-result evidence.

State mutation, not attention, is the ANE cliff (§6.3). The stateless host-owned-cache temporal step (MRT2TemporalBodyCarry: 48 K/V caches as ordinary inputs, one-token updates as ordinary outputs, no ct.StateType) compiles the full 12-layer stack to one ANE-resident graph (costWeights=ane:1.000, p99 14.991 ms on iPhone 12 Pro). Every in-graph ct.StateType variant fails ANECCompile() with error −14. Honesty (§6.7): the shipped runtime still places temporal on .cpuAndGPU because ANE admission proved instance-fragile; the stateless graph is the proven escape to re-land.
Layout determines FP16 survival (§6.4). The decoder FP16 export is finite and ANE-resident after a channels-first (NCHW) rewrite plus an exact mid-network rescale. The naive channels-last FP16 export produces non-finite output (finite ratio 0.71) — "do not re-export at fp16" was wrong.
Weight bandwidth shapes the graph (§6.5). Twelve depth predictions per frame are weight-bandwidth-doomed no matter how little they compute, so the corrected depth body samples all 12 RVQ levels in one in-graph FP16 rollout fed by host-supplied Gumbel noise. FP32 rollout is token-exact (0/900); FP16 ships at 12.7 ms/frame on A14.

The authoritative artifact map is MODELS.md; the corrected device evidence is docs/validation-receipts.md.

Why surgery?

One graph cannot serve this model well. MRT2's generation step chains four computations with opposite hardware affinities — and the ANE compiler fails silently when you push it too far: the whole model as one graph reports success while quietly running on CPU at ~640 ms/frame, 16× over budget, no error raised. So the pipeline is cut at the joints and each piece lands on the silicon that wants it:

prompt text ──(compiled once, on a Mac)──▶ prompt vector
               │
               ▼
┌──────────────────────────────────┐
│  TEMPORAL  (carry, stateless)    │ ◀── ANE-clean (proven);
│  Predicts the next 40 ms of music│     shipped .cpuAndGPU; fp16
└──────────────┬───────────────────┘
               ▼
┌──────────────────────────────────┐
│  DEPTH  (in-graph rollout)       │ ◀── CPU/GPU/ANE (bandwidth)
│  Samples 12 RVQ levels in-graph  │     fp16, host Gumbel noise
└──────────────┬───────────────────┘
               ▼
┌──────────────────────────────────┐
│  RVQ GATHER  (Swift / C++)       │ ◀── CPU
│  Sum 12 codebook rows            │     data-dependent logic
└──────────────┬───────────────────┘
               ▼
┌──────────────────────────────────┐
│  DECODER  (SpectroStream, NCHW)  │ ◀── Neural Engine
│  Tokens → spectrogram frames     │     fp16, channels-first
└──────────────┬───────────────────┘
               ▼
┌──────────────────────────────────┐
│  iSTFT + OVERLAP-ADD  (C++)      │ ◀── CPU
│  Spectrogram → stereo PCM        │     cheap DSP
└──────────────┬───────────────────┘
               ▼
   48 kHz stereo audio — 25 frames per second

Sampling entropy (RNG, Gumbel noise) and the RVQ codebook gather stay on the host, so token-level parity is provable and seeds reproduce. CFG is baked at conditioning time — no runtime guidance machinery on-device.

What's in the download

Corrected artifacts (current)

File	Size	Precision	Compute target	Role
`MRT2TemporalBodyCarry.mlpackage`	349 MB	fp16	ANE-clean (proven); shipped runtime `.cpuAndGPU`	Stateless temporal step: 48 K/V caches in, 48 one-token updates out, host owns mutation. 2-frame bucket (`--frames 2`, `history_length 0`).
`MRT2DepthBodyRollout.f16.mlpackage`	71 MB	fp16	CPU/GPU/ANE (bandwidth-bound)	In-graph depth rollout: all 12 RVQ levels sampled in one prediction. The ship variant.
`MRT2DepthBodyRollout.f32.mlpackage`	141 MB	fp32	CPU/GPU	Token-exact reference (0/900 mismatches vs the reference sampler).
`SpectroStreamDecoder.f16.mlpackage`	68 MB	fp16 (fp16-safe rescale)	ANE (`.cpuAndNeuralEngine`)	NCHW-internal decoder — the only numerically finite FP16 variant. The ship variant.
`SpectroStreamDecoder.mlpackage`	136 MB	fp32	CPU/GPU	NCHW FP32 reference (SNR 118.85 dB vs MLX). Same graph as the fp16 ship variant, full precision.
`SpectroStreamRVQCodebooks.f32.bin` (+ `.json`)	12.6 MB	f32	CPU	12×1024×256 codebook table, host gather
`examples/test_vector_smooth_electronic.bin` (+ `.json`)	1 KB	f32	—	One certified conditioning vector
`metadata/`	—	—	—	Per-model export receipts (coremltools config, full I/O schemas)
`validation/`	—	—	—	Parity receipts backing every number on this card
`fixtures/`	1.3 MB	—	—	MLX reference tensors → validate without installing MLX

All models are mlprogram, minimum deployment target iOS 18 / macOS 15.

Device caveat, stated plainly (§6.7): the decoder's FP16 finiteness is compute-unit-dependent — on iPhone 12 Pro, .cpuAndNeuralEngine yields finite output (184,320/184,320 at 25 frames, p99 24.77 ms) while CPU-only and CPU+GPU produce non-finite output from the same artifact. Pin the fp16 decoder to .cpuAndNeuralEngine, or use the fp32 reference. And ANE admission for the temporal stack proved instance-fragile in a full app; the shipped runtime places temporal on .cpuAndGPU with the ANE-clean stateless graph as the documented escape.

Superseded artifacts (`superseded/`, negative-result evidence)

File	Size	Why superseded
`superseded/MRT2TemporalBody.mlpackage`	349 MB	Stateful (48 `ct.StateType`) temporal step. Every in-graph state-mutation variant fails `ANECCompile −14` on device (§6.3).
`superseded/MRT2DepthBody.mlpackage`	93 MB	FP32 logits + 12 host-side predictions per frame — weight-bandwidth-doomed (§6.5).

Do not treat these as the paper's shipped configuration. Their export receipts and validation receipts moved with them under superseded/.

Tensor contracts (corrected artifacts)

Model	Inputs	Outputs
MRT2TemporalBodyCarry	`temporal_inputs` `[1,2,1024]` f32 · `source_encoded` `[1,2,256]` f32 · 48 × `temporal_layer_XX_{self,cross}_{key,value}_cache_in` `[1,41,8,128]` fp16	`temporal_outputs` `[1,2,1024]` · 48 × `..._cache_updates` `[1,2,8,128]` fp16
MRT2DepthBodyRollout	`temporal_frame` `[1,1,1024]` f32 · `gumbel_noise` `[12,1024]` f32 · `inverse_temperature` `[1]` f32	`sampled_codes` `[12]` int32 · `temporal_feedback` `[1,1024]`
SpectroStreamDecoder (both precisions)	`decoder_embeddings` `[1,25,256]` f32 (host RVQ lookup sum)	`decoder_stft` `[1,96,480,4]` f32 (→ host iSTFT/overlap-add)

Everything is static-shaped. sampled_codes are codebook-local (0–1023); the unique id is 6 + level*1024 + code. temporal_feedback is the mean of the 12 sampled token embeddings (×32 scale baked in) — the next frame's temporal_inputs row. This shipped temporal package is the 2-frame bucket; re-export with --frames N for other buckets. Full I/O schemas are in metadata/.

Usage (Swift sketch)

The one-frame loop, structurally. The host owns the K/V ring buffer — the temporal graph is pure. A production runtime additionally needs a lock-free PCM ring and lookahead buffering sized from p99 inference time:

swift

import CoreML

let temporal = try MLModel(contentsOf: temporalCarryURL, configuration: cpuGpuConfig) // shipped .cpuAndGPU; ANE-clean graph, see §6.7
let depth    = try MLModel(contentsOf: depthRolloutURL,  configuration: anyConfig)    // fp16 in-graph rollout
let decoder  = try MLModel(contentsOf: decoderF16URL,    configuration: aneConfig)    // MUST be .cpuAndNeuralEngine for fp16
var kvCaches = HostKVCaches(layers: 12, window: 41)     // 48 fp16 buffers, host-owned
let codebooks = RVQCodebooks(contentsOf: codebooksURL)  // 12×1024×256 f32 table
let source    = loadConditioning("test_vector_smooth_electronic.bin")  // [1,256]

while generating {                                      // 25 Hz
  // 1. temporal step — caches go IN as inputs, one-token updates come OUT
  let t = try temporal.prediction(from: kvCaches.asInputs(
            ["temporal_inputs": prevFrameEmbedding, "source_encoded": source]))
  kvCaches.apply(updates: t)                            // host-owned ring-buffer write
  // 2. depth rollout — host supplies Gumbel noise, graph samples all 12 levels
  let d = try depth.prediction(from: ["temporal_frame": t.lastFrame,
                                      "gumbel_noise": gumbel(seed),
                                      "inverse_temperature": invTemp])
  // 3. codebook gather on the CPU → RVQ embeddings; feedback comes from the graph
  // 4. decoder → pre-iSTFT tensor → iSTFT + overlap-add on the CPU → PCM
  prevFrameEmbedding = d.temporalFeedback
}

The Python wrappers in the GitHub repo's exporters/mrt2_coreml/ are the executable specification for each host-side stage.

Verify our numbers

Every number on this card has a receipt in validation/. Quick local check:

bash

pip install coremltools numpy torch
git clone https://github.com/mattmireles/magenta-realtime-2-iphone && cd magenta-realtime-2-iphone
hf download mattmireles/magenta-realtime-2-iphone --local-dir hf

# Depth rollout, FP32: zero token mismatches required (gate: PASS)
PYTHONPATH=exporters python validation/validate_depth_body_rollout.py \
    --model-path hf/MRT2DepthBodyRollout.f32.mlpackage --float32

The temporal and decoder validators recompute the MLX reference from Google's checkpoint (requires the magenta_rt MLX stack on a Mac); fixtures/ contains pre-dumped MLX reference tensors for the fixture-based path. And every .mlpackage here is reproducible: run the exporters in the GitHub repo against mrt2_small.safetensors and compare — we regenerated every corrected package from the public exporters before uploading and the weight payloads matched the certified originals byte for byte.

Checksums (sha256), corrected artifacts

File	sha256
`MRT2TemporalBodyCarry.mlpackage/Data/com.apple.CoreML/weights/weight.bin`	`02657e24a19402fed8210e431e1a67baa0c8196d4d23f0c78dd931668a9c5a64`
`MRT2DepthBodyRollout.f16.mlpackage/Data/com.apple.CoreML/weights/weight.bin`	`6c69ebae8cd7401fcb9927e3ff558d5b240cebdae2e29972d731ee08bd81c4f8`
`MRT2DepthBodyRollout.f32.mlpackage/Data/com.apple.CoreML/weights/weight.bin`	`af3ec4f9fadb84991b5dabbac58acd44294df7514226af2db63aa79659e304a9`
`SpectroStreamDecoder.f16.mlpackage/Data/com.apple.CoreML/weights/weight.bin`	`ceb3ed7d5b0286e69a937a17db253ce5a95e6bfa5560a187eb4c17de9e080f5d`
`SpectroStreamDecoder.mlpackage/Data/com.apple.CoreML/weights/weight.bin`	`38cbdf5ca97fe1744fa4d5a5caee23d50170a7eeb18daed1c40fe3a73aed9852`

Conditioning

The temporal body cross-attends to a 256-dim compiled prompt (source_encoded). The exporter (export_conditioning.py) compiles any text prompt through MusicCoCa + the MRT2 conditioning encoder on a Mac, deterministically, with CFG baked at reference strength (3.0, 1.0, 1.0).

We ship exactly one conditioning vector — "smooth electronic", certified byte-reproducible — as a test vector, not a preset. We have not run the listening validation that would justify a preset library, and unvalidated presets are how on-device generation ends up sounding broken. Compile your own prompts; treat conditioning quality as experimental.

Requirements

iOS 18+ / macOS 15+ (mlprogram floor)
The fp16 decoder requires a device with a Neural Engine and .cpuAndNeuralEngine placement — it is non-finite on CPU and GPU. The fp32 decoder runs anywhere.
Proven on iPhone 15 Pro Max (A17 Pro) and iPhone 12 Pro (A14 Bionic, 2020) — both held a full 10-minute run with zero underruns and zero dropped frames. The older A14 needs a longer startup reservoir (≈15 s of PCM primed before playback) to absorb its higher per-frame tail.
coremltools ≥ 8 to re-export

License & attribution

The model weights in this repository are content-preserving derivatives (numerical precision and memory-layout transforms only — no fine-tuning, pruning, or distillation) of google/magenta-realtime-2 by Google DeepMind, and are redistributed under the same CC-BY-4.0 license. Use of the model remains subject to the upstream model's terms of use; you are responsible for what you generate.

Conversion code, validation harnesses, and docs are Apache-2.0 in the GitHub repo.

Credits

Google DeepMind — the Magenta team, for Magenta RealTime 2, SpectroStream, and MusicCoCa, and for releasing real on-device weights under a license that makes work like this possible. (magenta/magenta-realtime)
Apple's coremltools team.
Port by Matt Mireles. Prior art in the same spirit: kokoro-coreml.

Real-time music generation in your pocket, with the receipts to prove it.

mattmireles/magenta-realtime-2-iphone

Magenta RealTime 2 on the iPhone, Without the GPU