CoreML port of NVIDIA Sortformer for end-to-end speaker diarization on Apple Silicon. Runs on the Neural Engine. No separate embedding extraction or clustering — the model directly predicts per-frame speaker activity for up to 4 speakers, with streaming state (speaker cache + FIFO) carried across calls.
Model
Architecture
Sortformer (Sort Loss + 17-layer FastConformer + 18-layer Transformer)
Measured on M-series Apple Silicon (Neural Engine, warm):
Variant
Per-call latency
Realtime factor
default
one call per ~27 s of audio
~125–750×
balanced
one call per ~8 s
hundreds of ×
streaming
8.1 ms median per 480 ms step
~59× per step; ~39× end-to-end incremental
The streaming export is parity-gated: driven by NeMo's own streaming feature
loader and cache-update algorithm, it matches NeMo's native
forward_streaming loop at 100% frame-decision agreement (MAE 0.0005).
On a five-file VoxConverse-dev pilot, the incremental session reaches 8.1%
DER (collar 0.25) with correct speaker counts on all files.
Streaming interface
The streaming variant carries state through the CoreML interface; the
speaker-cache update runs host-side between calls.
