You can deploy this model and start earning money today!

investguy/
ultra_diar_streaming_sortformer_8spk_v1_onnx

Name: ultra_diar_streaming_sortformer_8spk_v1_onnx
Availability: InStock

onnx

Views

No views yet

Ultra Diar Streaming Sortformer 8spk — ONNX (fp32)

fp32 ONNX export of devsy0117/ultra_diar_streaming_sortformer_8spk_v1, extending NVIDIA diar_streaming_sortformer_4spk-v2.1 from 4 to 8 speakers.

Streaming speaker diarization: per-frame sigmoid activity for up to 8 speakers. The graph exports one streaming step (Sortformer cache + FIFO + chunk forward). Mel extraction and segment post-processing (median filter, hysteresis) are outside the ONNX file.

Files

File	Size (approx.)	Description
`ultra_diar_streaming_sortformer_8spk_v1.onnx`	~492 MB	Single fp32 streaming diarization graph

Streaming constants are stored in ONNX metadata (no separate config.json).

Model details

Property	Value
Base model	`nvidia/diar_streaming_sortformer_4spk-v2.1`
Extension	Ultra-Sortformer (4spk → 8spk)
Architecture	FastConformer encoder (17L, d=512) + Transformer (18L, d=192) + Sortformer streaming cache
Max speakers	8
Sample rate	16 kHz mono (mel computed externally)
Mel features	128 (log-mel, NeMo-compatible)
Subsampling	8× (mel frames → model frames)
Model frame duration	80 ms
Embedding dim	512
Streaming latency	~30.4 s `(chunk_len + right_context) × 80 ms`
Export streaming params	`chunk_len=340`, `right_context=40`, `fifo_len=40`, `spkcache_len=376`
Mel frames per feed	3040 `(340 + 40) × 8`
Model frames per step	340 (output predictions per chunk)
Audio samples per feed	486,400 @ 16 kHz `(340 + 40) × 8 × 160 hop`
Audio samples per stride	435,200 @ 16 kHz `340 × 8 × 160 hop`

ONNX metadata

Key	Value
`model`	`ultra_diar_streaming_sortformer_8spk_v1`
`num_speakers`	`8`
`chunk_len`	`340`
`right_context`	`40`
`fifo_len`	`40`
`spkcache_len`	`376`

Graph I/O

Inputs

Name	Shape	Dtype	Description
`chunk`	`[batch, time_chunk, 128]`	float32	Log-mel features for current window
`chunk_lengths`	`[batch]`	int64	Valid mel frames in `chunk`
`spkcache`	`[batch, time_cache, 512]`	float32	Speaker cache embeddings
`spkcache_lengths`	`[batch]`	int64	Valid frames in `spkcache`
`fifo`	`[batch, time_fifo, 512]`	float32	FIFO context embeddings
`fifo_lengths`	`[batch]`	int64	Valid frames in `fifo`

Outputs

Name	Shape	Dtype	Description
`spkcache_fifo_chunk_preds`	`[batch, time_out, 8]`	float32	Per-frame speaker activity (sigmoid)
`chunk_pre_encode_embs`	`[batch, time_pre_encode, 512]`	float32	Pre-encode embeddings for the chunk
`chunk_pre_encode_lengths`	`[batch]`	int64	Valid pre-encode frames

Use zero-length spkcache / fifo (length 0) at stream start, then update state each step per the Sortformer streaming protocol (see NeMo / parakeet-rs sortformer.rs).

How this was produced

bash

conda activate onnx   # NeMo + torch + onnx + onnxruntime
pip install "nemo_toolkit[asr]" onnx onnxruntime

python parakeet-rs/scripts/export_ultra_diar_8spk.py \
  ultra_diar_streaming_sortformer_8spk_v1/ultra_diar_streaming_sortformer_8spk_v1.nemo \
  out_diar_8spk/ultra_diar_streaming_sortformer_8spk_v1.onnx

The export script runs an ONNX Runtime smoke test after writing the file. Override streaming params with --chunk-len, --right-context, --fifo-len, --spkcache-len if needed; re-export for a different latency trade-off.

NeMo may warn that spkcache_update_period (300) is less than chunk_len (340); the effective update period is 340.

Usage

Preprocessor (outside ONNX)

Match NeMo / parakeet-rs Sortformer frontend:

16 kHz mono
n_fft=512, win_length=400, hop_length=160
128 mel bins, log with zero guard 2^-24
Pre-emphasis 0.97

ONNX Runtime (single step)

python

import numpy as np
import onnxruntime as ort

CHUNK_LEN = 340
RIGHT_CONTEXT = 40
FIFO_LEN = 40
SPKCACHE_LEN = 376
SUBSAMPLING = 8
MEL_FRAMES = (CHUNK_LEN + RIGHT_CONTEXT) * SUBSAMPLING  # 3040

session = ort.InferenceSession(
    "ultra_diar_streaming_sortformer_8spk_v1.onnx",
    providers=["CPUExecutionProvider"],
)

chunk = np.zeros((1, MEL_FRAMES, 128), dtype=np.float32)
chunk_lengths = np.array([MEL_FRAMES], dtype=np.int64)
spkcache = np.zeros((1, 0, 512), dtype=np.float32)
spkcache_lengths = np.array([0], dtype=np.int64)
fifo = np.zeros((1, 0, 512), dtype=np.float32)
fifo_lengths = np.array([0], dtype=np.int64)

preds, embs, emb_lens = session.run(
    None,
    {
        "chunk": chunk,
        "chunk_lengths": chunk_lengths,
        "spkcache": spkcache,
        "spkcache_lengths": spkcache_lengths,
        "fifo": fifo,
        "fifo_lengths": fifo_lengths,
    },
)

# preds: [1, time_out, 8] — slice chunk predictions from cache/fifo region
print(preds.shape)

A full streaming loop must maintain FIFO, speaker cache, and smart cache compression between steps (see NVIDIA NeMo Sortformer streaming or parakeet-rs sortformer.rs).

parakeet-rs

Load with the sortformer feature. Speaker count and streaming params are read from ONNX metadata automatically:

parakeet-rs = { path = "../parakeet-rs", features = ["sortformer"] }

rust

use parakeet_rs::sortformer::{Sortformer, DiarizationConfig};

let mut sortformer = Sortformer::with_config(
    "ultra_diar_streaming_sortformer_8spk_v1.onnx",
    None,
    DiarizationConfig::callhome(),
)?;

println!("speakers={}", sortformer.num_speakers()); // 8
println!("latency={:.1}s", sortformer.latency());   // ~30.4s

// Buffered streaming (recommended for 8spk latency)
for chunk in audio.chunks(320) {
    for seg in sortformer.feed(chunk)? {
        println!("Speaker {} [{:.2}s - {:.2}s]", seg.speaker_id, ...);
    }
}
sortformer.flush()?;

Example: