This is a modified ONNX export of the Mimi audio codec encoder.

It bypasses the Residual Vector Quantization (RVQ) to directly output continuous feature representations instead of discrete tokens. This preserves fine-grained acoustic details, making it ideal for downstream tasks like real-time dialogue analysis or speech emotion recognition.

Figure: Architecture showing the RVQ bypass for continuous latents. (Adapted from the Moshi technical report)

This model is a derivative work based on the "Mimi" model.

Numbers below are illustrative (desktop CPU, FP32 ONNX vs PyTorch kyutai/mimi, streaming chunk emit=1280 @ 16 kHz, inference.py --compare --mic). Your machine, thread counts, and ORT / PyTorch builds will shift them.

Speed — RTF (processing time ÷ real-time audio length per chunk)
In repeated 1-second windows, Torch often lands around RTF ≈ 0.22–0.32 and FP32 ONNX around RTF ≈ 0.14–0.18 — i.e. ONNX is commonly ~1.5–2× less wall time than Torch for the same window. Both RTF values stay below 1 in those traces, so either path can run faster than real time for this configuration.

Accuracy — Torch vs FP32 ONNX embeddings
With identical windows fed to both backends, many seconds show max absolute differences ~1e−6 … 1e−3 and cosine similarity ≳ 0.9999. During loud transients or speech onsets, max_abs can briefly reach ~1e−2 … 5e−2 while mean_abs stays much smaller; cosine may dip toward ~0.97 in the worst second before tightening again. The published FP32 ONNX pair is the intended match to Torch; INT8 trades more numerical error for CPU throughput.

Use Python 3.10+ and install the following (versions should match your CUDA/CPU setup for PyTorch).

Recommended: transformers==5.5.3 (streaming Mimi + cache behaviour used by the ONNX export and inference.py are validated against this release).

Example install for CPU ONNX + Torch:

For GPU ONNX, replace onnxruntime with onnxruntime-gpu and install a matching torch build. For live microphone streaming, add pyaudio when your platform supports it.

The Torch encoder path downloads kyutai/mimi weights via Hugging Face on first use. The ONNX path, if you do not pass local paths, downloads continuous_mimi_fp32.onnx / .json (or INT8 counterparts) from this Hugging Face repository using huggingface_hub.

inference.py provides a small CLI around the streaming Mimi encoder: PyTorch (kyutai/mimi) or ONNX, optional 16 kHz microphone streaming, and an optional Torch–ONNX comparison mode. Run it with python inference.py ... after installing the packages above.

Requires pyaudio. Prints RTF (processing time ÷ real-time audio duration) and per-chunk timing on a fixed wall-clock interval (default 1 s).

Initializes both backends and feeds identical streaming windows; reports numeric differences and per-backend timing / RTF.

Each [1s] line aggregates every encoder step in that wall-clock second: embedding agreement (max_abs, mean_abs, cos) plus per-chunk mean inference time and RTF for Torch and ONNX separately.

When --compare is set, --backend is ignored (both paths run). With --compare and without --mic, --compare-steps sets how many synthetic chunks to run.