NVIDIA Canary-SpeechLM (MLX Port)

Features

• No PyTorch at Inference: Pure MLX implementation for optimal performance and memory on macOS.
• Fast Transcription: RTF of 0.067x (runs 14.8x faster than real-time on Apple Silicon).
• High-Fidelity Alignment: Intermediate outputs are validated to match PyTorch/NeMo reference feature maps within float16/float32 precision limits.

Performance Statistics

• Audio Duration: 3.88s
• Feature Extraction + Conformer Encoding: 0.0506s
• Prefill/Time-to-First-Token (TTFT): 0.0247s (2551.55 tok/s)
• Decode Loop Generation Speed: 58.99 tok/s (up to 80.71 tok/s raw)
• Real-Time Factor (RTF): 0.0674x (14.8x faster than real-time)

Installation & Setup

1. Clone this repository:
   bash

   1git clone https://huggingface.co/Santosh-bold/speechlm-canary 2cd speechlm-canary
2. Install dependencies:
   pip install mlx mlx-lm librosa transformers soundfile
3. Ensure you have the base Qwen3-1.7B model downloaded (which contains the base tokenizer and weights):
   python -c "from huggingface_hub import snapshot_download; snapshot_download('Qwen/Qwen3-1.7B')"

Quick Usage

Record & Transcribe from Microphone

Technical Details

1. ConvSubsampling 8x downsampling module.
2. Conformer Block featuring depthwise 1D convolutions and relative multi-head self-attention.
3. Transformer-XL dynamic Relative Positional Encoding (RelPositionalEncoding).
4. LoRA adapter weight overlay on top of Qwen Causal LM.