Whisper-Sarashina ASR Stage 2 GroupLR
This repository contains the final inference artifacts from
outputs/whisper-sarashina-stage2-grouplr.
The model is a custom ASR stack:
- Whisper large-v3-turbo encoder
- Conv1d projector from Whisper encoder states to Sarashina hidden states
- Sarashina2.2 0.5B causal LM with a LoRA adapter
It is not directly loadable with AutoModel.from_pretrained. Use the included
realtime_infer_whisper_sarashina.py script or equivalent custom loading code.
Files
projector.pt: trained audio-to-LLM projectorllm_lora/: Sarashina LoRA adaptertokenizer/: tokenizer snapshot used for training and inferencemetadata.json and train_config.json: model architecture and training config*_results.json: training/evaluation summaries
Checkpoint directories and optimizer states are intentionally omitted from the
Hub upload. They are not required for inference and are much larger than the
final artifacts.
Usage
Install the project dependencies, plus sounddevice for microphone input:
On Linux/WSL, PortAudio must also be installed by the OS package manager
for microphone capture, for example sudo apt-get install libportaudio2.
Run microphone inference:
1uv run python realtime_infer_whisper_sarashina.py \
2 --model Kendamarron/whisper-sarashina-stage2-grouplr \
3 --window-seconds 8 \
4 --hop-seconds 2 \
5 --device auto
Run file inference:
1uv run python realtime_infer_whisper_sarashina.py \
2 --model Kendamarron/whisper-sarashina-stage2-grouplr \
3 --audio-file path/to/audio.wav
The real-time mode performs repeated sliding-window decoding. A CUDA GPU is
recommended for low-latency use.
Evaluation Snapshot
Final training/evaluation metrics:
- train loss: 0.7798
- eval loss: 1.1813 on 500 validation samples
Partial checkpoint CER/WER evaluation is stored in eval_all_checkpoints.json.
