SenseVoice is an audio foundation model with audio understanding capabilities, including Automatic Speech Recognition (ASR), Language Identification (LID), Speech Emotion Recognition (SER), and Acoustic Event Classification (AEC) or Acoustic Event Detection (AED).
Currently, SenseVoice-small supports multilingual speech recognition, emotion recognition, and event detection for Chinese, Cantonese, English, Japanese, and Korean, with extremely low inference latency.
Inference speed (RKNN2): About 20x real-time on a single NPU core of RK3588 (processing 20 seconds of audio per second), approximately 6 times faster than the official whisper model provided in the rknn-model-zoo.
If you find that recognition is not working correctly when testing with your own audio files, you may need to convert them to 16kHz, 16-bit, mono WAV format in advance.
You need to install rknn-toolkit2 in advance. The tested working version is 2.3.3a25, which can be downloaded from https://console.zbox.filez.com/l/I00fc3 (password: "rknn").
The model file should be named 'sense-voice-encoder.onnx' and placed in the same directory as the conversion script.
Convert to RKNN model
python convert_rknn.py ./sense-voice-encoder.onnx
Known Issues
When using fp16 inference with RKNN2, overflow may occur, resulting in inf values. You can try modifying the scaling ratio of the input data to resolve this.
Set SPEECH_SCALE to a smaller value in sensevoice_rknn.py. (This issue should now be resolved by inserting scaling operators inside the model.)
