facebook/wav2vec2-xls-r-300m with two-step, extended fine-tuning. The model was originally adapted to Finnish ASR with 1500 hours of speech from the Lahjoita puhetta (Donate Speech) corpus, followed by adding randomly initialized weights and bias terms in the final linear layer (language modeling head) for the 12 new characters introduced by the target Sámi data and fine-tuning on 20 hours of Sámi Parliament speech data on 16kHz sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.
Model description
The Sámi Wav2Vec2 Large has the same architecture and uses the same training objective as the English and multilingual one described in Paper.
You can read more about the pre-trained model from this paper. The training scripts are available on GitHub
Intended uses & limitations
You can use this model for Sámi ASR (speech-to-text).
How to use
To transcribe audio files the model can be used as a standalone acoustic model as follows:
from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC
from datasets import load_dataset
import torch
# load model and processor
processor = Wav2Vec2Processor.from_pretrained("GetmanY1/wav2vec2-xls-r-300m-sami-parl-ext-ft")
model = Wav2Vec2ForCTC.from_pretrained("GetmanY1/wav2vec2-xls-r-300m-sami-parl-ext-ft")
# load dummy dataset and read soundfiles
ds = load_dataset("mozilla-foundation/common_voice_16_1", "fi", split='test')
# tokenize
input_values = processor(ds[0]["audio"]["array"], return_tensors="pt", padding="longest").input_values # Batch size 1
# retrieve logits
logits = model(input_values).logits
# take argmax and decode
predicted_ids = torch.argmax(logits, dim=-1)
transcription = processor.batch_decode(predicted_ids)
Limitations and bias
This model was fine-tuned with audio samples whose maximum length was 30 seconds so this model most likely works the best for short audios of similar length. However, you can try this model with a lot longer audios too and see how it works. If you encounter out of memory errors with very long audio files you can use the audio chunking method introduced in this blog post.
The model was fine-tuned on the data from the Sámi Parliament speech data so this model might have biases towards formal Sámi.
Citation
If you use our models or scripts, please cite our article as:
bibtex
1@inproceedings{getman24b_interspeech,
2 author={Yaroslav Getman and Tamas Grosz and Katri Hiovain-Asikainen and Mikko Kurimo},
3 title={{Exploring adaptation techniques of large speech foundation models for low-resource ASR: a case study on Northern Sámi}},
4 year=2024,
5 booktitle={Proc. INTERSPEECH 2024},
6 pages={2539--2543},
7 doi={10.21437/Interspeech.2024-479},
8 issn={2958-1796}
9}
