WhisperD is a fine-tuned version of whisper-large-v2 that is able to transcribe multi-speaker, conversational speech. It was used to generate synthetic transcriptions for training
Parakeet. Diarization is performed implicitly by the model, where "[S1]", "[S2]", etc. denote speaker identity. WhisperD is (often) able to transcribe non-speech events, e.g. "(coughs)", "(laughs)". Outputs include disfluencies.
More details can be found in the
WhisperD blog post.
This model has only been tested on segments up to 30 seconds in length. It may be unable to handle conditioning on previous text, as this was not included during fine-tuning. Thus, if a pipeline / codebase uses this feature in order to transcribe audio with duration over 30 seconds, generation quality may be poor.
1import torch
2import torchaudio
3from transformers import WhisperForConditionalGeneration, WhisperProcessor, WhisperTokenizer
4
5model = WhisperForConditionalGeneration.from_pretrained('jordand/whisper-d-v1a', torch_dtype=torch.float16).cuda()
6processor = WhisperProcessor.from_pretrained('openai/whisper-large-v2')
7tokenizer = WhisperTokenizer.from_pretrained('openai/whisper-large-v2')
8
9model.generation_config.suppress_tokens = None
10model.generation_config.forced_decoder_ids = None
11
12audio, sr = torchaudio.load('PATH_TO_AUDIO_FILE')
13audio = audio.mean(dim=0, keepdim=True)
14audio = torchaudio.transforms.Resample(sr, 16000)(audio)
15audio = audio[0, :16000*30] # whisper-d-v1 only can handle up to 30 seconds of audio
16
17inputs = processor(audio, return_tensors="pt")
18model_out = model.generate(inputs['input_features'].cuda().half())
19text = tokenizer.decode(model_out[0], skip_special_tokens=True)
20print(text)
1@misc{darefsky2024parakeet,
2 author = {Darefsky, Jordan and Zhu, Ge and Duan, Zhiyao},
3 title = {Parakeet},
4 year = {2024},
5 url = {https://jordandarefsky.com/blog/2024/parakeet/}
6}
