Using this open-source model in production?
Consider switching to
pyannoteAI for better and faster options.
This model ingests 10 seconds of mono audio sampled at 16kHz and outputs speaker diarization as a (num_frames, num_classes) matrix where the 7 classes are non-speech, speaker #1, speaker #2, speaker #3, speakers #1 and #2, speakers #1 and #3, and speakers #2 and #3.
1# waveform (first row)
2duration, sample_rate, num_channels = 10, 16000, 1
3waveform = torch.randn(batch_size, num_channels, duration * sample_rate)
4
5# powerset multi-class encoding (second row)
6powerset_encoding = model(waveform)
7
8# multi-label encoding (third row)
9from pyannote.audio.utils.powerset import Powerset
10max_speakers_per_chunk, max_speakers_per_frame = 3, 2
11to_multilabel = Powerset(
12 max_speakers_per_chunk,
13 max_speakers_per_frame).to_multilabel
14multilabel_encoding = to_multilabel(powerset_encoding)
The various concepts behind this model are described in details in this
paper.
It has been trained by Séverin Baroudi with
pyannote.audio 3.0.0 using the combination of the training sets of AISHELL, AliMeeting, AMI, AVA-AVD, DIHARD, Ego4D, MSDWild, REPERE, and VoxConverse.
This
companion repository by
Alexis Plaquet also provides instructions on how to train or finetune such a model on your own data.
1# instantiate the model
2from pyannote.audio import Model
3model = Model.from_pretrained(
4 "pyannote/segmentation-3.0",
5 use_auth_token="HUGGINGFACE_ACCESS_TOKEN_GOES_HERE")
This model cannot be used to perform speaker diarization of full recordings on its own (it only processes 10s chunks).
See
pyannote/speaker-diarization-3.0 pipeline that uses an additional speaker embedding model to perform full recording speaker diarization.
1from pyannote.audio.pipelines import VoiceActivityDetection
2pipeline = VoiceActivityDetection(segmentation=model)
3HYPER_PARAMETERS = {
4 # remove speech regions shorter than that many seconds.
5 "min_duration_on": 0.0,
6 # fill non-speech regions shorter than that many seconds.
7 "min_duration_off": 0.0
8}
9pipeline.instantiate(HYPER_PARAMETERS)
10vad = pipeline("audio.wav")
11# `vad` is a pyannote.core.Annotation instance containing speech regions
1from pyannote.audio.pipelines import OverlappedSpeechDetection
2pipeline = OverlappedSpeechDetection(segmentation=model)
3HYPER_PARAMETERS = {
4 # remove overlapped speech regions shorter than that many seconds.
5 "min_duration_on": 0.0,
6 # fill non-overlapped speech regions shorter than that many seconds.
7 "min_duration_off": 0.0
8}
9pipeline.instantiate(HYPER_PARAMETERS)
10osd = pipeline("audio.wav")
11# `osd` is a pyannote.core.Annotation instance containing overlapped speech regions
1@inproceedings{Plaquet23,
2 author={Alexis Plaquet and Hervé Bredin},
3 title={{Powerset multi-class cross entropy loss for neural speaker diarization}},
4 year=2023,
5 booktitle={Proc. INTERSPEECH 2023},
6}
1@inproceedings{Bredin23,
2 author={Hervé Bredin},
3 title={{pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe}},
4 year=2023,
5 booktitle={Proc. INTERSPEECH 2023},
6}
