We are open-sourcing our Conformer-based
W2v-BERT 2.0 speech encoder as described in Section 3.2.1 of the
paper, which is at the core of our Seamless models.
This model was pre-trained on 4.5M hours of unlabeled audio data covering more than 143 languages. It requires finetuning to be used for downstream tasks such as Automatic Speech Recognition (ASR), or Audio Classification.
This is a bare checkpoint without any modeling head, and thus requires finetuning to be used for downstream tasks such as ASR. You can however use it to extract audio embeddings from the top layer with this code snippet:
1from transformers import AutoFeatureExtractor, Wav2Vec2BertModel
2import torch
3from datasets import load_dataset
4
5dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
6dataset = dataset.sort("id")
7sampling_rate = dataset.features["audio"].sampling_rate
8
9processor = AutoProcessor.from_pretrained("facebook/w2v-bert-2.0")
10model = Wav2Vec2BertModel.from_pretrained("facebook/w2v-bert-2.0")
11
12# audio file is decoded on the fly
13inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
14with torch.no_grad():
15 outputs = model(**inputs)
This model can be used in
Seamless Communication, where it was released.
Here's how to make a forward pass through the voice encoder, after having completed the
installation steps:
1import torch
2
3from fairseq2.data.audio import AudioDecoder, WaveformToFbankConverter
4from fairseq2.memory import MemoryBlock
5from fairseq2.nn.padding import get_seqs_and_padding_mask
6from pathlib import Path
7from seamless_communication.models.conformer_shaw import load_conformer_shaw_model
8
9
10audio_wav_path, device, dtype = ...
11audio_decoder = AudioDecoder(dtype=torch.float32, device=device)
12fbank_converter = WaveformToFbankConverter(
13 num_mel_bins=80,
14 waveform_scale=2**15,
15 channel_last=True,
16 standardize=True,
17 device=device,
18 dtype=dtype,
19)
20collater = Collater(pad_value=1)
21
22model = load_conformer_shaw_model("conformer_shaw", device=device, dtype=dtype)
23model.eval()
24
25with Path(audio_wav_path).open("rb") as fb:
26 block = MemoryBlock(fb.read())
27
28decoded_audio = audio_decoder(block)
29src = collater(fbank_converter(decoded_audio))["fbank"]
30seqs, padding_mask = get_seqs_and_padding_mask(src)
31
32with torch.inference_mode():
33 seqs, padding_mask = model.encoder_frontend(seqs, padding_mask)
34 seqs, padding_mask = model.encoder(seqs, padding_mask)
