Allophant is a multilingual phoneme recognizer trained on spoken sentences in 34 languages, capable of generalizing zero-shot to unseen phoneme inventories.
The model is based on
facebook/wav2vec2-xls-r-300m and was pre-trained on a subset of the
Common Voice Corpus 10.0 transcribed with
eSpeak NG.
Note that our baseline models were trained without phonetic feature classifiers and therefore only support phoneme recognition.
1from allophant.estimator import Estimator
2
3device = "cpu"
4model, attribute_indexer = Estimator.restore("kgnlp/allophant", device=device)
5supported_features = attribute_indexer.feature_names
6# The phonetic feature categories supported by the model, including "phonemes"
7print(supported_features)
Allophant supports decoding custom phoneme inventories, which can be constructed in multiple ways:
1# 1. For a single language:
2inventory = attribute_indexer.phoneme_inventory("es")
3# 2. For multiple languages, e.g. in code-switching scenarios
4inventory = attribute_indexer.phoneme_inventory(["es", "it"])
5# 3. Any custom selection of phones for which features are available in the Allophoible database
6inventory = ['a', 'ai̯', 'au̯', 'b', 'e', 'eu̯', 'f', 'ɡ', 'l', 'ʎ', 'm', 'ɲ', 'o', 'p', 'ɾ', 's', 't̠ʃ']
Audio files can then be loaded, resampled and transcribed using the given
inventory by first computing the log probabilities for each classifier:
1import torch
2import torchaudio
3from allophant.dataset_processing import Batch
4
5# Load an audio file and resample the first channel to the sample rate used by the model
6audio, sample_rate = torchaudio.load("utterance.wav")
7audio = torchaudio.functional.resample(audio[:1], sample_rate, model.sample_rate)
8
9# Construct a batch of 0-padded single channel audio, lengths and language IDs
10# Language ID can be 0 for inference
11batch = Batch(audio, torch.tensor([audio.shape[1]]), torch.zeros(1))
12model_outputs = model.predict(
13 batch.to(device),
14 attribute_indexer.composition_feature_matrix(inventory).to(device)
15)
Finally, the log probabilities can be decoded into the recognized phonemes or phonetic features:
1from allophant import predictions
2
3# Create a feature mapping for your inventory and CTC decoders for the desired feature set
4inventory_indexer = attribute_indexer.attributes.subset(inventory)
5ctc_decoders = predictions.feature_decoders(inventory_indexer, feature_names=supported_features)
6
7for feature_name, decoder in ctc_decoders.items():
8 decoded = decoder(model_outputs.outputs[feature_name].transpose(1, 0), model_outputs.lengths)
9 # Print the feature name and values for each utterance in the batch
10 for [hypothesis] in decoded:
11 # NOTE: token indices are offset by one due to the <BLANK> token used during decoding
12 recognized = inventory_indexer.feature_values(feature_name, hypothesis.tokens - 1)
13 print(feature_name, recognized)
1@inproceedings{glocker2023allophant,
2 title={Allophant: Cross-lingual Phoneme Recognition with Articulatory Attributes},
3 author={Glocker, Kevin and Herygers, Aaricia and Georges, Munir},
4 year={2023},
5 booktitle={{Proc. Interspeech 2023}},
6 month={8}}
