Recently, large pre-trained multilingual speech models have shown potential in scaling Automatic Speech Recogni- tion (ASR) to many low-resource languages. Some of these models employ language adapters in their formulation, which helps to improve monolingual performance and avoids some of the drawbacks of multi-lingual modeling on resource-rich languages. However, this formulation restricts the usability of these models on code-switched speech, where two lan- guages are mixed together in the same utterance. In this work, we propose ways to effectively fine-tune such mod- els on code-switched speech, by assimilating information from both language adapters at each language adaptation point in the network. We also model code-switching as a sequence of latent binary sequences that can be used to guide the flow of information from each language adapter at the frame level. The proposed approaches are evaluated on three code-switched datasets encompassing Arabic, Mandarin, and Hindi languages paired with English, showing consistent im- provements in code-switching performance with at least 10% absolute reduction in CER across all test sets.

We modify the Wav2Vec2 transformer blocks used in MMS to use 2 pretrained adapter modules corresponding to the matrix and embedded languages to incorporate information from both. Based on this modification, we propose two code-switching approaches:

We add a Post-Adapyter-Code-Switcher network (PACS) inside every transformer block after the 2 adapter modules (see Figure 1a) . Output from the adapter modules is concatenated and fed to PACS which learns to assimilate information from both. The base model and the 2 pretrained adapter modules are kept frozen during the training hence only PACS and the output layer is trainable. PACS follows the same architectures as the adapter modules used in MMS: two feedforward layers with a LayerNorm layer and a linear projection to 16 dimensions with ReLU activation

We use a transformer network with sigmoid as output activation as a Transformer Code Switcher (TCS). It learns to predict a code-switch-sequence O _CS using output of the Wav2Vec2 Feature Projection block (Figure 1b). The code-switch-sequence is a latent binary sequence that helps to identify code-switching boundaries at frame level. It regulates the flow of information from two adapters to enable the network to handle code-switched speech by dynamically masking out one of the languages as per the switching equation :

We use a threshold value of 0.5 to the output of the sigmoid activation to create binarized latent codes O _CS. The base model and adapter are kept frozen, only TCS and the output layers are trained on code-switched data.

Clone this repository

NOTE: This repo includes the entire codebase of hugging face transformers. We write our modifications on top of their codebase. Most of our modified code is in this file.

First we recommend creating a new conda environment especially if you have transformers already installed. We will be installing modified code for the transformers library from this repo which can cause conflicts with your existing installation. Hence create and activate new environment using

We also provide MMS checkpoints after finetuning matrix-language adapters on the 3 datasets. NOTE: In order to do inference on these finetuned checkpoints, one should use standard implementation of MMS from huggingface instead of our modified transformers code.

Use main branch for Transformer Code Switching (TCS) and post-adapter-switching branch for Post Adapter Code Swtiching (PACS).

Check demo.ipynb for inference demo.

We also share transcripts generated by our proposed systems on the 3 datasets in generated_transcripts/.