You can deploy this model and start earning money today!

llm-lab/
CLASP

automatic-speech-recognition

mozilla-foundation/common_voice_4_0

Views

No views yet

arXiv

GitHub

CLASP (Contrastive Language-Speech Pretraining) is a novel, lightweight, multilingual, multimodal representation designed for audio-text retrieval.
To learn more about our proposed model, please refer to this paper, which is published at ECIR 2025. All code is available on this GitHub page.
The newly introduced dataset, SpeechBrown, which we created for training this model, can be found on this page

CLASP creates powerful and meaningful semantic embeddings for raw speech in a 768-dimensional multilingual representation space. These embeddings can be used in various tasks such as speech retrieval or classification.

Here, we have uploaded different versions of the CLASP and LASP models we trained:

CLASP_Concat_Final_Fusion_Encoder.pt: The best model we trained based on retrieval and classification metrics. It uses the concatenation fusion encoder strategy and is trained with contrastive loss.
CLASP_Gating.pt: Trained with contrastive loss and employs a gating algorithm.
LASP_Concat.pt: Trained with Huber loss and employs the concatenation strategy.
LASP_Gating.pt: Trained with Huber loss and employs the gating algorithm.

To use these models or train your own on custom datasets, please refer to our GitHub page. The clasp-inference.ipynb notebook provides an example of loading and using the model.

Steps for Inference with CLASP:

Load our model with the specified architecture.
Load the EfficientNet encoder for spectrogram encoding.
Load the HuBERT encoder for self-supervised speech encoding.
Use our scripts to generate embeddings for your audio files.
Use these embeddings for tasks like classification or speech retrieval. For speech retrieval, you can load the LaBSE sentence transformer to compute the cosine similarity between query and speech embeddings.

Architecture Overview

Contributions

We introduce CLASP (Contrastive Language-Speech Pretraining), a novel, lightweight, multilingual, multimodal representation designed for audio-text retrieval.
We present a diverse paired speech-text dataset (Speech Brown) in 15 categories, covering a wide range of topics from fiction to religion.
We demonstrate that combining audio spectrograms with a pre-trained self-supervised speech model enhances audio encoding in retrieval applications.
Evaluations in multiple languages show that CLASP achieves new benchmarks in HITS@1, Mean Reciprocal Rank (MRR), and Mean Rank (meanR) metrics.

Citations

If you find our paper, code, data, or models useful, please cite the paper:

@inproceedings{10.1007/978-3-031-88717-8_2,
                author = {Abootorabi, Mohammad Mahdi and Asgari, Ehsaneddin},
                title = {CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrieval},
                year = {2025},
                isbn = {978-3-031-88716-1},
                publisher = {Springer-Verlag},
                address = {Berlin, Heidelberg},
                url = {https://doi.org/10.1007/978-3-031-88717-8_2},
                doi = {10.1007/978-3-031-88717-8_2},
                abstract = {This study introduces CLASP (Contrastive Language-Speech Pretraining), a multilingual, multimodal representation tailored for audio-text information retrieval. CLASP leverages the synergy between spoken content and textual data. During training, we utilize our newly introduced speech-text dataset, which encompasses 15 diverse categories ranging from fiction to religion. CLASP’s audio component integrates audio spectrograms with a pre-trained self-supervised speech model, while its language encoding counterpart employs a sentence encoder pre-trained on over 100 languages. This unified lightweight model bridges the gap between various modalities and languages, enhancing its effectiveness in handling and retrieving multilingual and multimodal data. Our evaluations across multiple languages demonstrate that CLASP establishes new benchmarks in HITS@1, MRR, and meanR metrics, outperforming traditional ASR-based retrieval methods that rely on transcribing speech into text for subsequent text retrieval, especially in specific scenarios.},
                booktitle = {Advances in Information Retrieval: 47th European Conference on Information Retrieval, ECIR 2025, Lucca, Italy, April 6–10, 2025, Proceedings, Part IV},
                pages = {10–20},
                numpages = {11},
                keywords = {Multimodal IR, Speech Retrieval, Contrastive Learning},
                location = {Lucca, Italy}
}

Contact

If you have questions, please email mahdi.abootorabi2@gmail.com or asgari@berkeley.edu.

llm-lab/CLASP