Views
No views yet
pip install glap_modelscore_glap audio_input_file text1;text2;text31import torch
2from glap_model import glap_inference
3
4audio = torch.randn(1, 160000).tanh() # 10s of heavy noise
5
6glap_model = glap_inference()
7
8score = glap_model.score_forward(audio, text=["the sound of noise","a car is driving","a person is speaking"])
9print(score)| Task | Prompt |
|---|---|
| Speech | {label} |
| Music | The music in the style of {label}. |
| Sound | The sound of {label} can be heard. |
1import torch
2from glap_model import glap_inference
3
4glap_model = glap_inference()
5audio = torch.randn(1, 64000).tanh()
6prefix = "The sound of"
7labels = [ f"{prefix} {label}" for label in ("Cat","Dog","Water","Noise")]
8text_embeds = glap_model.encode_text(labels)
9audio_embeds = glap_model.encode_audio(audio)
10scores = glap_model.score(audio_embeds, text_embeds)
11for label_name, score in zip(labels, scores):
12 print(label_name,score)
13
141git clone https://github.com/xiaomi-research/GLAP
2cd GLAP
3uv venv --python 3.10
4source activate .venv/bin/activate
5uv sync
6
7#python3 -m pip install .
8# Additionally, sndfile is needed
9# conda install -c conda-forge libsndfile==1.0.311git clone https://github.com/xiaomi-research/GLAP
2cd GLAP
3python3 -m pip install .
4# Additionally, sndfile is needed
5# conda install -c conda-forge libsndfile==1.0.31
6# Or if you have root, use your package managertar/tar.gz format: # tar -tf a.tar
908-31957-0013.flac
908-31957-0013.json
2961-960-0013.flac
2961-960-0013.json.json should have one of three fields caption, captions or text.
Data preparation can be done using the wavlist_to_tar script, which is provided in the dasheng dependency.
Further information how to process data can be seen here.configs/train/multilingual_dasheng_asr_sound2_sigmoidloss_balanced.yaml.accelerate launch --mixed-precision='fp16' run.py train configs/train/multilingual_dasheng_asr_sound2_sigmoidloss_balanced.yaml1# There ; is a separator for different text keys
2python3 run.py zeroshot pretrained_checkpoint/glap_checkpoint.pt PATH_TO_WAV_FLAC_MP3_SAMPLE.wav "The sound of a horse;Car;Mama;The sound of music;somebody is speaking;The sound of ein Pferd;一只马;Music is played;音乐的声音;Musik ist zu hoeren";Zero;One;Two;Three"1# Should be run on a single GPU
2accelerate launch --mixed-precision='fp16' run.py evaluate PATH_TO_CHECKPOINTresample=True is not recommendedpython3 run.py translate_sonar data/WavCaps/freesound/freesound_train_sample_0000* --output_path data/translations/WavCaps/freesound/accelerate launch run.py translate_sonar data/WavCaps/freesound/freesound_train_sample_0000* --output_path data/translations/WavCaps/freesound/1@inproceedings{dinkel2025glap,
2 title={GLAP: General contrastive audio-text pretraining across domains and languages},
3 year={2025}
4}