This is a sentence-transformers model finetuned from laion/clap-htsat-fused on the librispeech_asr dataset. It maps sentences & paragraphs to a None-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: laion/clap-htsat-fused
Maximum Sequence Length: None tokens
Output Dimensionality: None dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- librispeech_asr
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'get_text_features', 'method_output_name': None}, 'audio': {'method': 'get_audio_features', 'method_output_name': None}}, 'module_output_name': 'sentence_embedding', 'architecture': 'ClapModel'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/clap-htsat-fused-librispeech")
# Run inference
sentences = [
    'THERE ARE NATURES TOO TO WHOSE SENSE OF JUSTICE THE PRICE EXACTED LOOMS UP MONSTROUSLY ENORMOUS ODIOUS OPPRESSIVE WORRYING HUMILIATING EXTORTIONATE INTOLERABLE THOSE ARE THE FANATICS',
    'HE BEGAN TO WISH THAT HE HAD COMPROMISED IN SOME WAY OR OTHER THAT HE HAD SENT THE MONEY PERHAPS HE COULD DO IT UP HERE',
    'HERE THE HOLY PRELATE OF FERNS MET HIM AND RELATED A VISION IN WHICH HE HAD BEEN INSTRUCTED TO DEMAND THE ABOLITION OF THE IMPOST',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000, -0.4742, -0.2719],
#         [-0.4742,  1.0000,  0.8206],
#         [-0.2719,  0.8206,  1.0000]])

Evaluation

Metrics

Information Retrieval

Datasets: librispeech-eval and librispeech-test
Evaluated with InformationRetrievalEvaluator

Metric	librispeech-eval	librispeech-test
cosine_accuracy@1	0.108	0.151
cosine_accuracy@3	0.196	0.288
cosine_accuracy@5	0.272	0.371
cosine_accuracy@10	0.438	0.518
cosine_precision@1	0.108	0.151
cosine_precision@3	0.0653	0.096
cosine_precision@5	0.0544	0.0742
cosine_precision@10	0.0438	0.0518
cosine_recall@1	0.108	0.151
cosine_recall@3	0.196	0.288
cosine_recall@5	0.272	0.371
cosine_recall@10	0.438	0.518
cosine_ndcg@10	0.2432	0.3132
cosine_mrr@10	0.1849	0.2505
cosine_map@100	0.206	0.2694

Training Details

Training Dataset

librispeech_asr

Dataset: librispeech_asr at 71cacbf
Size: 132,553 training samples
Columns: audio and text
Approximate statistics based on the first 1000 samples:
audio text
type dict string
details
min: 20 characters
mean: 189.15 characters
max: 294 characters

Samples:

audio	text
`{'path': '374-180298-0000.flac', 'array': array([ 6.92203816e-04, 8.04404495e-04, 8.03834875e-04, ...,[object Object] -3.02505396e-05, -6.59527450e-06, 1.11444592e-06]), 'sampling_rate': 48000}`	`CHAPTER SIXTEEN I MIGHT HAVE TOLD YOU OF THE BEGINNING OF THIS LIAISON IN A FEW LINES BUT I WANTED YOU TO SEE EVERY STEP BY WHICH WE CAME I TO AGREE TO WHATEVER MARGUERITE WISHED`
`{'path': '374-180298-0001.flac', 'array': array([-9.33515839e-05, -1.25754057e-04, -1.44482241e-04, ...,[object Object] -2.66165182e-04, -2.03228556e-04, -1.03404833e-04]), 'sampling_rate': 48000}`	`MARGUERITE TO BE UNABLE TO LIVE APART FROM ME IT WAS THE DAY AFTER THE EVENING WHEN SHE CAME TO SEE ME THAT I SENT HER MANON LESCAUT FROM THAT TIME SEEING THAT I COULD NOT CHANGE MY MISTRESS'S LIFE I CHANGED MY OWN`
`{'path': '374-180298-0002.flac', 'array': array([-2.47883319e-04, -2.91854434e-04, -2.82971043e-04, ...,[object Object] -1.43931946e-04, -1.17829914e-04, -6.32331648e-05]), 'sampling_rate': 48000}`	`I WISHED ABOVE ALL NOT TO LEAVE MYSELF TIME TO THINK OVER THE POSITION I HAD ACCEPTED FOR IN SPITE OF MYSELF IT WAS A GREAT DISTRESS TO ME THUS MY LIFE GENERALLY SO CALM`

Loss: MultipleNegativesSymmetricRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim", 4 "gather_across_devices": false 5}

Evaluation Dataset

librispeech_asr

Dataset: librispeech_asr at 71cacbf
Size: 1,000 evaluation samples
Columns: audio and text
Approximate statistics based on the first 1000 samples:
audio text
type dict string
details
min: 8 characters
mean: 104.62 characters
max: 516 characters

Samples:

audio	text
`{'path': '2277-149896-0000.flac', 'array': array([ 0.00179741, 0.00170625, 0.00120927, ..., -0.00144462,[object Object] -0.00102732, -0.00048062]), 'sampling_rate': 48000}`	`HE WAS IN A FEVERED STATE OF MIND OWING TO THE BLIGHT HIS WIFE'S ACTION THREATENED TO CAST UPON HIS ENTIRE FUTURE`
`{'path': '2277-149896-0001.flac', 'array': array([ 0.00111104, 0.00081758, 0.00021103, ..., -0.00138193,[object Object] -0.0009173 , -0.00041702]), 'sampling_rate': 48000}`	`HE WOULD HAVE TO PAY HER THE MONEY WHICH SHE WOULD NOW REGULARLY DEMAND OR THERE WOULD BE TROUBLE IT DID NOT MATTER WHAT HE DID`
`{'path': '2277-149896-0002.flac', 'array': array([0.00080266, 0.00088462, 0.00083408, ..., 0.00105488, 0.00083673,[object Object] 0.00043296]), 'sampling_rate': 48000}`	`HURSTWOOD WALKED THE FLOOR MENTALLY ARRANGING THE CHIEF POINTS OF HIS SITUATION`

Loss: MultipleNegativesSymmetricRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim", 4 "gather_across_devices": false 5}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 16
per_device_eval_batch_size: 16
learning_rate: 2e-05
num_train_epochs: 1
warmup_ratio: 0.1
bf16: True
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 16
per_device_eval_batch_size: 16
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 1
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
use_cpu: False
seed: 42
data_seed: None
jit_mode_eval: False
bf16: True
fp16: False
half_precision_backend: None
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_for_metrics: []
eval_do_concat_batches: True
mp_parameters:
auto_find_batch_size: False
full_determinism: False
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss	librispeech-eval_cosine_ndcg@10	librispeech-test_cosine_ndcg@10
-1	-1	-	-	0.0114	-
0.0100	83	3.5908	-	-	-
0.0200	166	2.5371	-	-	-
0.0301	249	2.1799	-	-	-
0.0401	332	2.0415	-	-	-
0.0501	415	1.9394	-	-	-
0.0601	498	1.8167	-	-	-
0.0701	581	1.7589	-	-	-
0.0801	664	1.7262	-	-	-
0.0902	747	1.7585	-	-	-
0.1001	829	-	1.5991	0.0335	-
0.1002	830	1.7521	-	-	-
0.1102	913	1.6822	-	-	-
0.1202	996	1.6176	-	-	-
0.1302	1079	1.6391	-	-	-
0.1403	1162	1.6931	-	-	-
0.1503	1245	1.4626	-	-	-
0.1603	1328	1.4305	-	-	-
0.1703	1411	1.4998	-	-	-
0.1803	1494	1.4073	-	-	-
0.1903	1577	1.3843	-	-	-
0.2001	1658	-	1.2227	0.0925	-
0.2004	1660	1.3371	-	-	-
0.2104	1743	1.3908	-	-	-
0.2204	1826	1.2835	-	-	-
0.2304	1909	1.3203	-	-	-
0.2404	1992	1.2549	-	-	-
0.2505	2075	1.2384	-	-	-
0.2605	2158	1.2189	-	-	-
0.2705	2241	1.1658	-	-	-
0.2805	2324	1.1771	-	-	-
0.2905	2407	1.2068	-	-	-
0.3002	2487	-	1.0471	0.1318	-
0.3005	2490	1.1708	-	-	-
0.3106	2573	1.1389	-	-	-
0.3206	2656	1.0786	-	-	-
0.3306	2739	1.0792	-	-	-
0.3406	2822	1.0562	-	-	-
0.3506	2905	0.98	-	-	-
0.3607	2988	1.1153	-	-	-
0.3707	3071	0.9987	-	-	-
0.3807	3154	1.0002	-	-	-
0.3907	3237	1.0017	-	-	-
0.4002	3316	-	0.8901	0.1589	-
0.4007	3320	0.9364	-	-	-
0.4107	3403	0.9394	-	-	-
0.4208	3486	0.9459	-	-	-
0.4308	3569	0.9604	-	-	-
0.4408	3652	0.9491	-	-	-
0.4508	3735	0.9295	-	-	-
0.4608	3818	0.9508	-	-	-
0.4709	3901	0.9122	-	-	-
0.4809	3984	0.8483	-	-	-
0.4909	4067	0.8443	-	-	-
0.5003	4145	-	0.7955	0.1908	-
0.5009	4150	0.8838	-	-	-
0.5109	4233	0.8367	-	-	-
0.5209	4316	0.8516	-	-	-
0.5310	4399	0.8112	-	-	-
0.5410	4482	0.8368	-	-	-
0.5510	4565	0.873	-	-	-
0.5610	4648	0.8156	-	-	-
0.5710	4731	0.8864	-	-	-
0.5811	4814	0.8278	-	-	-
0.5911	4897	0.8006	-	-	-
0.6004	4974	-	0.7649	0.1874	-
0.6011	4980	0.8199	-	-	-
0.6111	5063	0.7475	-	-	-
0.6211	5146	0.7345	-	-	-
0.6311	5229	0.7301	-	-	-
0.6412	5312	0.774	-	-	-
0.6512	5395	0.7391	-	-	-
0.6612	5478	0.6929	-	-	-
0.6712	5561	0.7218	-	-	-
0.6812	5644	0.7071	-	-	-
0.6912	5727	0.7024	-	-	-
0.7004	5803	-	0.6712	0.2419	-
0.7013	5810	0.6428	-	-	-
0.7113	5893	0.6719	-	-	-
0.7213	5976	0.6972	-	-	-
0.7313	6059	0.7043	-	-	-
0.7413	6142	0.663	-	-	-
0.7514	6225	0.6963	-	-	-
0.7614	6308	0.6591	-	-	-
0.7714	6391	0.6736	-	-	-
0.7814	6474	0.7033	-	-	-
0.7914	6557	0.6314	-	-	-
0.8005	6632	-	0.6806	0.2319	-
0.8014	6640	0.6508	-	-	-
0.8115	6723	0.6532	-	-	-
0.8215	6806	0.6788	-	-	-
0.8315	6889	0.6038	-	-	-
0.8415	6972	0.658	-	-	-
0.8515	7055	0.656	-	-	-
0.8616	7138	0.6533	-	-	-
0.8716	7221	0.601	-	-	-
0.8816	7304	0.6243	-	-	-
0.8916	7387	0.6315	-	-	-
0.9005	7461	-	0.6526	0.2432	-
0.9016	7470	0.5707	-	-	-
0.9116	7553	0.5778	-	-	-
0.9217	7636	0.5736	-	-	-
0.9317	7719	0.615	-	-	-
0.9417	7802	0.5756	-	-	-
0.9517	7885	0.5724	-	-	-
0.9617	7968	0.5678	-	-	-
0.9718	8051	0.5661	-	-	-
0.9818	8134	0.6162	-	-	-
0.9918	8217	0.5766	-	-	-
-1	-1	-	-	-	0.3132

Environmental Impact

Carbon emissions were measured using CodeCarbon.

Energy Consumed: 0.429 kWh
Carbon Emitted: 0.115 kg of CO2
Hours Used: 2.094 hours

Training Hardware

On Cloud: No
GPU Model: 1 x NVIDIA GeForce RTX 3090
CPU Model: 13th Gen Intel(R) Core(TM) i7-13700K
RAM Size: 31.78 GB

Framework Versions

Python: 3.11.6
Sentence Transformers: 5.2.0.dev0
Transformers: 4.57.0.dev0
PyTorch: 2.8.0+cu128
Accelerate: 1.6.0
Datasets: 3.6.0
Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

	audio	text
type	dict	string
details		min: 20 characters mean: 189.15 characters max: 294 characters