This is a sentence-transformers model finetuned from sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

License

The base model (paraphrase-multilingual-MiniLM-L12-v2) is Apache-2.0. However, this checkpoint was fine-tuned on 2ADT-Consulting/susu-parallel, which is dominated by Bible-derived text retrieved from YouVersion and subject to YouVersion's terms of service independently of any model license. Out of caution, and for consistency with our other Susu models trained on the same corpus (nllb-susu-v1/v2, both CC-BY-NC-4.0), we release these weights under CC-BY-NC-4.0 pending clearer guidance on whether fine-tuning on restricted-license text carries those restrictions to the resulting weights. Review the dataset's license notes before commercial use.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
Maximum Sequence Length: 128 tokens
Output Dimensionality: 384 dimensions
Similarity Function: Cosine Similarity
Supported Modality: Text

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Les navires de Hiram, qui apportèrent de lor dOphir, amenèrent aussi dOphir une grande quantité de bois de sandal et des pierres précieuses.',
    'Xirami nun Sulemani xa walikɛe naxee fa xɛɛma ra kelife Ofiri bɔxi ma, nee man naxa fa wuri nun gɛmɛ tofanyie ra.',
    'Won a kolon won nun Ala na a ra, barima a bara a Xaxili fi won ma.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6592, 0.1617],
#         [0.6592, 1.0000, 0.0717],
#         [0.1617, 0.0717, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

Size: 225,516 training samples
Columns: anchor and positive
Approximate statistics based on the first 100 samples:
anchor positive
type string string
modality text text
details
min: 13 tokens
mean: 38.24 tokens
max: 93 tokens
min: 10 tokens
mean: 50.44 tokens
max: 115 tokens

Samples:

anchor	positive
`He made it according to the instructions that Moses, the Lord 's servant, had given the Israelites, as it says in the Law of Moses: an altar made of stones which have not been cut with iron tools. On it they offered burnt sacrifices to the Lord , and they also presented their fellowship offerings.`	`alɔ Alatala xa konyi Munsa a fala Isirayilakae bɛ ki naxɛ. E naxa sɛrɛxɛbade gɛmɛ daaxi ti, wure mu nu din naxan na, alɔ a sɛbɛxi Tawureta Munsa kitaabui kui ki naxɛ. E naxa sɛrɛxɛ gan daaxie nun xanunteya sɛrɛxɛe ba Alatala bɛ na fari.`
`Celle-ci ordonna à Hathac de rapporter sa réponse à Mardochée :.`	`Esita to na mɛ, a man naxa Hataki xɛɛ Morodekayi xɔn ma, a xa sa yi fala a bɛ.`
`But there is something you do that is right—you hate the things that the Nicolaitans do. I also hate what they do.`	`«Kɔnɔ wo bara fe nde raba naxan fan. Wo bara Nikolasi xa ɲama xa fe xɔn, alɔ n fan a xɔnxi ki naxɛ.».`

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim", 4 "gather_across_devices": false, 5 "directions": [ 6 "query_to_doc" 7 ], 8 "partition_mode": "joint", 9 "hardness_mode": null, 10 "hardness_strength": 0.0 11}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 64
learning_rate: 2e-05
num_train_epochs: 4
warmup_steps: 0.1
fp16: True

All Hyperparameters

Click to expand

do_predict: False
prediction_loss_only: True
per_device_train_batch_size: 64
per_device_eval_batch_size: 8
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 4
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_ratio: None
warmup_steps: 0.1
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
enable_jit_checkpoint: False
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
use_cpu: False
seed: 42
data_seed: None
bf16: False
fp16: True
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: -1
ddp_backend: None
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
group_by_length: False
length_column_name: length
project: huggingface
trackio_space_id: trackio
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_for_metrics: []
eval_do_concat_batches: True
auto_find_batch_size: False
full_determinism: False
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
use_cache: False
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0284	50	5.0351
0.0568	100	4.5387
0.0851	150	4.0088
0.1135	200	3.2828
0.1419	250	2.6326
0.1703	300	2.0852
0.1986	350	1.6870
0.2270	400	1.4379
0.2554	450	1.2296
0.2838	500	1.0465
0.3121	550	0.8908
0.3405	600	0.8758
0.3689	650	0.7199
0.3973	700	0.6701
0.4257	750	0.6193
0.4540	800	0.5435
0.4824	850	0.5279
0.5108	900	0.4810
0.5392	950	0.4594
0.5675	1000	0.4344
0.5959	1050	0.3961
0.6243	1100	0.3767
0.6527	1150	0.3601
0.6810	1200	0.3690
0.7094	1250	0.3448
0.7378	1300	0.3396
0.7662	1350	0.3242
0.7946	1400	0.3236
0.8229	1450	0.2961
0.8513	1500	0.2853
0.8797	1550	0.2933
0.9081	1600	0.2625
0.9364	1650	0.2758
0.9648	1700	0.2617
0.9932	1750	0.2720
1.0216	1800	0.2535
1.0499	1850	0.2366
1.0783	1900	0.2097
1.1067	1950	0.2183
1.1351	2000	0.2201
1.1635	2050	0.2284
1.1918	2100	0.2259
1.2202	2150	0.2125
1.2486	2200	0.2059
1.2770	2250	0.1950
1.3053	2300	0.2066
1.3337	2350	0.1944
1.3621	2400	0.2019
1.3905	2450	0.2051
1.4188	2500	0.1903
1.4472	2550	0.1958
1.4756	2600	0.1869
1.5040	2650	0.1827
1.5323	2700	0.1804
1.5607	2750	0.1692
1.5891	2800	0.2033
1.6175	2850	0.1740
1.6459	2900	0.1810
1.6742	2950	0.1785
1.7026	3000	0.1737
1.7310	3050	0.1914
1.7594	3100	0.1779
1.7877	3150	0.1670
1.8161	3200	0.1744
1.8445	3250	0.1647
1.8729	3300	0.1720
1.9012	3350	0.1746
1.9296	3400	0.1559
1.9580	3450	0.1571
1.9864	3500	0.1655
2.0148	3550	0.1342
2.0431	3600	0.1304
2.0715	3650	0.1335
2.0999	3700	0.1290
2.1283	3750	0.1486
2.1566	3800	0.1274
2.1850	3850	0.1384
2.2134	3900	0.1257
2.2418	3950	0.1277
2.2701	4000	0.1372
2.2985	4050	0.1329
2.3269	4100	0.1347
2.3553	4150	0.1332
2.3837	4200	0.1322
2.4120	4250	0.1208
2.4404	4300	0.1311
2.4688	4350	0.1248
2.4972	4400	0.1246
2.5255	4450	0.1155
2.5539	4500	0.1243
2.5823	4550	0.1160
2.6107	4600	0.1143
2.6390	4650	0.1275
2.6674	4700	0.1258
2.6958	4750	0.1196
2.7242	4800	0.1068
2.7526	4850	0.1167
2.7809	4900	0.1181
2.8093	4950	0.1057
2.8377	5000	0.1169
2.8661	5050	0.1287
2.8944	5100	0.1108
2.9228	5150	0.1110
2.9512	5200	0.1145
2.9796	5250	0.1161
3.0079	5300	0.1172
3.0363	5350	0.1005
3.0647	5400	0.0977
3.0931	5450	0.1045
3.1215	5500	0.1021
3.1498	5550	0.1059
3.1782	5600	0.1026
3.2066	5650	0.0998
3.2350	5700	0.0982
3.2633	5750	0.1003
3.2917	5800	0.1015
3.3201	5850	0.0966
3.3485	5900	0.0971
3.3768	5950	0.1033
3.4052	6000	0.1001
3.4336	6050	0.0942
3.4620	6100	0.1028
3.4904	6150	0.0934
3.5187	6200	0.0918
3.5471	6250	0.0993
3.5755	6300	0.0943
3.6039	6350	0.1046
3.6322	6400	0.0941
3.6606	6450	0.0999
3.6890	6500	0.0998
3.7174	6550	0.0987
3.7457	6600	0.1078
3.7741	6650	0.0972
3.8025	6700	0.1008
3.8309	6750	0.1070
3.8593	6800	0.0878
3.8876	6850	0.0966
3.9160	6900	0.0909
3.9444	6950	0.0991
3.9728	7000	0.0969

Training Time

Training: 2.5 hours

Framework Versions

Python: 3.12.13
Sentence Transformers: 5.6.1
Transformers: 5.0.0
PyTorch: 2.10.0+cu128
Accelerate: 1.14.0
Datasets: 5.0.1
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss