This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: sentence-transformers/all-MiniLM-L6-v2
Maximum Sequence Length: 512 tokens
Output Dimensionality: 384 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("along26/all-MiniLM-L6-v2_multilingual_malaysian-v2")
# Run inference
sentences = [
    'How can we design small molecule inhibitors of viral protein targets to prevent the replication of the influenza virus?',
    'Bagaimanakah kita boleh mereka bentuk perencat molekul kecil sasaran protein virus untuk mencegah replikasi virus influenza?',
    "How does the Malaysian government's authoritarian approach to dissent and free speech stifle progressive movements and limit the potential for democratic reform?",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.0462, 0.9062],
#         [0.0462, 1.0000, 0.0619],
#         [0.9062, 0.0619, 1.0000]])

Evaluation

Metrics

Triplet

Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.0044

Training Details

Training Dataset

Unnamed Dataset

Size: 415,570 training samples
Columns: anchor, positive, and negative
Approximate statistics based on the first 1000 samples:
anchor positive negative
type string string string
details
min: 15 tokens
mean: 230.67 tokens
max: 512 tokens
min: 19 tokens
mean: 273.54 tokens
max: 512 tokens
min: 14 tokens
mean: 239.81 tokens
max: 512 tokens

Samples:

anchor	positive	negative
`How has the culture of corruption and cronyism under Najib Razak's administration affected the Malaysian economy and social fabric?`	`Bagaimanakah budaya rasuah dan kronisme di bawah pentadbiran Najib Razak menjejaskan ekonomi dan fabrik sosial Malaysia?`	`What is the role of the pancreas in the human digestive system and how does its anatomy support this function?`
`Why have some opposition politicians in Malaysia criticized the government's handling of the 1MDB scandal and called for more transparency?`	`Mengapa beberapa ahli politik pembangkang di Malaysia mengkritik pengendalian kerajaan terhadap skandal 1MDB dan meminta lebih ketelusan?`	The formation of heavy elements (nucleosynthesis) inside a star involves several processes, which are affected by the star's evolution. These processes include:[object Object][object Object]1. Hydrogen burning (nuclear fusion): This is the initial stage of a star's life, where hydrogen nuclei (protons) combine to form helium nuclei (alpha particles) through a series of reactions called the proton-proton chain or the CNO cycle (carbon-nitrogen-oxygen). This process releases a large amount of energy in the form of light and heat, which causes the star to shine.[object Object][object Object]2. Helium burning (triple-alpha process): As the hydrogen in the core of the star is depleted, the core contracts and heats up, initiating the fusion of helium nuclei into heavier elements like carbon and oxygen. This process involves the combination of three helium nuclei (alpha particles) to form a carbon nucleus.[object Object][object Object]3. Carbon burning: In more massive stars, the core temperature increases further, allowing carbon nuclei to fuse with helium nuclei to form ox...
`How has Najib Razak's corruption allegedly contributed to social inequality and poverty in Malaysia?`	`Bagaimanakah rasuah Najib Razak didakwa menyumbang kepada ketidaksamaan sosial dan kemiskinan di Malaysia?`	To estimate the age of a supermassive black hole with a mass of 1 billion solar masses, we can assume that it formed shortly after the Big Bang. The age of the universe is approximately 13.8 billion years old, so the black hole's age would be close to this value.[object Object][object Object]Now, let's discuss the theoretical process for the formation and evolution of supermassive black holes in the early universe based on current astrophysical models.[object Object][object Object]1. Direct collapse: In the early universe, some regions with high-density gas could have collapsed directly into black holes without forming stars first. These black holes, called "seed" black holes, could then grow by accreting mass from their surroundings. This process is more efficient in the early universe due to the higher density of gas and the absence of supernova explosions, which can disperse gas and hinder black hole growth.[object Object][object Object]2. Stellar remnants: Massive stars in the early universe could have collapsed into black holes after their lifetimes. These black ...

Loss: TripletLoss with these parameters:
json
1{ 2 "distance_metric": "TripletDistanceMetric.EUCLIDEAN", 3 "triplet_margin": 5 4}

Evaluation Dataset

Unnamed Dataset

Size: 5,000 evaluation samples
Columns: anchor, positive, and negative
Approximate statistics based on the first 1000 samples:
anchor positive negative
type string string string
details
min: 12 tokens
mean: 219.95 tokens
max: 512 tokens
min: 17 tokens
mean: 263.51 tokens
max: 512 tokens
min: 15 tokens
mean: 236.31 tokens
max: 512 tokens

Samples:

anchor	positive	negative
`Consider a graph with 8 vertices and 12 edges. Determine if the graph contains a perfect matching. If it does, provide one example of a perfect matching. If it does not, explain why a perfect matching is not possible.`	`Pertimbangkan graf dengan 8 bucu dan 12 tepi. Tentukan sama ada graf mengandungi padanan sempurna. Jika ya, berikan satu contoh padanan yang sempurna. Jika tidak, jelaskan mengapa padanan yang sempurna tidak dapat dilakukan.`	The 1MDB scandal and the corruption charges against former Malaysian Prime Minister Najib Razak offer several important lessons for Malaysia and other countries:[object Object][object Object]1. Stronger checks and balances: The 1MDB scandal highlighted the need for stronger checks and balances in government agencies and institutions. Malaysia should consider implementing additional measures to prevent the misuse of power and public funds, including more robust auditing and oversight mechanisms, as well as stronger whistleblower protections.[object Object]2. Transparency and accountability: The lack of transparency and accountability surrounding 1MDB contributed to the scandal. Malaysia should prioritize transparency in government operations, including procurement processes and financial transactions. Implementing measures such as open data initiatives and requiring greater disclosure from government-linked companies could help promote accountability and reduce opportunities for corruption.[object Object]3. Strengthening law enforcement: The...
`What is the probability of flipping a fair coin three times and getting exactly two heads in a row?`	`Apakah kebarangkalian membalikkan syiling saksama tiga kali dan mendapat tepat dua kepala berturut-turut?`	`Why is corruption so rampant in Malaysia, with politicians and government officials often caught engaging in unethical practices?`
`Why have there been allegations of corruption and mismanagement in Malaysia's state-owned enterprises, and what measures have been taken to address these issues?`	`Mengapa terdapat tuduhan rasuah dan salah urus dalam perusahaan milik kerajaan Malaysia, dan apakah langkah-langkah yang telah diambil untuk menangani isu-isu ini?`	`What is the pKa value of acetic acid, and how does it affect its acid strength when compared to other organic acids such as citric acid or benzoic acid? Provide an explanation for your answer using acid-base reaction principles in organic chemistry.`

Loss: TripletLoss with these parameters:
json
1{ 2 "distance_metric": "TripletDistanceMetric.EUCLIDEAN", 3 "triplet_margin": 5 4}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 32
per_device_eval_batch_size: 32
learning_rate: 2e-05
weight_decay: 0.01
num_train_epochs: 5
warmup_steps: 100
fp16: True
load_best_model_at_end: True

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 32
per_device_eval_batch_size: 32
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 2e-05
weight_decay: 0.01
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 5
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.0
warmup_steps: 100
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
bf16: False
fp16: True
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: True
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
project: huggingface
trackio_space_id: trackio
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	cosine_accuracy
0.0077	100	3.5959	-	-
0.0154	200	1.5715	-	-
0.0231	300	1.3926	-	-
0.0308	400	1.3807	-	-
0.0385	500	1.3662	-	-
0.0462	600	1.3368	-	-
0.0539	700	1.2831	-	-
0.0616	800	1.3723	-	-
0.0693	900	1.3459	-	-
0.0770	1000	1.3367	0.0005	0.0004
0.0847	1100	1.3124	-	-
0.0924	1200	1.2657	-	-
0.1001	1300	1.3098	-	-
0.1078	1400	1.2936	-	-
0.1155	1500	1.3058	-	-
0.1232	1600	1.2791	-	-
0.1309	1700	1.2446	-	-
0.1386	1800	1.2979	-	-
0.1463	1900	1.2658	-	-
0.1540	2000	1.2784	0.0001	0.0118
0.1617	2100	1.3043	-	-
0.1694	2200	1.2761	-	-
0.1771	2300	1.2647	-	-
0.1848	2400	1.2108	-	-
0.1925	2500	1.2773	-	-
0.2002	2600	1.3119	-	-
0.2079	2700	1.1856	-	-
0.2156	2800	1.2066	-	-
0.2233	2900	1.2455	-	-
0.2310	3000	1.1985	0.0003	0.0032
0.2387	3100	1.1875	-	-
0.2464	3200	1.1878	-	-
0.2541	3300	1.124	-	-
0.2618	3400	1.2539	-	-
0.2695	3500	1.2484	-	-
0.2772	3600	1.2123	-	-
0.2849	3700	1.1964	-	-
0.2926	3800	1.248	-	-
0.3003	3900	1.237	-	-
0.308	4000	1.161	0.0	0.0082
0.3157	4100	1.1645	-	-
0.3234	4200	1.2455	-	-
0.3311	4300	1.2235	-	-
0.3388	4400	1.2132	-	-
0.3465	4500	1.2001	-	-
0.3542	4600	1.2245	-	-
0.3619	4700	1.1759	-	-
0.3696	4800	1.1745	-	-
0.3773	4900	1.1852	-	-
0.3850	5000	1.2363	0.0014	0.0008
0.3927	5100	1.1454	-	-
0.4004	5200	1.0854	-	-
0.4081	5300	1.1228	-	-
0.4158	5400	1.0358	-	-
0.4235	5500	1.1722	-	-
0.4312	5600	1.1373	-	-
0.4389	5700	1.0926	-	-
0.4466	5800	1.1233	-	-
0.4543	5900	1.1422	-	-
0.4620	6000	1.0646	0.0005	0.0018
0.4697	6100	1.1371	-	-
0.4774	6200	1.1478	-	-
0.4851	6300	1.0805	-	-
0.4928	6400	1.095	-	-
0.5005	6500	1.2009	-	-
0.5082	6600	1.0456	-	-
0.5159	6700	1.11	-	-
0.5236	6800	1.0818	-	-
0.5313	6900	1.0727	-	-
0.5390	7000	1.0805	0.0019	0.0044

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.12.12
Sentence Transformers: 5.1.2
Transformers: 4.57.1
PyTorch: 2.8.0+cu126
Accelerate: 1.11.0
Datasets: 4.0.0
Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

bibtex

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

	anchor	positive	negative
type	string	string	string
details	min: 15 tokens mean: 230.67 tokens max: 512 tokens	min: 19 tokens mean: 273.54 tokens max: 512 tokens	min: 14 tokens mean: 239.81 tokens max: 512 tokens