This is a sentence-transformers model finetuned from google/bert_uncased_L-2_H-128_A-2. It maps sentences & paragraphs to a 128-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: google/bert_uncased_L-2_H-128_A-2
Maximum Sequence Length: 128 tokens
Output Dimensionality: 128 dimensions
Similarity Function: Cosine Similarity
Supported Modality: Text

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 128, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("swardiantara/bert-tiny-snli-k10-fixed-cosine")
# Run inference
sentences = [
    'The school is having a special event in order to show the american culture on how other cultures are dealt with in parties. [SEP] A school is hosting an event.',
    'A man is sitting on the side of a sidewalk with a paper in his hands. [SEP] A man is waiting for the bus.',
    'A man is sitting on the side of a sidewalk with a paper in his hands. [SEP] A man is waiting for the bus.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 128]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.3092, 0.3092],
#         [0.3092, 1.0000, 1.0000],
#         [0.3092, 1.0000, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

Size: 11,537,277 training samples
Columns: text_a, text_b, and label
Approximate statistics based on the first 100 samples:
text_a text_b label
type string string list
modality text text
details
min: 13 tokens
mean: 21.08 tokens
max: 28 tokens
min: 20 tokens
mean: 26.1 tokens
max: 34 tokens
size: 2 elements

Samples:

text_a	text_b	label
`A person on a horse jumps over a broken down airplane. [SEP] A person is training his horse for a competition.`	`The man is on a black and white bike. [SEP] The man is going to be in a bicycle race.`	`[1.0, 0.0]`
`A person on a horse jumps over a broken down airplane. [SEP] A person is training his horse for a competition.`	`A man, on a basketball team, dunking the ball through the hoop. [SEP] The man is playing basketball.`	`[0.0, 0.5]`
`A person on a horse jumps over a broken down airplane. [SEP] A person is training his horse for a competition.`	`A young man wearing blue jeans and a t-shirt sits in the grass, with a ball in the air. [SEP] The young man is throwing a ball.`	`[0.0, 0.5]`

Loss: [object Object].OrdinalProxyContrastiveLoss

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 1024
num_train_epochs: 10
learning_rate: 2e-05
load_best_model_at_end: True

All Hyperparameters

Click to expand

per_device_train_batch_size: 1024
num_train_epochs: 10
max_steps: -1
learning_rate: 2e-05
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_steps: 0
optim: adamw_torch
optim_args: None
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
optim_target_modules: None
gradient_accumulation_steps: 1
average_tokens_across_devices: True
max_grad_norm: 1.0
label_smoothing_factor: 0.0
bf16: False
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
use_liger_kernel: False
liger_kernel_config: None
use_cache: False
neftune_noise_alpha: None
torch_empty_cache_steps: None
auto_find_batch_size: False
log_on_each_node: True
logging_nan_inf_filter: True
include_num_input_tokens_seen: no
log_level: passive
log_level_replica: warning
disable_tqdm: False
project: huggingface
trackio_space_id: None
trackio_bucket_id: None
trackio_static_space_id: None
per_device_eval_batch_size: 8
prediction_loss_only: True
eval_on_start: False
eval_do_concat_batches: True
eval_use_gather_object: False
eval_accumulation_steps: None
include_for_metrics: []
batch_eval_metrics: False
save_only_model: False
save_on_each_node: False
enable_jit_checkpoint: False
push_to_hub: False
hub_private_repo: None
hub_model_id: None
hub_strategy: every_save
hub_always_push: False
hub_revision: None
load_best_model_at_end: True
ignore_data_skip: False
restore_callback_states_from_checkpoint: False
full_determinism: False
seed: 42
data_seed: None
use_cpu: False
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_pin_memory: True
dataloader_persistent_workers: False
dataloader_prefetch_factor: None
remove_unused_columns: True
label_names: None
train_sampling_strategy: random
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
ddp_static_graph: None
ddp_backend: None
ddp_timeout: 1800
fsdp: None
fsdp_config: None
deepspeed: None
debug: []
skip_memory_metrics: True
do_predict: False
resume_from_checkpoint: None
warmup_ratio: None
local_rank: -1
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0444	500	0.0125
0.0888	1000	0.0095
0.1331	1500	0.0093
0.1775	2000	0.0092
0.2219	2500	0.0092
0.2663	3000	0.0091
0.3106	3500	0.0089
0.3550	4000	0.0088
0.3994	4500	0.0086
0.4438	5000	0.0085
0.4882	5500	0.0082
0.5325	6000	0.0080
0.5769	6500	0.0079
0.6213	7000	0.0077
0.6657	7500	0.0076
0.7100	8000	0.0075
0.7544	8500	0.0074
0.7988	9000	0.0073
0.8432	9500	0.0071
0.8875	10000	0.0070
0.9319	10500	0.0068
0.9763	11000	0.0069
1.0	11267	-
1.0207	11500	0.0069
1.0651	12000	0.0066
1.1094	12500	0.0067
1.1538	13000	0.0066
1.1982	13500	0.0065
1.2426	14000	0.0064
1.2869	14500	0.0065
1.3313	15000	0.0064
1.3757	15500	0.0063
1.4201	16000	0.0063
1.4645	16500	0.0063
1.5088	17000	0.0062
1.5532	17500	0.0063
1.5976	18000	0.0062
1.6420	18500	0.0061
1.6863	19000	0.0062
1.7307	19500	0.0060
1.7751	20000	0.0061
1.8195	20500	0.0060
1.8639	21000	0.0061
1.9082	21500	0.0060
1.9526	22000	0.0060
1.9970	22500	0.0060
2.0	22534	-
2.0414	23000	0.0058
2.0857	23500	0.0058
2.1301	24000	0.0058
2.1745	24500	0.0057
2.2189	25000	0.0058
2.2632	25500	0.0057
2.3076	26000	0.0057
2.3520	26500	0.0057
2.3964	27000	0.0057
2.4408	27500	0.0057
2.4851	28000	0.0056
2.5295	28500	0.0056
2.5739	29000	0.0057
2.6183	29500	0.0056
2.6626	30000	0.0056
2.7070	30500	0.0055
2.7514	31000	0.0055
2.7958	31500	0.0056
2.8402	32000	0.0055
2.8845	32500	0.0055
2.9289	33000	0.0055
2.9733	33500	0.0055
3.0	33801	-
3.0177	34000	0.0055
3.0620	34500	0.0054
3.1064	35000	0.0054
3.1508	35500	0.0053
3.1952	36000	0.0053
3.2395	36500	0.0054
3.2839	37000	0.0054
3.3283	37500	0.0053
3.3727	38000	0.0053
3.4171	38500	0.0053
3.4614	39000	0.0053
3.5058	39500	0.0053
3.5502	40000	0.0053
3.5946	40500	0.0053
3.6389	41000	0.0053
3.6833	41500	0.0052
3.7277	42000	0.0052
3.7721	42500	0.0052
3.8165	43000	0.0052
3.8608	43500	0.0052
3.9052	44000	0.0051
3.9496	44500	0.0052
3.9940	45000	0.0051
4.0	45068	-
4.0383	45500	0.0052
4.0827	46000	0.0051
4.1271	46500	0.0051
4.1715	47000	0.0051
4.2159	47500	0.0051
4.2602	48000	0.0051
4.3046	48500	0.0051
4.3490	49000	0.0051
4.3934	49500	0.0050
4.4377	50000	0.0050
4.4821	50500	0.0050
4.5265	51000	0.0050
4.5709	51500	0.0050
4.6152	52000	0.0050
4.6596	52500	0.0050
4.7040	53000	0.0050
4.7484	53500	0.0050
4.7928	54000	0.0050
4.8371	54500	0.0050
4.8815	55000	0.0050
4.9259	55500	0.0049
4.9703	56000	0.0049
5.0	56335	-
5.0146	56500	0.0050
5.0590	57000	0.0049
5.1034	57500	0.0049
5.1478	58000	0.0049
5.1922	58500	0.0049
5.2365	59000	0.0049
5.2809	59500	0.0049
5.3253	60000	0.0049
5.3697	60500	0.0048
5.4140	61000	0.0049
5.4584	61500	0.0049
5.5028	62000	0.0049
5.5472	62500	0.0049
5.5916	63000	0.0048
5.6359	63500	0.0049
5.6803	64000	0.0048
5.7247	64500	0.0049
5.7691	65000	0.0048
5.8134	65500	0.0048
5.8578	66000	0.0048
5.9022	66500	0.0048
5.9466	67000	0.0049
5.9909	67500	0.0048
6.0	67602	-
6.0353	68000	0.0048
6.0797	68500	0.0048
6.1241	69000	0.0047
6.1685	69500	0.0047
6.2128	70000	0.0048
6.2572	70500	0.0047
6.3016	71000	0.0048
6.3460	71500	0.0048
6.3903	72000	0.0048
6.4347	72500	0.0047
6.4791	73000	0.0047
6.5235	73500	0.0048
6.5679	74000	0.0047
6.6122	74500	0.0047
6.6566	75000	0.0047
6.7010	75500	0.0048
6.7454	76000	0.0048
6.7897	76500	0.0048
6.8341	77000	0.0047
6.8785	77500	0.0047
6.9229	78000	0.0047
6.9672	78500	0.0047
7.0	78869	-
7.0116	79000	0.0047
7.0560	79500	0.0047
7.1004	80000	0.0046
7.1448	80500	0.0047
7.1891	81000	0.0047
7.2335	81500	0.0047
7.2779	82000	0.0047
7.3223	82500	0.0047
7.3666	83000	0.0046
7.4110	83500	0.0047
7.4554	84000	0.0046
7.4998	84500	0.0047
7.5442	85000	0.0046
7.5885	85500	0.0046
7.6329	86000	0.0046
7.6773	86500	0.0046
7.7217	87000	0.0047
7.7660	87500	0.0046
7.8104	88000	0.0046
7.8548	88500	0.0046
7.8992	89000	0.0046
7.9436	89500	0.0047
7.9879	90000	0.0047
8.0	90136	-
8.0323	90500	0.0046
8.0767	91000	0.0046
8.1211	91500	0.0045
8.1654	92000	0.0046
8.2098	92500	0.0046
8.2542	93000	0.0046
8.2986	93500	0.0046
8.3429	94000	0.0046
8.3873	94500	0.0046
8.4317	95000	0.0046
8.4761	95500	0.0046
8.5205	96000	0.0046
8.5648	96500	0.0046
8.6092	97000	0.0046
8.6536	97500	0.0046
8.6980	98000	0.0046
8.7423	98500	0.0046
8.7867	99000	0.0046
8.8311	99500	0.0046
8.8755	100000	0.0046
8.9199	100500	0.0046
8.9642	101000	0.0046
9.0	101403	-
9.0086	101500	0.0046
9.0530	102000	0.0046
9.0974	102500	0.0046
9.1417	103000	0.0045
9.1861	103500	0.0046
9.2305	104000	0.0046
9.2749	104500	0.0046
9.3193	105000	0.0045
9.3636	105500	0.0044
9.4080	106000	0.0046
9.4524	106500	0.0046
9.4968	107000	0.0046
9.5411	107500	0.0045
9.5855	108000	0.0045
9.6299	108500	0.0046
9.6743	109000	0.0045
9.7186	109500	0.0046
9.7630	110000	0.0046
9.8074	110500	0.0046
9.8518	111000	0.0046
9.8962	111500	0.0046
9.9405	112000	0.0045
9.9849	112500	0.0045
10.0	112670	-

The bold row denotes the saved checkpoint.

Training Time

Training: 4.4 hours
Evaluation: 52.3 seconds
Total: 4.4 hours

Framework Versions

Python: 3.12.4
Sentence Transformers: 5.5.1
Transformers: 5.11.0
PyTorch: 2.5.1+cu121
Accelerate: 1.13.0
Datasets: 2.21.0
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

	text_a	text_b	label
type	string	string	list
modality	text	text
details	min: 13 tokens mean: 21.08 tokens max: 28 tokens	min: 20 tokens mean: 26.1 tokens max: 34 tokens	size: 2 elements