This is a sentence-transformers model finetuned from google-bert/bert-base-cased on the csv dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: google-bert/bert-base-cased
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- csv

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Jimmy-Ooi/TTM_1239_12_10_0.0001_AdamW")
# Run inference
sentences = [
    'Cc1ccccc1C(=O)N1CCN(Cc2ccc(F)cc2)CC1',
    'CC(C)(C)c1ccc(O)cc1',
    'Oc1ccc(/N=N/c2ccccc2O)cc1',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.6928, -0.8911],
#         [ 0.6928,  1.0000, -0.3628],
#         [-0.8911, -0.3628,  1.0000]])

Training Details

Training Dataset

csv

Dataset: csv
Size: 286,816 training samples
Columns: premise, hypothesis, and label
Approximate statistics based on the first 1000 samples:
premise hypothesis label
type string string int
details
min: 8 tokens
mean: 38.42 tokens
max: 213 tokens
min: 8 tokens
mean: 37.37 tokens
max: 213 tokens
0: ~49.40%
2: ~50.60%

Samples:

premise	hypothesis	label
`CNC(=S)N/N=C(\C)c1ccc(OC)cc1O`	`O=C(NO)Nc1ccc(NC(=O)NO)cc1`	`0`
`COc1ccc(C(=O)OCc2cc(=O)c(O)co2)c(O)c1`	`Cn1c2ccccc2c2cc(/C=C/C(=O)c3cccc(NC(=O)c4cccc(F)c4)c3)ccc21`	`0`
`Oc1ccc(CCc2ccccc2)c(O)c1`	`O=C(NO)Nc1ccc(O)cc1`	`0`

Loss: SoftmaxLoss

Evaluation Dataset

csv

Dataset: csv
Size: 50,615 evaluation samples
Columns: premise, hypothesis, and label
Approximate statistics based on the first 1000 samples:
premise hypothesis label
type string string int
details
min: 8 tokens
mean: 39.16 tokens
max: 213 tokens
min: 8 tokens
mean: 37.92 tokens
max: 207 tokens
0: ~51.70%
2: ~48.30%

Samples:

premise	hypothesis	label
`O=c1c2ccccc2oc2cccc(O)c12`	`COc1cc(C=O)ccc1OC(=O)COC(=O)/C=C/c1ccc(O)cc1`	`0`
`CCCCOc1cccc2c1C(=O)c1c(OCCCC)cc(CO)cc1C2=O`	`COc1ccc(C(=O)OC[C@H]2O[C@H]3c4c(cc(OC)c(OC)c4OC)C(=O)O[C@@H]3[object Object][C@@H]2O)cc1O`	`0`
`O=c1cc(CSc2nncn2/N=C/c2ccc(O)cc2O)occ1O`	`CC(C)[object Object]C(=O)O`	`0`

Loss: SoftmaxLoss

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 64
num_train_epochs: 10
warmup_steps: 100
optim: adamw_torch
weight_decay: 0.001
fp16: True
per_device_eval_batch_size: 64

All Hyperparameters

Click to expand

per_device_train_batch_size: 64
num_train_epochs: 10
max_steps: -1
learning_rate: 5e-05
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_steps: 100
optim: adamw_torch
optim_args: None
weight_decay: 0.001
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
optim_target_modules: None
gradient_accumulation_steps: 1
average_tokens_across_devices: True
max_grad_norm: 1.0
label_smoothing_factor: 0.0
bf16: False
fp16: True
bf16_full_eval: False
fp16_full_eval: False
tf32: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
use_liger_kernel: False
liger_kernel_config: None
use_cache: False
neftune_noise_alpha: None
torch_empty_cache_steps: None
auto_find_batch_size: False
log_on_each_node: True
logging_nan_inf_filter: True
include_num_input_tokens_seen: no
log_level: passive
log_level_replica: warning
disable_tqdm: False
project: huggingface
trackio_space_id: trackio
eval_strategy: no
per_device_eval_batch_size: 64
prediction_loss_only: True
eval_on_start: False
eval_do_concat_batches: True
eval_use_gather_object: False
eval_accumulation_steps: None
include_for_metrics: []
batch_eval_metrics: False
save_only_model: False
save_on_each_node: False
enable_jit_checkpoint: False
push_to_hub: False
hub_private_repo: None
hub_model_id: None
hub_strategy: every_save
hub_always_push: False
hub_revision: None
load_best_model_at_end: False
ignore_data_skip: False
restore_callback_states_from_checkpoint: False
full_determinism: False
seed: 42
data_seed: None
use_cpu: False
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_pin_memory: True
dataloader_persistent_workers: False
dataloader_prefetch_factor: None
remove_unused_columns: True
label_names: None
train_sampling_strategy: random
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
ddp_backend: None
ddp_timeout: 1800
fsdp: []
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
deepspeed: None
debug: []
skip_memory_metrics: True
do_predict: False
resume_from_checkpoint: None
warmup_ratio: None
local_rank: -1
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0223	100	0.9842
0.0446	200	0.6990
0.0669	300	0.6739
0.0892	400	0.6522
0.1116	500	0.6240
0.1339	600	0.6113
0.1562	700	0.6017
0.1785	800	0.5886
0.2008	900	0.5891
0.2231	1000	0.5909
0.2454	1100	0.5788
0.2677	1200	0.5752
0.2900	1300	0.5704
0.3124	1400	0.5749
0.3347	1500	0.5697
0.3570	1600	0.5773
0.3793	1700	0.5703
0.4016	1800	0.5589
0.4239	1900	0.5788
0.4462	2000	0.5570
0.4685	2100	0.5512
0.4909	2200	0.5567
0.5132	2300	0.5568
0.5355	2400	0.5558
0.5578	2500	0.5461
0.5801	2600	0.5492
0.6024	2700	0.5604
0.6247	2800	0.5492
0.6470	2900	0.5465
0.6693	3000	0.5395
0.6917	3100	0.5435
0.7140	3200	0.5417
0.7363	3300	0.5413
0.7586	3400	0.5508
0.7809	3500	0.5490
0.8032	3600	0.5453
0.8255	3700	0.5421
0.8478	3800	0.5411
0.8701	3900	0.5229
0.8925	4000	0.5346
0.9148	4100	0.5437
0.9371	4200	0.5430
0.9594	4300	0.5319
0.9817	4400	0.5303
1.0040	4500	0.5357
1.0263	4600	0.5349
1.0486	4700	0.5397
1.0710	4800	0.5335
1.0933	4900	0.5327
1.1156	5000	0.5346
1.1379	5100	0.5228
1.1602	5200	0.5275
1.1825	5300	0.5389
1.2048	5400	0.5186
1.2271	5500	0.5260
1.2494	5600	0.5305
1.2718	5700	0.5264
1.2941	5800	0.5337
1.3164	5900	0.5287
1.3387	6000	0.5211
1.3610	6100	0.5265
1.3833	6200	0.5199
1.4056	6300	0.5414
1.4279	6400	0.5221
1.4502	6500	0.5278
1.4726	6600	0.5276
1.4949	6700	0.5258
1.5172	6800	0.5259
1.5395	6900	0.5279
1.5618	7000	0.5246
1.5841	7100	0.5300
1.6064	7200	0.5179
1.6287	7300	0.5269
1.6510	7400	0.5310
1.6734	7500	0.5210
1.6957	7600	0.5207
1.7180	7700	0.5186
1.7403	7800	0.5266
1.7626	7900	0.5111
1.7849	8000	0.5197
1.8072	8100	0.5245
1.8295	8200	0.5106
1.8519	8300	0.5272
1.8742	8400	0.5185
1.8965	8500	0.5137
1.9188	8600	0.5134
1.9411	8700	0.5201
1.9634	8800	0.5128
1.9857	8900	0.5224
2.0080	9000	0.5264
2.0303	9100	0.5115
2.0527	9200	0.5207
2.0750	9300	0.5229
2.0973	9400	0.5154
2.1196	9500	0.5125
2.1419	9600	0.5205
2.1642	9700	0.5177
2.1865	9800	0.5166
2.2088	9900	0.5206
2.2311	10000	0.5169
2.2535	10100	0.5204
2.2758	10200	0.5107
2.2981	10300	0.5180
2.3204	10400	0.5083
2.3427	10500	0.5166
2.3650	10600	0.5053
2.3873	10700	0.5212
2.4096	10800	0.5223
2.4320	10900	0.5137
2.4543	11000	0.5071
2.4766	11100	0.5171
2.4989	11200	0.5101
2.5212	11300	0.5145
2.5435	11400	0.5183
2.5658	11500	0.5118
2.5881	11600	0.5122
2.6104	11700	0.5083
2.6328	11800	0.5077
2.6551	11900	0.5044
2.6774	12000	0.5220
2.6997	12100	0.5116
2.7220	12200	0.5109
2.7443	12300	0.5157
2.7666	12400	0.5117
2.7889	12500	0.5152
2.8112	12600	0.5184
2.8336	12700	0.5161
2.8559	12800	0.5104
2.8782	12900	0.5123
2.9005	13000	0.5101
2.9228	13100	0.5088
2.9451	13200	0.5061
2.9674	13300	0.5026
2.9897	13400	0.5203
3.0120	13500	0.5197
3.0344	13600	0.5019
3.0567	13700	0.5087
3.0790	13800	0.5079
3.1013	13900	0.4874
3.1236	14000	0.5047
3.1459	14100	0.5234
3.1682	14200	0.5010
3.1905	14300	0.5076
3.2129	14400	0.5079
3.2352	14500	0.5098
3.2575	14600	0.5092
3.2798	14700	0.5061
3.3021	14800	0.5040
3.3244	14900	0.5066
3.3467	15000	0.5003
3.3690	15100	0.5132
3.3913	15200	0.5180
3.4137	15300	0.5080
3.4360	15400	0.4996
3.4583	15500	0.5089
3.4806	15600	0.5081
3.5029	15700	0.5101
3.5252	15800	0.5137
3.5475	15900	0.5114
3.5698	16000	0.5133
3.5921	16100	0.5035
3.6145	16200	0.5051
3.6368	16300	0.5109
3.6591	16400	0.5029
3.6814	16500	0.5096
3.7037	16600	0.5088
3.7260	16700	0.5045
3.7483	16800	0.5042
3.7706	16900	0.5098
3.7929	17000	0.4972
3.8153	17100	0.5131
3.8376	17200	0.5052
3.8599	17300	0.5026
3.8822	17400	0.5046
3.9045	17500	0.5070
3.9268	17600	0.5136
3.9491	17700	0.5101
3.9714	17800	0.5001
3.9938	17900	0.4979
4.0161	18000	0.5054
4.0384	18100	0.5063
4.0607	18200	0.5109
4.0830	18300	0.5109
4.1053	18400	0.4944
4.1276	18500	0.4959
4.1499	18600	0.4939
4.1722	18700	0.5119
4.1946	18800	0.5114
4.2169	18900	0.5169
4.2392	19000	0.5052
4.2615	19100	0.5033
4.2838	19200	0.4980
4.3061	19300	0.5010
4.3284	19400	0.5005
4.3507	19500	0.5016
4.3730	19600	0.5098
4.3954	19700	0.5111
4.4177	19800	0.4979
4.4400	19900	0.4945
4.4623	20000	0.4950
4.4846	20100	0.5015
4.5069	20200	0.5060
4.5292	20300	0.5043
4.5515	20400	0.5102
4.5739	20500	0.5129
4.5962	20600	0.5017
4.6185	20700	0.5025
4.6408	20800	0.4955
4.6631	20900	0.5019
4.6854	21000	0.5004
4.7077	21100	0.4975
4.7300	21200	0.4952
4.7523	21300	0.5022
4.7747	21400	0.4919
4.7970	21500	0.4968
4.8193	21600	0.5047
4.8416	21700	0.4976
4.8639	21800	0.4999
4.8862	21900	0.5012
4.9085	22000	0.5055
4.9308	22100	0.5011
4.9531	22200	0.5028
4.9755	22300	0.4959
4.9978	22400	0.4966
5.0201	22500	0.4969
5.0424	22600	0.5037
5.0647	22700	0.4939
5.0870	22800	0.5012
5.1093	22900	0.5023
5.1316	23000	0.5010
5.1539	23100	0.4971
5.1763	23200	0.4943
5.1986	23300	0.5051
5.2209	23400	0.4954
5.2432	23500	0.4995
5.2655	23600	0.5028
5.2878	23700	0.4972
5.3101	23800	0.5034
5.3324	23900	0.4968
5.3548	24000	0.4898
5.3771	24100	0.4946
5.3994	24200	0.5000
5.4217	24300	0.4956
5.4440	24400	0.4915
5.4663	24500	0.5039
5.4886	24600	0.5026
5.5109	24700	0.4981
5.5332	24800	0.5019
5.5556	24900	0.4997
5.5779	25000	0.4919
5.6002	25100	0.5132
5.6225	25200	0.5166
5.6448	25300	0.5112
5.6671	25400	0.5031
5.6894	25500	0.4902
5.7117	25600	0.5068
5.7340	25700	0.4956
5.7564	25800	0.4923
5.7787	25900	0.4879
5.8010	26000	0.4954
5.8233	26100	0.5059
5.8456	26200	0.4965
5.8679	26300	0.4927
5.8902	26400	0.5036
5.9125	26500	0.5012
5.9349	26600	0.4949
5.9572	26700	0.4989
5.9795	26800	0.5058
6.0018	26900	0.5029
6.0241	27000	0.4992
6.0464	27100	0.4892
6.0687	27200	0.4904
6.0910	27300	0.4943
6.1133	27400	0.4949
6.1357	27500	0.4981
6.1580	27600	0.4969
6.1803	27700	0.4968
6.2026	27800	0.5005
6.2249	27900	0.4844
6.2472	28000	0.4987
6.2695	28100	0.4944
6.2918	28200	0.4874
6.3141	28300	0.4938
6.3365	28400	0.4998
6.3588	28500	0.4944
6.3811	28600	0.4973
6.4034	28700	0.4882
6.4257	28800	0.4918
6.4480	28900	0.4909
6.4703	29000	0.4952
6.4926	29100	0.5013
6.5149	29200	0.4974
6.5373	29300	0.4985
6.5596	29400	0.4886
6.5819	29500	0.4945
6.6042	29600	0.4934
6.6265	29700	0.4974
6.6488	29800	0.4900
6.6711	29900	0.5017
6.6934	30000	0.4948
6.7158	30100	0.4901
6.7381	30200	0.4984
6.7604	30300	0.4973
6.7827	30400	0.4965
6.8050	30500	0.4934
6.8273	30600	0.4938
6.8496	30700	0.5009
6.8719	30800	0.4972
6.8942	30900	0.4808
6.9166	31000	0.4864
6.9389	31100	0.4950
6.9612	31200	0.5028
6.9835	31300	0.4945
7.0058	31400	0.5002
7.0281	31500	0.4988
7.0504	31600	0.4905
7.0727	31700	0.4891
7.0950	31800	0.4928
7.1174	31900	0.4899
7.1397	32000	0.5027
7.1620	32100	0.4931
7.1843	32200	0.4995
7.2066	32300	0.4994
7.2289	32400	0.4864
7.2512	32500	0.4946
7.2735	32600	0.4927
7.2959	32700	0.4879
7.3182	32800	0.4962
7.3405	32900	0.4886
7.3628	33000	0.4835
7.3851	33100	0.4948
7.4074	33200	0.4926
7.4297	33300	0.4922
7.4520	33400	0.4895
7.4743	33500	0.4917
7.4967	33600	0.4948
7.5190	33700	0.4890
7.5413	33800	0.4878
7.5636	33900	0.4937
7.5859	34000	0.4907
7.6082	34100	0.4983
7.6305	34200	0.4904
7.6528	34300	0.4904
7.6751	34400	0.4951
7.6975	34500	0.4928
7.7198	34600	0.4908
7.7421	34700	0.4953
7.7644	34800	0.4911
7.7867	34900	0.4983
7.8090	35000	0.4898
7.8313	35100	0.4989
7.8536	35200	0.4975
7.8759	35300	0.4929
7.8983	35400	0.4991
7.9206	35500	0.4919
7.9429	35600	0.4971
7.9652	35700	0.4881
7.9875	35800	0.4868
8.0098	35900	0.4969
8.0321	36000	0.5019
8.0544	36100	0.4885
8.0768	36200	0.4929
8.0991	36300	0.4967
8.1214	36400	0.4979
8.1437	36500	0.4917
8.1660	36600	0.4973
8.1883	36700	0.4836
8.2106	36800	0.4886
8.2329	36900	0.4962
8.2552	37000	0.4929
8.2776	37100	0.4871
8.2999	37200	0.4900
8.3222	37300	0.4877
8.3445	37400	0.4907
8.3668	37500	0.4964
8.3891	37600	0.4825
8.4114	37700	0.4933
8.4337	37800	0.4884
8.4560	37900	0.4905
8.4784	38000	0.4965
8.5007	38100	0.4905
8.5230	38200	0.4890
8.5453	38300	0.4903
8.5676	38400	0.4880
8.5899	38500	0.4860
8.6122	38600	0.4911
8.6345	38700	0.4867
8.6568	38800	0.4822
8.6792	38900	0.4909
8.7015	39000	0.4914
8.7238	39100	0.4937
8.7461	39200	0.4951
8.7684	39300	0.4925
8.7907	39400	0.4949
8.8130	39500	0.4833
8.8353	39600	0.4969
8.8577	39700	0.4910
8.8800	39800	0.4883
8.9023	39900	0.4937
8.9246	40000	0.4962
8.9469	40100	0.4891
8.9692	40200	0.4858
8.9915	40300	0.4914
9.0138	40400	0.4932
9.0361	40500	0.4905
9.0585	40600	0.4928
9.0808	40700	0.4946
9.1031	40800	0.4842
9.1254	40900	0.4982
9.1477	41000	0.4910
9.1700	41100	0.4882
9.1923	41200	0.4965
9.2146	41300	0.4927
9.2369	41400	0.4864
9.2593	41500	0.4897
9.2816	41600	0.4889
9.3039	41700	0.4948
9.3262	41800	0.4855
9.3485	41900	0.4864
9.3708	42000	0.4902
9.3931	42100	0.4819
9.4154	42200	0.4874
9.4378	42300	0.4986
9.4601	42400	0.4911
9.4824	42500	0.4937
9.5047	42600	0.4948
9.5270	42700	0.4866
9.5493	42800	0.4885
9.5716	42900	0.4928
9.5939	43000	0.4998
9.6162	43100	0.4824
9.6386	43200	0.4905
9.6609	43300	0.4930
9.6832	43400	0.4947
9.7055	43500	0.4866
9.7278	43600	0.4888
9.7501	43700	0.4845
9.7724	43800	0.4952
9.7947	43900	0.4855
9.8170	44000	0.4857
9.8394	44100	0.4852
9.8617	44200	0.4890
9.8840	44300	0.4904
9.9063	44400	0.4861
9.9286	44500	0.4928
9.9509	44600	0.4955
9.9732	44700	0.4932
9.9955	44800	0.4951

Framework Versions

Python: 3.12.12
Sentence Transformers: 5.3.0
Transformers: 5.3.0
PyTorch: 2.10.0+cu128
Accelerate: 1.13.0
Datasets: 4.0.0
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers and SoftmaxLoss

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

	premise	hypothesis	label
type	string	string	int
details	min: 8 tokens mean: 38.42 tokens max: 213 tokens	min: 8 tokens mean: 37.37 tokens max: 213 tokens	0: ~49.40% 2: ~50.60%