SentenceTransformer based on google/bert_uncased_L-6_H-512_A-8

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/bert_uncased_L-6_H-512_A-8
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 512 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • parquet

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 512, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

Training Details

Training Dataset

parquet

• Dataset: parquet
• Size: 3,810,976 training samples
• Columns: text and label
• Approximate statistics based on the first 100 samples:

  	text	label
  type	string	list
  modality	text
  details	min: 99 tokens mean: 314.97 tokens max: 505 tokens	size: 1024 elements

• Samples:

  text	label
  Negligent homicide is a criminal charge brought against a person who, through criminal negligence, allows another person to die. Other times, an intentional killing may be negotiated down to this lesser charge as a compromised resolution of a murder case, as might occur in the context of the intentional shooting of an unarmed man after a traffic altercation. Negligent homicide can be distinguished from involuntary manslaughter by its mens rea requirement: negligent homicide requires criminal negligence, while manslaughter requires recklessness.	[0.07659912109375, -0.01461029052734375, 0.008819580078125, -0.062469482421875, -0.0020580291748046875, ...]
  Latin trap is a subgenre of Latin hip hop music that originated in Puerto Rico. A direct descendant of southern hip hop, and influenced by reggaeton, it gained popularity after 2007, and has since spread throughout Latin America. The trap is slang for a place where drugs are sold. Latin trap is similar to mainstream trap with lyrics about life on la calle (the street), drugs, sex and violence.[object Object]Characteristics[object Object]Latin trap is a subgenre of Latin hip hop, taking influence from Southern hip hop as well as Puerto Rican genres like reggaeton and dembow. Vocals include a bend of rapping and singing using synthesizers and voice distorted autotune, often in Spanish, while still maintaining the trap style sonic circuitry. The lyrics in Latin trap are often about street life, violence, sex, drugs, and people who live on the other side of the law and are proud of it.[object Object]History	[-0.0191802978515625, -0.0106353759765625, -0.036407470703125, 0.029571533203125, -0.004840850830078125, ...]
  Surface 3 is a 2-in-1 detachable from the Microsoft Surface series, introduced by Microsoft in 2015. Unlike its predecessor, the Surface 2, Surface 3 utilizes an x86 Intel Atom system-on-chip architecture, or SoC, rather than a processor with ARM architecture such as the Nvidia Tegra that powered the Surface 2, and runs standard versions of Windows 8.1 or Windows 10.[object Object]History[object Object]The older, original Surface (also known as Surface RT) and Surface 2, with their ARM-based processors and Windows RT operating system, were designed to compete with the iPad and other tablets. The first Surface was criticized for performance issues, which were reduced in the succeeding Surface 2. Due to the differing processors, these devices were incompatible with the vast number of Windows programs written for x86-based computers, running only those written and compiled for Windows RT, loaded from Microsoft's application store.	[-0.04119873046875, -0.016265869140625, -0.0126800537109375, 0.0103912353515625, -0.0557861328125, ...]

• Loss: EmbedDistillLoss with these parameters:
  json

  1{ 2 "distance_metric": "cosine", 3 "projection_dim": 1024 4}

Evaluation Dataset

parquet

• Dataset: parquet
• Size: 256 evaluation samples
• Columns: text and label
• Approximate statistics based on the first 100 samples:

  	text	label
  type	string	list
  modality	text
  details	min: 53 tokens mean: 308.2 tokens max: 512 tokens	size: 1024 elements

• Samples:

  text	label
  Revolution Pro Wrestling (2022–present) Slater made his debut in Revolution Pro Wrestling at RevPro Live In London 64 on August 7, 2022, where he unsuccessfully challenged Luke Jacobs for the Undisputed British Cruiserweight Championship. He continued to make regular appearances in the company's signature events such as the British J-Cup, where he made his first appearance at the 202 edition where he defeated Lio Rush in the first rounds but fell short to Robbie X, Lee Hunter and Will Kaven in a four-way match in the finals which occurred on the same night. One year later, Slater succeeded in winning the cup by defeating Will Kaven in the first rounds ans then Harrison Bennett, Mascara Dorada and Wild Boar in the four-way match of the finals. At RevPro Revolution Rumble 2023 on March 26, Slater competed in the traditional royal rumble match, bout won by Michael Oku and also involving various other notable opponents such as Big Damo, Callum Newman, Eddie Dennis, Gabriel Kidd, Francesco ...	[-0.026824951171875, 0.03338623046875, 0.011566162109375, 0.01247406005859375, 0.002338409423828125, ...]
  On 9 April 2015, Masagos was promoted to the rank of full Minister but did not have a portfolio yet, so he was a Minister in the Prime Minister's Office. This was the first time in Singaporean history when there were two Malay ministers in the Cabinet, the other being Yaacob Ibrahim. Masagos was also promoted to Second Minister for Home Affairs and Second Minister for Foreign Affairs, and was put in charge of leading the PAP team in Tampines GRC. On the same day, Prime Minister Lee Hsien Loong said that Masagos's elevation to full Minister reflected the "progress of the Malay community" in Singapore. Masagos also said that he was honoured to have been appointed and that "[h]aving two Malay full ministers in the Cabinet for the first time in our nation's history reflects the [Government's] trust and recognition of the good progress made by the Malay-Muslim community".[object Object]On 1 October 2015, Masagos took up the portfolio of Minister for Environment and Water Resources. From 1 May 2018, he wa...	[0.039398193359375, -0.022613525390625, 0.006793975830078125, 0.00878143310546875, -0.005939483642578125, ...]
  In the late 15th century the Clan Mackay and Clan Ross had long been at feud. This resulted in the Battle of Tarbat in 1486 where the Mackays were defeated by the Rosses and chief Angus Roy Mackay, 9th of Strathnaver was killed. This was followed by the Battle of Aldy Charrish where the Rosses were defeated by the Mackays and the Ross chief was killed along with many of his clan. According to 17th-century historian Sir Robert Gordon, who was a younger son of Alexander Gordon, 12th Earl of Sutherland, the Clan Sutherland joined the side of the Clan Mackay at this battle. However, 19th-century historian Angus Mackay disputes the Sutherland's presence at the battle stating that it would be unlikely that the Earl of Sutherland at the time would have assisted against the Rosses as he was married to a daughter of the Ross chief of Balnagowan, and also that the feudal superiority of the Sutherlands over the Mackays "nowhere existed save in his own fertile imagination".[object Object]16th century and clan c...	[0.06927490234375, 0.021881103515625, 0.01392364501953125, -0.044342041015625, 0.058258056640625, ...]

• Loss: EmbedDistillLoss with these parameters:
  json

  1{ 2 "distance_metric": "cosine", 3 "projection_dim": 1024 4}

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• max_steps: 100000
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• warmup_steps: 2000
• fp16: True
• per_device_eval_batch_size: 32
• seed: 12

All Hyperparameters

Training Logs

Training Time

• Training: 11.3 minutes
• Evaluation: 1.0 seconds
• Total: 11.3 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.7.0
• Transformers: 5.13.1
• PyTorch: 2.10.0+cu128
• Accelerate: 1.14.0
• Datasets: 5.0.1
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

EmbedDistillLoss