Bert base fine-tuned with Cantonese and English mixed STS dataset

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: hon9kon9ize/bert-large-cantonese-sts
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

• Language: yue
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Information Retrieval

• Datasets: NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with InformationRetrievalEvaluator

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 129,371 training samples
• Columns: query and answer
• Approximate statistics based on the first 1000 samples:

  	query	answer
  type	string	string
  details	min: 13 tokens mean: 22.58 tokens max: 134 tokens	min: 7 tokens mean: 169.6 tokens max: 512 tokens

• Samples:

  query	answer
  query: hotel and restaurant employees and bartenders international union	document: Hotel Employees and Restaurant Employees Union The Hotel Employees and Restaurant Employees Union (HERE) was a United States labor union representing workers of the hospitality industry, formed in 1891. In 2004, HERE merged with the Union of Needletrades, Industrial, and Textile Employees (UNITE) to form UNITE HERE. HERE notably organized the staff of Yale University in 1984. Other major employers that contracted with this union included several large casinos (Harrah's, Caesars Palace, and Wynn Resorts); hotels (Hilton, Hyatt and Starwood), and Walt Disney World. HERE was affiliated with the AFL-CIO.
  query: 多肢离断伤的并发症是什么？	document: 失血性休克；血循环危象；急性肾功能衰竭
  query: who is the father of kelly taylor's son on 90210	document: Kelly Taylor (90210) In 2008, Kelly Taylor returned in the spin-off 90210, now working as a guidance counselor at her alma mater West Beverly Hills High School. It was revealed that in the intervening years, she attained a master's degree and had a son named Sammy with Dylan. She and Dylan ended their relationship soon after. It was also revealed that West Beverly principal Harry Wilson was Kelly's neighbor growing up.[39]

• Loss: CachedGISTEmbedLoss with these parameters:
  json

  1{'guide': SentenceTransformer( 2 (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: NewModel 3 (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) 4 (2): Normalize() 5), 'temperature': 0.01}

Evaluation Dataset

Unnamed Dataset

• Size: 1,000 evaluation samples
• Columns: query and answer
• Approximate statistics based on the first 1000 samples:

  	query	answer
  type	string	string
  details	min: 11 tokens mean: 22.61 tokens max: 146 tokens	min: 7 tokens mean: 164.27 tokens max: 512 tokens

• Samples:

  query	answer
  query: 微创经皮肾镜手术的推荐药有些什么？	document: 阿司匹林
  query: why are the fires in ca called the thomas fires	document: Thomas Fire On December 4, 2017, the Thomas Fire was reported at 6:26 p.m. PST,[36] to the north of Santa Paula, near Steckel Park and Thomas Aquinas College,[3][24] after which the fire is named.[37] That night, the small brush fire exploded in size and raced through the rugged mountain terrain that lies west of Santa Paula, between Ventura and Ojai.[19][38] Officials blamed strong Santa Ana winds that gusted up to 60 miles per hour (97 km/h) for the sudden expansion.[28][39] Soon after the fire had started, a second blaze was ignited nearly 30 minutes later, about 4 miles (6.4 km) to the north in Upper Ojai at the top of Koenigstein Road.[40] According to eyewitnesses, this second fire was sparked by an explosion in the power line over the area. The second fire was rapidly expanded by the strong Santa Ana winds, and soon merged into the Thomas Fire later that night.[40]
  query: which mountain man rediscovered south pass and brought back important information about this trail	document: Jedediah Smith Jedediah Strong Smith (January 6, 1799 – May 27, 1831), was a clerk, frontiersman, hunter, trapper, author, cartographer, and explorer of the Rocky Mountains, the North American West, and the Southwest during the early 19th century. After 75 years of obscurity following his death, Smith was rediscovered as the American whose explorations led to the use of the 20-mile (32 km)-wide South Pass as the dominant point of crossing the Continental Divide for pioneers on the Oregon Trail.

• Loss: CachedGISTEmbedLoss with these parameters:
  json

  1{'guide': SentenceTransformer( 2 (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: NewModel 3 (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) 4 (2): Normalize() 5), 'temperature': 0.01}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• learning_rate: 2e-05
• num_train_epochs: 2
• warmup_ratio: 0.05
• seed: 12
• bf16: True
• prompts: {'query': 'query: ', 'answer': 'document: '}
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs