Sparse CSR model trained on Natural Questions

Model Details

Model Description

• Model Type: CSR Sparse Encoder
• Base model: mixedbread-ai/mxbai-embed-large-v1
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 4096 dimensions (trained with 256 maximum active dimensions)
• Similarity Function: Cosine Similarity
• Training Dataset:
  • natural-questions
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): CSRSparsity({'input_dim': 1024, 'hidden_dim': 4096, 'k': 256, 'k_aux': 512, 'normalize': False, 'dead_threshold': 30})
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Sparse Information Retrieval

• Dataset: nq_eval_4
• Evaluated with SparseInformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "max_active_dims": 4 3}

Sparse Information Retrieval

• Dataset: nq_eval_8
• Evaluated with SparseInformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "max_active_dims": 8 3}

Sparse Information Retrieval

• Dataset: nq_eval_16
• Evaluated with SparseInformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "max_active_dims": 16 3}

Sparse Information Retrieval

• Dataset: nq_eval_32
• Evaluated with SparseInformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "max_active_dims": 32 3}

Sparse Information Retrieval

• Dataset: nq_eval_64
• Evaluated with SparseInformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "max_active_dims": 64 3}

Sparse Information Retrieval

• Dataset: nq_eval_128
• Evaluated with SparseInformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "max_active_dims": 128 3}

Sparse Information Retrieval

• Dataset: nq_eval_256
• Evaluated with SparseInformationRetrievalEvaluator with these parameters:
  json

  1{ 2 "max_active_dims": 256 3}

Training Details

Training Dataset

natural-questions

• Dataset: natural-questions at f9e894e
• Size: 99,000 training samples
• Columns: query and answer
• Approximate statistics based on the first 1000 samples:

  	query	answer
  type	string	string
  details	min: 10 tokens mean: 11.71 tokens max: 26 tokens	min: 4 tokens mean: 131.81 tokens max: 450 tokens

• Samples:

  query	answer
  who played the father in papa don't preach	Alex McArthur Alex McArthur (born March 6, 1957) is an American actor.
  where was the location of the battle of hastings	Battle of Hastings The Battle of Hastings[a] was fought on 14 October 1066 between the Norman-French army of William, the Duke of Normandy, and an English army under the Anglo-Saxon King Harold Godwinson, beginning the Norman conquest of England. It took place approximately 7 miles (11 kilometres) northwest of Hastings, close to the present-day town of Battle, East Sussex, and was a decisive Norman victory.
  how many puppies can a dog give birth to	Canine reproduction The largest litter size to date was set by a Neapolitan Mastiff in Manea, Cambridgeshire, UK on November 29, 2004; the litter was 24 puppies.[22]

• Loss: CSRLoss with these parameters:
  json

  1{ 2 "beta": 0.1, 3 "gamma": 0.1, 4 "loss": "SparseMultipleNegativesRankingLoss(scale=20.0, similarity_fct='cos_sim')" 5}

Evaluation Dataset

natural-questions

• Dataset: natural-questions at f9e894e
• Size: 1,000 evaluation samples
• Columns: query and answer
• Approximate statistics based on the first 1000 samples:

  	query	answer
  type	string	string
  details	min: 10 tokens mean: 11.69 tokens max: 23 tokens	min: 15 tokens mean: 134.01 tokens max: 512 tokens

• Samples:

  query	answer
  where is the tiber river located in italy	Tiber The Tiber (/ˈtaɪbər/, Latin: Tiberis,[1] Italian: Tevere [ˈteːvere])[2] is the third-longest river in Italy, rising in the Apennine Mountains in Emilia-Romagna and flowing 406 kilometres (252 mi) through Tuscany, Umbria and Lazio, where it is joined by the river Aniene, to the Tyrrhenian Sea, between Ostia and Fiumicino.[3] It drains a basin estimated at 17,375 square kilometres (6,709 sq mi). The river has achieved lasting fame as the main watercourse of the city of Rome, founded on its eastern banks.
  what kind of car does jay gatsby drive	Jay Gatsby At the Buchanan home, Jordan Baker, Nick, Jay, and the Buchanans decide to visit New York City. Tom borrows Gatsby's yellow Rolls Royce to drive up to the city. On the way to New York City, Tom makes a detour at a gas station in "the Valley of Ashes", a run-down part of Long Island. The owner, George Wilson, shares his concern that his wife, Myrtle, may be having an affair. This unnerves Tom, who has been having an affair with Myrtle, and he leaves in a hurry.
  who sings if i can dream about you	I Can Dream About You "I Can Dream About You" is a song performed by American singer Dan Hartman on the soundtrack album of the film Streets of Fire. Released in 1984 as a single from the soundtrack, and included on Hartman's album I Can Dream About You, it reached number 6 on the Billboard Hot 100.[1]

• Loss: CSRLoss with these parameters:
  json

  1{ 2 "beta": 0.1, 3 "gamma": 0.1, 4 "loss": "SparseMultipleNegativesRankingLoss(scale=20.0, similarity_fct='cos_sim')" 5}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• learning_rate: 4e-05
• num_train_epochs: 1
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

Environmental Impact

• Energy Consumed: 0.104 kWh
• Carbon Emitted: 0.040 kg of CO2
• Hours Used: 0.26 hours

Training Hardware

• On Cloud: No
• GPU Model: 1 x NVIDIA GeForce RTX 3090
• CPU Model: 13th Gen Intel(R) Core(TM) i7-13700K
• RAM Size: 31.78 GB

Framework Versions

• Python: 3.11.6
• Sentence Transformers: 4.2.0.dev0
• Transformers: 4.52.4
• PyTorch: 2.7.1+cu126
• Accelerate: 1.5.1
• Datasets: 2.21.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

CSRLoss

SparseMultipleNegativesRankingLoss