SPLADE BERT-tiny trained on Natural-Questions tuples

Model Details

Model Description

• Model Type: SPLADE Sparse Encoder

• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 30522 dimensions
• Similarity Function: Dot Product
• Training Dataset:
  • natural-questions
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False}) with MLMTransformer model: BertForMaskedLM 
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Sparse Information Retrieval

• Datasets: NanoMSMARCO, NanoNFCorpus and NanoNQ
• Evaluated with SparseInformationRetrievalEvaluator

Sparse Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with SparseNanoBEIREvaluator with these parameters:
  json

  1{ 2 "dataset_names": [ 3 "msmarco", 4 "nfcorpus", 5 "nq" 6 ] 7}

Training Details

Training Dataset

natural-questions

• Dataset: natural-questions at f9e894e
• Size: 99,000 training samples
• Columns: query and answer
• Approximate statistics based on the first 1000 samples:

  	query	answer
  type	string	string
  details	min: 29 characters mean: 46.96 characters max: 93 characters	min: 10 characters mean: 582.13 characters max: 2141 characters

• Samples:

  query	answer
  who played the father in papa don't preach	Alex McArthur Alex McArthur (born March 6, 1957) is an American actor.
  where was the location of the battle of hastings	Battle of Hastings The Battle of Hastings[a] was fought on 14 October 1066 between the Norman-French army of William, the Duke of Normandy, and an English army under the Anglo-Saxon King Harold Godwinson, beginning the Norman conquest of England. It took place approximately 7 miles (11 kilometres) northwest of Hastings, close to the present-day town of Battle, East Sussex, and was a decisive Norman victory.
  how many puppies can a dog give birth to	Canine reproduction The largest litter size to date was set by a Neapolitan Mastiff in Manea, Cambridgeshire, UK on November 29, 2004; the litter was 24 puppies.[22]

• Loss: SpladeLoss with these parameters:
  json

  1{'loss': SparseMultipleNegativesRankingLoss( 2 (model): SparseEncoder( 3 (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False}) with MLMTransformer model: BertForMaskedLM 4 (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522}) 5 ) 6 (cross_entropy_loss): CrossEntropyLoss() 7), 'lambda_corpus': 3e-05, 'lambda_query': 5e-05, 'corpus_regularizer': FlopsLoss( 8 (model): SparseEncoder( 9 (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False}) with MLMTransformer model: BertForMaskedLM 10 (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522}) 11 ) 12), 'query_regularizer': FlopsLoss( 13 (model): SparseEncoder( 14 (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False}) with MLMTransformer model: BertForMaskedLM 15 (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522}) 16 ) 17)}

Evaluation Dataset

natural-questions

• Dataset: natural-questions at f9e894e
• Size: 1,000 evaluation samples
• Columns: query and answer
• Approximate statistics based on the first 1000 samples:

  	query	answer
  type	string	string
  details	min: 30 characters mean: 47.2 characters max: 96 characters	min: 58 characters mean: 598.96 characters max: 2480 characters

• Samples:

  query	answer
  where is the tiber river located in italy	Tiber The Tiber (/ˈtaɪbər/, Latin: Tiberis,[1] Italian: Tevere [ˈteːvere])[2] is the third-longest river in Italy, rising in the Apennine Mountains in Emilia-Romagna and flowing 406 kilometres (252 mi) through Tuscany, Umbria and Lazio, where it is joined by the river Aniene, to the Tyrrhenian Sea, between Ostia and Fiumicino.[3] It drains a basin estimated at 17,375 square kilometres (6,709 sq mi). The river has achieved lasting fame as the main watercourse of the city of Rome, founded on its eastern banks.
  what kind of car does jay gatsby drive	Jay Gatsby At the Buchanan home, Jordan Baker, Nick, Jay, and the Buchanans decide to visit New York City. Tom borrows Gatsby's yellow Rolls Royce to drive up to the city. On the way to New York City, Tom makes a detour at a gas station in "the Valley of Ashes", a run-down part of Long Island. The owner, George Wilson, shares his concern that his wife, Myrtle, may be having an affair. This unnerves Tom, who has been having an affair with Myrtle, and he leaves in a hurry.
  who sings if i can dream about you	I Can Dream About You "I Can Dream About You" is a song performed by American singer Dan Hartman on the soundtrack album of the film Streets of Fire. Released in 1984 as a single from the soundtrack, and included on Hartman's album I Can Dream About You, it reached number 6 on the Billboard Hot 100.[1]

• Loss: SpladeLoss with these parameters:
  json

  1{'loss': SparseMultipleNegativesRankingLoss( 2 (model): SparseEncoder( 3 (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False}) with MLMTransformer model: BertForMaskedLM 4 (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522}) 5 ) 6 (cross_entropy_loss): CrossEntropyLoss() 7), 'lambda_corpus': 3e-05, 'lambda_query': 5e-05, 'corpus_regularizer': FlopsLoss( 8 (model): SparseEncoder( 9 (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False}) with MLMTransformer model: BertForMaskedLM 10 (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522}) 11 ) 12), 'query_regularizer': FlopsLoss( 13 (model): SparseEncoder( 14 (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False}) with MLMTransformer model: BertForMaskedLM 15 (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522}) 16 ) 17)}

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

Environmental Impact

• Energy Consumed: 0.027 kWh
• Carbon Emitted: 0.011 kg of CO2
• Hours Used: 0.082 hours

Training Hardware

• On Cloud: No
• GPU Model: 1 x NVIDIA GeForce RTX 3090
• CPU Model: 13th Gen Intel(R) Core(TM) i7-13700K
• RAM Size: 31.78 GB

Framework Versions

• Python: 3.11.6
• Sentence Transformers: 4.2.0.dev0
• Transformers: 4.49.0
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.1
• Datasets: 2.21.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

SpladeLoss