ColBERT ModernBERT-base distilled from BGE on MS MARCO

Model Details

Model Description

• Model Type: Multi-Vector Encoder
• Base model: answerdotai/ModernBERT-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: MaxSim
• Supported Modality: Text
• Training Dataset:
  • ms-marco-en-bge
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Multi-Vector Encoders on Hugging Face

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'query_length': 32, 'document_length': 180, 'do_query_expansion': True, 'attend_to_expansion_tokens': False, 'architecture': 'ModernBertModel'})
  (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings', 'use_residual': False})
  (2): MultiVectorMask({'skiplist_words': ['!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~']})
  (3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Multi Vector Information Retrieval

• Datasets: NanoMSMARCO, NanoNQ, NanoFiQA2018, NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with MultiVectorInformationRetrievalEvaluator

Multi Vector Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with MultiVectorNanoBEIREvaluator with these parameters:
  json

  1{ 2 "dataset_names": [ 3 "msmarco", 4 "nq", 5 "fiqa2018" 6 ], 7 "dataset_id": "sentence-transformers/NanoBEIR-en" 8}

Multi Vector Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with MultiVectorNanoBEIREvaluator with these parameters:
  json

  1{ 2 "dataset_names": [ 3 "climatefever", 4 "dbpedia", 5 "fever", 6 "fiqa2018", 7 "hotpotqa", 8 "msmarco", 9 "nfcorpus", 10 "nq", 11 "quoraretrieval", 12 "scidocs", 13 "arguana", 14 "scifact", 15 "touche2020" 16 ], 17 "dataset_id": "sentence-transformers/NanoBEIR-en" 18}

Training Details

Training Dataset

ms-marco-en-bge

• Dataset: ms-marco-en-bge at ad24729
• Size: 20,000 training samples
• Columns: query, documents, and scores
• Approximate statistics based on the first 100 samples:

  	query	documents	scores
  type	string	list	list
  modality	text
  details	min: 4 tokens mean: 9.92 tokens max: 21 tokens	size: 32 elements	size: 32 elements

• Samples:

  query	documents	scores
  define extreme	['extremist. 1 AN EXTREMIST IS SOMEONE WHO SUPPORTS AN IDEA, CAUSE, OR SET OF VALUES SO ADAMANTLY AND WITHOUT COMPROMISE THAT SAID PERSON WILL USE THEIR IDEAS TO JUSTIFY ANYTHING THEY DO.', "at the extreme end meaning, at the extreme end definition	English Cobuild dictionary. extreme. 1 adj Extreme means very great in degree or intensity. The girls were afraid of snakes and picked their way along with extreme caution., ...people living in extreme poverty., ...the author's extreme reluctance to generalise.", 'extremity (plural extremities) 1 The most extreme or furthest point of something. 2 An extreme measure. 3 A hand or foot. A limb (major appendage of human or animal such as a leg an arm or a wing)', ': extreme in a way that is not normal or that shows an illness or mental problem. medical: relating to or caused by disease.: of or relating to the study of diseases: relating to pathology. extreme in a way that is not normal or that shows an illness or mental problem. medical: relating to or caused by disease.', 'Definition of extreme. 1a : existing in a very high degree extreme povertyb : going to great or exaggerated lengths : radical went on an extreme dietc : exceeding the ordinary, usual, or expected extreme weather conditions. 2 archaic : last.', ...]
  what does chattel mean on credit history	["Duhaime's Law Dictionary. Chattel Mortgage Definition: Related Terms: Chattel, Mortgage. When a lien is given on goods, chattels, moveable or personal property (other than real property in which case it is referred to as just a mortgage), in writing, to guarantee the payment of a debt or the execution of some action.", 'From Wikipedia, the free encyclopedia. Chattel mortgage, sometimes abbreviated CM, is the legal term for a type of loan contract used in some states with legal systems derived from English law. Under a typical chattel mortgage, the purchaser borrows funds for the purchase of movable personal property (the chattel) from the lender. The lender then secures the loan with a mortgage over the chattel.', 'Chattel Mortgages In Detail. A Chattel Mortgage uses your vehicle or some other (non-real estate) property as the security on the loan meaning you can access a low interest rate.ncidentally, these loans can be used for other purposes such as business equipment. If you have a preference for Chattel Mortgage, ask the team at 360 Finance. Term / Length of the loan â\x80\x93 the life of the loan or the time you have to pay it off.', 'A chattel mortgage is a mortgage that provides for a security interest in assets other than real estate to secure the loan. In the event of a default in payments, the lender has a lien in the assets used as collateral for the loan. In most states, a security agreement has replaced the use of chattel mortgages. chattel mortgage is a mortgage that provides for a security interest in assets other than real estate to secure the loan. In the event of a default in payments, the lender has a lien in the assets used as collateral for the loan. In most states, a security agreement has replaced the use of chattel mortgages.', 'A Chattel Mortgage is a type of loan contract that allows the buyer to take ownership of a vehicle at the time of purchase. The lender provides the buyer with the total loan amount to cover the price of the vehicle (chattel) so that it can be bought outright.', ...]	[0.7124203443527222, 0.7379189729690552, 0.5786551237106323, 0.6142299175262451, 0.6755089163780212, ...]
  what was the great leap forward brainly	['It was a clever scheme that was hatched soon after the 1949 revolution. The first phase was to send spies to the west during the great leap forward in the 1950s to plant falsified basic science into our western understanding of physics.', 'Great Leap Forward Devolution Into the Great Famine . Yang Jisheng, the author of Tombstone , wrote in the New York Times, â\x80\x9cThe Great Leap Forward that Mao began in 1958 set ambitious goals without the means to meet them. A vicious cycle ensued; exaggerated production reports from below emboldened the higher-ups to set even loftier targets.', 'In 1958 Mao introduced a second five year plan which became known as the â\x80\x98Great Leap Forwardâ\x80\x99 (GLF). He believed it was possible for China to overtake Britain as a leading industrial power within seven years and the USA soon after.n 1958 Mao introduced a second five year plan which became known as the â\x80\x98Great Leap Forwardâ\x80\x99 (GLF). He believed it was possible for China to overtake Britain as a leading industrial power within seven years and the USA soon after.', 'The Great Leap Forward approach was epitomized by the development of small backyard steel furnaces in every village and urban neighbourhood, which were intended to accelerate the industrialization process.', 'The Great Leap Forward was begun in 1957 by Chairman Mao Zedong to bring the nation quickly into the forefront of economic development. Mao wanted China to become a leading industrial power, and to accomplish his goals he and his colleagues pushed for the construction of steel plants across the country.', ...]	[0.6462352871894836, 0.7880821228027344, 0.791019856929779, 0.7709633111953735, 0.8284491300582886, ...]

• Loss: MultiVectorDistillKLDivLoss with these parameters:
  json

  1{ 2 "score_metric": "colbert_kd_scores", 3 "normalize_scores": true, 4 "temperature": 1.0, 5 "size_average": true 6}

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 4
• num_train_epochs: 1
• learning_rate: 3e-05
• warmup_steps: 0.05
• bf16: True
• per_device_eval_batch_size: 4
• load_best_model_at_end: True

All Hyperparameters

Training Logs

Training Time

• Training: 1.2 hours
• Evaluation: 27.5 minutes
• Total: 1.7 hours

Framework Versions

• Python: 3.11.6
• Sentence Transformers: 5.6.0.dev0
• Transformers: 5.8.0.dev0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0.dev0
• Datasets: 4.8.4
• Tokenizers: 0.22.2

Additional Resources

• Sentence Transformers Documentation: the full documentation site, including training, evaluation, and pre-trained model catalogs.
• PyLate: the upstream library whose features were absorbed into Sentence Transformers for multi-vector / late-interaction models.

Citation

BibTeX

Sentence Transformers

MultiVectorDistillKLDivLoss