You can deploy this model and start earning money today!

tomaarsen/
multivector-ModernBERT-base-msmarco-cached-contrastive-no-query-expansion

Name: multivector-ModernBERT-base-msmarco-cached-contrastive-no-query-expansion
Availability: InStock

Views

No views yet

ColBERT ModernBERT-base trained on MS MARCO triplets with GradCache

This is a Multi-Vector Encoder model finetuned from answerdotai/ModernBERT-base on the msmarco-bm25 dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

Model Details

Model Description

Model Type: Multi-Vector Encoder
Base model: answerdotai/ModernBERT-base
Maximum Sequence Length: 8192 tokens
Output Dimensionality: 128 dimensions
Similarity Function: maxsim
Supported Modality: Text
Training Dataset:
- msmarco-bm25
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Documentation: Multi-Vector Encoder Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Multi-Vector Encoders on Hugging Face

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
  (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
  (2): MultiVectorMask({'skiplist_words': [], 'keep_only_token_ids': None})
  (3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import MultiVectorEncoder

# Download from the 🤗 Hub
model = MultiVectorEncoder("tomaarsen/multivector-ModernBERT-base-msmarco-cached-contrastive")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
    'what is territorial sovereignty',
]
documents = [
    'territorial sovereignty. Exclusive right of a state to exercise its powers within the boundaries of its territory.',
    'Territorial preservation, as Agnew explains is merely one aspect of a states territorial integrity (2005). The lack of territorial sovereignty is often a key characteristic of so-called failed states where effective monopoly over the internal means of violence is lost.',
    '1 Active Transportï\x82§ Active Transport requires the cell to use energy, usually in the form of ATP.ï\x82§ Active Transport creates a charge gradient in the cell membrane. 2  For example in the mitochondrion, hydrogen ion pumps pump hydrogen ions into the intermembrane space of the organelle as part of making ATP. 3  15.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (6, 128) (24, 128)

# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[5.6691, 5.5729, 1.3079]])

Evaluation

Metrics

Multi Vector Information Retrieval

Datasets: NanoMSMARCO, NanoNQ, NanoFiQA2018, NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
Evaluated with MultiVectorInformationRetrievalEvaluator

Metric	NanoMSMARCO	NanoNQ	NanoFiQA2018	NanoClimateFEVER	NanoDBPedia	NanoFEVER	NanoHotpotQA	NanoNFCorpus	NanoQuoraRetrieval	NanoSCIDOCS	NanoArguAna	NanoSciFact	NanoTouche2020
maxsim_accuracy@1	0.34	0.38	0.3	0.2	0.56	0.58	0.68	0.42	0.78	0.34	0.14	0.48	0.449
maxsim_accuracy@3	0.5	0.5	0.4	0.38	0.78	0.9	0.76	0.5	0.88	0.54	0.46	0.6	0.7551
maxsim_accuracy@5	0.6	0.6	0.44	0.46	0.82	0.94	0.78	0.54	0.9	0.62	0.62	0.64	0.8367
maxsim_accuracy@10	0.72	0.7	0.52	0.54	0.86	0.98	0.84	0.56	0.94	0.76	0.76	0.7	0.9796
maxsim_precision@1	0.34	0.38	0.3	0.2	0.56	0.58	0.68	0.42	0.78	0.34	0.14	0.48	0.449
maxsim_precision@3	0.1667	0.1733	0.1667	0.1267	0.5	0.3067	0.28	0.3333	0.3667	0.2667	0.1533	0.2267	0.4558
maxsim_precision@5	0.12	0.124	0.124	0.096	0.448	0.192	0.192	0.296	0.232	0.216	0.124	0.148	0.4286
maxsim_precision@10	0.072	0.072	0.078	0.064	0.374	0.102	0.108	0.234	0.124	0.152	0.076	0.08	0.398
maxsim_recall@1	0.34	0.36	0.1934	0.0917	0.0646	0.5467	0.34	0.0279	0.6807	0.0717	0.14	0.445	0.0375
maxsim_recall@3	0.5	0.49	0.2849	0.1707	0.1224	0.8467	0.42	0.0967	0.8353	0.1647	0.46	0.595	0.1122
maxsim_recall@5	0.6	0.58	0.3257	0.2063	0.1721	0.8867	0.48	0.1135	0.872	0.2217	0.62	0.64	0.1684
maxsim_recall@10	0.72	0.65	0.3724	0.2703	0.2484	0.9333	0.54	0.136	0.9227	0.3107	0.76	0.7	0.2823
maxsim_ndcg@10	0.5197	0.4985	0.3225	0.2154	0.4666	0.7635	0.5294	0.3066	0.8471	0.2974	0.4229	0.5832	0.4341
maxsim_mrr@10	0.4572	0.4622	0.3641	0.3007	0.6739	0.7285	0.7264	0.4662	0.8372	0.465	0.3171	0.5505	0.6234
maxsim_map@100	0.4703	0.4559	0.285	0.1658	0.3213	0.6973	0.4468	0.141	0.8197	0.2265	0.3249	0.5527	0.3793

Multi Vector Nano BEIR

Dataset: NanoBEIR_mean
Evaluated with MultiVectorNanoBEIREvaluator with these parameters:
json
1{ 2 "dataset_names": [ 3 "msmarco", 4 "nq", 5 "fiqa2018" 6 ], 7 "dataset_id": "sentence-transformers/NanoBEIR-en" 8}

Metric	Value
maxsim_accuracy@1	0.32
maxsim_accuracy@3	0.46
maxsim_accuracy@5	0.5333
maxsim_accuracy@10	0.6267
maxsim_precision@1	0.32
maxsim_precision@3	0.1689
maxsim_precision@5	0.1173
maxsim_precision@10	0.0713
maxsim_recall@1	0.2778
maxsim_recall@3	0.425
maxsim_recall@5	0.4843
maxsim_recall@10	0.5628
maxsim_ndcg@10	0.4339
maxsim_mrr@10	0.4159
maxsim_map@100	0.3931

Multi Vector Nano BEIR

Dataset: NanoBEIR_mean
Evaluated with MultiVectorNanoBEIREvaluator with these parameters:
json
1{ 2 "dataset_names": [ 3 "climatefever", 4 "dbpedia", 5 "fever", 6 "fiqa2018", 7 "hotpotqa", 8 "msmarco", 9 "nfcorpus", 10 "nq", 11 "quoraretrieval", 12 "scidocs", 13 "arguana", 14 "scifact", 15 "touche2020" 16 ], 17 "dataset_id": "sentence-transformers/NanoBEIR-en" 18}

Metric	Value
maxsim_accuracy@1	0.4345
maxsim_accuracy@3	0.6119
maxsim_accuracy@5	0.6767
maxsim_accuracy@10	0.7584
maxsim_precision@1	0.4345
maxsim_precision@3	0.271
maxsim_precision@5	0.2108
maxsim_precision@10	0.1488
maxsim_recall@1	0.2569
maxsim_recall@3	0.3922
maxsim_recall@5	0.4528
maxsim_recall@10	0.5266
maxsim_ndcg@10	0.4774
maxsim_mrr@10	0.5363
maxsim_map@100	0.4066

Training Details

Training Dataset

msmarco-bm25

Dataset: msmarco-bm25 at ce8a493
Size: 99,000 training samples
Columns: query, positive, and negative

Approximate statistics based on the first 100 samples:

	query	positive	negative
type	string	string	string
modality	text	text	text
details	min: 5 tokens mean: 9.4 tokens max: 18 tokens	min: 28 tokens mean: 83.3 tokens max: 197 tokens	min: 24 tokens mean: 76.0 tokens max: 226 tokens

Samples:

query	positive	negative
`what is an agate made of`	`Agate is the name given to a group of silicate minerals that are made up primarily of chalcedony. Chalcedony is a member of the quartz family of minerals.`	`What is an agate, though, and what are the properties and feng shui meaning of agate? Let's find out. WHAT IS THE MEANING OF AGATE? As a form of chalcedony (type of quartz) agate exhibits a variety of colours , shapes, as well as an often present gentle iridescence.`
`what is the socratic method?`	`The Socratic Learning Method (SLM) is a constructivist learning approach consisting of four key. steps: eliciting relevant preconceptions, clarifying preconceptions, testing ones own. hypotheses or encountered propositions, and deciding whether to accept the hypotheses or. propositions.`	`The Socratic Learning Method and the Inquiry-Based Learning Method. Since the Socratic dialogues are among the earliest documented instances of learning through. inquiry, it is reasonable to argue that what is now known as inquiry-based learning can trace its. origin to the Socratic Learning Method.`
`what ditto means`	`â¢ DITTO (noun). The noun DITTO has 1 sense: 1. a mark used to indicate the word above it should be repeated. Familiarity information: DITTO used as a noun is very rare. â¢ DITTO (verb). The verb DITTO has 1 sense: 1. repeat an action or statement. Familiarity information: DITTO used as a verb is very rare.`	`Valerie Hill 0 I had texted one of my Auntie's and she sent the word Ditto back to me I was like what do that mean. Now I know!!! Acronym. DITTO in text means the same, or me too, or I agree.`

Loss: CachedMultiVectorMultipleNegativesRankingLoss with these parameters:
json
1{ 2 "score_metric": "colbert_scores", 3 "mini_batch_size": 32, 4 "mini_batch_num_tokens": null, 5 "score_mini_batch_size": 32, 6 "scale": 1.0, 7 "size_average": true, 8 "gather_across_devices": false 9}

Evaluation Dataset

msmarco-bm25

Dataset: msmarco-bm25 at ce8a493
Size: 1,000 evaluation samples
Columns: query, positive, and negative

Approximate statistics based on the first 100 samples:

	query	positive	negative
type	string	string	string
modality	text	text	text
details	min: 4 tokens mean: 9.18 tokens max: 21 tokens	min: 22 tokens mean: 76.76 tokens max: 210 tokens	min: 30 tokens mean: 84.36 tokens max: 210 tokens

Samples:

query	positive	negative
`expected return is a function of`	`security market line Security market line (SML) is the representation of the capital asset pricing model. It displays the expected rate of return of an individual security as a function of systematic, non-diversifiable risk (its beta). beta Average sensitivity of a security's price to overall securities market prices.`	`A portfolio's expected return is the sum of the weighted average of each asset's expected return. Calculate a portfolio's expected return. To calculate the expected return of a portfolio, you need to know the expected return and weight of each asset in a portfolio.`
`what smoker temperature for barbecue chicken`	`Chickens smoked hot and fast over indirect heat on the grill. But for pulled chicken, I wanted a slightly more intense smokiness to balance the barbecue sauce. I decided to do a side-by-side comparison, cooking one on the grill over indirect heat at around 375Â°F, and one in the smoker at 225Â°F.`	`Cook for an hour and 15 minutes. Check your smoked chicken breasts to make sure the temperature is still holding at about 250 degrees. Flip the chicken breasts and close the lid again. After 30 minutes, check the internal temperature of the chicken with a meat thermometer. You are looking for a temperature of 160 degrees before you can pull them off the smoker. Serve your smoked chicken breasts with a side of barbecue sauce for dipping.`
`in classification of matter what is an element`	`Matter can be in the same phase or in two different phases for this separation to take place. Key Terms. mixture: Something that consists of diverse, non-bonded elements or molecules. element: A chemical substance that is made up of a particular kind of atom and cannot be broken down or transformed by a chemical reaction.`	2. What four elements make up 96% of all living matter? The four elements that make up 96% of all living matter are oxygen, carbon, hydrogen and nitrogen. 3. What is the difference between an essential element and a trace element? An essential element is an element that an organism needs to live a healthy life and reproduce. A trace element is required by an organism in only minute quantities. Section 2 4. Sketch a model of an atom of helium, showing the electrons, protons, neutrons, and atomic nucleus. Neutrons Protons Electrons 5.

Loss: CachedMultiVectorMultipleNegativesRankingLoss with these parameters:
json
1{ 2 "score_metric": "colbert_scores", 3 "mini_batch_size": 32, 4 "mini_batch_num_tokens": null, 5 "score_mini_batch_size": 32, 6 "scale": 1.0, 7 "size_average": true, 8 "gather_across_devices": false 9}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 256
num_train_epochs: 1
learning_rate: 3e-05
warmup_steps: 0.05
bf16: True
per_device_eval_batch_size: 32
load_best_model_at_end: True
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

per_device_train_batch_size: 256
num_train_epochs: 1
max_steps: -1
learning_rate: 3e-05
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_steps: 0.05
optim: adamw_torch_fused
optim_args: None
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
optim_target_modules: None
gradient_accumulation_steps: 1
average_tokens_across_devices: True
max_grad_norm: 1.0
label_smoothing_factor: 0.0
bf16: True
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
use_liger_kernel: False
liger_kernel_config: None
use_cache: False
neftune_noise_alpha: None
torch_empty_cache_steps: None
auto_find_batch_size: False
log_on_each_node: True
logging_nan_inf_filter: True
include_num_input_tokens_seen: no
log_level: passive
log_level_replica: warning
disable_tqdm: False
project: huggingface
trackio_space_id: None
trackio_bucket_id: None
trackio_static_space_id: None
per_device_eval_batch_size: 32
prediction_loss_only: True
eval_on_start: False
eval_do_concat_batches: True
eval_use_gather_object: False
eval_accumulation_steps: None
include_for_metrics: []
batch_eval_metrics: False
save_only_model: False
save_on_each_node: False
enable_jit_checkpoint: False
push_to_hub: False
hub_private_repo: None
hub_model_id: None
hub_strategy: every_save
hub_always_push: False
hub_revision: None
load_best_model_at_end: True
ignore_data_skip: False
restore_callback_states_from_checkpoint: False
full_determinism: False
seed: 42
data_seed: None
use_cpu: False
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_pin_memory: True
dataloader_persistent_workers: False
dataloader_prefetch_factor: None
remove_unused_columns: True
label_names: None
train_sampling_strategy: random
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
ddp_static_graph: None
ddp_backend: None
ddp_timeout: 1800
fsdp: None
fsdp_config: None
deepspeed: None
debug: []
skip_memory_metrics: True
do_predict: False
resume_from_checkpoint: None
warmup_ratio: None
local_rank: -1
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}
max_length: None

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss	NanoMSMARCO_maxsim_ndcg@10	NanoNQ_maxsim_ndcg@10	NanoFiQA2018_maxsim_ndcg@10	NanoBEIR_mean_maxsim_ndcg@10	NanoClimateFEVER_maxsim_ndcg@10	NanoDBPedia_maxsim_ndcg@10	NanoFEVER_maxsim_ndcg@10	NanoHotpotQA_maxsim_ndcg@10	NanoNFCorpus_maxsim_ndcg@10	NanoQuoraRetrieval_maxsim_ndcg@10	NanoSCIDOCS_maxsim_ndcg@10	NanoArguAna_maxsim_ndcg@10	NanoSciFact_maxsim_ndcg@10	NanoTouche2020_maxsim_ndcg@10
-1	-1	-	-	0.1572	0.1851	0.1355	0.1593	-	-	-	-	-	-	-	-	-	-
0.0103	4	5.9065	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0207	8	5.7869	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0310	12	5.5644	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0413	16	5.2117	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0517	20	4.7197	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0620	24	4.0890	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0724	28	3.5701	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0827	32	3.2990	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0930	36	3.0834	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1008	39	-	1.5038	0.3511	0.2964	0.2598	0.3025	-	-	-	-	-	-	-	-	-	-
0.1034	40	2.9494	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1137	44	2.7276	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1240	48	2.4742	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1344	52	2.3246	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1447	56	2.2679	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1550	60	2.1709	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1654	64	2.1077	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1757	68	2.0022	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1860	72	1.9609	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1964	76	1.9054	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2016	78	-	1.0499	0.4417	0.4054	0.2641	0.3704	-	-	-	-	-	-	-	-	-	-
0.2067	80	1.9625	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2171	84	1.9317	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2274	88	1.9280	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2377	92	1.7935	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2481	96	1.8347	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2584	100	1.8208	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2687	104	1.8619	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2791	108	1.8128	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2894	112	1.7879	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2997	116	1.7014	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3023	117	-	0.9225	0.4534	0.4303	0.3673	0.4170	-	-	-	-	-	-	-	-	-	-
0.3101	120	1.6731	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3204	124	1.6961	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3307	128	1.7301	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3411	132	1.6529	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3514	136	1.6703	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3618	140	1.6882	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3721	144	1.6177	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3824	148	1.6462	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3928	152	1.6391	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4031	156	1.5817	0.8342	0.4936	0.4750	0.3217	0.4301	-	-	-	-	-	-	-	-	-	-
0.4134	160	1.6026	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4238	164	1.6237	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4341	168	1.5942	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4444	172	1.6211	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4548	176	1.6076	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4651	180	1.5197	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4755	184	1.5760	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4858	188	1.5535	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4961	192	1.5019	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5039	195	-	0.8004	0.5052	0.4891	0.3107	0.4350	-	-	-	-	-	-	-	-	-	-
0.5065	196	1.5746	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5168	200	1.5618	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5271	204	1.5410	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5375	208	1.5333	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5478	212	1.5448	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5581	216	1.5221	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5685	220	1.5687	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5788	224	1.5448	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5891	228	1.4613	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5995	232	1.5090	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6047	234	-	0.7690	0.4905	0.4702	0.3187	0.4265	-	-	-	-	-	-	-	-	-	-
0.6098	236	1.4572	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6202	240	1.5143	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6305	244	1.4898	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6408	248	1.4234	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6512	252	1.3648	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6615	256	1.3490	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6718	260	1.3034	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6822	264	1.3108	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6925	268	1.2706	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7028	272	1.3064	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7054	273	-	0.6826	0.4836	0.4642	0.2856	0.4112	-	-	-	-	-	-	-	-	-	-
0.7132	276	1.2720	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7235	280	1.2861	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7339	284	1.2676	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7442	288	1.2693	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7545	292	1.3310	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7649	296	1.3174	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7752	300	1.2657	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7855	304	1.2438	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7959	308	1.2717	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8062	312	1.2444	0.6657	0.5069	0.4660	0.3159	0.4296	-	-	-	-	-	-	-	-	-	-
0.8165	316	1.2162	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8269	320	1.2765	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8372	324	1.2029	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8475	328	1.2716	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8579	332	1.2241	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8682	336	1.2276	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8786	340	1.2421	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8889	344	1.2431	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8992	348	1.2105	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.907	351	-	0.6562	0.5197	0.4985	0.3225	0.4469	-	-	-	-	-	-	-	-	-	-
0.9096	352	1.2252	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9199	356	1.2255	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9302	360	1.2393	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9406	364	1.1916	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9509	368	1.1789	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9612	372	1.2024	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9716	376	1.2324	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9819	380	1.2035	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9922	384	1.2166	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.0	387	-	0.6495	0.5036	0.4702	0.3279	0.4339	-	-	-	-	-	-	-	-	-	-
-1	-1	-	-	0.5197	0.4985	0.3225	0.4774	0.2154	0.4666	0.7635	0.5294	0.3066	0.8471	0.2974	0.4229	0.5832	0.4341

The bold row denotes the saved checkpoint.

Training Time

Training: 41.5 minutes
Evaluation: 7.2 minutes
Total: 48.7 minutes

Framework Versions

Python: 3.11.6
Sentence Transformers: 5.7.0.dev0
Transformers: 5.13.1
PyTorch: 2.10.0+cu128
Accelerate: 1.14.0
Datasets: 4.8.4
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultiVectorMultipleNegativesRankingLoss

bibtex

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

tomaarsen/multivector-ModernBERT-base-msmarco-cached-contrastive-no-query-expansion

ColBERT ModernBERT-base trained on MS MARCO triplets with GradCache

Model Details

Model Description

Model Sources

Full Model Architecture

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Multi Vector Information Retrieval

Multi Vector Nano BEIR

Multi Vector Nano BEIR

Training Details

Training Dataset

msmarco-bm25

Evaluation Dataset

msmarco-bm25

Training Hyperparameters

Non-Default Hyperparameters

All Hyperparameters

Training Logs

Training Time

Framework Versions

Citation

BibTeX

Sentence Transformers

CachedMultiVectorMultipleNegativesRankingLoss

tomaarsen/
multivector-ModernBERT-base-msmarco-cached-contrastive-no-query-expansion