Continuous Pretraining with 4.6 billion multilingual tokens using knowledge distillation from state-of-the-art reranker models. GTE-ModernColBERT Foundation building upon the English-focused lightonai/GTE-ModernColBERT-v1 model. Multilingual Enhancement extending English capabilities to European languages through targeted multilingual training.

🎯 Core Features and Innovations:

Multilingual Continuous Pretraining: Enhanced with 4,641,714,000 multilingual tokens covering 7 European languages while learning from powerful reranker models
English-to-Multilingual Transfer: Successfully extends the strong English performance of GTE-ModernColBERT to European languages
Compressed Architecture: Maintains the efficient 149M parameter design of ModernColBERT

💪 From English Excellence to Multilingual Mastery

Starting from the strong GTE-ModernColBERT-v1 foundation – a model optimized for English retrieval – we've expanded its capabilities through:

4.6 billion multilingual tokens covering 7 European languages
Knowledge distillation from state-of-the-art reranker models
Continuous pretraining that preserves English strength while adding multilingual capabilities

This creates a truly multilingual retriever that maintains exceptional English performance while delivering strong results across European languages.

Model Overview

Model: VAGOsolutions/SauerkrautLM-Multi-ModernColBERT
Base: Continuous pretrained from lightonai/GTE-ModernColBERT-v1 using knowledge distillation
Architecture: PyLate / ColBERT (Late Interaction) with ModernBERT backbone
Languages: Multilingual (optimized for 7 European languages: German, English, Spanish, French, Italian, Dutch, Portuguese)
License: Apache 2.0
Model Size: 149M parameters Additional Training: 4.6B multilingual tokens via knowledge distillation

Model Description

Model Type: Multi-vector embedding model with innovative Late Interaction architecture
Document Length: 8192 tokens (32× longer than traditional BERT models)
Query Length: 256 tokens (optimized for complex, multi-part queries)
Output Dimensionality: 128 tokens (efficient vector representation)
Similarity Function: MaxSim (enables precise token-level matching)
Training Method: Continuous pretraining with knowledge distillation

Architecture

ColBERT(
  (0): Transformer(CompressedModernBertModel)
  (1): Dense(384 -> 128 dim, no bias)
)

🔬 Technical Innovations in Detail

Multilingual Continuous Pretraining

Our approach transforms an English-specialized model into a multilingual powerhouse:

Base Model Selection: Starting with GTE-ModernColBERT-v1, which provides state-of-the-art English retrieval
Multilingual Enhancement: 4,641,714,000 tokens across 7 European languages
Knowledge Distillation: Learning from state-of-the-art reranker models throughout the training
Balanced Training: Ensuring strong multilingual capabilities without degrading English performance

Architectural Advantages

SauerkrautLM-Multi-ModernColBERT leverages:

ModernBERT Efficiency: Compressed architecture with 149M parameters
Late Interaction Benefits: Token-level matching for precise retrieval
Cross-lingual Transfer: Successfully extends English capabilities to multiple languages
Maintained Performance: Preserves the strong English foundation while adding languages

This architecture combines the efficiency of ModernColBERT with true multilingual capabilities.

🔬 Benchmarks: Multilingual Retrieval Performance

Our evaluation demonstrates strong multilingual retrieval performance, successfully extending GTE-ModernColBERT's English excellence to European languages.

NanoBEIR Europe (multilingual retrieval)

Average nDCG@10 across seven European languages, showing the effectiveness of our multilingual continuous pretraining:

Language	nDCG@10	Performance Notes
en	67.70	Maintains exceptional English performance from base model
de	51.21	Strong german language transfer
es	54.73	Excellent spanish language capabilities
fr	54.44	Consistent cross-lingual performance
it	53.87	Balanced multilingual representation
nl	52.15	Effective on closely related languages
pt	53.80	Maintains quality across language families

Key Observations:

Preserved English Excellence: The continuous pretraining maintains the exceptional English performance (67.70 nDCG@10) from GTE-ModernColBERT
Strong Multilingual Addition: All non-English languages achieve strong performance (51-55 nDCG@10)
Successful Transfer: The model effectively transfers English capabilities to European languages
Balanced Performance: Consistent results across different language families

Why SauerkrautLM-Multi-ModernColBERT Matters for Production

Strong language capabilities for european languages: Maintains state-of-the-art English while adding languages
Efficient Architecture: 149M parameters deployable on standard infrastructure
True Multilingual: Single model for 7 European languages
Knowledge Distillation Benefits: Learns from models many times its size
Drop-in Replacement: Can replace English-only ColBERT models with multilingual support

This model serves as an excellent solution for:

Organizations expanding from English to European markets
Multilingual search systems requiring strong English
Cross-lingual retrieval applications
Systems needing efficient multilingual models

Real-World Applications

The combination of strong English foundation and multilingual capabilities enables:

Global Search Systems: Single model for international deployments
E-commerce Expansion: English-first companies entering European markets
Multilingual Documentation: Technical documentation search across languages
Customer Support: Unified search across multilingual knowledge bases
Research Applications: Cross-lingual academic literature retrieval

📈 Summary: English Excellence, Multilingual Capability

SauerkrautLM-Multi-ModernColBERT demonstrates how continuous pretraining can successfully extend an English-specialized model to multiple languages. By combining:

GTE-ModernColBERT's strong English foundation
4.6 billion tokens of multilingual training
Knowledge distillation from advanced rerankers
Efficient ModernBERT architecture

We've created a model that excels in English (67.70 nDCG@10) while delivering strong performance across all European languages. This makes it an ideal choice for organizations that need both exceptional English retrieval and comprehensive multilingual support in a single, efficient model.

Model

This is a multi-vector (ColBERT-style late interaction) embedding model. It maps sentences & paragraphs to sequences of 128-dimensional dense vectors and can be used for semantic textual similarity using the MaxSim operator.

Usage

Sentence Transformers

This model can be used with Sentence Transformers as a multi-vector (ColBERT-style late interaction) retriever via the MultiVectorEncoder:

pip install "sentence-transformers>=6.0.0"

python

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("VAGOsolutions/SauerkrautLM-Multi-ModernColBERT")

query = "Welcher Planet ist als der Rote Planet bekannt?"
documents = [
    "Venus wird wegen ihrer ähnlichen Größe und Nähe oft als Erdzwilling bezeichnet.",
    "Mars, bekannt für sein rötliches Aussehen, wird oft als der Rote Planet bezeichnet.",
    "Jupiter, der größte Planet in unserem Sonnensystem, hat einen markanten roten Fleck.",
    "Saturn, berühmt für seine Ringe, wird manchmal für den Roten Planeten gehalten.",
]

query_embeddings = model.encode_query(query)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings[0].shape)
# (32, 128) (29, 128)

# MaxSim late-interaction scoring (higher is more relevant)
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
# tensor([[28.7305, 29.5820, 29.0117, 29.1172]])

PyLate

First install the PyLate library:

pip install -U pylate

Retrieval

PyLate provides a streamlined interface to index and retrieve documents using ColBERT models. The index leverages the Voyager HNSW index to efficiently handle document embeddings and enable fast retrieval.

Indexing documents

First, load the ColBERT model and initialize the Voyager index, then encode and index your documents:

python

from pylate import indexes, models, retrieve

# Step 1: Load the ColBERT model
model = models.ColBERT(
    model_name_or_path="VAGOsolutions/SauerkrautLM-Multi-ModernColBERT",
)

# Step 2: Initialize the Voyager index
index = indexes.Voyager(
    index_folder="pylate-index",
    index_name="index",
    override=True,  # This overwrites the existing index if any
)

# Step 3: Encode the documents
documents_ids = ["1", "2", "3"]
documents = ["document 1 text", "document 2 text", "document 3 text"]

documents_embeddings = model.encode(
    documents,
    batch_size=32,
    is_query=False,  # Ensure that it is set to False to indicate that these are documents, not queries
    show_progress_bar=True,
)

# Step 4: Add document embeddings to the index by providing embeddings and corresponding ids
index.add_documents(
    documents_ids=documents_ids,
    documents_embeddings=documents_embeddings,
)

Note that you do not have to recreate the index and encode the documents every time. Once you have created an index and added the documents, you can re-use the index later by loading it:

python

# To load an index, simply instantiate it with the correct folder/name and without overriding it
index = indexes.Voyager(
    index_folder="pylate-index",
    index_name="index",
)

Retrieving top-k documents for queries

Once the documents are indexed, you can retrieve the top-k most relevant documents for a given set of queries. To do so, initialize the ColBERT retriever with the index you want to search in, encode the queries and then retrieve the top-k documents to get the top matches ids and relevance scores:

python

# Step 1: Initialize the ColBERT retriever
retriever = retrieve.ColBERT(index=index)

# Step 2: Encode the queries
queries_embeddings = model.encode(
    ["query for document 3", "query for document 1"],
    batch_size=32,
    is_query=True,  #  # Ensure that it is set to False to indicate that these are queries
    show_progress_bar=True,
)

# Step 3: Retrieve top-k documents
scores = retriever.retrieve(
    queries_embeddings=queries_embeddings,
    k=10,  # Retrieve the top 10 matches for each query
)

Reranking

If you only want to use the ColBERT model to perform reranking on top of your first-stage retrieval pipeline without building an index, you can simply use rank function and pass the queries and documents to rerank:

python

from pylate import rank, models

queries = [
    "query A",
    "query B",
]

documents = [
    ["document A", "document B"],
    ["document 1", "document C", "document B"],
]

documents_ids = [
    [1, 2],
    [1, 3, 2],
]

model = models.ColBERT(
    model_name_or_path="VAGOsolutions/SauerkrautLM-Multi-ModernColBERT",
)

queries_embeddings = model.encode(
    queries,
    is_query=True,
)

documents_embeddings = model.encode(
    documents,
    is_query=False,
)

reranked_documents = rank.rerank(
    documents_ids=documents_ids,
    queries_embeddings=queries_embeddings,
    documents_embeddings=documents_embeddings,
)

Citation

BibTeX

SauerkrautLM‑Multi‑ModernColBERT

bibtex

@misc{SauerkrautLM-Multi-ModernColBERT,
  title={SauerkrautLM-Multi-ModernColBERT},
  author={David Golchinfar},
  url={https://huggingface.co/VAGOsolutions/SauerkrautLM-Multi-ModernColBERT},
  year={2025}
}

GTE-ModernColBERT

bibtex

@misc{GTE-ModernColBERT,
title={GTE-ModernColBERT},
author={Chaffin, Antoine},
url={https://huggingface.co/lightonai/GTE-ModernColBERT-v1},
year={2025}
}

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
  title = {Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
  author = {Reimers, Nils and Gurevych, Iryna},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing},
  month = {11},
  year = {2019},
  publisher = {Association for Computational Linguistics},
  url = {https://arxiv.org/abs/1908.10084}
}

PyLate

bibtex

@misc{PyLate,
  title={PyLate: Flexible Training and Retrieval for Late Interaction Models},
  author={Chaffin, Antoine and Sourty, Raphaël},
  url={https://github.com/lightonai/pylate},
  year={2024}
}

Acknowledgements

We thank the PyLate team for providing the training framework that made this work possible, and the LightOn AI team for creating the excellent GTE-ModernColBERT base model.