You can deploy this model and start earning money today!

illuin-conteb/
modernbert-large-insent

Name: modernbert-large-insent
Availability: InStock

sentence-transformers

answerdotai/ModernBERT-large

finetune

autotrain_compatible

text-embeddings-inference

endpoints_compatible

Views

No views yet

ModernBERT-embed-large + InSeNT

GitHub

Hugging Face

This is a contextual model finetuned from modernbert-embed-large on the ConTEB training dataset. It was trained using the InSeNT training approach, detailed in the corresponding paper.

[!WARNING] This experimental model stems from the paper Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings. While results are promising, we have seen regression on standard embedding tasks, and using it in production will probably require further work on extending the training set to improve robustness and OOD generalization.

Usage

Direct Usage

First install the contextual-embeddings package:

pip install git+https://github.com/illuin-tech/contextual-embeddings

To run inference with a contextual model, you can use the following examples:

python

from contextual_embeddings import LongContextEmbeddingModel
from sentence_transformers import SentenceTransformer

documents = [
    [
        "The old lighthouse keeper trimmed his lamp, its beam cutting a lonely path through the fog.",
        "He remembered nights of violent storms, when the ocean seemed to swallow the sky whole.",
        "Still, he found comfort in his duty, a silent guardian against the treacherous sea."
    ],
    [
        "A curious fox cub, all rust and wonder, ventured out from its den for the first time.",
        "Each rustle of leaves, every chirping bird, was a new symphony to its tiny ears.",
        "Under the watchful eye of its mother, it began to learn the secrets of the whispering forest."
    ]
]
base_model = SentenceTransformer("illuin-conteb/modernbert-large-insent")
contextual_model = LongContextEmbeddingModel(
    base_model=base_model,
    add_prefix=True
)
embeddings = contextual_model.embed_documents(documents)
print("Length of embeddings:", len(embeddings)) # 2
print("Length of first document embedding:", len(embeddings[0])) # 3
print(f"Shape of first chunk embedding: {embeddings[0][0].shape}") # torch.Size([768])

Model Details

Model Description

Model Type: Sentence Transformer
Maximum Sequence Length: 8192 tokens
Output Dimensionality: 768 dimensions
Similarity Function: cosine
Training Dataset:
- train

Model Sources

Repository: Contextual Embeddings
Hugging Face: Contextual Embeddings

Citation

bibtex

@misc{conti2025contextgoldgoldpassage,
      title={Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings}, 
      author={Max Conti and Manuel Faysse and Gautier Viaud and Antoine Bosselut and Céline Hudelot and Pierre Colombo},
      year={2025},
      eprint={2505.24782},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2505.24782}, 
}