You can deploy this model and start earning money today!

lightonai/
DenseOn-unsupervised

Name: DenseOn-unsupervised
Rating: 2.5 (250 reviews)

sentence-transformers

text-embeddings-inference

endpoints_compatible

Views

No views yet

Website

📚 Collection | 📝 Blog

DenseOn-unsupervised

State-of-the-Art Dense Retrieval Model by LightOn

DenseOn | LateOn | PyLate | FastPLAID

🎯 TL;DR: The intermediate dense checkpoint produced by Stage 1 only of the DenseOn pipeline: large-scale unsupervised contrastive pre-training on filtered query-document pairs. Released as a strong starting point for your own supervised fine-tuning, knowledge distillation, or downstream adaptation.

About the LateOn / DenseOn Family

State-of-the-art retrieval is increasingly dominated by closed models, either hidden behind APIs or trained on undisclosed data. This blocks reproducibility, prevents study of possible data leakage, and gatekeeps progress to a handful of private labs. We thus decided to gather and curate a large amount of data and explore various mixtures. We release all the data used in our explorations:

Gathered pre-training data, 1.4B query-documents pairs alongside annotations used for non-destructive filtering (structural filtering, deduplication, cross-encoder pair relevancy)
Best pre-training mixture found with already applied filters
Fine-tuning datasets with query, positive and 2048 mined documents alongside their scores for 1.88M samples.

Based on our findings, we trained LateOn (multi-vector/ColBERT) and DenseOn (single vector/dense) models on a proprietary Apache 2.0-compatible training dataset and release those models as well. Both are built on the ModernBERT backbone at 149M parameters, a size we believe sits at the sweet spot: large enough to handle real-world queries and documents, small enough to serve at high throughput in latency-sensitive production systems. For more information, please read our blogpost.

DenseOn-unsupervised

DenseOn-unsupervised is the output of the first stage of the DenseOn training pipeline. It has been pre-trained on a large, filtered corpus of query-document pairs using in-batch contrastive learning, but has not yet been fine-tuned with mined hard negatives.

For most production use cases, you should use the fully-trained DenseOn instead. This unsupervised checkpoint is intended for:

Researchers studying what each pipeline stage contributes
Practitioners who want to fine-tune on their own domain-specific data
Distillation experiments where you want to start from a strong but un-aligned base
Anyone running their own ablations on hard-negative mining strategies

Results

BEIR (14 datasets, NDCG@10)

Model	Average	Size	Emb dim	ArguAna	CQADupstackRetrieval	ClimateFEVER	DBPedia	FEVER	FiQA2018	HotpotQA	MSMARCO	NFCorpus	NQ	QuoraRetrieval	SCIDOCS	SciFact	TRECCOVID	Touche2020
modernbert-embed-base	52.89	149	768	48.96	42.08	35.67	41.50	87.35	40.59	67.11	41.47	33.40	62.15	88.85	18.59	69.63	84.15	31.91
bge-large-en-v1.5	54.34	335	1024	64.52	42.23	36.57	44.11	87.18	45.02	74.10	42.49	38.06	55.03	89.07	22.63	74.64	74.70	24.81
gte-modernbert-base	55.19	149	768	74.56	42.64	45.90	41.39	93.98	49.54	70.39	39.93	34.32	56.10	88.57	20.44	76.41	75.75	17.97
snowflake-arctic-embed-l-v2.0	55.22	568	1024	59.11	45.88	41.82	43.40	91.54	45.35	68.15	44.86	35.08	63.67	88.75	20.28	70.90	83.63	25.89
jina-embeddings-v5-text-nano	56.06	239	768	65.70	44.66	39.60	45.26	89.51	47.85	69.07	41.64	38.69	63.38	88.87	22.60	75.78	77.60	30.70
Qwen3-Embedding-0.6B	55.52	600	1024	70.97	46.03	42.11	39.48	88.15	46.61	65.74	37.99	36.71	53.46	87.78	24.41	69.72	90.52	33.18
pplx-embed-v1-0.6b	56.70	600	1024	60.45	45.96	39.82	44.30	90.66	52.05	74.41	43.86	35.80	62.04	88.96	22.84	74.78	85.63	28.98
DenseOn-unsupervised	49.05	149	768	54.94	46.28	18.20	37.39	70.68	52.34	59.77	29.30	37.92	50.62	88.98	23.05	76.35	68.12	21.87
DenseOn	56.20	149	768	54.65	46.89	37.49	44.65	90.69	53.86	74.51	43.58	39.03	59.25	89.31	22.35	75.95	82.33	28.43

DenseOn reaches 56.20 NDCG@10 on BEIR, making it the top base-size dense retriever and the first sub-150M model to clear the 56 bar. At 149M parameters it decisively beats GTE-ModernBERT (55.19) at the same size, and more tellingly outperforms snowflake-arctic-embed-l-v2.0 (55.22, 568M) and Qwen3-Embedding-0.6B (55.52, 595M) despite being roughly 4× smaller. DenseOn also stays within half a point of the strongest current-generation dense baselines, pplx-embed-v1-0.6B (56.70, 596M) and jina-embeddings-v5-text-nano (56.08, 239M), both substantially larger.

Decontaminated BEIR (12 datasets, NDCG@10)

Standard benchmarks risk overestimating model quality when training data overlaps with evaluation corpora. This is a non-negligible risk in our case, as our mixture explorations are mostly built on BEIR evaluation. To quantify this and ensure that our model has not memorized possible leakage, we built decontaminated versions of the BEIR datasets by removing samples found in both the mGTE training dataset and in our internal training datasets. Since many retrieval models draw from similar public sources (Wikipedia, MS MARCO, Common Crawl, academic corpora), we expect significant overlap across models and believe the decontaminated benchmarks provide a meaningful, if imperfect, stress test. The decontaminated datasets are publicly available on HuggingFace.

Despite being in the toughest position (as the decontamination is based on our data), LateOn and DenseOn stay consistent under decontamination. LateOn keeps its #1 position and DenseOn stays in the top four (only falling behind our other strong multi-vector model, ColBERT-Zero). Neither model flinches, which is direct evidence of generalization rather than overfitting.

More broadly, ColBERT models seems to generalize better under decontamination: all three ColBERT models hold or improve their ranking, and they take 2 of the top 3 decontaminated positions. Although DenseOn holds strong, some dense models are hit harder, for example GTE-ModernBERT, dropping from 8th to last, which is particularly interesting considering our base mixture is derived from theirs. This highlights the strength of our curation methodology. While other models such as Qwen3-Embedding-0.6B also drop some ranks, hinting at an overlap with the BEIR evaluation, it is worth noting that newer models, such as the new jina-embeddings-v5 and pplx-embed-v1-0.6b seems to exhibit stronger evidence of generalization rather than overfitting.

Related Checkpoints

Model	Stage	Link
DenseOn-unsupervised (this card)	Pre-training only	`lightonai/DenseOn-unsupervised`
DenseOn	Pre-training + fine-tuning	`lightonai/DenseOn`
LateOn-unsupervised	Multi-vector counterpart, pre-training only	`lightonai/LateOn-unsupervised`
LateOn	Multi-vector counterpart, full pipeline	`lightonai/LateOn`

Model Details

Model Description

Model Type: Sentence Transformer
Base model: ModernBERT-base
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Language: English
License: Apache 2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("lightonai/DenseOn-unsupervised")
# Run inference
queries = [
    "Which planet is known as the Red Planet?",
]
documents = [
    "Venus is often called Earth's twin because of its similar size and proximity.",
    'Mars, known for its reddish appearance, is often referred to as the Red Planet.',
    'Saturn, famous for its rings, is sometimes mistaken for the Red Planet.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.3464, 0.4823, 0.5147]])

Training Details

Framework Versions

Python: 3.11.10
Sentence Transformers: 5.1.1
Transformers: 4.57.5
PyTorch: 2.9.0+cu128
Accelerate: 1.12.0
Datasets: 3.6.0
Tokenizers: 0.22.1

Citation

BibTeX

DenseOn and LateOn

bibtex

@misc{sourty2026denseonlateon,
  title={DenseOn with the LateOn: Open State-of-the-Art Single and Multi-Vector Models},
  author={Sourty, Raphael and Chaffin, Antoine and Weller, Orion and Demoura, Paulo and Chatelain, Amelie},
  year={2026},
  howpublished={\url{https://huggingface.co/blog/lightonai/denseon-lateon}},
}

PyLate

bibtex

@inproceedings{DBLP:conf/cikm/ChaffinS25,
  author       = {Antoine Chaffin and
                  Rapha{\"{e}}l Sourty},
  editor       = {Meeyoung Cha and
                  Chanyoung Park and
                  Noseong Park and
                  Carl Yang and
                  Senjuti Basu Roy and
                  Jessie Li and
                  Jaap Kamps and
                  Kijung Shin and
                  Bryan Hooi and
                  Lifang He},
  title        = {PyLate: Flexible Training and Retrieval for Late Interaction Models},
  booktitle    = {Proceedings of the 34th {ACM} International Conference on Information
                  and Knowledge Management, {CIKM} 2025, Seoul, Republic of Korea, November
                  10-14, 2025},
  pages        = {6334--6339},
  publisher    = {{ACM}},
  year         = {2025},
  url          = {https://github.com/lightonai/pylate},
  doi          = {10.1145/3746252.3761608},
}

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084"
}

Acknowledgements

We thank Xin Zhang, Zach Nussbaum, Tom Aarsen, Bo Wang, Eugene Yang, Benjamin Clavié, Nandan Thakur, Oskar Hallström and Iacopo Poli for their valuable contributions and feedback. We are grateful to the teams behind Sentence Transformers and the BEIR benchmark, and to the open-source retrieval community, in particular the authors of Nomic Embed.

This work was granted access to the HPC resources of IDRIS under GENCI allocations AS011016449, A0181016214, and A0171015706 (Jean Zay supercomputer). We also acknowledge the Barcelona Supercomputing Center (BSC-CNS) for providing access to MareNostrum 5 under EuroHPC AI Factory Fast Lane project EHPC-AIF-2025FL01-445.