LateOn-hpool-regularized is a ColBERT-style multi-vector retrieval model trained by LightOn using PyLate. It is built on the LateOn family and regularized specifically to make hierarchical pooling much more effective.
Late interaction models are powerful because they store one embedding per document token and score with MaxSim. The downside is index size: long documents require many vectors. We previously introduced hierarchical pooling to reduce this footprint by clustering and merging similar token embeddings inside each document. This model is trained so that such pooling preserves retrieval quality even at aggressive compression ratios.
The headline result is simple:
LateOn-hpool-regularized reaches 98.7% retention at a 32-token document budget, corresponding to roughly 5× compression, with negligible impact on full-token retrieval performance.
This model is the recommended LateOn checkpoint when you want a versatile ColBERT that works well across all compression levels. It is trained with multiple pooling budgets ([4, 8, 16, 32, 64, 128, 300]) so you can set the compression ratio at inference time. If you know your deployment target, training specifically for that budget pushes retention even further (see Targeted Training below).
If you only use full-token PLAID retrieval without pooling, LateOn or LateOn-regularized remain strong choices.
For the complete methodology and analysis, refer to our blog post.
Method
Hierarchical pooling contains a discrete operation: cluster assignment. Tokens are merged according to a hierarchical clustering tree, and this assignment is not differentiable. To train through it, we use a Straight-Through Estimator.
During training:
The model encodes query and document tokens normally.
Document token embeddings are hierarchically clustered and pooled in the forward pass.
Retrieval scores are computed on the pooled document representation.
The loss combines the standard full-token MaxSim loss with a pooled-representation loss.
During the backward pass, the Straight-Through Estimator lets gradients flow through the non-differentiable pooling decision.
The total loss is a weighted sum of the full-token MaxSim contrastive loss and the pooled-representation contrastive loss.
This released checkpoint uses multi-budget training with targets [4, 8, 16, 32, 64, 128, 300] at each step, which encourages the model to become broadly more poolable instead of overfitting to one compression point.
[!IMPORTANT]
Important note: while we converged towards this approach very naturally due to our previous projection work and having introduced the hierarchical pooling approach, it is important to note that an independent researcher (Stefan Josef) did the same kind of study a few months ago. Although we did not have this work in mind while converging back to the idea, we were totally aware of the work and actually chatted with the author during the LIR workshop. Thus, we would like to highlight that the paternity of the idea should be given to Stefan and not to us. You can see these notes as an independent rediscovery, highlighting how much the idea makes sense from the beginning, and most importantly, a scaling up of the study to more data. Stefan is an independent researcher whose compute capacity was limited (as stated in the paper). When scaled to LateOn data, the results become even stronger, so we are happy to share them (alongside the model), but again, please have a look at the original work
Results
All of the results can be found in this space. The results are only on a subset of the 7 smaller BEIR datasets to make experiments easier and match AGC setup (SciFact, NFCorpus, ArguAna, TREC-Covid, FiQa, Touché, SciDocs). Given the original results of the hpool paper, we do not expect much different from running the whole BEIR suit.
Hierarchical Pooling Retention at 32 Tokens
The table below reports retention at a 32-token document budget, measured against the full-token LateOn reference.
Model / Configuration
Retention @ 32 Tokens
Full NDCG@10
LateOn, no regularization
77.1%
0.506
LateOn-regularized
88.3%
0.508
+ hpool regularization, λ=0.5
94.9%
0.512
+ hpool regularization, λ=0.7
95.8%
0.510
+ hpool regularization, λ=1.0
96.7%
0.507
+ hpool regularization, λ=1.5
97.9%
0.504
The regularization strongly improves pooling retention while preserving full-token retrieval quality. Even at λ=1.5 (this model), where the model is optimized aggressively for pooling, full-token performance remains essentially unchanged.
Targeted Training
If you know your deployment compression budget, you can train specifically for that target instead of using multi-budget training. With targeted r=32 training at λ=0.7:
Config
Retention @ 32 Tokens
Full NDCG@10
multi-r λ=0.7 (r∈{4..300})
95.8%
0.510
single-r λ=0.7 (r=32 only)
97.4%
0.509
Training at the deployment target gains 1.6 percentage points of retention with virtually no cost to full-token quality. The trade-off: the model is less versatile at other (higher) compression levels.
How does this compare to learned pooling?
A fair question: if we're going to train for pooling anyway, why not learn the pooling function itself? The recent Attention-Guided Clustering (AGC) method does exactly that, replacing hierarchical clustering with a learned attention mechanism that decides which tokens to merge.
On the 7 BEIR datasets they evaluated, AGC reaches an average retention of 97.4% at 32 tokens. Our hierarchical pooling regularization at λ=1.5 reaches 97.9%, while being trained for all cutoffs and thus being very versatile.
A few caveats: the training data and base model differ, so the comparison is not apples-to-apples. Our base model is also stronger (0.506 vs 0.462 NDCG@10). Take the numbers with a grain of salt, the point is not to claim superiority but to show that a simpler, more flexible method can reach comparable territory.
Hierarchical pooling sets the budget at inference time. You can choose to compress more or less your documents with the same model and methodology, to set the quality-storage trade off. You compute the dendrogram and put the cut point where you want.
Adaptive Pooling
Not all documents carry the same density of information. With adaptive pooling, easy-to-compress documents receive fewer vectors while harder documents retain more. The average budget remains the same, but the token allocation is more efficient.
Model / Configuration
Pooling Strategy
Retention @ 32 Tokens
hpool reg, λ=1.5 (this model)
Fixed budget
97.9%
hpool reg, λ=1.5 (this model)
Adaptive budget
98.7%
targeted r=32, λ=0.7
Fixed budget
97.4%
targeted r=32, λ=0.7
Adaptive budget
99.4%
This model with adaptive pooling reaches 98.7% retention at ~5× compression. Combining targeted training with adaptive pooling pushes this to 99.4%.
Cumulative Improvement
Step
Retention @ 32 Tokens
Improvement
Method
LateOn, no regularization
77.1%
—
Base model
+ MUVERA-style regularization
88.3%
+11.2 pp
General geometry regularization
+ hpool reg, λ=0.7 multi-r
95.8%
+7.5 pp
Pooling-specific regularization
+ targeted r=32 training
97.4%
+1.6 pp
Training at the deployment target
+ adaptive budgeting
99.4%
+2.0 pp
Adaptive inference-time allocation
Total improvement
+22.3 pp
Overall, the full stack improves retention by 22.3 percentage points at the 32-token compression point. Note that, although the gain from MUVERA regularization seems very large, directly training the base model for hierarchical pooling yields very similar results, although slightly worse.
When Should You Use This Model?
Use LateOn-hpool-regularized if you want a versatile ColBERT model that works well across all compression levels via hierarchical pooling, with the ability to set the budget at inference time.
For maximum retention at a known deployment target, consider training a targeted single-budget variant using the training code in PyLate. See the blog post for details.
Use LateOn-regularized instead if your main goal is compatibility with projection-based candidate generation methods such as MUVERA or SMVE.
Use LateOn if you only care about full-token PLAID retrieval and do not need pooling or projection-oriented compression.
Related Checkpoints
Model
Description
Link
LateOn-hpool-regularized
STE-regularized for hierarchical pooling and reduced index footprint
This model can be used with Sentence Transformers as a multi-vector (ColBERT-style late interaction) retriever via the MultiVectorEncoder:
pip install "sentence-transformers>=6.0.0"
python
1from sentence_transformers import MultiVectorEncoder
23model = MultiVectorEncoder("lightonai/LateOn-hpool-regularized")45query ="Which planet is known as the Red Planet?"6documents =[7"Venus is often called Earth's twin because of its similar size and proximity.",8"Mars, known for its reddish appearance, is often referred to as the Red Planet.",9"Jupiter, the largest planet in our solar system, has a prominent red spot.",10"Saturn, famous for its rings, is sometimes mistaken for the Red Planet.",11]1213query_embeddings = model.encode_query(query)14document_embeddings = model.encode_document(documents)15print(query_embeddings.shape, document_embeddings[0].shape)16# (12, 128) (18, 128)1718# MaxSim late-interaction scoring (higher is more relevant)19scores = model.similarity(query_embeddings, document_embeddings)20print(scores)21# tensor([[11.3723, 11.6385, 11.5022, 11.6186]])
PyLate
First install PyLate:
pip install -U pylate
Retrieval with PLAID
Load the ColBERT model and initialize a PLAID index.
1@misc{chaffin2026hpoolregularization,
2 title={After the party comes the free lunch: regularizing ColBERT models to enhance pooling capabilities and reduce index footprint},
3 author={Chaffin, Antoine},
4 year={2026},
5 howpublished={\url{https://huggingface.co/blog/lightonai/lateon-hpool-regularization}},
6}
Learn to Pool
bibtex
1@inproceedings{learntopool,
2 author = {Stefan Josef},
3 title = {Learn to Pool: Lightweight Fine-Tuning for Flexible Multi-Vector Compression},
4 booktitle = {Proceedings the 1st Late Interaction Workshop (LIR) @ ECIR 2026},
5 year = {2025},
6}
LateOn Regularization
bibtex
1@misc{chaffin2026lateonregularization,
2 title={Party is over: regularizing ColBERT models to fix efficient ANN methods},
3 author={Chaffin, Antoine},
4 year={2026},
5 howpublished={\url{https://huggingface.co/blog/lightonai/lateon-regularization}},
6}
Hierarchical Pooling
bibtex
1@misc{clavie2024reducingfootprintmultivectorretrieval,
2 title={Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling},
3 author={Benjamin Clavié and Antoine Chaffin and Griffin Adams},
4 year={2024},
5 eprint={2409.14683},
6 archivePrefix={arXiv},
7 primaryClass={cs.IR},
8 url={https://arxiv.org/abs/2409.14683},
9}
DenseOn and LateOn
bibtex
1@misc{sourty2026denseonlateon,
2 title={DenseOn with the LateOn: Open State-of-the-Art Single and Multi-Vector Models},
3 author={Sourty, Raphael and Chaffin, Antoine and Weller, Orion and Demoura, Paulo and Chatelain, Amelie},
4 year={2026},
5 howpublished={\url{https://huggingface.co/blog/lightonai/denseon-lateon}},
6}
PyLate
bibtex
1@inproceedings{DBLP:conf/cikm/ChaffinS25,
2 author = {Antoine Chaffin and Raphaël Sourty},
3 title = {PyLate: Flexible Training and Retrieval for Late Interaction Models},
4 booktitle = {Proceedings of the 34th ACM International Conference on Information and Knowledge Management},
5 year = {2025},
6 url = {https://github.com/lightonai/pylate},
7 doi = {10.1145/3746252.3761608},
8}
