Training data: this model was contrastively pretrained on capemox/denseon-pretrain-50m-balanced, a 50,000,000-pair sample drawn from the DenseOn corpus lightonai/embeddings-pre-training-curated (665M curated query–document pairs across 34 sources). Pairs are sampled with T=2 temperature weighting + iterative equal-redistribution capping so that no single source dominates the mix.

This is a Stage 1 (pretraining-only) checkpoint — it has not been fine-tuned on supervised retrieval data. Use it as a strong starting point for Stage 2 fine-tuning, or as a zero-shot retrieval encoder.

Training recipe


Base model	`jhu-clsp/ettin-encoder-32m` (ModernBERT, ~32M params)
Training data	`capemox/denseon-pretrain-50m-balanced` (50M pairs, DenseOn split)
Loss	`MultipleNegativesRankingLoss` (full, in-batch negatives)
Batch size	1024 (1023 in-batch negatives/anchor)
Per-source batching	each batch drawn from one source dataset (DenseOn recipe)
Learning rate	3e-5, linear decay, 5% warmup
Epochs	1 (~48,827 steps)
Precision	bf16 + tf32, SDPA, torch.compile
Hardware	1× A100-80GB, ~9h 55m

Evaluation (Stage 1 checkpoint, NDCG@10)

BEIR subset (zero-shot, no fine-tuning):

Dataset	NDCG@10
ArguAna	0.4810
FiQA2018	0.3107
NFCorpus	0.2841
SCIDOCS	0.1796
SciFact	0.6448
TRECCOVID	0.5814
Mean	0.4136

NanoBEIR (13-dataset aggregate, end of training): 0.5300

Downstream value

When this checkpoint is fine-tuned on MS MARCO hard negatives (tomaarsen/msmarco-Qwen3-Reranker-0.6B), it reaches 0.3225 mean BEIR NDCG@10 — versus 0.2319 for the same fine-tune starting from the raw base model (+39% from this pretraining step).

Usage

python

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("capemox/ettin-encoder-32m-pretrain-50m")

queries = ["What is the capital of France?"]
docs    = ["Paris is the capital and largest city of France."]

q = model.encode(queries, normalize_embeddings=True)
d = model.encode(docs,    normalize_embeddings=True)
print(q @ d.T)