ProtSent-V2 ESM-2 35M
Contrastively fine-tuned
ESM-2 35M producing fixed-length protein
embeddings where biological similarity maps to embedding proximity. Intended for
retrieval, clustering, and nearest-neighbour transfer.
Retrained on a corpus decontaminated against the benchmark test sets. Predecessor:
oriel9p/protsent-esm2-35M.
Other scale:
GrimSqueaker/ProtSent-V2-150M.
Usage
1from sentence_transformers import SentenceTransformer
2from sentence_transformers.util import cos_sim
3
4model = SentenceTransformer("GrimSqueaker/ProtSent-V2-35M")
5emb = model.encode([
6 "MKTLLLTLVVVTIVCLDLGYT",
7 "MKTLLLTLVVVTIVCLDLGYN",
8 "AGWYRSPQEGLKPVDTFKDIV",
9])
10print(cos_sim(emb[0], emb[1:]))
Embeddings are mean-pooled over the final layer, dimension 480. Matryoshka heads at
64/128/256 are available by truncating the embedding.
Training data
Three sources, all decontaminated (see below). No DMS/ProteinGym component — unlike
the V1 release, which included ProteinGym DMS pairs under a CoSENT loss.
| source | pairs used |
|---|
| Pfam families | 777,306 |
| AlphaFold DB (Foldseek clusters) | 18,987,468 |
| STRING-DB v12 PPI | 15,000,000 |
| total | 34,764,774 |
Pfam and AFDB pairs are sampled within clusters; STRING is a fixed 15M-pair subsample
(seed 42) of the filtered pair table.
Decontamination
Every source was searched against the benchmark test sequences with MMseqs2 easy-search
(corpus as query, 40% identity, 80% coverage, --cov-mode 1) and matching sequences
removed before training.
| corpus | rows before | rows after | removed |
|---|
| Pfam | 28,530,684 | 27,929,772 | 2.11% |
| AlphaFold DB | 135,404,259 | 126,301,607 | 6.72% |
| STRING | 76,070,154 | 71,891,417 | 5.49% |
Filter targets: biomap-research/fold_prediction (remote homology) and
Synthyra/bernett_gold_ppi (PPI) test splits. The result was verified by semi-joining each
training file against the removal lists: zero flagged sequences remained.
SCOPe-40 was not a filter target — it has no train/test split, so filtering against it
would remove nearly all domain sequences from the corpus.
Training configuration
| setting | value |
|---|
| backbone | ESM-2 35M (480 hidden, 12 layers) |
| loss | CachedMultipleNegativesRankingLoss |
| contrastive batch | 1024 per device |
| gather across devices | off |
| synthetic hard negatives | off |
| multi-dataset sampler | proportional |
| Matryoshka dims | 64 / 128 / 256 |
| max sequence length | 512 |
| optimiser | AdamW, LR 2e-4, cosine_with_min_lr |
| precision / attention | bf16, flash-attention-2 |
| hardware | 7x NVIDIA B300 |
| steps | 4,850 (one epoch) |
| gradient-cache mini-batch | 256 |
| warmup | 1,000 steps |
| wall clock | 10 h 53 m |
Training code:
github.com/oriel9p/ProtSent,
train_esm2_35m.sh.
Results
SCOPe-40 structural retrieval, test split, self excluded, no-hit queries counted as
failures. Restricted to the 1,693 of 2,207 queries that have a non-self same-family protein
in the gallery.
| method | R@1 | R@10 | MAP |
|---|
| ESM-2 35M | 0.4991 | 0.7614 | 0.4210 |
MMseqs2 (-s 7.5) | 0.6556 | 0.7348 | 0.4041 |
| ProtSent-V1 35M | 0.5854 | 0.8512 | 0.5509 |
| ProtSent-V2 35M | 0.6852 | 0.9220 | 0.6459 |
Paired bootstrap over queries (10,000 resamples), V2 − V1: R@1 +0.0986 [+0.0762, +0.1211],
MAP +0.0943 [+0.0814, +0.1074]. Against a maximally sensitive HMMER (phmmer) this model is
statistically tied at top-1 and ahead at R@10 and MAP.
Remote homology (the task the corpus was filtered against), test split:
| model | 3-NN accuracy | linear-probe accuracy |
|---|
| ESM-2 35M | 0.5835 | 0.6868 |
| ProtSent-V1 35M | 0.6587 | 0.6899 |
| ProtSent-V2 35M | 0.6668 | 0.7016 |
Limitations
- Under a trained linear probe on the final layer, this model is roughly neutral to
slightly worse than the stock ESM-2 backbone across a 23-task suite. The advantage is in
nearest-neighbour geometry, not in information a trained readout can extract.
- The final layer is not the best pooling layer for property prediction. In a layer sweep
on remote homology, an intermediate layer (~2/3 depth) scored higher for every model
tested, including the stock backbone.
- V2 differs from V1 in more than decontamination: no hard negatives, proportional
sampling, no DMS source, larger effective batch. It is not a controlled ablation of
filtering alone.
- Only the remote-homology and PPI test sets were decontamination targets; other benchmark
test sets were not filtered against.
Citation
