HELM-BERT is built upon the DeBERTa architecture, pre-trained on ~75k peptides from four databases (ChEMBL, CREMP, CycPeptMPDB, Propedia) using Masked Language Modeling (MLM) with a Warmup-Stable-Decay (WSD) learning rate schedule.

Disentangled Attention: Decomposes attention into content-content and content-position terms
Enhanced Mask Decoder (EMD): Injects absolute position embeddings at the decoder stage
Span Masking: Contiguous token masking with geometric distribution
nGiE: n-gram Induced Encoding layer (1D convolution, kernel size 3)

Model Specifications

Parameter	Value
Parameters	54.8M
Hidden size	768
Layers	6
Attention heads	12
Vocab size	78
Max token length	512
Pre-training data	~75k peptides (ChEMBL, CREMP, CycPeptMPDB, Propedia)
Pre-training objective	MLM (span masking, p=0.15)
LR schedule	Warmup-Stable-Decay (WSD)

How to Use

python

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("Flansma/helm-bert", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("Flansma/helm-bert", trust_remote_code=True)

# Cyclosporine A
inputs = tokenizer("PEPTIDE1{[Abu].[Sar].[meL].V.[meL].A.[dA].[meL].[meL].[meV].[Me_Bmt(E)]}$PEPTIDE1,PEPTIDE1,1:R1-11:R2$$$", return_tensors="pt")
outputs = model(**inputs)
embeddings = outputs.last_hidden_state

Training Data

Pre-trained on deduplicated peptide sequences from:

ChEMBL: Bioactive molecules database
CREMP: Cyclic peptide conformational ensemble database
CycPeptMPDB: Cyclic peptide membrane permeability database
Propedia: Protein-peptide interaction database

Downstream Performance

Permeability Regression (CycPeptMPDB)

Single-Assay (mixed PAMPA/Caco-2 target):

Split	R²	Pearson	RMSE	MAE
Random	0.658	0.817	0.471	0.300
Scaffold	0.502	0.723	0.450	0.324

Per-Assay (separate models for PAMPA and Caco-2):

Split	Assay	R²	Pearson	RMSE	MAE
Random	PAMPA	0.800	0.895	0.355	0.253
Random	Caco-2	0.747	0.866	0.388	0.289
Scaffold	PAMPA	0.529	0.739	0.412	0.295
Scaffold	Caco-2	0.637	0.874	0.405	0.334

Train/test 9:1, val 10% from train. Scaffold split by Murcko scaffolds.

PPI Classification (Propedia v2)

Split	ROC-AUC	PR-AUC	F1	MCC	Balanced Acc
Random	0.968	0.901	0.847	0.808	0.906
aCSM	0.862	0.683	0.587	0.522	0.722

Train/test 8:2, val 10% from train, 1:4 positive:negative ratio.

Random: random split
aCSM: clustering-based split on aCSM-ALL complex signatures with protein overlap pruning

PPI Classification (ChEMBL)

Split	ROC-AUC	PR-AUC	F1	MCC	Balanced Acc
Random	0.992	0.975	0.948	0.936	0.969
Family	0.786	0.449	0.267	0.222	0.570

Val 10% from train.

Citation

bibtex

@article{lee2026helmbert,
  title={HELM-BERT: Topology-Aware Representations for Chemically Modified Peptides},
  author={Lee, Seungeon and Koyama, Takuto and Maeda, Itsuki and Matsumoto, Shigeyuki and Okuno, Yasushi},
  journal={Journal of Chemical Information and Modeling},
  year={2026},
  doi={10.1021/acs.jcim.6c00451},
  publisher={American Chemical Society},
  url={https://pubs.acs.org/doi/10.1021/acs.jcim.6c00451}
}

License

MIT License