Antibody language models (AbLMs) help decode immune repertoires for therapeutic discovery, but optimal scaling strategies remain unclear. We trained
ESM-2-based AbLMs across five sizes (8M–650M parameters) and three paired Ab sequences dataset scales (~1.6M total sequences). Results described in our preprint on biorxiv. Effective AbLM scaling requires balancing model size with data availability to avoid diminishing returns.Datasets used for pre-training are avaliable on [Zenodo] and code is avaliable on ...
1from transformers import EsmTokenizer, EsmForMaskedLM
2
3model = EsmForMaskedLM.from_pretrained("brineylab/35M_full_checkpoint-500000")
4tokenizer = EsmTokenizer.from_pretrained("facebook/esm2_t30_150M_UR50D")
The model can be finetuned for classification tasks (such as specificity and pair classification in the paper) by loading the model with a sequence classification head:
1from transformers import EsmForSequenceClassification
2
3model = EsmForSequenceClassification.from_pretrained("brineylab/35M_full_checkpoint-500000")
4
5# freeze the base model weights prior to finetuning
6for param in model.base_model.parameters():
7 param.requires_grad = False
