from sentence_transformers import models, SentenceTransformer
from transformers import AutoModel

transformer = models.Transformer("entropy/roberta_zinc_480m", 
                                 max_seq_length=256, 
                                 model_args={"add_pooling_layer": False})

pooling = models.Pooling(transformer.get_word_embedding_dimension(), 
                         pooling_mode="mean")

roberta_zinc = SentenceTransformer(modules=[transformer, pooling])
erbb1_mlp = AutoModel.from_pretrained("entropy/erbb1_mlp", trust_remote_code=True)

# smiles should be canonicalized
smiles = [
    "Brc1cc2c(NCc3ccccc3)ncnc2s1",
    "Brc1cc2c(NCc3ccccn3)ncnc2s1",
    "Brc1cc2c(NCc3cccs3)ncnc2s1",
    "Brc1cc2c(NCc3ccncc3)ncnc2s1",
    "Brc1cc2c(Nc3ccccc3)ncnc2s1"
]

embeddings = roberta_zinc.encode(smiles, convert_to_tensor=True)
predictions = erbb1_mlp(embeddings).predictions

Training Procedure

Preprocessing

ErbB1 ligands were downloaded from ChEMBL (target_chembl_id="CHEMBL203", type="IC50", relation="=", assay_type="B"). Results were filtered for assays with IC50 values in nM for homo sapiens, canonicalized and deduplicated. IC50 values were converted to pIC50 values. The final dataset contains 7327 data points.

Prior to training, pIC50 values were normalized. The model was trained on normalized values, and uses the saved mean/variance of the dataset to denormalize predictions.

Training Hyperparameters

The model was trained for 30 epochs with a batch size of 32, learing rate of 1e-3, weight decay of 1e-4 and cosine learning rate scheduling.

Model Card Authors

Karl Heyer

Model Card Contact

karl@darmatterai.xyz

entropy/erbb1_mlp

Model Card for ErbB1 MLP

Model Description

Direct Use

Training Procedure

Preprocessing

Training Hyperparameters

Model Card Authors

Model Card Contact

license: mit

entropy/
erbb1_mlp