ChemBERTa-700k-Augmented-SMILES
A RoBERTa-based transformer model pre-trained on 70k unique nonisomeric SMILES from PubChem, augmented 10x to 700k training samples using masked language modeling (MLM). This model is designed for molecular property prediction and chemical SMILES understanding tasks.
Model Details
Model Description
- Model Type: RoBERTa (RobertaForMaskedLM)
- Architecture: Transformer-based encoder
- Task: Masked Language Modeling (MLM)
- Language: SMILES (Simplified Molecular Input Line Entry System)
Model Architecture
- Hidden Size: 768
- Number of Hidden Layers: 6
- Number of Attention Heads: 12
- Intermediate Size: 12288
- Vocabulary Size: 600
- Max Position Embeddings: 515
- Activation Function: GELU
- Hidden Dropout: 0.1
- Attention Dropout: 0.1
Training Details
Training Data
- Dataset: 70k unique random nonisomeric SMILES from PubChem, augmented 10x to 700k total training samples
- Augmentation: 10x augmentation using random SMILES generation
- Starting with 70k unique nonisomeric SMILES strings
- Each SMILES string was augmented by generating 10 random (non-canonical) SMILES variants
- Final training set contains 700k SMILES strings (70k × 10)
- Augmentation helps the model learn invariant representations across different SMILES representations of the same molecule
- SMILES Type: Nonisomeric SMILES (no stereochemical information)
- Pre-processing: SMILES strings tokenized using a custom SMILES tokenizer
Training Procedure
- Training Method: Masked Language Modeling (MLM)
- Batch Size: 4 per device
- Number of Epochs: 10
- Optimizer: AdamW
- Learning Rate: (default RoBERTa settings)
Training Infrastructure
Trained using Hugging Face Transformers library with distributed training capabilities.
Note: This model was pre-trained from scratch (not fine-tuned). The model architecture and training approach are inspired by ChemBERTa, and
seyonec/SMILES_tokenized_PubChem_shard00_160k is used as a baseline for comparison. The tokenizer from
seyonec/SMILES_tokenized_PubChem_shard00_160k should be used with this model as they share the same vocabulary.
Usage
Direct Use
This model can be used for masked language modeling on SMILES strings to understand molecular structure representations. It can be fine-tuned on downstream tasks such as:
- Molecular property prediction (classification and regression)
- Drug discovery tasks
- Toxicity prediction
- Bioavailability prediction
- Flavor prediction
Downstream Use
The model should be fine-tuned on specific molecular property prediction tasks. Example tasks include:
- Classification: BBBP (Blood-Brain Barrier Penetration), Tox21 (Toxicity), Clintox (Clinical Toxicity)
- Regression: Delaney (Aqueous Solubility), Lipo (Lipophilicity), Clearance (Molecular Clearance)
Out-of-Scope Use
This model was pre-trained on molecular SMILES data and is not designed for:
- Natural language understanding tasks
- Non-molecular chemical representations
- Tasks requiring 3D molecular geometry
How to Get Started with the Model
1from transformers import AutoModelForMaskedLM, AutoTokenizer
2from transformers import pipeline
3
4# Load model and tokenizer
5model = AutoModelForMaskedLM.from_pretrained("terrytwk/chemberta-700k-augmented-smiles")
6tokenizer = AutoTokenizer.from_pretrained("seyonec/SMILES_tokenized_PubChem_shard00_160k")
7
8# Create fill-mask pipeline
9fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)
10
11# Example: predict masked token in a SMILES string
12results = fill_mask("CC(=O)OC1=CC=CC=C1C(=O)O [MASK]")
13print(results)
Fine-tuning Example
1from transformers import AutoModelForSequenceClassification, AutoTokenizer
2from transformers import TrainingArguments, Trainer
3
4# Load pre-trained model for sequence classification
5model = AutoModelForSequenceClassification.from_pretrained(
6 "terrytwk/chemberta-700k-augmented-smiles",
7 num_labels=2 # Adjust based on your task
8)
9tokenizer = AutoTokenizer.from_pretrained("seyonec/SMILES_tokenized_PubChem_shard00_160k")
10
11# Fine-tune on your molecular property prediction dataset
12# ... (add your training code)
Limitations and Bias
- The model was trained on 70k unique molecules (700k augmented samples) from PubChem, which may not represent all chemical space
- Performance may vary on molecules with rare or unusual substructures
- SMILES augmentation introduces some variance in representations
- Model performance is limited by the vocabulary size (600 tokens)
Evaluation
FART (Flavor and Aroma Recognition Task) Evaluation
This model was evaluated on the FART dataset for molecular taste prediction. On this benchmark, the model achieved an evaluation accuracy of 0.8895, performing better or on par with the baseline model seyonec/SMILES_tokenized_PubChem_shard00_160k.
Notably, this performance was achieved despite:
- Training on 7% of the data volume compared to the baseline (700k vs ~10M training samples)
- Training on 0.7% of unique molecules compared to the baseline (70k unique molecules vs ~10M unique molecules)
- Using 10x SMILES augmentation instead of training on a larger dataset
This demonstrates the effectiveness of SMILES augmentation as a data-efficient training strategy for molecular language models.
For evaluation results on other molecular property prediction benchmarks, please refer to fine-tuning experiments on datasets such as:
- MoleculeNet benchmarks (BBBP, Tox21, ClinTox, etc.)
Citation
If you use this model in your research, please cite:
This Model
1@misc{chemberta700kaugmented2024,
2 title={ChemBERTa-700k-Augmented-SMILES: Improved Molecular Property Prediction with SMILES Augmentation},
3 author={Terry Kim},
4 year={2024},
5 howpublished={\url{https://huggingface.co/terrytwk/chemberta-700k-augmented-smiles}}
6}
FART Evaluation
1@article{zimmermann2025chemical,
2 title={A chemical language model for molecular taste prediction},
3 author={Zimmermann, Yoel and Sieben, Leif and Seng, Henrik and Pestlin, Philipp and G{\"o}rlich, Franz},
4 journal={npj Science of Food},
5 volume={9},
6 number={1},
7 pages={122},
8 year={2025},
9 publisher={Nature Publishing Group},
10 doi={10.1038/s41538-025-00474-z},
11 url={https://www.nature.com/articles/s41538-025-00474-z}
12}
Baseline Model for Comparison
- seyonec/SMILES_tokenized_PubChem_shard00_160k - Baseline model used for comparison (hypothesized to be trained on ~10M SMILES)
Related Research
1@article{chemberta2020,
2 title={ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction},
3 author={Chithrananda, Seyone and Grand, Gabriel and Ramsundar, Bharath},
4 journal={arXiv preprint arXiv:2010.09885},
5 year={2020}
6}
Model Card Contact
For questions or issues regarding this model, please open an issue in the repository.
