You can deploy this model and start earning money today!

omshrivastava/
omnibimol-chemprot-lora

Name: omnibimol-chemprot-lora
Availability: InStock

safetensors

2106.09685

Views

No views yet

OmniBiMol ChemProt LoRA: Chemical-Protein Interaction Ranker

A lightweight LoRA adapter for drug repurposing and target prioritization, fine-tuned on ChemProt for OmniBiMol.

Model Description

This is a LoRA adapter for BiomedBERT fine-tuned on the ChemProt chemical-protein relation extraction dataset for binary classification of chemical-protein interactions.

Task: Given a chemical name, protein name, and PubMed abstract context, predict whether the chemical interacts with the protein (binary classification).

Use case: Drug repurposing candidate scoring, target prioritization, evidence-based chemical-protein interaction ranking for the OmniBiMol bioinformatics platform.

Experiment Results

Experiment	AUROC	Avg Precision	F1	Precision	Recall	NDCG@10	Time
Exp1: Zero-Shot BGE Embedding	0.503	0.538	0.520	0.538	0.504	0.548	181s
Exp2: LoRA BiomedBERT (this model)	0.702	0.697	0.721	0.636	0.833	0.609	6703s
Exp3: Evidence-Weighted Ensemble	0.712	0.706	0.697	0.665	0.732	0.668	1s

Key Findings

LoRA fine-tuning improves AUROC by +39.6% over zero-shot embedding baseline (0.503 → 0.702)
Ensemble adds +1.4% AUROC over LoRA alone by incorporating text evidence features
Only 296,450 trainable parameters (0.27% of 110M total) — 1.2MB adapter file
Fully CPU-trained on HF free tier (cpu-basic) in ~1.7 hours

Training Details

Base model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext (110M params)
LoRA config: r=8, alpha=16, dropout=0.1, target_modules=["query", "value"]
Dataset: bigbio/chemprot (chemprot_full_source) — BioCreative VI chemical-protein RE
- Train: 7,389 pairs (balanced), Val: 4,134, Test: 6,495
- Positive relations: CPR:3 (upregulator), CPR:4 (downregulator), CPR:5 (agonist), CPR:6 (antagonist), CPR:9 (substrate)
Hyperparameters: lr=2e-4, 3 epochs, batch=16×2 grad_accum=32 effective, linear warmup 100 steps
Compute: CPU-only (HF free tier, cpu-basic sandbox)
Training loss: 1.33 → 1.09 (epoch 1→3)
Val metrics across epochs:
- Epoch 1: AUROC=0.671, F1=0.721
- Epoch 2: AUROC=0.713, F1=0.750
- Epoch 3: AUROC=0.719, F1=0.755

Usage

python

from transformers import AutoModelForSequenceClassification, AutoTokenizer
from peft import PeftModel
import torch

# Load base model + LoRA adapter
base_model = AutoModelForSequenceClassification.from_pretrained(
    "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext",
    num_labels=2
)
model = PeftModel.from_pretrained(base_model, "omshrivastava/omnibimol-chemprot-lora")
tokenizer = AutoTokenizer.from_pretrained("omshrivastava/omnibimol-chemprot-lora")

model.eval()

# Score a chemical-protein interaction
chemical = "metformin"
protein = "ampk"
context = "Metformin activates AMP-activated protein kinase (AMPK) in hepatocytes, leading to decreased hepatic glucose production."

inputs = tokenizer(
    f"{chemical} interacts with {protein}",
    context[:256],
    truncation=True, max_length=256, return_tensors="pt"
)

with torch.no_grad():
    logits = model(**inputs).logits
    prob = torch.softmax(logits, dim=-1)[0, 1].item()

print(f"Interaction probability: {prob:.4f}")
# Output: ~0.85 (high confidence interaction)

Evidence-Weighted Ensemble

For best AUROC (0.712), combine LoRA predictions with text evidence features:

python

# After getting LoRA prob:
import numpy as np
from sklearn.linear_model import LogisticRegression

# Text evidence features (from abstract)
features = {
    "bge_cosine_sim": 0.72,     # BGE embedding similarity
    "lora_prob": prob,           # This model's prediction
    "chem_in_context": 1,       # Chemical mentioned in abstract
    "prot_in_context": 1,       # Protein mentioned in abstract
    "cooccurrence": 1,          # Both in same sentence
    "keyword_count": 3,         # Interaction keywords found
    "context_length": 25,       # Word count
    "entity_distance": 0.15     # Normalized distance in text
}
# Feature weights from trained ensemble:
# lora_prob: +0.8713 (dominant signal)
# cooccur: +0.3643, kw_count: +0.1169, distance: +0.1089

Citations

bibtex

@inproceedings{hu2022lora,
  title={LoRA: Low-Rank Adaptation of Large Language Models},
  author={Hu, Edward and others},
  booktitle={ICLR},
  year={2022}
}

@article{gu2021pubmedbert,
  title={Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing},
  author={Gu, Yu and others},
  journal={ACM THBI},
  year={2021}
}

@inproceedings{krallinger2017chemprot,
  title={Overview of the BioCreative VI chemical-protein interaction Track},
  author={Krallinger, Martin and others},
  booktitle={BioCreative VI Workshop},
  year={2017}
}

OmniBiMol Integration

This model serves as the evidence scoring layer in OmniBiMol's drug repurposing pipeline:

Input: User queries a drug-disease pair
Retrieval: Fetch relevant PubMed abstracts mentioning the drug and potential protein targets
Scoring: This LoRA model scores each (chemical, protein, abstract) triple
Ranking: Sort targets by interaction probability; use ensemble features for re-ranking
Output: Ranked list of protein targets with evidence citations

Free-tier constraints

Model adapter: 1.2MB (fits in any free HF repo)
Inference: CPU-only, ~15 samples/sec
Training: Reproducible on HF free tier cpu-basic in ~1.7 hours
No GPU, no paid API, no external infrastructure required

License

MIT (same as base model)