Eleuthia is a binary protein variant classifier that predicts whether a mutated protein sequence is likely Pathogenic or Benign. It is fine-tuned from ESM-2 and intended for research support in variant prioritization workflows.
Model Details
Model Description
Eleuthia takes an amino acid sequence containing a variant and returns class probabilities for:
Benign (label 0)
Pathogenic (label 1)
The model uses a maximum sequence length of 1022 tokens (ESM-2 limit). For longer sequences, inference applies a centered sliding window (optionally around the mutation position).
Developed by: Hariz Rafie
Funded by : Self Funded Final Year Project (academic project)
Shared by : RizFie
Model type: Protein sequence classification (binary)
Training data was built from cancer-associated genes using EBI Proteins variation endpoints with ClinVar-style significance filtering. Variants were filtered to keep unambiguous single amino-acid substitutions with binary labels:
A dataset CSV is included in the project as Genetic_Variants_Dataset.csv.
Training Procedure
Preprocessing [optional]
Construct mutated sequence from reference sequence + substitution.
Enforce ESM-2 max length (1022) using mutation-centered sliding window when needed.
Train/test split with stratification (test_size=0.25, random_state=42).
Oversample the benign class in the training split to balance classes.
Tokenize with AutoTokenizer from the base ESM-2 checkpoint.
Training Hyperparameters
Training regime: fp16 mixed precision
Base model:facebook/esm2_t30_150M_UR50D
Optimizer/scheduler: default Trainer optimizer + cosine LR scheduler
Learning rate:1e-5
Batch size:8 (train), 8 (eval)
Epochs:15
Weight decay:0.01
Early stopping: patience 3
Loss: Focal Loss (alpha=0.25, gamma=2.0)
Model selection metric:f1
Speeds, Sizes, Times [optional]
Please add your measured training wall-clock time, throughput, and final checkpoint size.
Evaluation
Testing Data, Factors & Metrics
Testing Data
Held-out stratified test split from the collected variant dataset (25%), with 2864 total samples.
Factors
Current evaluation is aggregate binary classification. Subgroup analyses (by gene, protein length, mutation type, domain) are recommended and currently not fully reported.
Metrics
F1 score
Accuracy
Precision
Recall
ROC-AUC
Results
Final evaluation on the held-out test set:
Overall Accuracy: 0.7105 (71.05%)
ROC-AUC: 0.7731
Macro F1: 0.68
Weighted F1: 0.72
Per-class performance:
Class
Precision
Recall
F1-score
Support
Benign (0)
0.49
0.69
0.57
806
Pathogenic (1)
0.85
0.72
0.78
2058
Aggregate report:
Macro avg: precision 0.67, recall 0.70, F1 0.68
Weighted avg: precision 0.75, recall 0.71, F1 0.72
Summary
Eleuthia demonstrates feasible discrimination between benign and pathogenic variants on a cancer-focused held-out split, with stronger precision on pathogenic calls and moderate overall calibration. External validation on independent datasets is still required for broader use.
Model Examination [optional]
Evaluation visualizations from the held-out test split:
Confusion Matrix
Eleuthia Confusion Matrix
ROC Curve
Eleuthia ROC Curve
Additional recommended analyses for future versions:
Per-gene performance breakdown
Calibration plots
Precision-recall curve (especially for class imbalance)
