ENBED DNA Mutation Prediction (512bp, Checkpoint-200)

Model Details

• Base model: malusare/dna-base-512-pretrained (~581M params, 18 encoder / 6 decoder layers)
• Fine-tuning task: Seq2seq DNA mutation prediction (parent → child sequence)
• Training data: 4,143 mutation pairs across 9 domains (Bacteria, Archaea, Viruses, Retroviridae, Viridiplantae, Metazoa, Viroids, Satellites, Lichenes HGT)
• Best checkpoint: Step 200 (eval loss 1.250)
• Context length: 512 tokens (byte-level: 1 base = 1 token)
• Hardware: Intel Hades Canyon NUC + GTX 1080 8GB eGPU

Training Configuration

• Optimizer: Adafactor
• Batch size: 1 (gradient accumulation: 16, effective batch = 16)
• Learning rate: 1e-4
• Precision: fp16 (model loaded in fp16, no AMP scaler)
• Gradient checkpointing: enabled
• Epochs: 5 (best at epoch ~0.8)

Eval Loss Progression

Intended Use

• Cross-domain DNA mutation prediction
• Horizontal gene transfer pattern analysis
• Synthetic mutation data generation for underrepresented domains
• Research baseline for DNA foundation model comparisons

Limitations

• Overfitting: 581M params on 4K pairs → overfits quickly. Best used as a baseline or synthetic data generator.
• Context: 512bp limit truncates longer sequences
• Domains with few examples (Bacteria, Archaea) will have poor performance
• HGT events (large insertions) are the hardest prediction task and likely near-random

Citation

License