Qwen3-1.7B Plant Disease (Continued Pretraining)

Model Details

• Base model: Qwen/Qwen3-1.7B
• Method: Continued Pretraining (causal language modeling)
• Domain: Plant pathology, fungal/bacterial/viral plant diseases
• Training tokens: ~25.3M
• Epochs: 4
• Learning rate: 2e-05
• Effective batch size: 32
• Sequence length: 2048

Performance

• Baseline val PPL: 13.6
• Final val PPL: 8.9
• Improvement: 34.9%

Anti-Catastrophic-Forgetting Safeguards

1. Replay buffer: 10% of training mix is general English text (C4)
2. Conservative LR: 2e-05 (low to preserve base knowledge)
3. Cosine schedule with 3% warmup
4. Gradient clipping at 1.0
5. Best-K checkpointing: kept top-3 checkpoints by validation loss

Use Case

• ✅ Further fine-tuning (SFT) on plant disease tasks
• ✅ Continued language modeling on plant disease text
• ✅ Embeddings for plant disease retrieval
• ❌ NOT directly suitable for chat/instruction (no SFT applied)

Dataset

• Wikipedia articles on plant diseases and pathogens (~19.4M tokens)
• BioProtection Portal resources on integrated pest management
• General English text mixed in (anti-forgetting buffer)

Citation

@misc{qwen3_plantdisease_cpt_2026,
  title     = {Qwen3-1.7B Plant Disease CPT},
  author    = {Muhammad, Ismail},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/deadbear34/qwen3-1.7b-plantdisease-cpt}
}

License