T5 Biological Sequence + English Mixed Model
A T5-small model was trained on a mixture of DNA, protein sequences, and English text data, primarily for downstream fine-tuning tasks such as sequence function prediction.
Tokenizer Training
T5 uses the Unigram tokenizer. The input data consists of DNA sequences, protein sequences, and English text.
The specific training script is: t5_token_gene_eng.py.
Tokenizer training requires more than 128GB of memory and can be time-consuming.
You may use the pre-trained tokenizer directly:
trained_t5_gene_eng_tokenizer
Pre-training the T5 Model
A T5-large model was trained from scratch on a mixed dataset of DNA, protein sequences, and English text. The steps are as follows:
- Obtain the T5 configuration by running
get_t5_config.ipynb.
- Prepare the mixed training data by running
combine_data.ipynb.
- Launch the pre-training script
./run_pt.sh.
Training takes approximately 5 hours using 8x NVIDIA 4090 GPUs.
Fine-tuning the T5 Model
- Protein Function Prediction:
t5_gene_eng_abstract_ft_protein_fun.ipynb
- Amazon Review Summarization (for reference):
t5_gene_eng_abstract_ft_review.ipynb
- CNN Article Summarization (for reference):
t5_gene_eng_abstract_ft_cnn.ipynb
- DNA-Protein Coding Prediction (experimental, poor performance, for reference only):
t5_gene_eng_abstract_ft_dna_protein.ipynb
Additional Experiments
Directory: multi_trans_lab
This contains experimental tasks exploring cross-modal and cross-lingual transfer capabilities, such as English-to-Spanish summarization and even English-to-DNA sequence generation. These are research-oriented and provided for academic reference only.
NC_000001.11_chapter_1.fna.p1: Partial human genome sequence data.get_dna_summary.ipynb: Generates summaries for genomic DNA sequences (can use different fine-tuned models; see fine-tuning section above).get_gene_summary.ipynb: Generates summaries for coding DNA regions (model can be swapped).dna_abstract_search_bench.ipynb: Indirectly evaluates summary quality via search-based methods. Results are currently poor; ongoing research.abstract_trans_en_es.ipynb: Baseline test for transferring English summarization capability to Spanish.
