readme.md

T5 Biological Sequence + English Mixed Model

A T5-small model was trained on a mixture of DNA, protein sequences, and English text data, primarily for downstream fine-tuning tasks such as sequence function prediction.

Tokenizer Training

T5 uses the Unigram tokenizer. The input data consists of DNA sequences, protein sequences, and English text.

The specific training script is: t5_token_gene_eng.py.

Tokenizer training requires more than 128GB of memory and can be time-consuming.
You may use the pre-trained tokenizer directly:

trained_t5_gene_eng_tokenizer

Pre-training the T5 Model

A T5-large model was trained from scratch on a mixed dataset of DNA, protein sequences, and English text. The steps are as follows:

1. Obtain the T5 configuration by running get_t5_config.ipynb.
2. Prepare the mixed training data by running combine_data.ipynb.
3. Launch the pre-training script ./run_pt.sh.
   Training takes approximately 5 hours using 8x NVIDIA 4090 GPUs.

Fine-tuning the T5 Model

1. Protein Function Prediction: t5_gene_eng_abstract_ft_protein_fun.ipynb
2. Amazon Review Summarization (for reference): t5_gene_eng_abstract_ft_review.ipynb
3. CNN Article Summarization (for reference): t5_gene_eng_abstract_ft_cnn.ipynb
4. DNA-Protein Coding Prediction (experimental, poor performance, for reference only): t5_gene_eng_abstract_ft_dna_protein.ipynb

Additional Experiments

Directory: multi_trans_lab
This contains experimental tasks exploring cross-modal and cross-lingual transfer capabilities, such as English-to-Spanish summarization and even English-to-DNA sequence generation. These are research-oriented and provided for academic reference only.

• NC_000001.11_chapter_1.fna.p1: Partial human genome sequence data.
• get_dna_summary.ipynb: Generates summaries for genomic DNA sequences (can use different fine-tuned models; see fine-tuning section above).
• get_gene_summary.ipynb: Generates summaries for coding DNA regions (model can be swapped).
• dna_abstract_search_bench.ipynb: Indirectly evaluates summary quality via search-based methods. Results are currently poor; ongoing research.
• abstract_trans_en_es.ipynb: Baseline test for transferring English summarization capability to Spanish.

---

T5 生物序列+英文混合模型/Chinese

主要由DNA、蛋白质序列加上英文数据，混合训练了一个T5-small模型，用于序列功能预测等微调测试

分词器训练

T5 使用unigram分词器，输入为DNA、蛋白质序列加上英文数据

具体代码为：t5_token_gene_eng.py

训练时间比较上，需要内存大于128G

可以使用训练好的分词器

trained_t5_gene_eng_tokenizer

预训练T5模型

DNA、蛋白质序列加上英文数据从头训练一个T5大模型，具体步骤：

1. 获得T5的配置，执行get_t5_config.ipynb即可
2. 混合训练数据，执行combine_data.ipynb即可
3. 执行训练脚本./run_pt.sh. 8 卡4090大致需要5个小时左右。

微调T5模型

1. 蛋白质序列功能预测：t5_gene_eng_abstract_ft_protein_fun.ipynb
2. amazon评论摘要，主要做参考：t5_gene_eng_abstract_ft_review.ipynb
3. cnn文章摘要，主要做参考：t5_gene_eng_abstract_ft_cnn.ipynb
4. dna蛋白质编码预测，这个效果不行，仅做参考：t5_gene_eng_abstract_ft_dna_protein.ipynb

一些额外的实验

multi_trans_lab目录 主要做一些摘要的语言能力迁移实验，包括英语到西班牙语的 以及英语到DNA序列的，为论文科研性质的，仅供参考。

• NC_000001.11_chapter_1.fna.p1 部分人类基因组数据
• get_dna_summary.ipynb 获得基因组dna序列的摘要，可以更换微调模型，具体见前面微调模型部分
• get_gene_summary.ipynb 获得编码区dan序列的摘要，可以更换微调模型
• dna_abastract_search_bench.ipynb 通过搜索的方式间接评价摘要获取的效果，结果很差，持续科研中
• abstact_trans_en_es.ipynb, 对照测试，英文摘要能力迁移到西班牙文