---
license: apache-2.0
---
# T5 Biological Sequence + English Mixed Model

A T5-small model was trained on a mixture of DNA, protein sequences, and English text data, primarily for downstream fine-tuning tasks such as sequence function prediction.

## Tokenizer Training
T5 uses the Unigram tokenizer. The input data consists of DNA sequences, protein sequences, and English text.

The specific training script is: `t5_token_gene_eng.py`.

Tokenizer training requires more than 128GB of memory and can be time-consuming.  
You may use the pre-trained tokenizer directly:

**trained_t5_gene_eng_tokenizer**

## Pre-training the T5 Model
A T5-large model was trained from scratch on a mixed dataset of DNA, protein sequences, and English text. The steps are as follows:
1. Obtain the T5 configuration by running `get_t5_config.ipynb`.
2. Prepare the mixed training data by running `combine_data.ipynb`.
3. Launch the pre-training script `./run_pt.sh`.  
   Training takes approximately 5 hours using 8x NVIDIA 4090 GPUs.

## Fine-tuning the T5 Model
1. **Protein Function Prediction**: `t5_gene_eng_abstract_ft_protein_fun.ipynb`  
2. **Amazon Review Summarization** (for reference): `t5_gene_eng_abstract_ft_review.ipynb`  
3. **CNN Article Summarization** (for reference): `t5_gene_eng_abstract_ft_cnn.ipynb`  
4. **DNA-Protein Coding Prediction** (experimental, poor performance, for reference only): `t5_gene_eng_abstract_ft_dna_protein.ipynb`

## Additional Experiments
Directory: `multi_trans_lab`  
This contains experimental tasks exploring cross-modal and cross-lingual transfer capabilities, such as English-to-Spanish summarization and even English-to-DNA sequence generation. These are research-oriented and provided for academic reference only.

- `NC_000001.11_chapter_1.fna.p1`: Partial human genome sequence data.
- `get_dna_summary.ipynb`: Generates summaries for genomic DNA sequences (can use different fine-tuned models; see fine-tuning section above).
- `get_gene_summary.ipynb`: Generates summaries for coding DNA regions (model can be swapped).
- `dna_abstract_search_bench.ipynb`: Indirectly evaluates summary quality via search-based methods. Results are currently poor; ongoing research.
- `abstract_trans_en_es.ipynb`: Baseline test for transferring English summarization capability to Spanish.