| | --- |
| | license: cc-by-sa-4.0 |
| | tags: |
| | - DNA |
| | - biology |
| | - genomics |
| | - protein |
| | - kmer |
| | - cancer |
| | - gleason-grade-group |
| | --- |
| | ## Project Description |
| | This repository contains the trained model for our paper: **Fine-tuning a Sentence Transformer for DNA & Protein tasks** that is currently under review at BMC Bioinformatics. This model, called **simcse-dna**; is based on the original implementation of **SimCSE [1]**. The original model was adapted for DNA downstream tasks by training it on a small sample size k-mer tokens generated from the human reference genome, and can be used to generate sentence embeddings for DNA tasks. |
| |
|
| | ### Prerequisites |
| | ----------- |
| | Please see the original [SimCSE](https://github.com/princeton-nlp/SimCSE) for installation details. The model will als be hosted on Zenodo (DOI: 10.5281/zenodo.11046580). |
| |
|
| | ### Usage |
| |
|
| | Run the following code to get the sentence embeddings: |
| |
|
| | ```python |
| | |
| | import torch |
| | from transformers import AutoModel, AutoTokenizer |
| | |
| | # Import trained model and tokenizer |
| | tokenizer = AutoTokenizer.from_pretrained("dsfsi/simcse-dna") |
| | model = AutoModel.from_pretrained("dsfsi/simcse-dna") |
| | |
| | |
| | #sentences is your list of n DNA tokens of size 6 |
| | inputs = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt") |
| | |
| | # Get the embeddings |
| | with torch.no_grad(): |
| | embeddings = model(**inputs, output_hidden_states=True, return_dict=True).pooler_output |
| | |
| | |
| | ``` |
| | The retrieved embeddings can be utilized as input for a machine learning classifier to perform classification. |
| |
|
| | ## Performance on evaluation tasks |
| |
|
| | Find out more about the datasets and access in the paper **(TBA)** |
| |
|
| | ### Task 1: Detection of colorectal cancer cases (after oversampling) |
| |
|
| | | | 5-fold Cross Validation accuracy | Test accuracy | |
| | | --- | --- | ---| |
| | | LightGBM | 91 | 63 | |
| | | Random Forest | **94** | **71** | |
| | | XGBoost | 93 | 66 | |
| | | CNN | 42 | 52 | |
| |
|
| | | | 5-fold Cross Validation F1 | Test F1 | |
| | | --- | --- | ---| |
| | | LightGBM | 91 | 66 | |
| | | Random Forest | **94** | **72** | |
| | | XGBoost | 93 | 66 | |
| | | CNN | 41 | 60 | |
| |
|
| | ### Task 2: Prediction of the Gleason grade group (after oversampling) |
| |
|
| | | | 5-fold Cross Validation accuracy | Test accuracy | |
| | | --- | --- | ---| |
| | | LightGBM | 97 | 68 | |
| | | Random Forest | **98** | **78** | |
| | | XGBoost |97 | 70 | |
| | | CNN | 35 | 50 | |
| |
|
| | | | 5-fold Cross Validation F1 | Test F1 | |
| | | --- | --- | ---| |
| | | LightGBM | 97 | 70 | |
| | | Random Forest | **98** | **80** | |
| | | XGBoost |97 | 70 | |
| | | CNN | 33 | 59 | |
| |
|
| | ### Task 3: Detection of human TATA sequences (after oversampling) |
| |
|
| | | | 5-fold Cross Validation accuracy | Test accuracy | |
| | | --- | --- | ---| |
| | | LightGBM | 98 | 93 | |
| | | Random Forest | **99** | **96** | |
| | | XGBoost |**99** | 95 | |
| | | CNN | 38 | 59 | |
| |
|
| | | | 5-fold Cross Validation F1 | Test F1 | |
| | | --- | --- | ---| |
| | | LightGBM | 98 | 92 | |
| | | Random Forest | **99** | **95** | |
| | | XGBoost | **99** | 92 | |
| | | CNN | 58 | 10 | |
| |
|
| |
|
| | ## Authors |
| | ----------- |
| |
|
| | * Mpho Mokoatle, Vukosi Marivate, Darlington Mapiye, Riana Bornman, Vanessa M. Hayes |
| | * Contact details : u19394277@tuks.co.za |
| |
|
| | ## Citation |
| | ----------- |
| | Bibtex Reference **TBA** |
| |
|
| | ### References |
| |
|
| | <a id="1">[1]</a> |
| | Gao, Tianyu, Xingcheng Yao, and Danqi Chen. "Simcse: Simple contrastive learning of sentence embeddings." arXiv preprint arXiv:2104.08821 (2021). |
