Created ReadMe

README.md

@@ -0,0 +1,34 @@
+# Fine-Tuning RNA Language Models to Predict Branch Points
+This repository contains several fine-tuned RNA language models for predicting branch points within intronic sequences. The models are fine-tuned using the [MultiMolecule library](https://multimolecule.danling.org/) and evaluated on experimental datasets.
+
+The following RNA language models were fine-tuned:
+- SpliceBERT
+- RNABERT
+- RNA-FM
+- RNA-MSM
+- ERNIE-RNA
+- UTR-LM
+
+The dataset contains **177980 samples** and is an experimental-data only subset of the dataset used to train [BPHunter](https://www.pnas.org/doi/abs/10.1073/pnas.2211194119?url_ver=Z39.88-2003&rfr_id=ori%3Arid%3Acrossref.org&rfr_dat=cr_pub++0pubmed).
+
+It has been split into approximately **80/10/10 train/validation/test** by chromosome type:
+- Train: `chr1`, `chr2`, `chr3`, `chr4`, `chr5`, `chr6`, `chr7`, `chr12`, `chr13`, `chr14`, `chr15`, `chr16`, `chr17`, `chr18`, `chr19`, `chr20`, `chr21`, `chr22`, `chrX`, `chrY`,
+- Validation: `chr9`, `chr10`
+- Test: `chr8`, `chr11`
+
+## Training Details
+
+Each model was trained on the full dataset for 3 epochs with a batch size of 16, except for RNA-FM, which required a reduced batch size of 12 due to VRAM limitations. The following hyperparameters were used for most models, including RNABERT, RNA-FM, RNA-MSM, and UTR-LM:
+- Optimizer: AdamW
+- Learning rate: 3e-4
+- Weight decay: 0.001
+
+However, SpliceBERT and ERNIE-RNA failed to converge using these parameters. To address this, we adjusted the hyperparmeters to:
+- Learning rate: 2e-5
+- Weight decay of 0.01
+
+The adjustments were made based on empirical observations during early training.
+While ideally, comprehensive hyperparameter tuning would be done for each model to optimize perforance, this was not feasible within the scope of the project due to the high computational cost and training time required.
+
+## GitHub
+All code used to create and evaluate this model can be found at [this link](https://github.com/AliSaadatV/BP_LM/tree/main).