| --- |
| license: mit |
| tags: |
| - biology |
| - rna |
| - rna-design |
| - rna-generation |
| - de-novo-design |
| - generative-model |
| - language-model |
| - transformer |
| - gpt |
| - nucleotide |
| - bioinformatics |
| - computational-biology |
| - drug-discovery |
| - molecular-design |
| pipeline_tag: text-generation |
| --- |
| |
| # GenerRNA: A PFN™ Generative Language Model for *de novo* RNA Design |
|
|
| [](https://doi.org/10.1371/journal.pone.0310814) |
| [](https://doi.org/10.1101/2024.02.01.578496) |
| [](./LICENSE) |
| [](https://huggingface.co/pfnet/GenerRNA) |
|
|
| **GenerRNA is a PFN™ generative pre-trained language model for *de novo* RNA sequence design.** It is a Transformer (decoder-only, GPT-style) model that learns the "language" of RNA from millions of natural sequences and can generate novel, realistic RNA sequences **without any structural input, functional label, or sequence alignment**. To our knowledge, GenerRNA is the first application of a generative language model to RNA generation. |
|
|
| With GenerRNA you can: |
|
|
| - **Generate RNA in a zero-shot manner** to explore the RNA sequence space, or |
| - **Fine-tune on your own dataset** to generate RNAs belonging to a particular family or possessing specific characteristics (e.g., high binding affinity to a target protein). |
|
|
| > Developed by [Preferred Networks, Inc.](https://www.preferred.jp/en/) and The University of Tokyo. Introduced in *PLOS ONE* (2024): [GenerRNA: A generative pre-trained language model for *de novo* RNA design](https://doi.org/10.1371/journal.pone.0310814). |
|
|
| --- |
|
|
| ## Table of Contents |
|
|
| - [Model Summary](#model-summary) |
| - [Key Features](#key-features) |
| - [Model Details](#model-details) |
| - [Intended Use & Use Cases](#intended-use--use-cases) |
| - [Requirements](#requirements) |
| - [Quickstart](#quickstart) |
| - [Training & Fine-tuning](#training--fine-tuning) |
| - [Repository Structure](#repository-structure) |
| - [Training Data](#training-data) |
| - [Limitations](#limitations) |
| - [FAQ](#faq) |
| - [Citation](#citation) |
| - [License](#license) |
|
|
| --- |
|
|
| ## Model Summary |
|
|
| GenerRNA is a **Transformer decoder-only (GPT-style) language model** trained on RNA nucleotide sequences. By treating RNA as a sequence of tokens, it learns statistical and structural regularities of RNA directly from data and can then **sample entirely new sequences**. GenerRNA was pre-trained on ~16 million RNA sequences (16.09M), encompassing ~17.4 billion nucleotides. Generated RNAs are novel (distinct from training sequences) yet fold into stable secondary structures, and the model can be fine-tuned to design functional RNAs such as protein binders — all without requiring prior structural knowledge. |
|
|
| ## Key Features |
|
|
| - 🧬 **De novo RNA generation** — create novel RNA sequences from scratch; no structure, label, or alignment required. |
| - 🎯 **Zero-shot or fine-tuned** — explore RNA space out of the box, or specialize the model for a target family or function. |
| - 🔬 **Structurally plausible outputs** — generated sequences fold into stable secondary structures (low minimum free energy). |
| - 🧩 **Transformer / GPT architecture** — a familiar, scalable decoder-only design (~350M parameters). |
| - ⚡ **Two checkpoints provided** — an updated long-context model and the original historical model. |
| - 📖 **Open & reproducible** — MIT-licensed code, tokenizer, checkpoints, and the data behind the paper's figures. |
|
|
| ## Model Details |
|
|
| | | | |
| |---|---| |
| | **Model type** | Generative language model (decoder-only Transformer, GPT-style) | |
| | **Domain** | RNA / nucleotide sequences | |
| | **Parameters** | 350M (24 transformer layers, model dimension 1280) | |
| | **Context window** | 1024 tokens (~4000 nucleotides) | |
| | **Tokenizer** | Byte-Pair Encoding (BPE), vocabulary size 1024 | |
| | **Checkpoints** | `model_updated.pt` (recommended; longer context, deduplicated data) · original split model in `experiment_data/historical_version/` | |
| | **Framework** | PyTorch (≥ 2.0) | |
| | **License** | MIT | |
| | **Paper** | *PLOS ONE* 19(10):e0310814 (2024) · [doi:10.1371/journal.pone.0310814](https://doi.org/10.1371/journal.pone.0310814) | |
| | **Developed by** | Preferred Networks, Inc. & The University of Tokyo | |
|
|
| ## Intended Use & Use Cases |
|
|
| GenerRNA is intended for **research in RNA biology, synthetic biology, and RNA-based therapeutics / drug discovery**. Typical use cases include: |
|
|
| - Exploring the diversity of the RNA sequence space. |
| - Generating candidate RNAs from a target family by fine-tuning on family-specific data. |
| - Designing RNAs with desired functional properties, such as aptamers/binders with high affinity to a target protein (demonstrated for the RNA-binding proteins **ELAVL1** and **SRSF1** in the paper). |
| - Serving as a pre-trained backbone for downstream RNA modeling and design tasks. |
|
|
| ## Requirements |
|
|
| A CUDA environment with a minimum of **8 GB VRAM** is required. |
|
|
| ``` |
| torch>=2.0 |
| numpy |
| transformers==4.33.0.dev0 |
| datasets==2.14.4 |
| tqdm |
| ``` |
|
|
| ## Quickstart |
|
|
| Clone the repository (it ships with the recommended checkpoint `model_updated.pt` and its `tokenizer/`): |
|
|
| ```bash |
| git clone https://huggingface.co/pfnet/GenerRNA |
| cd GenerRNA |
| ``` |
|
|
| ### De novo generation (zero-shot) |
|
|
| ```bash |
| python sampling.py \ |
| --out_path {output_file_path} \ |
| --max_new_tokens 256 \ |
| --ckpt_path model_updated.pt \ |
| --tokenizer_path tokenizer |
| ``` |
|
|
| > **Want to use the original (historical) model instead?** It is stored as split files. Recombine it and use its dedicated tokenizer: |
| > |
| > ```bash |
| > cat experiment_data/historical_version/model.pt.part-* > model.pt |
| > python sampling.py \ |
| > --out_path {output_file_path} \ |
| > --max_new_tokens 256 \ |
| > --ckpt_path model.pt \ |
| > --tokenizer_path experiment_data/historical_version/tokenizer_bpe_1024 |
| > ``` |
| |
| ## Training & Fine-tuning |
| |
| **1. Tokenize your sequences** (one sequence per line, no header): |
| |
| ```bash |
| python tokenization.py \ |
| --data_dir {path_to_directory_containing_sequence_data} \ |
| --file_name {file_name_of_sequence_data} \ |
| --tokenizer_path tokenizer \ |
| --out_dir {directory_to_save_tokenized_data} \ |
| --block_size 256 |
| ``` |
| |
| **2. Create a config** based on `configs/example_pretraining.py` (training from scratch) or `configs/example_finetuning.py` (fine-tuning). |
|
|
| **3. Train / fine-tune:** |
|
|
| ```bash |
| python train.py --config {path_to_your_config_file} |
| ``` |
|
|
| ### Train your own tokenizer (optional) |
|
|
| ```bash |
| python train_BPE.py \ |
| --txt_file_path {path_to_training_file_one_sequence_per_line} \ |
| --vocab_size 50256 \ |
| --new_tokenizer_path {directory_to_save_trained_tokenizer} |
| ``` |
|
|
| ## Repository Structure |
|
|
| ``` |
| . |
| ├── LICENSE |
| ├── README.md |
| ├── CITATION.cff # machine-readable citation metadata |
| ├── model.py # model architecture (decoder-only Transformer) |
| ├── sampling.py # generate sequences from a trained model |
| ├── tokenization.py # tokenize sequence data for training |
| ├── train.py # pre-training / fine-tuning entry point |
| ├── train_BPE.py # train a new BPE tokenizer |
| ├── model_updated.pt # recommended checkpoint (longer context, deduplicated data) |
| ├── tokenizer/ # BPE tokenizer for model_updated.pt |
| ├── configs/ |
| │ ├── example_pretraining.py |
| │ └── example_finetuning.py |
| └── experiment_data/ |
| ├── *.csv # data underlying the paper's figures |
| ├── pretraining_data.sh # how the pre-training corpus was built (RNAcentral + MMseqs2) |
| └── historical_version/ # original model (split into parts) + its tokenizer |
| ├── model.pt.part-a{a,b,c,d} |
| └── tokenizer_bpe_1024/ |
| ``` |
|
|
| ## Training Data |
|
|
| GenerRNA was pre-trained on RNA sequences from **[RNAcentral](https://rnacentral.org/)** (release 22, which aggregates 51 expert databases). Starting from **34.39 million** raw sequences, deduplication with **[MMseqs2](https://github.com/soedinglab/MMseqs2)** at **80% sequence identity** yielded a pre-training corpus of **~16 million sequences (16.09M), encompassing ~17.4 billion nucleotides**. GenerRNA has a context window of **1024 tokens (~4000 nucleotides)**. The pre-processing pipeline is in [`experiment_data/pretraining_data.sh`](experiment_data/pretraining_data.sh), and the data underlying the paper's figures is provided in `experiment_data/`. See the [paper](https://doi.org/10.1371/journal.pone.0310814) for full dataset details. |
|
|
| ## Limitations |
|
|
| - GenerRNA models RNA **sequence**; it does not explicitly predict tertiary structure or function. Validate candidates with downstream structure/function tools and wet-lab experiments. |
| - A CUDA GPU is required for generation and training as provided. |
| - Zero-shot outputs reflect the natural distribution of the training data; targeting a specific family or property generally requires fine-tuning. |
| - Generated sequences are computational hypotheses and should be experimentally validated before any real-world application. |
|
|
| ## FAQ |
|
|
| **What is GenerRNA?** |
| GenerRNA is a generative, pre-trained language model (a decoder-only Transformer) that designs novel RNA sequences *de novo*, without requiring structural information, functional labels, or sequence alignments. |
|
|
| **How is GenerRNA different from other RNA models?** |
| Most RNA models are *discriminative* — they predict structure or properties from a given sequence. GenerRNA is *generative*: it samples entirely new sequences. To our knowledge, it is the first application of a generative language model to RNA generation. |
|
|
| **Do I need RNA structure or alignments as input?** |
| No. GenerRNA generates sequences directly from its learned distribution; no structure or alignment is needed. |
|
|
| **Can I generate RNAs from a specific family or with a specific function?** |
| Yes. Fine-tune GenerRNA on a family- or function-specific dataset. The paper demonstrates designing RNAs with high binding affinity to the proteins ELAVL1 and SRSF1. |
|
|
| **Which checkpoint should I use?** |
| Use `model_updated.pt` (longer context, trained on deduplicated data). The original split model is kept in `experiment_data/historical_version/` for reproducibility. |
|
|
| **Is GenerRNA free to use?** |
| Yes. The code and weights are released under the MIT License. Please cite the paper if you use GenerRNA in your work. |
|
|
| **How do I cite GenerRNA?** |
| See [Citation](#citation) below. |
|
|
| ## Citation |
|
|
| If you use GenerRNA, its checkpoints, or this repository in your research, please cite: |
|
|
| ```bibtex |
| @article{zhao2024generrna, |
| title = {GenerRNA: A generative pre-trained language model for de novo RNA design}, |
| author = {Zhao, Yichong and Oono, Kenta and Takizawa, Hiroki and Kotera, Masaaki}, |
| journal = {PLOS ONE}, |
| volume = {19}, |
| number = {10}, |
| pages = {e0310814}, |
| year = {2024}, |
| doi = {10.1371/journal.pone.0310814}, |
| publisher = {Public Library of Science} |
| } |
| ``` |
|
|
| **Plain text:** Zhao Y, Oono K, Takizawa H, Kotera M (2024) GenerRNA: A generative pre-trained language model for *de novo* RNA design. PLOS ONE 19(10): e0310814. https://doi.org/10.1371/journal.pone.0310814 |
|
|
| - 📄 **Paper (PLOS ONE):** https://doi.org/10.1371/journal.pone.0310814 |
| - 📝 **Preprint (bioRxiv):** https://doi.org/10.1101/2024.02.01.578496 |
| - 🤗 **Model:** https://huggingface.co/pfnet/GenerRNA |
| - 💻 **Code (GitHub):** https://github.com/ekkkkki/GenerRNA |
| - 🌐 **Project page:** https://ekkkkki.github.io/GenerRNA/ |
|
|
| ## License |
|
|
| The source code is licensed under the **MIT License** — see [`LICENSE`](LICENSE). © 2024 Yichong Zhao, Masaaki Kotera, Kenta Oono, Hiroki Takizawa. |
|
|
