README.md

---
license: other
license_name: ribospan-non-commercial-model-license
license_link: https://huggingface.co/SII-GAIR-NLP/RIBOSPAN-FM/blob/main/LICENSE
pipeline_tag: fill-mask

tags:
- biology
- rna
- genomics
- transcriptomics
- foundation-model
- masked-language-model

datasets:
- SII-GAIR-NLP/RIBOSPAN-FM-Corpus
---

# RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling

[![arXiv](https://img.shields.io/badge/arXiv-2608.22849-b31b1b.svg)](https://arxiv.org/abs/2608.22849)
[![GitHub](https://img.shields.io/badge/GitHub-181717?logo=github)](https://github.com/GAIR-NLP/RIBOSPAN-FM)
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97-Hugging%20Face-yellow)](https://huggingface.co/SII-GAIR-NLP/RIBOSPAN-FM)
[![Model License](https://img.shields.io/badge/Weights-RIBOSPAN_Non--Commercial-orange.svg)](https://huggingface.co/SII-GAIR-NLP/RIBOSPAN-FM/blob/main/LICENSE)

Many full-length RNAs, particularly mRNAs, exceed the ~1K context lengths used to pretrain representative dense RNA encoders, forcing long transcripts to be truncated and preventing their 5′ UTR, CDS, and 3′ UTR from being modeled jointly at single-nucleotide resolution. **RiboSpan** is a **1.61B-parameter bidirectional Transformer** with **single-nucleotide tokenization**, **dense self-attention in every layer**, and **native long-context pretraining at up to 10,240 nt**. It is trained on **67.6 million RNA sequences comprising 85.7 billion nucleotides** from RNAcentral, Ensembl, and Ensembl Genomes.

## ✨ Features

- **🧬 Single-nucleotide Resolution**: Each nucleotide occupies **one token position**, preserving high-resolution and position-aligned representations across complete RNA sequences.

- **🧠 Dense Bidirectional Transformer**: Every layer uses **all-to-all self-attention**, allowing each nucleotide to directly access upstream, downstream, and distal sequence context.

- **📏 Native 10K Pretraining**: The long-context RiboSpan models are **natively pretrained at 10,240 nt**, learning transcript-scale context during pretraining rather than relying on inference-time context extension.

- **🏆 SOTA Frozen Representations**: Without task-specific heads or fine-tuning, RiboSpan-10K achieves **state-of-the-art frozen RNA representation quality** among the evaluated models, with particularly strong performance on **long RNAs**.

- **🧩 Robust High-Masking Representations**: Continued pretraining with **40% masking** substantially improves reconstruction under heavy corruption while **preserving backbone representation quality** established during 15% MLM pretraining.

![RiboSpan architecture](docs/Model_Architecture.png)

## 🚀 Quick Start

The RIBOSPAN code and inference utilities are maintained in the [official GitHub repository](https://github.com/GAIR-NLP/RIBOSPAN-FM), while pretrained model weights are distributed through Hugging Face.

### Installation

```bash
git clone https://github.com/GAIR-NLP/RIBOSPAN-FM.git
cd RIBOSPAN-FM
pip install -e .
```

### Loading Pretrained Weights

Public RIBOSPAN checkpoints can be loaded directly from Hugging Face:

```python
from pathlib import Path

import torch
import ribospan
from ribospan import RiboSpanForMaskedLM, RiboSpanTokenizer

ckpt = "SII-GAIR-NLP/RIBOSPAN-10K-15"

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = RiboSpanTokenizer(str(Path(ribospan.__file__).with_name("vocab.txt")))

model = RiboSpanForMaskedLM.from_pretrained(ckpt).eval().to(device)
```

`from_pretrained()` automatically downloads the checkpoint from Hugging Face and caches it locally, so no manual model-weight placement is required. Alternatively, the checkpoint can be downloaded manually to a user-specified local directory and loaded from that path:

```python
ckpt = "path/to/RIBOSPAN-10K-15"
```

For restricted-access checkpoints, **request access** through the corresponding Hugging Face model page. Once access is approved, the checkpoint can be loaded using an authenticated Hugging Face account.

For additional inference examples and usage details, see the [official GitHub repository](https://github.com/GAIR-NLP/RIBOSPAN-FM).

## 🏗️ Model and Checkpoints

RiboSpan uses a **32-layer pre-norm bidirectional Transformer encoder** with dense multi-head self-attention and SwiGLU feed-forward networks.

| Setting | Configuration | Setting | Configuration |
|---|---|---|---|
| Transformer Layers | 32 | Model Dimension | 2,048 |
| FFN Intermediate Size | 5,440 | Attention Heads | 32 |
| Activation | SwiGLU | Normalization | LayerNorm |
| Position Encoding | RoPE (rotary dim = 64) | Head Dimension | 64 |
| Vocabulary Size | 16 | Token Unit | nucleotide |
| Maximum Native Context | 10,240 tokens | Parameters | 1.61B |

The RIBOSPAN model family currently consists of four matched checkpoints:

| Checkpoint | Native Context | Masking | Availability |
|---|---:|---:|---|
| **RIBOSPAN-1K-15** | 1,024 | 15% | ✅ [Public](https://huggingface.co/SII-GAIR-NLP/RIBOSPAN-1K-15) |
| **RIBOSPAN-1K-40** | 1,024 | 40% | ✅ [Public](https://huggingface.co/SII-GAIR-NLP/RIBOSPAN-1K-40) |
| **RIBOSPAN-10K-15** | 10,240 | 15% | 📝 [Access upon Request](https://huggingface.co/SII-GAIR-NLP/RIBOSPAN-10K-15) |
| **RIBOSPAN-10K-40** | 10,240 | 40% | ⏳ Coming Soon |

The 40% checkpoints continue from the corresponding 15% runs, **improving reconstruction robustness under heavy corruption while preserving backbone representation quality**.

## 🧬 Pretraining

The RiboSpan pretraining corpus combines diverse RNA sequences from **RNAcentral v26.0** with quality-controlled protein-coding transcripts from **Ensembl release 115** and **Ensembl Genomes release 62**. Ensembl-derived transcripts retain complete CDS and UTR annotations, providing full-transcript examples for learning dependencies across coding and untranslated regions.

After source-specific filtering, sequence normalization, and exact deduplication, the final corpus contains **67.6 million RNA sequences** comprising **85.7 billion nucleotide tokens**.

The long-context models are pretrained with masked language modeling at a **native context length of 10,240 nt**. Training begins with **15% masking** and is subsequently continued under **40% masking**. Matched 1K models use the same corpus and backbone architecture with a native context length of 1,024 nt.

![Pretraining corpus](docs/Pretraining_Corpus.png)

## 📊 Evaluation

We evaluate RiboSpan from three complementary perspectives: **nucleotide reconstruction**, **controlled long-context representation behavior**, and **frozen RNA-type representation quality**.

### 🧩 Long-Context Reconstruction

Native 10K RiboSpan models maintain **strong nucleotide reconstruction at 10,240 nt**, whereas short-context models degrade substantially when extrapolated far beyond their pretrained context. Continued pretraining with **40% masking further improves reconstruction under heavy corruption** while remaining closely matched to the 15% checkpoint under standard masking.

### 🔭 Long-Context Representation

We introduce a **controlled long-context representation benchmark** using complete mRNAs spanning **1,024 to 10,240 nt**. A short central interval is rearranged while preserving its nucleotide composition, allowing **contextual responsiveness**, **region-specific differentiation**, and the **propagation of representation changes** to be evaluated jointly.

Native 10K RiboSpan exhibits a distinct long-context representation profile. Compared with short-context dense encoders extrapolated beyond their pretrained range, it preserves **substantially stronger contextual organization at transcript-scale lengths**. Compared with long-sequence models based on hybrid sequence-mixing architectures, it shows **stronger context-dependent differentiation** while maintaining **highly controlled distal propagation**.

Together, these results show that native long-context pretraining enables dense bidirectional attention to combine **interaction flexibility with long-range calibration**, supporting **strong context-dependent reorganization** while suppressing broad, non-selective propagation across the transcript.

![Long-context representation](docs/Long_Context_Benchmark.png)

### 🗺️ SOTA Frozen RNA-Type Representations

Final-layer hidden states are mean-pooled into sequence representations and evaluated directly using **leave-one-out cosine 10-NN label recovery** and **neighborhood purity**, with **no classifier, projection head, or downstream fine-tuning**.

RiboSpan-10K achieves **state-of-the-art frozen RNA representation quality** among the evaluated foundation models, including the strongest **Overall Biotype** and **Long RNA** results. Its advantage is particularly clear for **RNAs longer than 1,024 nt**, where native long-context modeling preserves substantially more complete sequence information. The 15% and 40% 10K checkpoints remain closely matched, showing that **high-masking continued pretraining preserves the learned representation structure**.

![RNA-type representations](docs/RNA_Type_Benchmark.png)

## 📄 License

The RIBOSPAN model weights are provided for **non-commercial research use** under the [RIBOSPAN Non-Commercial Model License 1.0](https://huggingface.co/SII-GAIR-NLP/RIBOSPAN-FM/blob/main/LICENSE).

## 🙏 Acknowledgements

We thank [RNAcentral](https://rnacentral.org/), [Ensembl](https://www.ensembl.org/), and [Ensembl Genomes](https://ensemblgenomes.org/) for providing the sequence resources used to construct the RiboSpan pretraining corpus.

We also acknowledge [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [Hugging Face Transformers](https://github.com/huggingface/transformers), whose open-source infrastructure supported the development and implementation of RiboSpan.

## 📖 Citation

If you find RiboSpan useful in your research, please cite:

```bibtex
@misc{wang2026ribospan,
  title         = {{RIBOSPAN}: A Long-Context {RNA} Foundation Model for Versatile {RNA} Modeling},
  author        = {Wang, Ziyuan and Tang, Bohao and Zhang, Fei and Han, Shuo and Liu, Pengfei},
  year          = {2026},
  eprint        = {2608.22849},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG}
}
```

---

**RiboSpan** — Single-nucleotide Pretraining Across Native-context.