SII-GAIR-NLP/RIBOSPAN-FM

RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling

Many full-length RNAs, particularly mRNAs, exceed the ~1K context lengths used to pretrain representative dense RNA encoders, forcing long transcripts to be truncated and preventing their 5′ UTR, CDS, and 3′ UTR from being modeled jointly at single-nucleotide resolution. RiboSpan is a 1.61B-parameter bidirectional Transformer with single-nucleotide tokenization, dense self-attention in every layer, and native long-context pretraining at up to 10,240 nt. It is trained on 67.6 million RNA sequences comprising 85.7 billion nucleotides from RNAcentral, Ensembl, and Ensembl Genomes.

✨ Features

• 🧬 Single-nucleotide Resolution: Each nucleotide occupies one token position, preserving high-resolution and position-aligned representations across complete RNA sequences.

• 🧠 Dense Bidirectional Transformer: Every layer uses all-to-all self-attention, allowing each nucleotide to directly access upstream, downstream, and distal sequence context.

• 📏 Native 10K Pretraining: The long-context RiboSpan models are natively pretrained at 10,240 nt, learning transcript-scale context during pretraining rather than relying on inference-time context extension.

• 🏆 SOTA Frozen Representations: Without task-specific heads or fine-tuning, RiboSpan-10K achieves state-of-the-art frozen RNA representation quality among the evaluated models, with particularly strong performance on long RNAs.

• 🧩 Robust High-Masking Representations: Continued pretraining with 40% masking substantially improves reconstruction under heavy corruption while preserving backbone representation quality established during 15% MLM pretraining.

🚀 Quick Start

The RIBOSPAN code and inference utilities are maintained in the official GitHub repository, while pretrained model weights are distributed through Hugging Face.

Installation

git clone https://github.com/GAIR-NLP/RIBOSPAN-FM.git
cd RIBOSPAN-FM
pip install -e .

Loading Pretrained Weights

Public RIBOSPAN checkpoints can be loaded directly from Hugging Face:

from pathlib import Path

import torch
import ribospan
from ribospan import RiboSpanForMaskedLM, RiboSpanTokenizer

ckpt = "SII-GAIR-NLP/RIBOSPAN-10K-15"

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = RiboSpanTokenizer(str(Path(ribospan.__file__).with_name("vocab.txt")))

model = RiboSpanForMaskedLM.from_pretrained(ckpt).eval().to(device)

from_pretrained() automatically downloads the checkpoint from Hugging Face and caches it locally, so no manual model-weight placement is required. Alternatively, the checkpoint can be downloaded manually to a user-specified local directory and loaded from that path:

ckpt = "path/to/RIBOSPAN-10K-15"

For restricted-access checkpoints, request access through the corresponding Hugging Face model page. Once access is approved, the checkpoint can be loaded using an authenticated Hugging Face account.

For additional inference examples and usage details, see the official GitHub repository.

🏗️ Model and Checkpoints

RiboSpan uses a 32-layer pre-norm bidirectional Transformer encoder with dense multi-head self-attention and SwiGLU feed-forward networks.

Setting	Configuration	Setting	Configuration
Transformer Layers	32	Model Dimension	2,048
FFN Intermediate Size	5,440	Attention Heads	32
Activation	SwiGLU	Normalization	LayerNorm
Position Encoding	RoPE (rotary dim = 64)	Head Dimension	64
Vocabulary Size	16	Token Unit	nucleotide
Maximum Native Context	10,240 tokens	Parameters	1.61B

The RIBOSPAN model family currently consists of four matched checkpoints:

Checkpoint	Native Context	Masking	Availability
RIBOSPAN-1K-15	1,024	15%	✅ Public
RIBOSPAN-1K-40	1,024	40%	✅ Public
RIBOSPAN-10K-15	10,240	15%	📝 Access upon Request
RIBOSPAN-10K-40	10,240	40%	⏳ Coming Soon

The 40% checkpoints continue from the corresponding 15% runs, improving reconstruction robustness under heavy corruption while preserving backbone representation quality.

🧬 Pretraining

The RiboSpan pretraining corpus combines diverse RNA sequences from RNAcentral v26.0 with quality-controlled protein-coding transcripts from Ensembl release 115 and Ensembl Genomes release 62. Ensembl-derived transcripts retain complete CDS and UTR annotations, providing full-transcript examples for learning dependencies across coding and untranslated regions.

After source-specific filtering, sequence normalization, and exact deduplication, the final corpus contains 67.6 million RNA sequences comprising 85.7 billion nucleotide tokens.

The long-context models are pretrained with masked language modeling at a native context length of 10,240 nt. Training begins with 15% masking and is subsequently continued under 40% masking. Matched 1K models use the same corpus and backbone architecture with a native context length of 1,024 nt.

📊 Evaluation

We evaluate RiboSpan from three complementary perspectives: nucleotide reconstruction, controlled long-context representation behavior, and frozen RNA-type representation quality.

🧩 Long-Context Reconstruction

Native 10K RiboSpan models maintain strong nucleotide reconstruction at 10,240 nt, whereas short-context models degrade substantially when extrapolated far beyond their pretrained context. Continued pretraining with 40% masking further improves reconstruction under heavy corruption while remaining closely matched to the 15% checkpoint under standard masking.

🔭 Long-Context Representation

We introduce a controlled long-context representation benchmark using complete mRNAs spanning 1,024 to 10,240 nt. A short central interval is rearranged while preserving its nucleotide composition, allowing contextual responsiveness, region-specific differentiation, and the propagation of representation changes to be evaluated jointly.

Native 10K RiboSpan exhibits a distinct long-context representation profile. Compared with short-context dense encoders extrapolated beyond their pretrained range, it preserves substantially stronger contextual organization at transcript-scale lengths. Compared with long-sequence models based on hybrid sequence-mixing architectures, it shows stronger context-dependent differentiation while maintaining highly controlled distal propagation.

Together, these results show that native long-context pretraining enables dense bidirectional attention to combine interaction flexibility with long-range calibration, supporting strong context-dependent reorganization while suppressing broad, non-selective propagation across the transcript.

🗺️ SOTA Frozen RNA-Type Representations

Final-layer hidden states are mean-pooled into sequence representations and evaluated directly using leave-one-out cosine 10-NN label recovery and neighborhood purity, with no classifier, projection head, or downstream fine-tuning.

RiboSpan-10K achieves state-of-the-art frozen RNA representation quality among the evaluated foundation models, including the strongest Overall Biotype and Long RNA results. Its advantage is particularly clear for RNAs longer than 1,024 nt, where native long-context modeling preserves substantially more complete sequence information. The 15% and 40% 10K checkpoints remain closely matched, showing that high-masking continued pretraining preserves the learned representation structure.

📄 License

The RIBOSPAN model weights are provided for non-commercial research use under the RIBOSPAN Non-Commercial Model License 1.0.

🙏 Acknowledgements

We thank RNAcentral, Ensembl, and Ensembl Genomes for providing the sequence resources used to construct the RiboSpan pretraining corpus.

We also acknowledge Megatron-LM and Hugging Face Transformers, whose open-source infrastructure supported the development and implementation of RiboSpan.

📖 Citation

If you find RiboSpan useful in your research, please cite:

@misc{wang2026ribospan,
  title         = {{RIBOSPAN}: A Long-Context {RNA} Foundation Model for Versatile {RNA} Modeling},
  author        = {Wang, Ziyuan and Tang, Bohao and Zhang, Fei and Han, Shuo and Liu, Pengfei},
  year          = {2026},
  eprint        = {2608.22849},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG}
}

---

RiboSpan — Single-nucleotide Pretraining Across Native-context.