GB.RNA-25M-MARS

25M-parameter model from the MARS training series. This is a standalone HuggingFace port that loads without the ModelGenerator package.

Renamed. GenBio AI renamed AIDO.RNA to GB.RNA; the upstream checkpoints now live at genbio-ai/GB.RNA-*. This repository was previously Taykhoom/AIDO.RNA-25M-MARS, and that id redirects here. The weights are unchanged.

Architecture

Parameter	Value
Layers	8
Attention heads	16
Embedding dimension	512
FFN hidden dimension	1344 (SwiGLU)
Vocabulary size	16
Positional encoding	RoPE (rotary_percent=1.0)
Normalization	LayerNorm
MLP activation	SwiGLU
Architecture	Pre-LN Transformer
Max sequence length	1024 (training truncation; RoPE has no hard limit)

Vocabulary: [PAD], [MASK], [CLS], [SEP], [UNK], A, G, C, T, U, N, [BOS], [EOS], [UNUSED1], [UNUSED2], [UNUSED3]

Inputs should use DNA notation (A, C, G, T), as in the upstream implementation. Convert U to T before tokenization; the tokenizer treats them as distinct vocabulary entries.

Pretraining

• Objective: Masked language modeling (MLM) on RNA sequences
• Data: ncRNA sequences (MARS dataset)
• Source checkpoint: genbio-ai/GB.RNA-25M-MARS

Checkpoint selection

Mid-size MARS-series checkpoint balancing speed and representation quality.

Parity Verification

Hidden-state representations compared against the original genbio-ai/GB.RNA-25M-MARS weights at all 9 representation levels (embedding + 8 transformer layers). Intermediate layer differences are floating-point accumulation noise normalised away by the final layer norm; the final output matches the original within 1e-5. max diff = 8.58e-06 (final output), 6.10e-05 (intermediate layers). Verified on GPU with PyTorch 2.7 / CUDA 12.

Related Models

See the full GB.RNA collection.

Model	Parameters	Data	Notes
GB.RNA-1M-MARS	1M	MARS ncRNA	Smallest MARS variant
GB.RNA-25M-MARS	25M	MARS ncRNA	Mid-size MARS variant
GB.RNA-300M-MARS	300M	MARS ncRNA	Large MARS variant
GB.RNA-650M	650M	42M ncRNA	Base model
GB.RNA-650M-CDS	650M	42M ncRNA + CDS	CDS-adapted
GB.RNA-1.6B	1.6B	42M ncRNA	Largest base model
GB.RNA-1.6B-CDS	1.6B	42M ncRNA + CDS	Largest CDS-adapted

Usage

Embedding generation

import torch
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("Taykhoom/GB.RNA-25M-MARS", trust_remote_code=True)
model = AutoModel.from_pretrained("Taykhoom/GB.RNA-25M-MARS", trust_remote_code=True)
model.eval()

sequences = ["ACGTGCTAGCTAGCTA", "ATGCTAGCTAGCTAGC"]
enc = tokenizer(sequences, return_tensors="pt", padding=True)

with torch.no_grad():
    out = model(**enc)

cls_emb   = out.last_hidden_state[:, 0, :]   # (batch, 512) -- CLS token
token_emb = out.last_hidden_state             # (batch, seq_len, 512)

# Intermediate layers
out_all = model(**enc, output_hidden_states=True)
layer3_emb = out_all.hidden_states[3]

MLM logits

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("Taykhoom/GB.RNA-25M-MARS", trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained("Taykhoom/GB.RNA-25M-MARS", trust_remote_code=True)
model.eval()

enc = tokenizer(["ACGT[MASK]GCTA"], return_tensors="pt")
with torch.no_grad():
    logits = model(**enc).logits   # (1, seq_len, 16)

Fine-tuning

Standard HF conventions. Use cls_emb = out.last_hidden_state[:, 0, :] (CLS token) as input to a task-specific head for sequence-level tasks.

Implementation Notes

The original genbio-ai/GB.RNA-25M-MARS checkpoint requires the ModelGenerator package to load. This port is a clean standalone re-implementation:

• All model logic is contained in modeling_gbrna.py and configuration_gbrna.py.
• The tokenizer normalizes U/u to T/t, matching the original checkpoint preprocessing.
• attn_implementation="sdpa" and attn_implementation="flash_attention_2" are added (not present in the original genbio-ai implementation).
• Architecture: pre-LN Transformer with SwiGLU MLP and RoPE positional embeddings.

Citation

@article{zou2024_gbrna,
  title   = {A Large-Scale Foundation Model for {RNA} Function and Structure Prediction},
  author  = {Zou, Shuxian and Tao, Tianhua and Mahbub, Sazan and Ellington, Caleb N. and Algayres, Robin and Li, Dian and Zhuang, Yonghao and Wang, Hongyi and Song, Le and Xing, Eric P.},
  journal = {bioRxiv},
  year    = {2024},
  doi     = {10.1101/2024.11.28.625345}
}

Credits

Original model and code by Zou et al. Source: GitHub. Hugging Face port maintained by Taykhoom Dalal.

License

GenBio AI Community License, following the original repository. See LICENSE for details.