---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- genomics
- rna
- nucleotide
- sequence-modeling
- biology
- bioinformatics
- electra
pipeline_tag: feature-extraction
---

# RNAElectra: Single-Nucleotide ELECTRA-Style Pre-training for RNA Representation Learning

RNAElectra is a nucleotide-resolution RNA language model trained using an ELECTRA-style objective for efficient and discriminative representation learning. The model produces contextualized embeddings for RNA sequences and is designed for downstream transcriptomic and regulatory modeling tasks.

## Model Details

- **Model Type**: Transformer-based discriminator model  
- **Training Objective**: ELECTRA-style replaced-token detection  
- **Resolution**: Single-nucleotide  
- **Domain**: RNA and transcriptomic sequences  
- **Architecture**: ModernBERT-style backbone adapted for nucleotide sequences  

RNAElectra focuses on efficient pre-training by learning to discriminate corrupted tokens rather than reconstruct them, leading to strong representations with improved training efficiency.

## Key Features

- Single-nucleotide tokenization  
- Contextual RNA sequence embeddings  
- ELECTRA-style discriminative pre-training  
- Suitable for RNA function prediction, RBP binding modeling, stability prediction, regulatory element analysis, and downstream fine-tuning tasks  

## Usage

### Basic Feature Extraction

```python
import torch
from transformers import AutoModel
from tokenizer import NucEL_Tokenizer

device = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModel.from_pretrained(
    "FreakingPotato/RNAElectra",
    trust_remote_code=True
).to(device)
model.eval()

tokenizer = NucEL_Tokenizer.from_pretrained(
    "FreakingPotato/RNAElectra",
    trust_remote_code=True
)

sequence = "AUGCAUGCAUGCAUGC"

inputs = tokenizer(sequence, return_tensors="pt")
inputs = {k: v.to(device) for k, v in inputs.items()}

with torch.no_grad():
    outputs = model(**inputs)

embeddings = outputs.last_hidden_state
print(f"Sequence embeddings shape: {embeddings.shape}")
```

## Installation

```bash
pip install transformers torch
```

## Requirements

- transformers >= 5.0.0
- torch >= 2.10.0
- Python >= 3.12.3

GPU is recommended for large-scale inference.

## Pre-training Overview

RNAElectra was trained using an ELECTRA-style generator–discriminator framework. A generator predicts corrupted tokens, and a discriminator learns to detect replaced tokens. Only the discriminator weights are released in this repository. This objective improves training efficiency compared to masked language modeling while preserving strong contextual representations.

## Intended Use

RNAElectra is intended for feature extraction, downstream fine-tuning, and representation learning in RNA and transcriptomic modeling tasks. It is not intended for clinical decision-making or medical diagnostics.

## License

This model is released under the Apache 2.0 License.