| ---
|
| language:
|
| - en
|
| license: apache-2.0
|
| library_name: transformers
|
| tags:
|
| - genomics
|
| - rna
|
| - nucleotide
|
| - sequence-modeling
|
| - biology
|
| - bioinformatics
|
| - electra
|
| pipeline_tag: feature-extraction
|
| ---
|
|
|
| # RNAElectra: Single-Nucleotide ELECTRA-Style Pre-training for RNA Representation Learning
|
|
|
| RNAElectra is a nucleotide-resolution RNA language model trained using an ELECTRA-style objective for efficient and discriminative representation learning. The model produces contextualized embeddings for RNA sequences and is designed for downstream transcriptomic and regulatory modeling tasks.
|
|
|
| ## Model Details
|
|
|
| - **Model Type**: Transformer-based discriminator model
|
| - **Training Objective**: ELECTRA-style replaced-token detection
|
| - **Resolution**: Single-nucleotide
|
| - **Domain**: RNA and transcriptomic sequences
|
| - **Architecture**: ModernBERT-style backbone adapted for nucleotide sequences
|
|
|
| RNAElectra focuses on efficient pre-training by learning to discriminate corrupted tokens rather than reconstruct them, leading to strong representations with improved training efficiency.
|
|
|
| ## Key Features
|
|
|
| - Single-nucleotide tokenization
|
| - Contextual RNA sequence embeddings
|
| - ELECTRA-style discriminative pre-training
|
| - Suitable for RNA function prediction, RBP binding modeling, stability prediction, regulatory element analysis, and downstream fine-tuning tasks
|
|
|
| ## Usage
|
|
|
| ### Basic Feature Extraction
|
|
|
| ```python
|
| import torch
|
| from transformers import AutoModel
|
| from tokenizer import NucEL_Tokenizer
|
|
|
| device = "cuda" if torch.cuda.is_available() else "cpu"
|
|
|
| model = AutoModel.from_pretrained(
|
| "FreakingPotato/RNAElectra",
|
| trust_remote_code=True
|
| ).to(device)
|
| model.eval()
|
|
|
| tokenizer = NucEL_Tokenizer.from_pretrained(
|
| "FreakingPotato/RNAElectra",
|
| trust_remote_code=True
|
| )
|
|
|
| sequence = "AUGCAUGCAUGCAUGC"
|
|
|
| inputs = tokenizer(sequence, return_tensors="pt")
|
| inputs = {k: v.to(device) for k, v in inputs.items()}
|
|
|
| with torch.no_grad():
|
| outputs = model(**inputs)
|
|
|
| embeddings = outputs.last_hidden_state
|
| print(f"Sequence embeddings shape: {embeddings.shape}")
|
| ```
|
|
|
| ## Installation
|
|
|
| ```bash
|
| pip install transformers torch
|
| ```
|
|
|
| ## Requirements
|
|
|
| - transformers >= 5.0.0
|
| - torch >= 2.10.0
|
| - Python >= 3.12.3
|
|
|
| GPU is recommended for large-scale inference.
|
|
|
| ## Pre-training Overview
|
|
|
| RNAElectra was trained using an ELECTRA-style generator–discriminator framework. A generator predicts corrupted tokens, and a discriminator learns to detect replaced tokens. Only the discriminator weights are released in this repository. This objective improves training efficiency compared to masked language modeling while preserving strong contextual representations.
|
|
|
| ## Intended Use
|
|
|
| RNAElectra is intended for feature extraction, downstream fine-tuning, and representation learning in RNA and transcriptomic modeling tasks. It is not intended for clinical decision-making or medical diagnostics.
|
|
|
| ## License
|
|
|
| This model is released under the Apache 2.0 License. |
