---
tags:
- dna
- genomics
- multispecies
- masked-lm
- bert
- genomics-foundation-model
- modernbert
---

# ModernGENA base
ModernGENA is a DNA foundation model based on **ModernBERT** (a modernized BERT-style encoder architecture) adapted for genomic sequence modeling.  
**ModernGENA base** is the 377M-parameter version introduced in the paper *Back to BERT in 2026: ModernGENA as a Strong, Efficient Baseline for DNA Foundation Models*.

How to load pre-trained model to fine-tune it on classification task: [GENA_LM repository](https://github.com/AIRI-Institute/GENA_LM/tree/main/examples/modernGENA)

## Technical features
- ModernBERT-based encoder architecture
- RoPE positional embeddings
- hybrid local/global attention
- pre-norm transformer blocks
- GeGLU feed-forward layers
- end-to-end unpadding
- FlashAttention-based efficient inference on compatible hardware
- `torch.compile` support

## Model strengths
- strong efficiency-quality trade-off
- higher inference throughput with FlashAttention-based implementations
- competitive downstream performance on the Nucleotide Transformer benchmark
- intended to support long genomic contexts

This makes it a practical baseline for genomic modeling experiments and future architectural comparisons.

## Tokenization


ModernGENA uses the [**32k BPE vocabulary (AIRI-Institute/gena-lm-bert-base-t2t)**](https://huggingface.co/AIRI-Institute/gena-lm-bert-base-t2t) introduced in GENA-LM, built over the DNA alphabet symbols `A/T/G/C/N`, with special tokens `[CLS]`, `[SEP]`, `[PAD]`, `[UNK]`, and `[MASK]`.

## Pretraining corpus

- **443 vertebrate genome assemblies**
- **353,574,093,776 bp** total
- Includes both forward strand and reverse complement sequences
- Excludes sequences containing ambiguous symbols other than `A/C/G/T`

To reduce overrepresentation of simple repeats and enrich biologically informative regions, training intervals were sampled around transcription start sites:

- window: **[-16 kbp, +8 kbp]** around each unique TSS
- overlapping intervals merged with BEDTools
- both strands included for each resulting region

## Load pretrained model

```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("AIRI-Institute/gena-lm-bert-base-t2t")
model = AutoModel.from_pretrained("AIRI-Institute/moderngena-base", trust_remote_code=True, attn_implementation="flash_attention_2")
```

## Evaluation
For evaluation results, see our paper: 

## Citation