---
title: NTv3 Foundation Models for Genomics
emoji: 🧬
colorFrom: indigo
colorTo: blue
sdk: static
pinned: false
license: apache-2.0
---

# 🧬 NTv3: Foundation Models for Genomics

A genomic foundation model that reads up to 1 Mb of DNA at single-base resolution. One architecture, three
capabilities: **functional-track prediction**, **gene annotation** and **regulatory sequence design**. Of the
paper's design models, the plant promoter one is released as an example of the generative models. This Space is the
companion hub, with runnable PyTorch notebooks.

> Looking for the 2025 models? See [Initial release (2025)](#-initial-release-2025) below.

## 📖 About NTv3

NTv3 reads DNA at single-base resolution across up to **1 Mb** of context in one pass. One architecture supports three
capabilities: **functional-track prediction**, **gene annotation** and **regulatory sequence design**. The backbone is
U-Net-style: a convolutional tower that reads the sequence, a Transformer tower that carries long-range context, and a
deconvolutional tower that returns predictions at base resolution.

They are acquired through a staged recipe, each stage building on the one before:

1. **Pre-training**, self-supervised. Masked-token prediction over unlabeled genomes from the OpenGenome2 corpus -> **NTv3-pretrained**.
2. **Tracks post-training**, supervised multi-task. 16,262 functional tracks across 15 species -> **NTv3-tracks**.
3. **Annotation fine-tuning**, supervised single-task. Reference gene annotations -> one **NTv3-annotate** model
   per clade (mammals, plants, insects).
4. **Generative post-training**, masked diffusion on MPRA activity datasets -> **NTv3-generative**. The plant
   promoter model is released as an example; the plant enhancer and fly enhancer models are not.

## 🧩 What it does

| Capability | Detail |
| --- | --- |
| Reads any genome | 1 Mb context, nucleotide resolution, base-pair-resolution prediction |
| Predicts functional tracks | 16,262 tracks, 15 species: 9,603 gene expression, 3,115 histone marks, 2,193 TF binding, 1,351 accessibility |
| Annotates genes | DNA alone to per-base posteriors over intergenic, intron, CDS and splice states, decoded to gene models on both strands |
| Designs regulatory DNA | Conditional generation of plant promoters, plant enhancers and fly enhancers, validated by measured activity |

## 🤖 The NTv3 models

Each capability is released as its own checkpoint, all sharing the same architecture and the same pretrained
representations, at **700M parameters**.

| Model | Stage | What it is for | Repository |
| --- | --- | --- | --- |
| NTv3-pretrained | Pretrained | Embeddings, masked-token prediction, your own fine-tuning | [`InstaDeepAI/NTv3_pretrained`](https://huggingface.co/InstaDeepAI/NTv3_pretrained) |
| NTv3-tracks | Tracks post-training | Functional-track prediction from 16 kb to 1 Mb inputs, and the backbone for downstream fine-tuning | [`InstaDeepAI/NTv3_tracks`](https://huggingface.co/InstaDeepAI/NTv3_tracks) |
| NTv3-tracks-1mb | Tracks post-training | The 1 Mb specialist behind the paper's results; 1 Mb inputs only | [`InstaDeepAI/NTv3_tracks_1mb`](https://huggingface.co/InstaDeepAI/NTv3_tracks_1mb) |
| NTv3-annotate | Annotation fine-tuning | *Ab initio* gene prediction, one model per clade | [`_mammals`](https://huggingface.co/InstaDeepAI/NTv3_annot_mammals) / [`_plants`](https://huggingface.co/InstaDeepAI/NTv3_annot_plants) / [`_insects`](https://huggingface.co/InstaDeepAI/NTv3_annot_insects) |
| NTv3-generative-plant-promoter | Generative post-training | Example generative model used in the paper: designs plant core promoters at a requested strength, and is what notebook 06 runs | [`InstaDeepAI/NTv3_generative_plant_promoter`](https://huggingface.co/InstaDeepAI/NTv3_generative_plant_promoter) |

Of the paper's design models, only the plant promoter one is released, as an example; the plant enhancer and fly
enhancer models are not. It is the model notebook 06 runs, and a worked example of the recipe notebook 05 teaches.
Read its model card before quoting anything from it: it is not species-aware.

**700M** is the figure quoted throughout, counted from the published weights rather than the internal registry, which
still calls the pretrained checkpoint 650M. Exactly: 703,349,146 for pretrained, 711,256,680 for tracks,
703,446,690 for each annotate clade, and 721,379,955 for the plant promoter example, which carries three condition
tables and conditioning layers in its trunk.

The [NTv3 collection](https://huggingface.co/collections/InstaDeepAI/ntv3) is the canonical index
of what is downloadable today.

## 🔐 Access

The models are gated: request access once on any of their model pages, and one approval, reviewed manually, covers
all of them. Then log in with `hf auth login`, or `notebook_login()` in a notebook. The datasets and this Space are
open.

## 📦 Install

```bash
pip install torch transformers accelerate safetensors huggingface_hub numpy
```

Works on **transformers 4.57.1 and 5.17.0**, which now agree exactly. Two faults that broke 5.x are fixed: the rotary
frequencies are rebuilt per call rather than restored from a buffer that 5.x zeroed, and the annotation checkpoints
store `residual_scale` as the scalar the model declares. Re-measured on chr19:6,700,000-6,716,384 across both
versions, the predictions are identical to the bit. Older 5.x releases may still carry the zeroed-buffer behaviour,
which fails silently, so pin `transformers>=4.57.1` and check against the fixtures in each repo if you stray.

Models use custom code, so pass `trust_remote_code=True`. Input sequence length must be a multiple of 128.

## 🚀 Quickstart with NTv3-pretrained

Both checkpoints are registered under `AutoModel`, and the forward takes the token tensor positionally rather than
the tokenizer dict, so `model(**batch)` will not work. The forward returns a plain `dict`, not a `ModelOutput`, so
read results with `out["logits"]` rather than `out.logits`.

```python
import torch
from transformers import AutoModel, AutoTokenizer

model_name = "InstaDeepAI/NTv3_pretrained"

tok = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True).eval()

# Batch only equal-length sequences: padding is not masked out
batch = tok(["ACGT" * 32, "TTGA" * 32], add_special_tokens=False, return_tensors="pt")

with torch.no_grad():
    out = model(batch["input_ids"])

print(out["logits"].shape)      # (B, L, V = 11)
print(out["embedding"].shape)   # (B, L, 1024)
```

Input length must be a multiple of 128, and **padded positions are not masked out of attention**. The same sequence
run alone and run padded to a longer length gives different values at its real positions, so batch only sequences of
equal length and run ragged ones separately.

## 📊 NTv3-tracks: functional-track prediction

The `ntv3-tracks` pipeline takes a genomic region and the tracks you want, fetches the sequence, conditions on the
species, predicts and plots.

```python
from transformers import pipeline

ntv3_tracks = pipeline("ntv3-tracks", model="InstaDeepAI/NTv3_tracks", trust_remote_code=True)

out = ntv3_tracks(
    {"chrom": "chr19", "start": 6_700_000, "end": 6_732_768, "species": "human"},
    plot=True,
    tracks_to_plot={"K562 RNA-seq": "ENCSR056HPM", "K562 CTCF": "ENCSR000AKO"},
)

print(out.tracks.shape)        # (12288, 2), the central 37.5 % of the window
print(out.coordinates)         # (6710240, 6722528), the span actually predicted
print(out.track("ENCSR000AKO").shape)  # (12288,): one track
```

Naming the tracks computes only those channels. The head is as wide as the largest species panel (hg38, 7,362
tracks), so a full forward at 1 Mb materialises about 11 GB, while a named subset is bit-identical and a few MB.
Species, assemblies and track order all come from the model's own `config.track_metadata`.

For the plain forward, which is where to start if you want to fine-tune rather than predict, `NTv3_tracks` takes a
species token alongside the sequence, and its nucleotide vocabulary is 16 tokens rather than the 11 of the pretrained
model, so it reads soft-masked (lowercase) sequence directly. Notebook 01 covers it.

The 2025 `ntv3-interpret` pipeline is not being carried over. Notebook 04 shows saliency and attention with a plain
`AutoModel` forward instead.

## 🏷️ NTv3-annotate: genome annotation

```python
import torch
from transformers import AutoModel, AutoTokenizer

CLADE = "mammals"  # or "plants", or "insects"
repo = f"InstaDeepAI/NTv3_annot_{CLADE}"

model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
batch = tok("ACGT" * 256, add_special_tokens=False, return_tensors="pt")

# Annotation is not conditioned on species, so all three clades take an empty list
with torch.no_grad():
    out = model(batch["input_ids"], conditions_tokens=[])

print(out["bed_tracks_logits"].shape)   # (1, 768, 1, 15); 768 = 1024 * 0.75
```

The output covers the central 75% of the window, the crop the model is served at. Logits alone are not an annotation:
notebook 03b decodes them into gene models (GFF3) with the HMM whose specification `config.json` carries under
`annotation`, and notebook 03c scores an annotation the way the benchmark does.

## 📓 Notebooks

The notebooks cover the workflows end to end, in `notebooks_tutorials/`:

| # | Notebook | File | Status |
| --- | --- | --- | --- |
| 00 | 🚀 Quickstart inference | [`00_quickstart_inference.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/00_quickstart_inference.ipynb) | ✅ |
| 01 | 📊 Tracks prediction | [`01_tracks_prediction.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/01_tracks_prediction.ipynb) | ✅ |
| 02 | 🎯 Fine-tune NTv3 on bigwig tracks | [`02_fine_tuning_bigwig_tracks.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/02_fine_tuning_bigwig_tracks.ipynb) | ✅ |
| 03a | 🏷️ Train a gene-annotation model | [`03a_genome_annotation_training.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/03a_genome_annotation_training.ipynb) | ✅ |
| 03b | 🏷️ From per-base predictions to gene models | [`03b_genome_annotation_decoding.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/03b_genome_annotation_decoding.ipynb) | ✅ |
| 03c | 🏷️ Evaluate a gene annotation against a reference | [`03c_genome_annotation_evaluation.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/03c_genome_annotation_evaluation.ipynb) | ✅ |
| 04 | 🔍 Model interpretation | [`04_model_interpretation.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/04_model_interpretation.ipynb) | ✅ |
| 05 | 🧪 Fine-tuning NTv3 into a diffusion model | [`05_diffusion_fine_tuning.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/05_diffusion_fine_tuning.ipynb) | ✅ |
| 06 | 🌱 Generating plant promoter sequences | [`06_plant_promoter_generation.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/06_plant_promoter_generation.ipynb) | ✅ |

All run against the published checkpoints.

## 🗄️ Initial release (2025)

The initial-release repositories are **retained unchanged** for reproducibility and backward compatibility. Existing
model IDs, files and interfaces continue to work. If you are reproducing published results or already have a working
pipeline, stay on these.

**Pretrained:** `NTv3_8M_pre`, `NTv3_100M_pre`, `NTv3_650M_pre`

**Post-trained:** `NTv3_100M_post`, `NTv3_650M_post`

**Context-length variants:** `NTv3_8M_pre_8kb`, `NTv3_100M_pre_8kb`, `NTv3_650M_pre_8kb`, `NTv3_100M_post_131kb`,
`NTv3_650M_post_131kb`

**Architecture ablations:** `NTv3_5downsample_pre`, `NTv3_5downsample_pre_8kb`, `NTv3_5downsample_post`,
`NTv3_5downsample_post_131kb`

**Generative:** `NTv3_generative`

All under the `InstaDeepAI/` namespace. Note that some earlier materials wrote the post-trained checkpoints as
`NTv3_100M_pos` / `NTv3_650M_pos`; the correct identifiers end in `_post`.

These models were trained with a *joint* post-training objective over tracks and annotation labels, so numbers quoted
in their materials do not always match the current paper. Where they differ, the current paper is authoritative.

## 🔗 Links

- 📄 **Paper:** [bioRxiv preprint](https://www.biorxiv.org/content/10.64898/2025.12.22.695963)
- 💻 **JAX model code:** [github.com/instadeepai/nucleotide-transformer](https://github.com/instadeepai/nucleotide-transformer)
- 🎯 **Model collection:** [NTv3 collection](https://huggingface.co/collections/InstaDeepAI/ntv3)
- 📚 **Benchmark dataset:** [`InstaDeepAI/NTv3_benchmark_dataset_revised`](https://huggingface.co/datasets/InstaDeepAI/NTv3_benchmark_dataset_revised)
- 🏷️ **Annotation dataset:** [`InstaDeepAI/NTv3_annot_dataset`](https://huggingface.co/datasets/InstaDeepAI/NTv3_annot_dataset)
- 🏆 **Benchmark leaderboard:** [`InstaDeepAI/ntv3_benchmark`](https://huggingface.co/spaces/InstaDeepAI/ntv3_benchmark)
- 🚀 **Interactive demo:** [`InstaDeepAI/ntv3_tracks`](https://huggingface.co/spaces/InstaDeepAI/ntv3_tracks)

## 📝 Citation

```bibtex
@article{ntv3,
  title   = {A multispecies foundation model for genome function prediction, gene annotation and sequence design},
  author  = {Boshar, Sam and Evans, Benjamin and Tang, Ziqi and Adel, Yanis and Bornachot, Yohan and Sidbon, Shawn and Lorbeer, Franziska K. and Picard, Armand and Rajesh, Chandana and Seitz, Evan and Emms, David and Mendoza-Revilla, Javier and Hernandez, Ariana and Al-Ani, Fatimah and Karch, Tristan and Schiff, Yair and Lopez, Marie and Laterre, Alexandre and Beguir, Karim and Koo, Peter and Kuleshov, Volodymyr and Stark, Alexander and de Almeida, Bernardo P. and Pierrot, Thomas},
  journal = {bioRxiv},
  year    = {2026},
  doi     = {10.64898/2025.12.22.695963},
  url     = {https://www.biorxiv.org/content/10.64898/2025.12.22.695963}
}
```

## 📜 License

**Model weights:** released under the InstaDeep Open Model Licence, which permits non-commercial use only and also
places conditions on the **outputs** of the models. See the `MODEL-LICENSE.md` shipped in each model repository for the
authoritative terms.

**Code and notebooks in this Space:** Apache-2.0.