---
title: NTv3 Foundation Models for Genomics
emoji: 🧬
colorFrom: indigo
colorTo: blue
sdk: static
pinned: false
---

# 🧬 NTv3: Foundation Models for Genomics

A genomic foundation model that reads up to 1 Mb of DNA at single-base resolution. One architecture, three
capabilities: **functional-track prediction**, **gene annotation** and **regulatory sequence design**. Of the
paper's design models, the plant promoter one is released as an example of the generative models. This Space is the
companion hub, with runnable PyTorch notebooks.

> **Release status.** This Space is being prepared and items still to be settled are marked `TBC` throughout.
> Looking for the 2025 models? See [Initial release (2025)](#-initial-release-2025) below.

## 📖 About NTv3

NTv3 reads DNA at single-base resolution across up to **1 Mb** of context in one pass. One architecture supports three
capabilities: **functional-track prediction**, **gene annotation** and **regulatory sequence design**. The backbone is
U-Net-style: a convolutional tower that reads the sequence, a Transformer tower that carries long-range context, and a
deconvolutional tower that returns predictions at base resolution.

They are acquired through a staged recipe, each stage building on the one before:

1. **Pre-training**, self-supervised. Masked-token prediction over unlabeled genomes from the OpenGenome2 corpus -> **NTv3-pretrained**.
2. **Tracks post-training**, supervised multi-task. 16,262 functional tracks across 15 species -> **NTv3-tracks**.
3. **Annotation fine-tuning**, supervised single-task. Reference gene annotations -> one **NTv3-annotate** model
   per clade (mammals, plants, insects).
4. **Generative post-training**, masked diffusion on MPRA activity datasets -> **NTv3-generative**. The plant
   promoter model is released as an example; the plant enhancer and fly enhancer models are not.

## 🧩 What it does

| Capability | Detail |
| --- | --- |
| Reads any genome | 1 Mb context, nucleotide resolution, base-pair-resolution prediction |
| Predicts functional tracks | 16,262 tracks, 15 species: 9,603 gene expression, 3,115 histone marks, 2,193 TF binding, 1,351 accessibility |
| Annotates genes | DNA alone to per-base posteriors over intergenic, intron, CDS and splice states, decoded to gene models on both strands |
| Designs regulatory DNA | Conditional generation of plant promoters, plant enhancers and fly enhancers, validated by measured activity |

## 🤖 The NTv3 models

Each capability is released as its own checkpoint, all sharing the same architecture and the same pretrained
representations, at **700M parameters**.

| Model | Stage | What it is for | Repository |
| --- | --- | --- | --- |
| NTv3-pretrained | Pretrained | Embeddings, masked-token prediction, your own fine-tuning | [`InstaDeepAI/NTv3_pretrained`](https://huggingface.co/InstaDeepAI/NTv3_pretrained) |
| NTv3-tracks | Tracks post-training | Functional-track prediction from 16 kb to 1 Mb inputs, and the backbone for downstream fine-tuning | [`InstaDeepAI/NTv3_tracks`](https://huggingface.co/InstaDeepAI/NTv3_tracks) |
| NTv3-tracks-1mb | Tracks post-training | The 1 Mb specialist behind the paper's results; 1 Mb inputs only | [`InstaDeepAI/NTv3_tracks_1mb`](https://huggingface.co/InstaDeepAI/NTv3_tracks_1mb) |
| NTv3-annotate | Annotation fine-tuning | *Ab initio* gene prediction, one model per clade | [`_mammals`](https://huggingface.co/InstaDeepAI/NTv3_annot_mammals) / [`_plants`](https://huggingface.co/InstaDeepAI/NTv3_annot_plants) / [`_insects`](https://huggingface.co/InstaDeepAI/NTv3_annot_insects) |
| NTv3-generative-plant-promoter | Generative post-training | Example generative model used in the paper: designs plant core promoters at a requested strength, and is what notebook 06 runs | [`InstaDeepAI/NTv3_generative_plant_promoter`](https://huggingface.co/InstaDeepAI/NTv3_generative_plant_promoter) |

Of the paper's design models, only the plant promoter one is released, as an example; the plant enhancer and fly
enhancer models are not. It is the model notebook 06 runs, and a worked example of the recipe notebook 05 teaches.
Read its model card before quoting anything from it: it is not species-aware.

**700M** is the figure quoted throughout, counted from the published weights rather than the internal registry, which
still calls the pretrained checkpoint 650M. Exactly: 703,349,146 for pretrained, 711,256,680 for tracks,
703,446,690 for each annotate clade, and 721,379,955 for the plant promoter example, which carries three condition
tables and conditioning layers in its trunk.

The [NTv3 collection](https://huggingface.co/collections/InstaDeepAI/ntv3) is the canonical index
of what is downloadable today.

## 🔐 Access

The models are gated: request access once on any of their model pages, and one approval, reviewed manually, covers
all of them. Then log in with `hf auth login`, or `notebook_login()` in a notebook. The datasets and this Space are
open.

## 📦 Install

```bash
pip install torch transformers accelerate safetensors huggingface_hub numpy
```

Works on **transformers 4.57.1 and 5.17.0**, which now agree exactly. Two faults that broke 5.x are fixed: the rotary
frequencies are rebuilt per call rather than restored from a buffer that 5.x zeroed, and the annotation checkpoints
store `residual_scale` as the scalar the model declares. Re-measured on chr19:6,700,000-6,716,384 across both
versions, the predictions are identical to the bit. Older 5.x releases may still carry the zeroed-buffer behaviour,
which fails silently, so pin `transformers>=4.57.1` and check against the fixtures in each repo if you stray.

Models use custom code, so pass `trust_remote_code=True`. Input sequence length must be a multiple of 128.

## 🚀 Quickstart with NTv3-pretrained

Both checkpoints are registered under `AutoModel`, and the forward takes the token tensor positionally rather than
the tokenizer dict, so `model(**batch)` will not work. The forward returns a plain `dict`, not a `ModelOutput`, so
read results with `out["logits"]` rather than `out.logits`.

```python
import torch
from transformers import AutoModel, AutoTokenizer

model_name = "InstaDeepAI/NTv3_pretrained"

tok = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True).eval()

# Batch only equal-length sequences: padding is not masked out
batch = tok(["ACGT" * 32, "TTGA" * 32], add_special_tokens=False, return_tensors="pt")

with torch.no_grad():
    out = model(batch["input_ids"])

print(out["logits"].shape)      # (B, L, V = 11)
print(out["embedding"].shape)   # (B, L, 1024)
```

Input length must be a multiple of 128, and **padded positions are not masked out of attention**. The same sequence
run alone and run padded to a longer length gives different values at its real positions, so batch only sequences of
equal length and run ragged ones separately.

## 📊 NTv3-tracks: functional-track prediction

The `ntv3-tracks` pipeline takes a genomic region and the tracks you want, fetches the sequence, conditions on the
species, predicts and plots.

```python
from transformers import pipeline

ntv3_tracks = pipeline("ntv3-tracks", model="InstaDeepAI/NTv3_tracks", trust_remote_code=True)

out = ntv3_tracks(
    {"chrom": "chr19", "start": 6_700_000, "end": 6_732_768, "species": "human"},
    plot=True,
    tracks_to_plot={"K562 RNA-seq": "ENCSR056HPM", "K562 CTCF": "ENCSR000AKO"},
)

print(out.tracks.shape)        # (12288, 2), the central 37.5 % of the window
print(out.coordinates)         # (6710240, 6722528), the span actually predicted
print(out.track("ENCSR000AKO").shape)  # (12288,): one track
```

Naming the tracks computes only those channels. The head is as wide as the largest species panel (hg38, 7,362
tracks), so a full forward at 1 Mb materialises about 11 GB, while a named subset is bit-identical and a few MB.
Species, assemblies and track order all come from the model's own `config.track_metadata`.

For the plain forward, which is where to start if you want to fine-tune rather than predict, `NTv3_tracks` takes a
species token alongside the sequence, and its nucleotide vocabulary is 16 tokens rather than the 11 of the pretrained
model, so it reads soft-masked (lowercase) sequence directly. Notebook 01 covers it.

The 2025 `ntv3-interpret` pipeline is not being carried over. Notebook 04 shows saliency and attention with a plain
`AutoModel` forward instead.

## 🏷️ NTv3-annotate: genome annotation

```python
import torch
from transformers import AutoModel, AutoTokenizer

CLADE = "mammals"  # or "plants", or "insects"
repo = f"InstaDeepAI/NTv3_annot_{CLADE}"

model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
batch = tok("ACGT" * 256, add_special_tokens=False, return_tensors="pt")

# Annotation is not conditioned on species, so all three clades take an empty list
with torch.no_grad():
    out = model(batch["input_ids"], conditions_tokens=[])

print(out["bed_tracks_logits"].shape)   # (1, 768, 1, 15); 768 = 1024 * 0.75
```

The output covers the central 75% of the window, the crop the model is served at. Logits alone are not an annotation:
notebook 03b decodes them into gene models (GFF3) with the HMM whose specification `config.json` carries under
`annotation`, and notebook 03c scores an annotation the way the benchmark does.

## 📓 Notebooks

The notebooks cover the workflows end to end, in `notebooks_tutorials/`:

| # | Notebook | File | Status |
| --- | --- | --- | --- |
| 00 | 🚀 Quickstart inference | [`00_quickstart_inference.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/00_quickstart_inference.ipynb) | ✅ |
| 01 | 📊 Tracks prediction | [`01_tracks_prediction.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/01_tracks_prediction.ipynb) | ✅ |
| 02 | 🎯 Fine-tune NTv3 on bigwig tracks | [`02_fine_tuning_bigwig_tracks.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/02_fine_tuning_bigwig_tracks.ipynb) | ✅ |
| 03a | 🏷️ Train a gene-annotation model | [`03a_genome_annotation_training.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/03a_genome_annotation_training.ipynb) | ✅ |
| 03b | 🏷️ From per-base predictions to gene models | [`03b_genome_annotation_decoding.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/03b_genome_annotation_decoding.ipynb) | ✅ |
| 03c | 🏷️ Evaluate a gene annotation against a reference | [`03c_genome_annotation_evaluation.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/03c_genome_annotation_evaluation.ipynb) | ✅ |
| 04 | 🔍 Model interpretation | [`04_model_interpretation.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/04_model_interpretation.ipynb) | ✅ |
| 05 | 🧪 Fine-tuning NTv3 into a diffusion model | [`05_diffusion_fine_tuning.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/05_diffusion_fine_tuning.ipynb) | ✅ |
| 06 | 🌱 Generating plant promoter sequences | [`06_plant_promoter_generation.ipynb`](https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/06_plant_promoter_generation.ipynb) | ✅ |

All run against the published checkpoints.

## 🗄️ Initial release (2025)

The initial-release repositories are **retained unchanged** for reproducibility and backward compatibility. Existing
model IDs, files and interfaces continue to work. If you are reproducing published results or already have a working
pipeline, stay on these.

**Pretrained:** `NTv3_8M_pre`, `NTv3_100M_pre`, `NTv3_650M_pre`

**Post-trained:** `NTv3_100M_post`, `NTv3_650M_post`

**Context-length variants:** `NTv3_8M_pre_8kb`, `NTv3_100M_pre_8kb`, `NTv3_650M_pre_8kb`, `NTv3_100M_post_131kb`,
`NTv3_650M_post_131kb`

**Architecture ablations:** `NTv3_5downsample_pre`, `NTv3_5downsample_pre_8kb`, `NTv3_5downsample_post`,
`NTv3_5downsample_post_131kb`

**Generative:** `NTv3_generative`

All under the `InstaDeepAI/` namespace. Note that some earlier materials wrote the post-trained checkpoints as
`NTv3_100M_pos` / `NTv3_650M_pos`; the correct identifiers end in `_post`.

These models were trained with a *joint* post-training objective over tracks and annotation labels, so numbers quoted
in their materials do not always match the current paper. Where they differ, the current paper is authoritative.

## 🔗 Links

- 📄 **Paper:** preprint `TBC`
- 💻 **JAX model code:** [github.com/instadeepai/nucleotide-transformer](https://github.com/instadeepai/nucleotide-transformer)
- 🎯 **Model collection:** [NTv3 collection](https://huggingface.co/collections/InstaDeepAI/ntv3)
- 📚 **Benchmark dataset:** [`InstaDeepAI/NTv3_benchmark_dataset_revised`](https://huggingface.co/datasets/InstaDeepAI/NTv3_benchmark_dataset_revised)
- 🏷️ **Annotation dataset:** [`InstaDeepAI/NTv3_annot_dataset`](https://huggingface.co/datasets/InstaDeepAI/NTv3_annot_dataset)
- 🏆 **Benchmark leaderboard:** [`InstaDeepAI/ntv3_benchmark_revised`](https://huggingface.co/spaces/InstaDeepAI/ntv3_benchmark_revised)
- 🚀 **Interactive demo:** [`InstaDeepAI/ntv3_tracks`](https://huggingface.co/spaces/InstaDeepAI/ntv3_tracks)

## 📝 Citation

`TBC`. To be added on publication.

```bibtex
@article{ntv3,
  title   = {…},
  author  = {…},
  journal = {…},
  year    = {…}
}
```

## 📜 License

**Model weights:** released under the InstaDeep Open Model Licence, which permits non-commercial use only and also
places conditions on the **outputs** of the models. See the `MODEL-LICENSE.md` shipped in each model repository for the
authoritative terms.

**Code and notebooks in this Space:** `TBC`