Add the access-request form (#6)

6966543 2 days ago

7.06 kB

---
library_name: transformers
tags:
  - genomics
  - dna
  - biology
  - ntv3
  - long-range
license: other
license_name: instadeep-open-model-licence
license_link: https://huggingface.co/InstaDeepAI/NTv3_pretrained/blob/main/MODEL-LICENSE.md
extra_gated_heading: Request access to the NTv3 models
extra_gated_description: >-
  One approval grants access to every gated NTv3 model. Requests are reviewed
  manually.
extra_gated_prompt: >-
  The NTv3 weights are released under the
  [InstaDeep Open Model Licence](https://huggingface.co/InstaDeepAI/NTv3_pretrained/blob/main/MODEL-LICENSE.md) for non-commercial use only, and its terms
  also cover the models' outputs. For commercial use, contact legal@instadeep.com.
extra_gated_fields:
  Full name: text
  Affiliation: text
  Institutional email: text
  Organisation type:
    type: select
    options:
      - University
      - Public research institute
      - Non-profit
      - Hospital
      - Company
      - Other
  Country: country
  Intended use: text
  I will use NTv3 and its outputs for non-commercial purposes only, under the InstaDeep Open Model Licence: checkbox
extra_gated_button_content: Request access
---
# NTv3 pretrained

The pretrained backbone of Nucleotide Transformer v3 (NTv3): a genomic foundation model
that reads DNA at single-base resolution across long context windows. Use it for
sequence embeddings, for masked-token prediction, and as the starting point for your own
fine-tuning.

This checkpoint is the backbone the other NTv3 capabilities are built on. The
task-adapted models — functional-track prediction and gene annotation — start from these
weights and resolve their modelling code from this repository.

## Architecture

A U-Net around a transformer bottleneck. A convolutional tower reads the sequence and
downsamples it by 128, a transformer tower carries long-range context at that reduced
resolution, and a deconvolutional tower returns predictions at base resolution, with
skip connections from the encoder at every scale.

The transformer uses grouped-query attention with per-head query, key and value
normalisation, rotary position embeddings on a frequency schedule chosen for genomic
distances, and a GeGLU feed-forward block.

| | |
| --- | --- |
| Parameters | 703,349,146 |
| Resolution | 1 bp |
| Input length | any multiple of 128 |
| Vocabulary | 11 tokens |
| Precision | float32 |

## Usage

```python
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer
repo = "InstaDeepAI/NTv3_pretrained"
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
# The input length must be a multiple of 128.
sequence = "ACGT" * 256
# add_special_tokens=False: the model reads a genomic window, so one base is one
# token and nothing is prepended.
batch = tokenizer([sequence], return_tensors="pt", add_special_tokens=False)
with torch.no_grad():
    out = model(**batch)
out.embedding.shape  # (1, 1024, 1024)  one vector per base
out.logits.shape     # (1, 1024, 11)    masked-token prediction
```

`AutoModel` works too, and outputs are reachable both as attributes
(`out.logits`) and as keys (`out["logits"]`).

`trust_remote_code=True` is required because the modelling code ships with the
repository rather than in `transformers`.

An `attention_mask` is accepted and **ignored**: the model derives its own mask
from the pad token at the resolution of its 128 bp blocks, so a per-token mask
has nowhere to apply. Pad with the pad token rather than masking.

### Outputs

| Key | Shape | What it is |
| --- | --- | --- |
| `embedding` | `(batch, length, 1024)` | Decoder output, one vector per base |
| `logits` | `(batch, length, 11)` | Masked-token prediction over the vocabulary |
| `after_transformer_embedding` | `(batch, length / 128, 2048)` | Trunk output, one vector per 128 bp block |
| `before_transformer_embedding` | `(batch, length / 128, 2048)` | Encoder output, before the transformer |

Use `skip_lm_head=True` to skip the language-modelling head when you only need
embeddings.

## Input requirements

**Two constraints, both of which fail quietly rather than loudly.**

**Length must be a multiple of 128.** That is the total downsampling factor of the
trunk. Pad or crop your window before calling the model.

**The alphabet is uppercase.** This checkpoint's vocabulary is `A C G T N` plus the
special tokens; lowercase is not in it. Most reference assemblies are distributed
soft-masked, with repeat regions in lowercase, and feeding one straight in turns nearly
every base into `<unk>`. The model will still return numbers. Upper-case your sequence
first, or use `InstaDeepAI/NTv3_tracks`, which reads a 16-symbol soft-masked alphabet.

Characters outside the vocabulary become `<unk>` and the sequence keeps its length.

## Intended use

Embeddings and masked-token prediction on genomic DNA, and fine-tuning on your own
task. The model was pretrained across a broad panel of species, so it is not specific to
one genome.

## Out of scope

Not a clinical or diagnostic tool, and not validated for variant interpretation in a
medical setting. Not designed for RNA, protein, or non-genomic sequence. Predictions on
sequence unlike anything in the pretraining distribution — synthetic constructs,
assemblies from very distant clades — should be treated as unvalidated rather than as
evidence.

## Limitations

The model sees sequence only. It has no access to cell type, developmental stage,
experimental condition, or any annotation, so nothing it outputs is conditioned on that
context.

Masked-token prediction reflects the statistics of the training genomes. High confidence
on a masked base means the surrounding sequence is predictive, not that the base is
functionally important.

Accuracy varies by species and by genomic region, and is not uniform across the genome.
Repetitive and low-complexity regions are harder, and this checkpoint cannot see the
soft-masking that would tell it where they are.

## Related models

| Model | For |
| --- | --- |
| [`NTv3_tracks`](https://huggingface.co/InstaDeepAI/NTv3_tracks) | Functional-track prediction, species-conditioned, from 16 kb to 1 Mb inputs |
| [`NTv3_tracks_1mb`](https://huggingface.co/InstaDeepAI/NTv3_tracks_1mb) | The same tracks from 1 Mb inputs only: the specialist behind the NTv3 paper's main results |
| NTv3_annot: [`_mammals`](https://huggingface.co/InstaDeepAI/NTv3_annot_mammals) / [`_plants`](https://huggingface.co/InstaDeepAI/NTv3_annot_plants) / [`_insects`](https://huggingface.co/InstaDeepAI/NTv3_annot_insects) | Ab initio gene annotation, one model per clade |
## Licence
Released under the [InstaDeep Open Model Licence](https://huggingface.co/InstaDeepAI/NTv3_pretrained/blob/main/MODEL-LICENSE.md). The licence restricts **output** as
well as weights, so predictions made with this model carry the same terms. Read it
before building on the model commercially.
## Citation
Citation to be added on publication.

	---
	library_name: transformers
	tags:
	- genomics
	- dna
	- biology
	- ntv3
	- long-range
	license: other
	license_name: instadeep-open-model-licence
	license_link: https://huggingface.co/InstaDeepAI/NTv3_pretrained/blob/main/MODEL-LICENSE.md
	extra_gated_heading: Request access to the NTv3 models
	extra_gated_description: >-
	One approval grants access to every gated NTv3 model. Requests are reviewed
	manually.
	extra_gated_prompt: >-
	The NTv3 weights are released under the
	[InstaDeep Open Model Licence](https://huggingface.co/InstaDeepAI/NTv3_pretrained/blob/main/MODEL-LICENSE.md) for non-commercial use only, and its terms
	also cover the models' outputs. For commercial use, contact legal@instadeep.com.
	extra_gated_fields:
	Full name: text
	Affiliation: text
	Institutional email: text
	Organisation type:
	type: select
	options:
	- University
	- Public research institute
	- Non-profit
	- Hospital
	- Company
	- Other
	Country: country
	Intended use: text
	I will use NTv3 and its outputs for non-commercial purposes only, under the InstaDeep Open Model Licence: checkbox
	extra_gated_button_content: Request access
	---

	# NTv3 pretrained

	The pretrained backbone of Nucleotide Transformer v3 (NTv3): a genomic foundation model
	that reads DNA at single-base resolution across long context windows. Use it for
	sequence embeddings, for masked-token prediction, and as the starting point for your own
	fine-tuning.

	This checkpoint is the backbone the other NTv3 capabilities are built on. The
	task-adapted models — functional-track prediction and gene annotation — start from these
	weights and resolve their modelling code from this repository.

	## Architecture

	A U-Net around a transformer bottleneck. A convolutional tower reads the sequence and
	downsamples it by 128, a transformer tower carries long-range context at that reduced
	resolution, and a deconvolutional tower returns predictions at base resolution, with
	skip connections from the encoder at every scale.

	The transformer uses grouped-query attention with per-head query, key and value
	normalisation, rotary position embeddings on a frequency schedule chosen for genomic
	distances, and a GeGLU feed-forward block.

	\| \| \|
	\| --- \| --- \|
	\| Parameters \| 703,349,146 \|
	\| Resolution \| 1 bp \|
	\| Input length \| any multiple of 128 \|
	\| Vocabulary \| 11 tokens \|
	\| Precision \| float32 \|

	## Usage

	```python
	import torch
	from transformers import AutoModelForMaskedLM, AutoTokenizer

	repo = "InstaDeepAI/NTv3_pretrained"
	model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True).eval()
	tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)

	# The input length must be a multiple of 128.
	sequence = "ACGT" * 256

	# add_special_tokens=False: the model reads a genomic window, so one base is one
	# token and nothing is prepended.
	batch = tokenizer([sequence], return_tensors="pt", add_special_tokens=False)

	with torch.no_grad():
	out = model(**batch)

	out.embedding.shape # (1, 1024, 1024) one vector per base
	out.logits.shape # (1, 1024, 11) masked-token prediction
	```

	`AutoModel` works too, and outputs are reachable both as attributes
	(`out.logits`) and as keys (`out["logits"]`).

	`trust_remote_code=True` is required because the modelling code ships with the
	repository rather than in `transformers`.

	An `attention_mask` is accepted and ignored: the model derives its own mask
	from the pad token at the resolution of its 128 bp blocks, so a per-token mask
	has nowhere to apply. Pad with the pad token rather than masking.

	### Outputs

	\| Key \| Shape \| What it is \|
	\| --- \| --- \| --- \|
	\| `embedding` \| `(batch, length, 1024)` \| Decoder output, one vector per base \|
	\| `logits` \| `(batch, length, 11)` \| Masked-token prediction over the vocabulary \|
	\| `after_transformer_embedding` \| `(batch, length / 128, 2048)` \| Trunk output, one vector per 128 bp block \|
	\| `before_transformer_embedding` \| `(batch, length / 128, 2048)` \| Encoder output, before the transformer \|

	Use `skip_lm_head=True` to skip the language-modelling head when you only need
	embeddings.

	## Input requirements

	Two constraints, both of which fail quietly rather than loudly.

	Length must be a multiple of 128. That is the total downsampling factor of the
	trunk. Pad or crop your window before calling the model.

	The alphabet is uppercase. This checkpoint's vocabulary is `A C G T N` plus the
	special tokens; lowercase is not in it. Most reference assemblies are distributed
	soft-masked, with repeat regions in lowercase, and feeding one straight in turns nearly
	every base into `<unk>`. The model will still return numbers. Upper-case your sequence
	first, or use `InstaDeepAI/NTv3_tracks`, which reads a 16-symbol soft-masked alphabet.

	Characters outside the vocabulary become `<unk>` and the sequence keeps its length.

	## Intended use

	Embeddings and masked-token prediction on genomic DNA, and fine-tuning on your own
	task. The model was pretrained across a broad panel of species, so it is not specific to
	one genome.

	## Out of scope

	Not a clinical or diagnostic tool, and not validated for variant interpretation in a
	medical setting. Not designed for RNA, protein, or non-genomic sequence. Predictions on
	sequence unlike anything in the pretraining distribution — synthetic constructs,
	assemblies from very distant clades — should be treated as unvalidated rather than as
	evidence.

	## Limitations

	The model sees sequence only. It has no access to cell type, developmental stage,
	experimental condition, or any annotation, so nothing it outputs is conditioned on that
	context.

	Masked-token prediction reflects the statistics of the training genomes. High confidence
	on a masked base means the surrounding sequence is predictive, not that the base is
	functionally important.

	Accuracy varies by species and by genomic region, and is not uniform across the genome.
	Repetitive and low-complexity regions are harder, and this checkpoint cannot see the
	soft-masking that would tell it where they are.

	## Related models

	\| Model \| For \|
	\| --- \| --- \|
	\| [`NTv3_tracks`](https://huggingface.co/InstaDeepAI/NTv3_tracks) \| Functional-track prediction, species-conditioned, from 16 kb to 1 Mb inputs \|
	\| [`NTv3_tracks_1mb`](https://huggingface.co/InstaDeepAI/NTv3_tracks_1mb) \| The same tracks from 1 Mb inputs only: the specialist behind the NTv3 paper's main results \|
	\| NTv3_annot: [`_mammals`](https://huggingface.co/InstaDeepAI/NTv3_annot_mammals) / [`_plants`](https://huggingface.co/InstaDeepAI/NTv3_annot_plants) / [`_insects`](https://huggingface.co/InstaDeepAI/NTv3_annot_insects) \| Ab initio gene annotation, one model per clade \|

	## Licence

	Released under the [InstaDeep Open Model Licence](https://huggingface.co/InstaDeepAI/NTv3_pretrained/blob/main/MODEL-LICENSE.md). The licence restricts output as
	well as weights, so predictions made with this model carry the same terms. Read it
	before building on the model commercially.

	## Citation

	Citation to be added on publication.