Blame for tabs/initial_release.html

<div class="notice">
  🗄️ <strong>This page describes the initial NTv3 release, from 2025.</strong> Those repositories are retained unchanged for
  reproducibility and backward compatibility. Existing model IDs, files and interfaces continue to work exactly as they
  did, and nothing on this page is being withdrawn. If you are starting something new, prefer the current models on the
  <strong>Home</strong> tab; if you are reproducing published results or already have a working pipeline, stay here.
</div>

<div class="summary">
  <h2>📖 About the initial release (2025)</h2>
  <p>
    The initial release described NTv3 as a single U-Net-style backbone pretrained on the OpenGenome2 corpus and then
    post-trained with a <em>joint</em> objective over functional tracks and annotation labels. It shipped three model sizes
    (8M, 100M, 650M), each in pretrained and post-trained form, plus context-length and architecture ablation variants.
  </p>
  <p>
    The current release changes that structure: tracks post-training is its own stage, and annotation becomes a separate
    per-clade fine-tune rather than a head trained jointly. Numbers quoted in these materials therefore do not always
    match the current paper. Where the two differ, the current paper is authoritative.
  </p>
</div>

<div class="paper-summary">
  <img src="assets/paper_summary.png" alt="NTv3 initial-release overview figure" />
</div>

<div class="grid">
  <div class="card" style="grid-column: span 12;">
    <h2>🤖 Initial-release checkpoints (2025)</h2>
    <p>
      All of the following are public on the Hub and unchanged. Sizes are the parameter counts as published.
    </p>
    <table>
      <thead>
        <tr>
          <th>Model</th>
          <th>Size</th>
          <th>Pretrained</th>
          <th>Post-trained</th>
          <th>Usage</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td><strong>NTv3-8M</strong></td>
          <td>8M params</td>
          <td><span class="checkmark">✅</span></td>
          <td>❌</td>
          <td>Embeddings, light inference</td>
        </tr>
        <tr>
          <td><strong>NTv3-100M</strong></td>
          <td>100M params</td>
          <td><span class="checkmark">✅</span></td>
          <td><span class="checkmark">✅</span></td>
          <td>Embeddings, tracks, annotation</td>
        </tr>
        <tr>
          <td><strong>NTv3-650M</strong></td>
          <td>650M params</td>
          <td><span class="checkmark">✅</span></td>
          <td><span class="checkmark">✅</span></td>
          <td>Embeddings, tracks, annotation, best accuracy</td>
        </tr>
      </tbody>
    </table>

    <h2 style="margin-top: 22px;">📦 Pretrained</h2>
    <ul>
      <li><a href="https://huggingface.co/InstaDeepAI/NTv3_8M_pre" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_8M_pre</code></a></li>
      <li><a href="https://huggingface.co/InstaDeepAI/NTv3_100M_pre" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_100M_pre</code></a></li>
      <li><a href="https://huggingface.co/InstaDeepAI/NTv3_650M_pre" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_650M_pre</code></a></li>
    </ul>

    <h2 style="margin-top: 18px;">🎯 Post-trained</h2>
    <ul>
      <li><a href="https://huggingface.co/InstaDeepAI/NTv3_100M_post" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_100M_post</code></a></li>
      <li><a href="https://huggingface.co/InstaDeepAI/NTv3_650M_post" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_650M_post</code></a></li>
    </ul>

    <h2 style="margin-top: 18px;">📏 Context-length variants</h2>
    <ul>
      <li><a href="https://huggingface.co/InstaDeepAI/NTv3_8M_pre_8kb" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_8M_pre_8kb</code></a></li>
      <li><a href="https://huggingface.co/InstaDeepAI/NTv3_100M_pre_8kb" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_100M_pre_8kb</code></a></li>
      <li><a href="https://huggingface.co/InstaDeepAI/NTv3_650M_pre_8kb" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_650M_pre_8kb</code></a></li>
      <li><a href="https://huggingface.co/InstaDeepAI/NTv3_100M_post_131kb" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_100M_post_131kb</code></a></li>
      <li><a href="https://huggingface.co/InstaDeepAI/NTv3_650M_post_131kb" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_650M_post_131kb</code></a></li>
    </ul>

    <h2 style="margin-top: 18px;">🔬 Architecture ablations (five-downsample variants)</h2>
    <ul>
      <li><a href="https://huggingface.co/InstaDeepAI/NTv3_5downsample_pre" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_5downsample_pre</code></a></li>
      <li><a href="https://huggingface.co/InstaDeepAI/NTv3_5downsample_pre_8kb" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_5downsample_pre_8kb</code></a></li>
      <li><a href="https://huggingface.co/InstaDeepAI/NTv3_5downsample_post" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_5downsample_post</code></a></li>
      <li><a href="https://huggingface.co/InstaDeepAI/NTv3_5downsample_post_131kb" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_5downsample_post_131kb</code></a></li>
    </ul>

    <h2 style="margin-top: 18px;">🧪 Generative</h2>
    <ul>
      <li><a href="https://huggingface.co/InstaDeepAI/NTv3_generative" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_generative</code></a></li>
    </ul>

    <div class="notice">
      <strong>Naming note.</strong> Some earlier materials referred to the post-trained checkpoints as
      <code>NTv3_100M_pos</code> and <code>NTv3_650M_pos</code>. The correct identifiers end in <code>_post</code>. If a
      script of yours 404s, that is the likely cause.
    </div>
  </div>

  <div class="card-stack">
    <div class="card">
      <h2>📓 Notebooks for the initial release (2025)</h2>
      <p>These notebooks were written and run against the initial-release checkpoints.</p>
      <ul>
        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/initial-release/notebooks_tutorials/00_quickstart_inference.ipynb" target="_blank" rel="noopener noreferrer">🚀 00 · Quickstart inference</a></li>
        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/initial-release/notebooks_tutorials/01_tracks_prediction.ipynb" target="_blank" rel="noopener noreferrer">📊 01 · Tracks prediction</a></li>
        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/initial-release/notebooks_tutorials/02_fine_tuning_pretrained_model_biwig.ipynb" target="_blank" rel="noopener noreferrer">🎯 02 · Fine-tune a pretrained model on bigwig tracks</a></li>
        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/initial-release/notebooks_tutorials/03_fine_tuning_posttrained_model_biwig.ipynb" target="_blank" rel="noopener noreferrer">🎯 03 · Fine-tune a post-trained model on bigwig tracks</a></li>
        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/initial-release/notebooks_tutorials/04_fine_tuning_pretrained_model_annotation.ipynb" target="_blank" rel="noopener noreferrer">🏷️ 04 · Fine-tune a pretrained model on annotations</a></li>
        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/initial-release/notebooks_tutorials/05_model_interpretation.ipynb" target="_blank" rel="noopener noreferrer">🔍 05 · Model interpretation</a></li>
        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/initial-release/notebooks_tutorials/06_NTv3_generative_training.ipynb" target="_blank" rel="noopener noreferrer">🧪 06 · Fine-tuning NTv3 into a diffusion model</a></li>
        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/initial-release/notebooks_tutorials/07_enhancer_generation.ipynb" target="_blank" rel="noopener noreferrer">🪰 07 · Generating enhancer sequences</a></li>
      </ul>
      <p style="margin-top: 12px;">Pipeline notebooks:</p>
      <ul>
        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/initial-release/notebooks_pipelines/01_functional_track_prediction.ipynb" target="_blank" rel="noopener noreferrer">🎯 01 · Generate bigwig predictions for given tracks</a></li>
        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/initial-release/notebooks_pipelines/02_model_interpretation.ipynb" target="_blank" rel="noopener noreferrer">🔍 02 · Interpret a given genomic region</a></li>
      </ul>
    </div>
    <div class="card">
      <h2>🔗 Initial-release links (2025)</h2>
      <ul>
        <li><a href="https://www.biorxiv.org/content/10.64898/2025.12.22.695963v1" target="_blank" rel="noopener noreferrer">📄 Initial preprint</a></li>
        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/tree/initial-release" target="_blank" rel="noopener noreferrer">🏠 Original NTv3 Space (files at tag <code>initial-release</code>)</a></li>
        <li><a href="https://github.com/instadeepai/nucleotide-transformer/tree/ntv3-initial-release" target="_blank" rel="noopener noreferrer">💻 JAX model code (GitHub, tag <code>ntv3-initial-release</code>)</a></li>
        <li><a href="https://huggingface.co/collections/InstaDeepAI/nucleotide-transformer-v3" target="_blank" rel="noopener noreferrer">🎯 HF model collection (2025)</a></li>
        <li><a href="https://huggingface.co/datasets/InstaDeepAI/NTv3_benchmark_dataset" target="_blank" rel="noopener noreferrer">📚 NTv3 benchmark dataset</a></li>
        <li><a href="https://huggingface.co/datasets/InstaDeepAI/NTv3_enhancer_generation" target="_blank" rel="noopener noreferrer">📚 NTv3 enhancer generation dataset</a></li>
        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3_benchmark" target="_blank" rel="noopener noreferrer">🏆 NTv3 benchmark leaderboard (2025)</a></li>
        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3_tracks" target="_blank" rel="noopener noreferrer">🚀 Interactive tracks demo</a></li>
      </ul>
    </div>
  </div>

  <div class="card">
    <h2>🤖 Load a pre-trained model</h2>
    <p>Here is an example of how to load and use a pre-trained NTv3 model.</p>
    <div class="code"><pre><code class="language-python">from transformers import AutoTokenizer, AutoModelForMaskedLM

model_name = "InstaDeepAI/NTv3_650M_pre"

model = AutoModelForMaskedLM.from_pretrained(model_name, trust_remote_code=True)
tok = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

batch = tok(["ATCGNATCG", "ACGT"], add_special_tokens=False, padding=True,
            pad_to_multiple_of=128, return_tensors="pt")

out = model(**batch)
print(out.logits.shape)       # (B, L, V = 11)
</code></pre></div>
    <p>
      To pin a specific version rather than tracking the branch, pass a tag:
      <code>from_pretrained(model_name, revision="v1.0", trust_remote_code=True)</code>
    </p>
  </div>

  <div class="card">
    <h2>🔍 Model interpretation</h2>
    <p>
      Here is an example of how to use the interpretation pipeline on the NTv3 post-trained model for multi-scale
      analysis of DNA sequences:
    </p>
    <div class="code"><pre><code class="language-python">from transformers import pipeline
import torch
import matplotlib.pyplot as plt

model_name = "InstaDeepAI/NTv3_650M_post"

# Build interpretation pipeline
ntv3_interpret = pipeline(
    "ntv3-interpret",
    model=model_name,
    trust_remote_code=True,
    device=0 if torch.cuda.is_available() else -1,
)

# Run interpretation on a given genomic region with tracks, annotations, attention, and saliency
result = ntv3_interpret(
    {"chrom": "chr11", "start": 5_253_561, "end": 5_286_329, "species": "human"},
    output_attention=True,
    output_saliency=True,
    saliency_track_id="ENCSR000EFT",  # K562 GATA1 ChIP-seq
    plot=True,  # plot predictions on tracks and annotations
    tracks_to_plot={"K562 RNA-seq": "ENCSR056HPM", "K562 GATA1": "ENCSR000EFT"},
    elements_to_plot=["exon", "promoter_Tissue_specific"],
)

# Access attention map results
result.plot_attention()  # attention map (last layer)
plt.show()

# Access saliency scores results
result.plot_saliency(window_size=128)
plt.show()</code></pre></div>
    <img src="assets/saliency_example.png" alt="Saliency example" style="max-width: 100%; margin-top: 20px;" />
  </div>

  <div class="card" style="grid-column: span 12;">
    <h2>💻 Use a post-trained model</h2>
    <p>
      Here is a quick example of how to use the post-trained NTv3 650M model to predict tracks for a human genomic
      window.
    </p>
    <div class="code"><pre><code class="language-python">from transformers import pipeline
import torch

model_name = "InstaDeepAI/NTv3_650M_post"

ntv3_tracks = pipeline(
    "ntv3-tracks",
    model=model_name,
    trust_remote_code=True,
    device=0 if torch.cuda.is_available() else -1,
)

# Run track prediction
out = ntv3_tracks(
  {
    "chrom": "chr19",
    "start": 6_700_000,
    "end": 6_831_072,
    "species": "human"
  }
)

# Print output shapes
# 7k human tracks over 37.5 % center region of the input sequence
print("bigwig_tracks_logits:", tuple(out.bigwig_tracks_logits.shape))
# Location of 21 genomic elements over 37.5 % center region of the input sequence
print("bed_tracks_logits:", tuple(out.bed_tracks_logits.shape))
# Language model logits for whole sequence over vocabulary
print("language model logits:", tuple(out.mlm_logits.shape))</code></pre></div>
    <p>Predictions can be plotted for a subset of tracks and genomic elements:</p>
    <div class="code"><pre><code class="language-python">tracks_to_plot = {
    "K562 RNA-seq": "ENCSR056HPM",
    "K562 DNAse": "ENCSR921NMD",
    "K562 H3k4me3": "ENCSR000DWD",
    "K562 CTCF": "ENCSR000AKO",
    "HepG2 RNA-seq": "ENCSR561FEE_P",
    "HepG2 DNAse": "ENCSR000EJV",
    "HepG2 H3k4me3": "ENCSR000AMP",
    "HepG2 CTCF": "ENCSR000BIE",
}
elements_to_plot = ["protein_coding_gene", "exon", "intron", "splice_donor", "splice_acceptor"]

out = ntv3_tracks(
    {"chrom": "chr19", "start": 6_700_000, "end": 6_831_072, "species": "human"},
    plot=True,
    tracks_to_plot=tracks_to_plot,
    elements_to_plot=elements_to_plot,
)</code></pre></div>
    <img src="assets/output_tracks.png" alt="Predicted tracks over a human genomic window" style="max-width: 100%; margin-top: 20px;" />
  </div>
</div>