Merge remote-tracking branch 'origin/main' into fine-tuning-notebook

.gitattributes

@@ -33,3 +33,14 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/paper_summary.jpg filter=lfs diff=lfs merge=lfs -text
+assets/paper_summary.png filter=lfs diff=lfs merge=lfs -text
+assets/output_tracks.png filter=lfs diff=lfs merge=lfs -text
+notebooks_pipelines/bigwig_outputs/K562_DNAse.bw filter=lfs diff=lfs merge=lfs -text
+notebooks_pipelines/bigwig_outputs/K562_H3k4me3.bw filter=lfs diff=lfs merge=lfs -text
+notebooks_pipelines/bigwig_outputs/K562_RNA_seq.bw filter=lfs diff=lfs merge=lfs -text
+notebooks_pipelines/bigwig_outputs/HepG2_CTCF.bw filter=lfs diff=lfs merge=lfs -text
+notebooks_pipelines/bigwig_outputs/HepG2_DNAse.bw filter=lfs diff=lfs merge=lfs -text
+notebooks_pipelines/bigwig_outputs/HepG2_H3k4me3.bw filter=lfs diff=lfs merge=lfs -text
+notebooks_pipelines/bigwig_outputs/HepG2_RNA_seq.bw filter=lfs diff=lfs merge=lfs -text
+notebooks_pipelines/bigwig_outputs/K562_CTCF.bw filter=lfs diff=lfs merge=lfs -text

README.md

@@ -7,63 +7,91 @@ sdk: static
 pinned: false
 ---
 
-# NTv3 — Foundation Models for Long-Range Genomics
+# 🧬 NTv3 — Foundation Models for Long-Range Genomics
 
 This Space is the companion hub for NTv3 checkpoints on the Hugging Face Hub. It provides PyTorch notebooks and minimal examples for inference, sequence-to-function prediction (functional tracks), genome annotation, fine-tuning, model interpretation and sequence generation.
 
-## Notebooks
+## 📖 About NTv3
+
+NTv3 is a multi-species genomic foundation model family that unifies representation learning, functional-track prediction, genome annotation, and controllable sequence generation within a single U-Net-style backbone. It models up to 1 Mb of DNA at single-base resolution, using a conv–Transformer–deconv architecture that efficiently captures both local motifs and long-range regulatory dependencies. NTv3 is first pretrained on ~9T base pairs from the OpenGenome2 corpus spanning >128k species using masked language modeling, and then post-trained with a joint objective on ~16k functional tracks and annotation labels across 24 animal and plant species, enabling state-of-the-art cross-species functional prediction and base-resolution genome annotation.
+
+Beyond prediction, NTv3 can be fine-tuned into a controllable generative model via masked-diffusion language modeling, allowing targeted design of regulatory sequences (for example, enhancers with specified activity and promoter selectivity) that have been validated experimentally.
+
+## 📓 Notebooks
 
 Notebooks live in `./notebooks/`:
 
-- `00_quickstart_inference.ipynb` — load a checkpoint + run inference
-- `01_tracks_prediction.ipynb` — sequence → functional tracks (+ plotting)
-- `02_genome_annotation_segmentation.ipynb` — sequence → annotation
-- `03_finetune_head.ipynb` — fine-tune on a bigwig track
-- `04_model_interpretation.ipynb` — interpretation of post-trained model
-- `05_sequence_generation.ipynb` — fine-tune NTv3 to generate enhancer sequences
+- 🚀 `00_quickstart_inference.ipynb` — load a checkpoint + run inference
+- 📊 `01_tracks_prediction.ipynb` — sequence → functional tracks (+ plotting)
+- 🏷️ `02_genome_annotation_segmentation.ipynb` — sequence → annotation
+- 🎯 `03_finetune_head.ipynb` — fine-tune on bigwig tracks
+- 🔍 `04_model_interpretation.ipynb` — interpretation of post-trained model
+- 🧪 `05_sequence_generation.ipynb` — fine-tune NTv3 to generate enhancer sequences
 
-## Install
+## 📦 Install
 
 ```bash
 pip install torch transformers accelerate safetensors huggingface_hub numpy
 ```
 
-## Load a model (To DO)
+## 🤖 Load a pre-trained model
 
 ```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
 
+repo = "InstaDeepAI/NTv3_650M_pre"
+tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
+model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True)
 
+batch = tok(["ATCGNATCG", "ACGT"], add_special_tokens=False, padding=True, pad_to_multiple_of=128, return_tensors="pt")
+out = model(**batch, output_hidden_states=True, output_attentions=True)
+
+print(out.logits.shape)       # (B, L, V = 11)
+print(len(out.hidden_states)) # convs + transformers + deconvs
+print(len(out.attentions))    # equals transformer layers = 12
 ```
 
-## Pipelines (To DO)
+## 💻 Pipelines
+
+Here is a quick example of how to use the post-trained NTv3 650M model on a human genomic window.
 
 ```python
-from transformers import pipeline
-import torch
-
-pipe = pipeline(
-    task="ntv3-tracks",
-    model="InstaDeepAI/ntv3_106M_7downsample_post_trained_1mb",
-    trust_remote_code=True,
-    device="cuda",
-    torch_dtype=torch.bfloat16,
+from transformers import AutoConfig
+
+model_name = "InstaDeepAI/NTv3_100M"
+
+# Load track prediction pipeline
+cfg = AutoConfig.from_pretrained(model_name, trust_remote_code=True, force_download=True)
+pipe = cfg.load_tracks_pipeline(model_name, device="auto")  # or "cpu"/"cuda"/"mps"
+
+# Run track prediction
+out = pipe(
+    {
+        "chrom": "chr19",
+        "start": 6_700_000,
+        "end": 6_831_072,
+        "species": "human"
+    }
 )
 
-out = pipe("ACGT...")
+print(out.bigwig_tracks_logits.shape)   # functional track predictions
+print(out.bed_tracks_logits.shape)      # genome annotation predictions
+print(out.mlm_logits.shape)             # MLM logits: (B, L, V = 11)
 ```
 
-## Checkpoints
+## 🤖 Checkpoints
 
-**Pre-trained:** `InstaDeepAI/ntv3_8M_7downsample_pretrained_le_1mb`, `InstaDeepAI/ntv3_106M_7downsample_pretrained_le_1mb`, `InstaDeepAI/ntv3_650M_7downsample_pretrained_le_1mb`
+**📦 Pre-trained:** `InstaDeepAI/NTv3_8M_pre`, `InstaDeepAI/NTv3_100M_pre`, `InstaDeepAI/NTv3_650M_pre`
 
-**Post-trained:** `InstaDeepAI/ntv3_650M_7downsample_post_trained_1mb`, `InstaDeepAI/ntv3_106M_7downsample_post_trained_1mb`
+**🎯 Post-trained:** `InstaDeepAI/NTv3_100M`, `InstaDeepAI/NTv3_650M`
 
-## Links
+## 🔗 Links
 
-- **Paper:** (add link)
-- **JAX research code (GitHub):** [https://github.com/instadeepai/nucleotide-transformer](https://github.com/instadeepai/nucleotide-transformer)
+- **📄 Paper:** (add link)
+- **💻 JAX research code (GitHub):** [https://github.com/instadeepai/nucleotide-transformer](https://github.com/instadeepai/nucleotide-transformer)
+- **🏆 NTv3 benchmark leaderboard: (add link)**
 
-## Citation
+## 📝 Citation
 
 ```bibtex
 @article{ntv3,
@@ -74,7 +102,7 @@ out = pipe("ACGT...")
 }
 ```
 
-## License
+## 📜 License
 
 **Code & notebooks in this Space:** (choose and add, e.g., Apache-2.0)
 

index.html

@@ -3,8 +3,9 @@
 <head>
   <meta charset="utf-8" />
   <meta name="viewport" content="width=device-width,initial-scale=1" />
-  <title>NTv3 — Foundation Models for Long-Range Genomics</title>
+  <title>NTv3 — Next-Gen Foundation Models for Genomics</title>
   <meta name="description" content="NTv3 companion hub: PyTorch notebooks for inference, fine-tuning, interpretation, and sequence generation on NTv3 models hosted on Hugging Face." />
+  <link href="https://cdnjs.cloudflare.com/ajax/libs/prism/1.29.0/themes/prism-tomorrow.min.css" rel="stylesheet" />
   <style>
     :root {
       --bg: #0b1020;
@@ -53,9 +54,49 @@
       border-radius: var(--radius);
       box-shadow: 0 6px 18px rgba(0,0,0,0.22);
     }
+    .card-stack {
+      grid-column: span 6;
+      display: flex;
+      flex-direction: column;
+      gap: 14px;
+    }
+    .card-stack .card {
+      grid-column: span 1;
+      margin: 0;
+    }
     .card h2 { margin: 0 0 10px 0; font-size: 16px; letter-spacing: 0.01em; }
     .card ul { margin: 0; padding-left: 18px; color: var(--muted); }
     .card li { margin: 8px 0; }
+    .card table {
+      width: 100%;
+      margin-top: 12px;
+      border-collapse: collapse;
+      font-size: 13px;
+    }
+    .card table th {
+      text-align: left;
+      padding: 10px 12px;
+      border-bottom: 2px solid var(--border);
+      color: var(--text);
+      font-weight: 600;
+      font-size: 12px;
+      text-transform: uppercase;
+      letter-spacing: 0.05em;
+    }
+    .card table td {
+      padding: 10px 12px;
+      border-bottom: 1px solid var(--border);
+      color: var(--muted);
+    }
+    .card table tr:last-child td {
+      border-bottom: none;
+    }
+    .card table tr:hover {
+      background: rgba(255, 255, 255, 0.02);
+    }
+    .card table td .checkmark {
+      color: #4ade80 !important;
+    }
     a { color: var(--link); text-decoration: none; }
     a:hover { text-decoration: underline; }
     .pillrow { display: flex; gap: 8px; flex-wrap: wrap; margin-top: 8px; }
@@ -85,10 +126,129 @@
       font-size: inherit;
       color: inherit;
     }
+    /* Prism.js theme overrides to match dark theme */
+    .code pre[class*="language-"] {
+      background: transparent;
+      margin: 0;
+      padding: 0;
+    }
+    .code code[class*="language-"] {
+      background: transparent;
+    }
+    .summary {
+      margin-top: 18px;
+      padding: 24px;
+      border: 1px solid var(--border);
+      background: var(--card);
+      border-radius: var(--radius);
+      box-shadow: var(--shadow);
+    }
+    .summary h2 {
+      margin: 0 0 16px 0;
+      font-size: 18px;
+      letter-spacing: 0.01em;
+    }
+    .summary p {
+      margin: 0 0 14px 0;
+      color: var(--muted);
+      line-height: 1.7;
+    }
+    .summary p:last-child {
+      margin-bottom: 0;
+    }
+    .why-ntv3 {
+      margin-top: 18px;
+      padding: 24px;
+      border: 1px solid var(--border);
+      background: var(--card);
+      border-radius: var(--radius);
+      box-shadow: var(--shadow);
+    }
+    .why-ntv3 h2 {
+      margin: 0 0 16px 0;
+      font-size: 18px;
+      letter-spacing: 0.01em;
+    }
+    .why-ntv3 ul {
+      margin: 0;
+      padding-left: 0;
+      list-style: none;
+      color: var(--muted);
+    }
+    .why-ntv3 li {
+      margin: 12px 0;
+      padding-left: 0;
+      line-height: 1.7;
+    }
+    .paper-summary {
+      margin-top: 12px;
+      padding: 24px;
+      border: 1px solid var(--border);
+      background: var(--card);
+      border-radius: var(--radius);
+      box-shadow: var(--shadow);
+    }
+    .paper-summary h2 {
+      text-align: center;
+      margin: 0 0 20px 0;
+    }
+    .paper-summary img {
+      width: 100%;
+      height: auto;
+      display: block;
+      border-radius: 12px;
+    }
     .footer { margin-top: 22px; color: var(--muted); font-size: 13px; }
+    
+    /* Tab navigation styles */
+    .tabs {
+      margin-top: 24px;
+      display: flex;
+      gap: 8px;
+      border-bottom: 2px solid var(--border);
+      overflow-x: auto;
+    }
+    .tab-button {
+      padding: 12px 20px;
+      background: transparent;
+      border: none;
+      border-bottom: 2px solid transparent;
+      color: var(--muted);
+      font-family: var(--sans);
+      font-size: 14px;
+      font-weight: 500;
+      cursor: pointer;
+      transition: all 0.2s ease;
+      white-space: nowrap;
+      margin-bottom: -2px;
+    }
+    .tab-button:hover {
+      color: var(--text);
+      background: rgba(255, 255, 255, 0.03);
+    }
+    .tab-button.active {
+      color: var(--link);
+      border-bottom-color: var(--link);
+    }
+    .tab-content {
+      display: none;
+      animation: fadeIn 0.3s ease;
+    }
+    .tab-content.active {
+      display: block;
+    }
+    @keyframes fadeIn {
+      from { opacity: 0; transform: translateY(8px); }
+      to { opacity: 1; transform: translateY(0); }
+    }
+    
     @media (max-width: 860px) {
       .card { grid-column: span 12; }
       h1 { font-size: 28px; }
+      .tab-button {
+        padding: 10px 16px;
+        font-size: 13px;
+      }
     }
   </style>
 </head>
@@ -96,77 +256,139 @@
 <body>
   <div class="wrap">
     <div class="hero">
-      <h1>NTv3 — Foundation Models for Long-Range Genomics</h1>
+      <h1>🧬 NTv3 — Next-Gen Foundation Models for Genomics</h1>
       <p>
         This Space is the companion hub for <strong>NTv3</strong> models: runnable notebooks for inference, fine-tuning, interpretation, and sequence generation.
       </p>
 
       <div class="pillrow">
-        <span class="pill">Long-context genomics</span>
-        <span class="pill">Torch notebooks</span>
-        <span class="pill">Inference • Fine-tune • Interpret • Generate</span>
+        <span class="pill">🤖 Foundation Models</span>
+		<span class="pill">🧬 Long-context genomics</span>
+		<span class="pill">🌍 Multi-species</span>
+        <span class="pill">⚡ Inference • Fine-tune • Interpret • Generate</span>
+        <span class="pill">📓 Torch notebooks</span>
       </div>
     </div>
 
-    <div class="grid">
-      <div class="card">
-        <h2>Notebooks</h2>
-        <ul>
-          <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/tree/main/notebooks" target="_blank" rel="noopener">Browse notebooks folder</a></li>
-          <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks/00_quickstart_inference.ipynb" target="_blank" rel="noopener">00 — Quickstart inference</a></li>
-          <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks/01_tracks_prediction.ipynb" target="_blank" rel="noopener">01 — Tracks prediction</a></li>
-          <li>02 — Genome annotation / segmentation</li>
-          <li>03 — Fine-tune a head</li>
-          <li>04 — Model interpretation</li>
-          <li>05 — Sequence generation</li>
-        </ul>
-      </div>
-
-      <div class="card">
-        <h2>Models</h2>
-        <ul>
-          <li>Pretrained checkpoints:
-            <div style="margin-top: 8px; margin-left: 0;">
-              <div><a href="https://huggingface.co/InstaDeepAI/ntv3_8M_7downsample_pretrained_le_1mb"><code>InstaDeepAI/ntv3_8M_7downsample_pretrained_le_1mb</code></a></div>
-              <div><a href="https://huggingface.co/InstaDeepAI/ntv3_106M_7downsample_pretrained_le_1mb"><code>InstaDeepAI/ntv3_106M_7downsample_pretrained_le_1mb</code></a></div>
-              <div><a href="https://huggingface.co/InstaDeepAI/ntv3_650M_7downsample_pretrained_le_1mb"><code>InstaDeepAI/ntv3_650M_7downsample_pretrained_le_1mb</code></a></div>
-            </div>
-          </li>
-          <li>Post-trained checkpoints:
-            <div style="margin-top: 8px; margin-left: 0;">
-              <div><a href="https://huggingface.co/InstaDeepAI/ntv3_650M_7downsample_post_trained_1mb"><code>InstaDeepAI/ntv3_650M_7downsample_post_trained_1mb</code></a></div>
-              <div><a href="https://huggingface.co/InstaDeepAI/ntv3_106M_7downsample_post_trained_1mb"><code>InstaDeepAI/ntv3_106M_7downsample_post_trained_1mb</code></a></div>
-            </div>
-          </li>
-        </ul>
-      </div>
+    <!-- Tab Navigation -->
+    <div class="tabs">
+      <button class="tab-button active" data-tab="home">🏠 Home</button>
+      <button class="tab-button" data-tab="demo">🚀 Live Demo</button>
+    </div>
 
-      <div class="card">
-        <h2>Model usage (to update)</h2>
-        <p>Here is a quick example of how to use NTv3 models.</p>
-        <div class="code"><code>from transformers import pipeline
-
-pipe = pipeline(
-    task="ntv3-tracks",
-    model="InstaDeepAI/ntv3_106M_7downsample_post_trained_1mb",
-    trust_remote_code=True,
-    device="cuda",
-    torch_dtype=torch.bfloat16,
-)</code></div>
-      </div>
+    <!-- Home Tab (Content loaded from tabs/home.html) -->
+    <div id="home" class="tab-content active">
+      <!-- Content will be loaded dynamically -->
+    </div>
 
-      <div class="card">
-        <h2>Links</h2>
-        <ul>
-          <li>Paper: (add link)</li>
-          <li><a href="https://github.com/instadeepai/nucleotide-transformer">JAX training code</a></li>
-        </ul>
-      </div>
+    <!-- Live Demo Tab (Content loaded from tabs/demo.html) -->
+    <div id="demo" class="tab-content">
+      <!-- Content will be loaded dynamically -->
     </div>
+          
+    <!-- <div class="paper-summary">
+		<h2>📄 A foundational model for joint sequence-function multi-species modeling at scale for long-range genomic prediction</h2>
+		<img src="assets/paper_summary.png" alt="NTv3 Paper Summary" />
+    </div> -->
 
     <p class="footer">
       © instadeep-ai — NTv3 companion Space.
     </p>
   </div>
+  <script src="https://cdnjs.cloudflare.com/ajax/libs/prism/1.29.0/components/prism-core.min.js"></script>
+  <script src="https://cdnjs.cloudflare.com/ajax/libs/prism/1.29.0/plugins/autoloader/prism-autoloader.min.js"></script>
+  <script>
+    // Tab content mapping
+    const tabFiles = {
+      'home': 'tabs/home.html',
+      'demo': 'tabs/demo.html',
+      'functional_tracks': 'tabs/functional_tracks.html',
+    };
+
+    // Cache for loaded tab content
+    const tabCache = {};
+
+    // Function to load tab content
+    async function loadTabContent(tabId) {
+      // Return cached content if available
+      if (tabCache[tabId]) {
+        return tabCache[tabId];
+      }
+
+      // Load content from file
+      const filePath = tabFiles[tabId];
+      if (!filePath) {
+        console.error(`No file path defined for tab: ${tabId}`);
+        return '';
+      }
+
+      try {
+        const response = await fetch(filePath);
+        if (!response.ok) {
+          throw new Error(`Failed to load ${filePath}: ${response.statusText}`);
+        }
+        const content = await response.text();
+        tabCache[tabId] = content;
+        return content;
+      } catch (error) {
+        console.error(`Error loading tab content for ${tabId}:`, error);
+        return `<div class="summary"><p>Error loading content. Please refresh the page.</p></div>`;
+      }
+    }
+
+    // Function to show a tab
+    async function showTab(tabId) {
+      const tabContent = document.getElementById(tabId);
+      if (!tabContent) {
+        console.error(`Tab element not found: ${tabId}`);
+        return;
+      }
+
+      // Load content if not already loaded
+      if (!tabContent.dataset.loaded) {
+        tabContent.innerHTML = await loadTabContent(tabId);
+        tabContent.dataset.loaded = 'true';
+        
+        // Re-run Prism.js syntax highlighting for code blocks in the loaded content
+        if (typeof Prism !== 'undefined') {
+          // Find all code blocks in the loaded content and highlight them
+          const codeBlocks = tabContent.querySelectorAll('code[class*="language-"]');
+          codeBlocks.forEach(block => {
+            Prism.highlightElement(block);
+          });
+        }
+      }
+    }
+
+    // Tab switching functionality
+    document.addEventListener('DOMContentLoaded', function() {
+      const tabButtons = document.querySelectorAll('.tab-button');
+      const tabContents = document.querySelectorAll('.tab-content');
+
+      // Load the default active tab (home)
+      const activeTab = document.querySelector('.tab-content.active');
+      if (activeTab) {
+        showTab(activeTab.id);
+      }
+
+      tabButtons.forEach(button => {
+        button.addEventListener('click', async () => {
+          const targetTab = button.getAttribute('data-tab');
+
+          // Remove active class from all buttons and contents
+          tabButtons.forEach(btn => btn.classList.remove('active'));
+          tabContents.forEach(content => content.classList.remove('active'));
+
+          // Add active class to clicked button and corresponding content
+          button.classList.add('active');
+          const tabElement = document.getElementById(targetTab);
+          tabElement.classList.add('active');
+
+          // Load and show the tab content
+          await showTab(targetTab);
+        });
+      });
+    });
+  </script>
 </body>
 </html>

{notebooks → notebooks_tutorials}/00_quickstart_inference.ipynb

@@ -5,33 +5,33 @@
       "id": "024bb8a8",
       "metadata": {},
       "source": [
-        "# NTv3 Quickstart — Pre-trained and Post-trained models\n",
+        "# 🚀 NTv3 Quickstart — Pre-trained and Post-trained models\n",
         "\n",
-        "This notebook demonstrates how to run **quick inference** with bothe pre- and post-trained NTv3 checkpoints:\n",
+        "This notebook demonstrates how to run **quick inference** with both the pre- and post-trained NTv3 checkpoints:\n",
         "\n",
-        "- **Pre-trained (MLM-focused):** `InstaDeepAI/ntv3_8M_7downsample_pretrained_le_1mb`, `InstaDeepAI/ntv3_106M_7downsample_pretrained_le_1mb`, `InstaDeepAI/ntv3_650M_ntv3_650M_7downsample_pretrained_le_1mb7downsample_pre_trained_1mb`\n",
-        "- **Post-trained (task heads):** `InstaDeepAI/ntv3_106M_7downsample_post_trained_1mb`, `InstaDeepAI/ntv3_650M_7downsample_post_trained_1mb`\n",
+        "- **Pre-trained (MLM-focused):** `InstaDeepAI/NTv3_8M_pre`, `InstaDeepAI/NTv3_100M_pre`, `InstaDeepAI/NTv3_650M_pre`\n",
+        "- **Post-trained (functional tracks and genome annotation):** `InstaDeepAI/NTv3_100M_post`, `InstaDeepAI/NTv3_650M_post`\n",
         "\n",
         "We show how to:\n",
         "\n",
         "1. Load tokenizers + models\n",
         "2. Run a forward pass on a DNA sequence window\n",
-        "3. Inspect key outputs"
+        "3. Inspect key outputs\n",
+        "\n",
+        "> 📝 **Note for Google Colab users:** This notebook is compatible with Colab! For faster inference, make sure to enable GPU: Runtime → Change runtime type → GPU (T4 or better recommended)."
       ]
     },
     {
       "cell_type": "markdown",
-      "id": "5d58bf1d",
+      "id": "5827af7e",
       "metadata": {},
       "source": [
-        "## 0) Install dependencies\n",
-        "\n",
-        "Skip if already installed."
+        "## 0) 📦 Imports + setup"
       ]
     },
     {
       "cell_type": "code",
-      "execution_count": null,
+      "execution_count": 1,
       "id": "38cc32a9",
       "metadata": {},
       "outputs": [],
@@ -39,17 +39,9 @@
         "!pip -q install \"transformers>=4.40\" \"huggingface_hub>=0.23\" safetensors torch numpy"
       ]
     },
-    {
-      "cell_type": "markdown",
-      "id": "5827af7e",
-      "metadata": {},
-      "source": [
-        "## 1) Imports + setup"
-      ]
-    },
     {
       "cell_type": "code",
-      "execution_count": 7,
+      "execution_count": 2,
       "id": "d56c105b",
       "metadata": {},
       "outputs": [
@@ -93,7 +85,7 @@
       "id": "82146876",
       "metadata": {},
       "source": [
-        "## 2) Pre-trained checkpoint (MLM-focused)\n",
+        "## 1) 🎯 Pre-trained checkpoint (MLM-focused)\n",
         "\n",
         "This shows the simplest usage: load model + tokenizer, then run a forward pass.\n",
         "\n",
@@ -103,7 +95,7 @@
     },
     {
       "cell_type": "code",
-      "execution_count": null,
+      "execution_count": 3,
       "id": "336bb40c",
       "metadata": {},
       "outputs": [
@@ -112,23 +104,12 @@
           "output_type": "stream",
           "text": [
             "torch.Size([2, 128, 11])\n",
-            "16\n",
-            "2\n",
             "MLM logits shape: (2, 128, 11)\n"
           ]
-        },
-        {
-          "name": "stderr",
-          "output_type": "stream",
-          "text": [
-            "/opt/anaconda3/envs/hf-finetune/lib/python3.10/site-packages/torch/amp/autocast_mode.py:283: UserWarning: In CPU autocast, but the target dtype is not supported. Disabling autocast.\n",
-            "CPU Autocast only supports dtype of torch.bfloat16, torch.float16 currently.\n",
-            "  warnings.warn(error_message)\n"
-          ]
         }
       ],
       "source": [
-        "pretrained_model_name = \"InstaDeepAI/ntv3_8M_7downsample_pretrained_le_1mb\"\n",
+        "pretrained_model_name = \"InstaDeepAI/NTv3_8M_pre\"\n",
         "\n",
         "# Load tokenizer/model\n",
         "tok_pre = AutoTokenizer.from_pretrained(pretrained_model_name, trust_remote_code=True)\n",
@@ -137,11 +118,9 @@
         "# Example: human sequence\n",
         "seqs = [\"ATCGNATCG\", \"ACGT\"]\n",
         "batch = tok_pre(seqs, add_special_tokens=False, padding=True, pad_to_multiple_of=128, return_tensors=\"pt\")\n",
-        "out = model_pre(**batch, output_hidden_states=True, output_attentions=True)\n",
+        "out = model_pre(**batch)\n",
         "\n",
         "print(out.logits.shape)       # (B, L, V = 11)\n",
-        "print(len(out.hidden_states)) # convs + transformers + deconvs\n",
-        "print(len(out.attentions))\n",
         "\n",
         "# Access MLM logits\n",
         "mlm_logits = out[\"logits\"]\n",
@@ -153,25 +132,19 @@
       "id": "60a01798",
       "metadata": {},
       "source": [
-        "## 3) Post-trained checkpoint (task heads: BigWig + BED)\n",
+        "## 2) 🧠 Post-trained checkpoint (task heads: BigWig + BED)\n",
         "\n",
-        "Post-trained checkpoints add task-specific heads.\n",
-        "\n",
-        "In particular:\n",
-        "- `condition_tokenizer` is used to tokenize a species condition like `\"human\"`\n",
-        "- `file_assembly_idx` selects the assembly (e.g., `hg38`)\n",
+        "Post-trained checkpoints add task-specific heads for functional track prediction and genome annotation.\n",
         "\n",
         "Expected outputs:\n",
-        "- `bigwig_tracks_logits`\n",
-        "- `bed_tracks_logits`\n",
-        "- `logits` (MLM)\n",
-        "\n",
-        "> If your post-trained checkpoint supports multiple assemblies, the config typically exposes a mapping like `cfg.bigwigs_per_file_assembly`."
+        "- `bigwig_tracks_logits`: functional track predictions\n",
+        "- `bed_tracks_logits`: genome annotation predictions\n",
+        "- `logits`: masked language modeling logits"
       ]
     },
     {
       "cell_type": "code",
-      "execution_count": null,
+      "execution_count": 4,
       "id": "6cc5f2df",
       "metadata": {},
       "outputs": [
@@ -179,45 +152,50 @@
           "name": "stdout",
           "output_type": "stream",
           "text": [
-            "torch.Size([1, 768, 7362])\n",
-            "torch.Size([1, 768, 21, 2])\n",
-            "torch.Size([1, 2048, 11])\n"
+            "Supported species: dict_keys(['<bos>', '<cls>', '<eos>', '<mask>', '<pad>', '<unk>', 'amphiprion_ocellaris', 'arabidopsis_thaliana', 'bison_bison_bison', 'caenorhabditis_elegans', 'canis_lupus_familiaris', 'chinchilla_lanigera', 'ciona_intestinalis', 'danio_rerio', 'drosophila_melanogaster', 'felis_catus', 'gallus_gallus', 'glycine_max', 'gorilla_gorilla', 'gossypium_hirsutum', 'human', 'macaca_nemestrina', 'mouse', 'oryza_sativa', 'rattus_norvegicus', 'salmo_trutta', 'serinus_canaria', 'tetraodon_nigroviridis', 'triticum_aestivum', 'zea_mays'])\n",
+            "bigwig_tracks_logits: (2, 48, 7362)\n",
+            "bed_tracks_logits: (2, 48, 21, 2)\n",
+            "language model logits: (2, 128, 11)\n"
           ]
         }
       ],
       "source": [
-        "posttrained_model_name = \"InstaDeepAI/ntv3_106M_7downsample_post_trained_1mb\"\n",
-        "\n",
-        "# Load config/tokenizers/model\n",
-        "cfg_pos = AutoConfig.from_pretrained(posttrained_model_name, trust_remote_code=True)\n",
-        "tok_pos = AutoTokenizer.from_pretrained(posttrained_model_name, trust_remote_code=True)\n",
-        "model_pos = AutoModel.from_pretrained(posttrained_model_name, trust_remote_code=True)\n",
-        "condition_tokenizer = AutoTokenizer.from_pretrained(\n",
-        "    posttrained_model_name, subfolder=\"condition_tokenizer\", trust_remote_code=True\n",
-        ")\n",
+        "# Load model\n",
+        "post_trained_model_name = \"InstaDeepAI/NTv3_100M_post\"\n",
         "\n",
-        "# Example: human sequence (sequence needs to be multiple of 128 due to 7 downsampling in model)\n",
-        "seq = \"ATCG\" * 512\n",
-        "batch = tok_pos([seq], add_special_tokens=False, return_tensors=\"pt\")\n",
-        "condition = condition_tokenizer([\"human\"], return_tensors=\"pt\")\n",
+        "tok_post = AutoTokenizer.from_pretrained(post_trained_model_name, trust_remote_code=True)\n",
+        "model_post = AutoModel.from_pretrained(post_trained_model_name, trust_remote_code=True)\n",
         "\n",
-        "# Get assembly index for human (hg38)\n",
-        "assemblies = list(cfg_pos.bigwigs_per_file_assembly.keys())\n",
-        "assembly_idx = torch.tensor([assemblies.index(\"hg38\")])\n",
+        "# Prepare inputs\n",
+        "batch = tok_post([\"ATCGNATCG\", \"ACGT\"], add_special_tokens=False, padding=True, pad_to_multiple_of=128, return_tensors=\"pt\")\n",
         "\n",
-        "out = model_pos(\n",
+        "# To show all supported species: \n",
+        "print(\"Supported species:\", model_post.config.species_to_token_id.keys())\n",
+        "# Species tokens\n",
+        "species = ['human', 'mouse']\n",
+        "species_ids = model_post.encode_species(species)\n",
+        "\n",
+        "# Forward pass\n",
+        "out = model_post(\n",
         "    input_ids=batch[\"input_ids\"],\n",
-        "    condition_ids=[condition[\"input_ids\"][0]],\n",
-        "    file_assembly_idx=assembly_idx,\n",
-        "    output_hidden_states=True,\n",
-        "    output_attentions=True,\n",
+        "    species_ids=species_ids,\n",
         ")\n",
         "\n",
-        "# Access model outputs\n",
-        "print(out[\"bigwig_tracks_logits\"].shape)  # per-assembly functional track predictions\n",
-        "print(out[\"bed_tracks_logits\"].shape)     # genomic element classifications\n",
-        "print(out[\"logits\"].shape)                # masked LM logits"
+        "# 7k human tracks over 37.5 % center region of the input sequence\n",
+        "print(\"bigwig_tracks_logits:\", tuple(out[\"bigwig_tracks_logits\"].shape))\n",
+        "# Location of 21 genomic elements over 37.5 % center region of the input sequence\n",
+        "print(\"bed_tracks_logits:\", tuple(out[\"bed_tracks_logits\"].shape))\n",
+        "# Language model logits for whole sequence over vocabulary\n",
+        "print(\"language model logits:\", tuple(out[\"logits\"].shape))\n"
       ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "037076cd",
+      "metadata": {},
+      "outputs": [],
+      "source": []
     }
   ],
   "metadata": {

tabs/demo.html

@@ -0,0 +1,33 @@
+<div class="grid">
+  <div class="card" style="grid-column: span 12; text-align: center; padding: 60px 40px;">
+    <h2 style="margin-bottom: 20px;">🚀 NTv3 Live Demo</h2>
+    <p style="margin-bottom: 30px; color: var(--muted); line-height: 1.7;">
+      Try out the NTv3 model interactively in this live demo. Predict functional tracks and genome annotation directly in your browser. Click the button below to open the interactive demo in a new window.
+    </p>
+    <a 
+      href="https://huggingface.co/spaces/InstaDeepAI/ntv3_tracks" 
+      target="_blank" 
+      rel="noopener noreferrer"
+      style="
+        display: inline-block;
+        padding: 16px 32px;
+        background: var(--link);
+        color: white;
+        text-decoration: none;
+        border-radius: 12px;
+        font-weight: 600;
+        font-size: 16px;
+        transition: all 0.2s ease;
+        box-shadow: 0 4px 12px rgba(125, 211, 252, 0.3);
+      "
+      onmouseover="this.style.background='#60b8e8'; this.style.transform='translateY(-2px)'; this.style.boxShadow='0 6px 16px rgba(125, 211, 252, 0.4)';"
+      onmouseout="this.style.background='var(--link)'; this.style.transform='translateY(0)'; this.style.boxShadow='0 4px 12px rgba(125, 211, 252, 0.3)';"
+    >
+      🚀 Open Live Demo →
+    </a>
+    <p style="margin-top: 30px; font-size: 13px; color: var(--muted);">
+      The demo will open in a new tab. Make sure pop-ups are enabled for this site.
+    </p>
+  </div>
+</div>
+

tabs/home.html

@@ -0,0 +1,193 @@
+<div class="summary">
+  <h2>📖 About NTv3</h2>
+  <p>
+    NTv3 is a multi-species genomic foundation model family that unifies representation learning, functional-track prediction, genome annotation, and controllable sequence generation within a single U-Net-style backbone. It models up to 1 Mb of DNA at single-base resolution, using a conv–Transformer–deconv architecture that efficiently captures both local motifs and long-range regulatory dependencies. NTv3 is first pretrained on ~9T base pairs from the OpenGenome2 corpus spanning >128k species using masked language modeling, and then post-trained with a joint objective on ~16k functional tracks and annotation labels across 24 animal and plant species, enabling state-of-the-art cross-species functional prediction and base-resolution genome annotation.
+  </p>
+  <p>
+    NTv3 also acts as a controllable generative model via masked-diffusion language modeling, allowing targeted design of regulatory sequences (for example, enhancers with specified activity and promoter selectivity) that have been validated experimentally.
+  </p>
+</div>
+
+<div class="paper-summary">
+  <!-- <h2>📄 A foundational model for joint sequence-function multi-species modeling at scale for long-range genomic prediction</h2> -->
+  <img src="assets/paper_summary.png" alt="NTv3 Paper Summary" />
+</div>
+
+<div class="why-ntv3">
+  <h2>✨ Why NTv3?</h2>
+  <ul>
+    <li>📏 <strong>1 Mb long context at nucleotide resolution</strong> — ~100× longer than typical genomics models.</li>
+    <li>🏗️ <strong>Unified architecture</strong> for: masked language modeling, functional-track prediction, genome annotation, and sequence generation.</li>
+    <li>🌍 <strong>Cross-species generalization</strong> across 24 animals + plants with a shared conditioned representation space.</li>
+    <li>⚡ <strong>U-Net–style architecture</strong> improves stability and GPU efficiency on very long sequences.</li>
+    <li>🎯 <strong>Controllable generative modeling</strong>, enabling targeted enhancer/promoter engineering validated by experimental assays.</li>
+  </ul>
+</div>
+
+<div class="grid">
+  <div class="card">
+    <h2>🤖 Models (see <a href="https://huggingface.co/collections/InstaDeepAI/nucleotide-transformer-v3" target="_blank" rel="noopener noreferrer">collection</a>)</h2>
+    <ul>
+      <li>📦 Pretrained checkpoints:
+        <div style="margin-top: 8px; margin-left: 0;">
+          <div><a href="https://huggingface.co/InstaDeepAI/NTv3_8M_pre" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_8M_pre</code></a></div>
+          <div><a href="https://huggingface.co/InstaDeepAI/NTv3_100M_pre" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_100M_pre</code></a></div>
+          <div><a href="https://huggingface.co/InstaDeepAI/NTv3_650M_pre" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_650M_pre</code></a></div>
+        </div>
+      </li>
+      <li>🎯 Post-trained checkpoints:
+        <div style="margin-top: 8px; margin-left: 0;">
+          <div><a href="https://huggingface.co/InstaDeepAI/NTv3_100M_pos" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_100M_pos</code></a></div>
+          <div><a href="https://huggingface.co/InstaDeepAI/NTv3_650M_pos" target="_blank" rel="noopener noreferrer"><code>InstaDeepAI/NTv3_650M_pos</code></a></div>
+        </div>
+      </li>
+    </ul>
+    <table>
+      <thead>
+        <tr>
+          <th>Model</th>
+          <th>Size</th>
+          <th>Pre-training</th>
+          <th>Post-training</th>
+          <th>Usage</th>
+        </tr>
+      </thead>
+      <tbody>
+        <tr>
+          <td><strong>NTv3-8M</strong></td>
+          <td>8M params</td>
+          <td><span class="checkmark">✅</span></td>
+          <td>❌</td>
+          <td>Embeddings, light inference</td>
+        </tr>
+        <tr>
+          <td><strong>NTv3-100M</strong></td>
+          <td>100M params</td>
+          <td><span class="checkmark">✅</span></td>
+          <td><span class="checkmark">✅</span></td>
+          <td>Embeddings, tracks, annotation</td>
+        </tr>
+        <tr>
+          <td><strong>NTv3-650M</strong></td>
+          <td>650M params</td>
+          <td><span class="checkmark">✅</span></td>
+          <td><span class="checkmark">✅</span></td>
+          <td>Embeddings, tracks, annotation, best accuracy</td>
+        </tr>
+      </tbody>
+    </table>
+  </div>
+
+  <div class="card-stack">
+    <div class="card">
+      <h2>📓 Tutorial notebooks (browse <a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/tree/main/notebooks_tutorials" target="_blank" rel="noopener noreferrer">folder</a>)</h2>
+      <ul>
+        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/00_quickstart_inference.ipynb" target="_blank" rel="noopener noreferrer">🚀 00 — Quickstart inference</a></li>
+        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/01_tracks_prediction.ipynb" target="_blank" rel="noopener noreferrer">📊 01 — Tracks prediction</a></li>
+        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/02_fine_tuning.ipynb" target="_blank" rel="noopener noreferrer">🎯 02 — Fine-tune on bigwig tracks</a></li>
+        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_tutorials/03_model_interpretation.ipynb" target="_blank" rel="noopener noreferrer">🔍 03 — Model interpretation</a></li>
+        <li>🧪 04 — Training NTv3-generative <em>(coming soon)</em></li>
+        <li>🪰 05 — Generating enhancer sequences <em>(coming soon)</em></li>
+      </ul>
+    </div>
+    <div class="card">
+      <h2>📓 Pipeline notebooks (browse <a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/tree/main/notebooks_pipelines" target="_blank" rel="noopener noreferrer">folder</a>)</h2>
+      <ul>
+        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/blob/main/notebooks_pipelines/01_functional_track_prediction.ipynb" target="_blank" rel="noopener noreferrer">🎯 01 — Generate bigwig predictions for certain tracks</a></li>
+        <li>🎯 02 — Fine-tune on bigwig tracks</li>
+        <li>🔍 03 — Interpret a given genomic region</li>
+        <li>🧪 04 — Sequence generation <em>(coming soon)</em></li>
+      </ul>
+    </div>
+    <div class="card">
+      <h2>🔗 Links</h2>
+      <ul>
+        <li>📄 Paper: (add link)</li>
+        <li><a href="https://github.com/instadeepai/nucleotide-transformer" target="_blank" rel="noopener noreferrer">💻 JAX model code (GitHub)</a></li>
+        <li><a href="https://huggingface.co/collections/InstaDeepAI/nucleotide-transformer-v3" target="_blank" rel="noopener noreferrer">🎯 HF Model Collection (all NTv3 models)</a></li>
+        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/tree/main/notebooks_tutorials" target="_blank" rel="noopener noreferrer">📚 Tutorial </a> and <a href="https://huggingface.co/spaces/InstaDeepAI/ntv3/tree/main/notebooks_pipelines" target="_blank" rel="noopener noreferrer">🔧 Pipeline</a> notebooks</li>
+        <li><a href="https://huggingface.co/spaces/InstaDeepAI/ntv3_benchmark" target="_blank" rel="noopener noreferrer">🏆 NTv3 benchmark leaderboard</a></li>
+      </ul>
+    </div>
+  </div>
+
+  <div class="card">
+    <h2>🤖 Load a pre-trained model</h2>
+    <p>Here is an example of how to load and use a pre-trained NTv3 model.</p>
+    <div class="code"><pre><code class="language-python">from transformers import AutoTokenizer, AutoModelForMaskedLM
+
+model_name = "InstaDeepAI/NTv3_650M_pre"
+
+# Load model and tokenizer
+model = AutoModelForMaskedLM.from_pretrained(model_name, trust_remote_code=True)
+tok = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+
+# Tokenize input sequences
+batch = tok(["ATCGNATCG", "ACGT"], add_special_tokens=False, padding=True, pad_to_multiple_of=128, return_tensors="pt")
+
+# Run model
+out = model(**batch)
+
+# Print output shapes
+print(out.logits.shape)       # (B, L, V = 11)
+</code></pre></div>
+    <p>Model embeddings can be used for fine-tuning on downstream tasks.</p>
+
+    <p style="margin-top: 40px;">TO DO: add pipeline for fine-tuning on functional tracks or genome annotation.</p>
+  </div>
+  
+  <div class="card">
+    <h2>💻 Use a post-trained model</h2>
+    <p>Here is a quick example of how to use the post-trained NTv3 650M model to predict tracks for a human genomic window.</p>
+    <div class="code"><pre><code class="language-python">from transformers import pipeline
+import torch
+
+model_name = "InstaDeepAI/NTv3_650M_pos"
+
+ntv3_tracks = pipeline(
+    "ntv3-tracks",
+    model=model_name,
+    trust_remote_code=True,
+    device=0 if torch.cuda.is_available() else -1,
+)
+
+# Run track prediction
+out = ntv3_tracks(
+  {
+    "chrom": "chr19",
+    "start": 6_700_000,
+    "end": 6_831_072,
+    "species": "human"
+  }
+)
+
+# Print output shapes
+# 7k human tracks over 37.5 % center region of the input sequence
+print("bigwig_tracks_logits:", tuple(out.bigwig_tracks_logits.shape))
+# Location of 21 genomic elements over 37.5 % center region of the input sequence
+print("bed_tracks_logits:", tuple(out.bed_tracks_logits.shape))
+# Language model logits for whole sequence over vocabulary
+print("language model logits:", tuple(out.mlm_logits.shape))</code></pre></div>
+    <p>Predictions can also be plotted for a subset of functional tracks and genomic elements:</p>
+    <div class="code"><pre><code class="language-python">tracks_to_plot = {
+    "K562 RNA-seq": "ENCSR056HPM",
+    "K562 DNAse": "ENCSR921NMD",
+    "K562 H3k4me3": "ENCSR000DWD",
+    "K562 CTCF": "ENCSR000AKO",
+    "HepG2 RNA-seq": "ENCSR561FEE_P",
+    "HepG2 DNAse": "ENCSR000EJV",
+    "HepG2 H3k4me3": "ENCSR000AMP",
+    "HepG2 CTCF": "ENCSR000BIE",
+}
+elements_to_plot = ["protein_coding_gene", "exon", "intron", "splice_donor", "splice_acceptor"]
+
+out = ntv3_tracks(
+    {"chrom": "chr19", "start": 6_700_000, "end": 6_831_072, "species": "human"},
+    plot=True,
+    tracks_to_plot=tracks_to_plot,
+    elements_to_plot=elements_to_plot,
+)</code></pre></div>
+    <img src="assets/output_tracks.png" alt="Output tracks visualization" style="max-width: 100%; margin-top: 20px;" />
+  </div>
+</div>
+