docs/specs/28-gradio-custom-component-niivue.md

Spec #28: Gradio Custom Component for NiiVue

Date: 2025-12-10 Status: REQUIRED - All gr.HTML hacks have failed (confirmed Dec 10) Blocks: Issue #24 (HF Spaces "Loading..." forever) Effort: Medium (2-3 days + 0.5-1 day buffer for HF Spaces quirks) Success Probability: 90% Audited: AUDIT_REPORT_2025_12_10.md - GO recommendation

---

Executive Summary

All gr.HTML + JavaScript approaches have FAILED. This is the only path forward.

Gradio maintainers have explicitly closed both:

• Issue #4511 - NIfTI/medical imaging support → "Not planned"
• Issue #7649 - WebGL canvas component → "Not planned"

Their official answer: "Create a Gradio Custom Component."

This spec documents what we need to build to properly integrate NiiVue (WebGL2 medical imaging viewer) into our Gradio app.

---

Why Current Approach Fails

What We've Tried

Attempt	Why It Failed
CDN import in js_on_load	HF Spaces CSP blocks external imports
Vendored NiiVue + dynamic import()	import() in js_on_load blocks Svelte hydration
head= parameter	Still uses ES module import, same problem
head_paths= parameter	Same as above
gr.set_static_paths()	File serving works, but JS loading mechanism broken

Root Cause

We're fighting gr.HTML's limitations, not Gradio itself. Gradio CAN do WebGL (proven by gradio-litmodel3d), but NOT via gr.HTML:

1. gr.HTML strips <script> tags (security)
2. js_on_load runs during component mount - async import() blocks Svelte hydration
3. Our A/B test proved: disabling js_on_load makes the app load perfectly

The gr.HTML + js_on_load + import() pattern is the blocker. Custom Components solve this by using Svelte's proper onMount lifecycle.

---

The Solution: Gradio Custom Component

What Is a Gradio Custom Component?

A Custom Component is a proper Svelte + Python component that integrates with Gradio's architecture:

gradio-niivue-viewer/
├── frontend/
│   ├── Index.svelte      # Svelte component (renders NiiVue)
│   ├── package.json      # Frontend deps (including niivue)
│   └── ...
├── backend/
│   └── gradio_niivue_viewer/
│       └── __init__.py   # Python component class
├── pyproject.toml        # Package definition
└── demo/
    └── app.py            # Example usage

Why This Works

1. Svelte-native: Component integrates with Gradio's lifecycle properly
2. Official pattern: Gradio maintainers recommend this for WebGL
3. Isolated loading: NiiVue loads within the component, not globally
4. Proper error handling: Failures don't block app initialization
5. Reusable: Can publish to PyPI for others to use

---

Prerequisites

Build Tooling Requirements

Tool	Version	Purpose
Node.js	>= 18.x	Required by gradio cc build
npm	>= 9.x	Package management for Svelte frontend
Python	>= 3.10	Backend component
Gradio	>= 5.0	Custom component framework

Verify installation:

node --version  # v18.x or higher
npm --version   # 9.x or higher
gradio --version  # 5.x or higher

Packaging Plan

Location: Monorepo subdirectory at packages/gradio-niivue-viewer/

This approach:

• Keeps component close to main app for easy iteration
• Allows pip install -e packages/gradio-niivue-viewer for local development
• No PyPI publishing required initially (can add later)

---

Value Schema

The component uses Gradio's file serving URLs (not base64) per Issue #19 optimization:

// Frontend (Svelte)
interface NiiVueViewerValue {
  background_url: string | null;  // e.g., "/gradio_api/file=/tmp/.../dwi.nii.gz"
  overlay_url: string | null;     // e.g., "/gradio_api/file=/tmp/.../mask.nii.gz"
}

# Backend (Python)
class NiiVueViewerData(GradioModel):
    background_url: str | None = None  # Gradio file URL
    overlay_url: str | None = None     # Gradio file URL

Critical: URLs must use /gradio_api/file= format, NOT base64. This reduces payload from ~65MB to <1KB.

---

Technical Approach

Phase 1: Scaffold Component (1 hour)

Use Gradio's CLI to create the component:

# From repository root
mkdir -p packages
cd packages

gradio cc create NiiVueViewer \
  --template Image \
  --overwrite

This creates the basic structure with Svelte frontend and Python backend.

Phase 2: Implement Svelte Frontend (4-6 hours)

2a. Install NiiVue dependency

cd packages/gradio-niivue-viewer/frontend
npm install @niivue/niivue@0.65.0 --save-exact

This pins the exact version 0.65.0 to match our tested vendored copy.

2b. Verify package.json

{
  "name": "gradio-niivue-viewer",
  "version": "0.1.0",
  "dependencies": {
    "@niivue/niivue": "0.65.0"
  }
}

2c. Modify frontend/Index.svelte:

<script lang="ts">
  import { onMount, onDestroy } from 'svelte';
  import { Niivue } from '@niivue/niivue';

  // Value schema: Gradio file URLs (not base64)
  export let value: {
    background_url: string | null;
    overlay_url: string | null;
  } | null = null;

  let container: HTMLDivElement;
  let canvas: HTMLCanvasElement;
  let nv: Niivue | null = null;
  let error: string | null = null;
  let loading: boolean = true;

  // WebGL2 capability check
  function checkWebGL2(): boolean {
    const testCanvas = document.createElement('canvas');
    const gl = testCanvas.getContext('webgl2');
    return gl !== null;
  }

  onMount(async () => {
    // Check WebGL2 support first
    if (!checkWebGL2()) {
      error = 'WebGL2 is not supported in this browser. Please use Chrome, Firefox, or Edge.';
      loading = false;
      return;
    }

    try {
      nv = new Niivue({
        backColor: [0, 0, 0, 1],
        show3Dcrosshair: true,
        logging: false,
      });
      await nv.attachToCanvas(canvas);

      // Handle WebGL context loss
      canvas.addEventListener('webglcontextlost', handleContextLost);
      canvas.addEventListener('webglcontextrestored', handleContextRestored);

      await loadVolumes();
      loading = false;
    } catch (e) {
      error = `Failed to initialize viewer: ${e instanceof Error ? e.message : 'Unknown error'}`;
      loading = false;
    }
  });

  onDestroy(() => {
    if (canvas) {
      canvas.removeEventListener('webglcontextlost', handleContextLost);
      canvas.removeEventListener('webglcontextrestored', handleContextRestored);
    }
    if (nv) nv.dispose();
  });

  function handleContextLost(event: Event) {
    event.preventDefault();
    error = 'WebGL context lost. Please refresh the page.';
  }

  function handleContextRestored() {
    error = null;
    if (nv && value) loadVolumes();
  }

  async function loadVolumes() {
    if (!nv) return;

    // Handle null/cleared value: clear the viewer
    if (!value || (!value.background_url && !value.overlay_url)) {
      try {
        // Clear all loaded volumes
        while (nv.volumes.length > 0) {
          nv.removeVolumeByIndex(0);
        }
        nv.drawScene();
      } catch (e) {
        console.warn('Failed to clear volumes:', e);
      }
      return;
    }

    try {
      loading = true;
      error = null;

      const volumes = [];
      if (value.background_url) {
        volumes.push({ url: value.background_url, name: 'background.nii.gz' });
      }
      if (value.overlay_url) {
        volumes.push({
          url: value.overlay_url,
          name: 'overlay.nii.gz',
          colorMap: 'red',
          opacity: 0.5,
        });
      }

      if (volumes.length > 0) {
        await nv.loadVolumes(volumes);
        // Configure view after loading
        nv.setSliceType(nv.sliceTypeMultiplanar);
        nv.setRenderAzimuthElevation(120, 10);
        nv.drawScene();
      }

      loading = false;
    } catch (e) {
      error = `Failed to load volumes: ${e instanceof Error ? e.message : 'Unknown error'}`;
      loading = false;
    }
  }

  // Reactive: reload when value changes (including null to clear)
  $: if (nv && !loading) loadVolumes();
</script>

<div bind:this={container} class="niivue-container">
  {#if error}
    <div class="error-message">{error}</div>
  {:else if loading}
    <div class="loading-message">Loading viewer...</div>
  {/if}
  <canvas bind:this={canvas} class:hidden={!!error}></canvas>
</div>

<style>
  .niivue-container {
    width: 100%;
    height: 500px;
    background: #000;
    position: relative;
    border-radius: 8px;
    overflow: hidden;
  }
  canvas {
    width: 100%;
    height: 100%;
  }
  canvas.hidden {
    display: none;
  }
  .error-message {
    position: absolute;
    top: 50%;
    left: 50%;
    transform: translate(-50%, -50%);
    color: #f66;
    text-align: center;
    padding: 20px;
    max-width: 80%;
  }
  .loading-message {
    position: absolute;
    top: 50%;
    left: 50%;
    transform: translate(-50%, -50%);
    color: #888;
    text-align: center;
  }
</style>

Key improvements from audit feedback:

• WebGL2 capability check before initialization
• WebGL context loss/restore handlers
• Proper error UI states
• Loading state management
• Reactive update when value changes

Phase 3: Implement Python Backend (2-3 hours)

# backend/gradio_niivue_viewer/__init__.py
from __future__ import annotations
from typing import Any
from gradio.components.base import Component
from gradio.data_classes import FileData, GradioModel

class NiiVueViewerData(GradioModel):
    background_url: str | None = None
    overlay_url: str | None = None

class NiiVueViewer(Component):
    """WebGL NIfTI viewer using NiiVue."""

    data_model = NiiVueViewerData

    def __init__(
        self,
        value: NiiVueViewerData | None = None,
        *,
        label: str | None = None,
        height: int = 500,
        **kwargs,
    ):
        self.height = height
        super().__init__(value=value, label=label, **kwargs)

    def preprocess(self, payload: NiiVueViewerData | None) -> dict[str, Any] | None:
        if payload is None:
            return None
        return {
            "background_url": payload.background_url,
            "overlay_url": payload.overlay_url,
        }

    def postprocess(self, value: dict[str, Any] | None) -> NiiVueViewerData | None:
        if value is None:
            return None
        return NiiVueViewerData(
            background_url=value.get("background_url"),
            overlay_url=value.get("overlay_url"),
        )

Phase 4: Build and Test (2-3 hours)

# Build the component
cd gradio-niivue-viewer
gradio cc build

# Install locally
pip install -e .

# Test in demo app
python demo/app.py

Phase 5: Integrate into stroke-deepisles-demo (1-2 hours)

Replace gr.HTML with the custom component:

# Before (broken)
from stroke_deepisles_demo.ui.viewer import create_niivue_html
viewer = gr.HTML(value="", elem_id="niivue-viewer")
# ... then set viewer.value = create_niivue_html(...)

# After (working)
from gradio_niivue_viewer import NiiVueViewer
viewer = NiiVueViewer(label="Interactive 3D Viewer")
# ... then set viewer.value = {"background_url": dwi_url, "overlay_url": mask_url}

Phase 6: HF Spaces Deployment (CRITICAL)

This phase is essential. HF Spaces runs pip install only - it does NOT run npm or gradio cc build.

6a. Commit build artifacts to git

cd packages/gradio-niivue-viewer

# Build the component (generates frontend/dist/ or templates/)
gradio cc build

# Force-add build artifacts (they may be gitignored by default)
git add -f gradio_niivue_viewer/templates/
# Or wherever the build output lands - check with:
# find . -name "*.js" -path "*/dist/*" -o -name "*.css" -path "*/dist/*"

git commit -m "chore: add compiled frontend assets for HF Spaces deployment"

Why: HF Spaces won't run npm/node build steps. The compiled JS/CSS must be in the repo.

6b. Update requirements.txt

Add the local component to the main requirements.txt:

# requirements.txt
gradio>=5.0
# ... other deps ...

# Local custom component (editable install)
-e ./packages/gradio-niivue-viewer

Alternative: If the component is at repo root:

-e .

6c. Verify .gitignore doesn't exclude build artifacts

Check that packages/gradio-niivue-viewer/.gitignore doesn't exclude:

• gradio_niivue_viewer/templates/
• frontend/dist/
• Any compiled .js or .css files needed at runtime

If they're excluded, either:

1. Remove those lines from .gitignore, OR
2. Use git add -f to force-add them

6d. Test deployment flow

# Simulate what HF Spaces does
pip install -r requirements.txt
python -m stroke_deepisles_demo.ui.app

# Should work WITHOUT running gradio cc build

---

Existing References

Working WebGL Custom Components

1. gradio-litmodel3d

   • WebGL Model3D viewer with HDR lighting
   • Source: https://github.com/gradio-app/gradio/tree/main/demo/model3d_component
   • Proof that WebGL works in Custom Components

2. gradio-molecule3d

   • 3D molecule viewer
   • Uses Three.js (WebGL)

Gradio Documentation

• Custom Components in 5 Minutes
• Gradio Components Documentation
• Custom Component Gallery

NiiVue Resources

• NiiVue GitHub
• NiiVue npm
• NiiVue Examples

---

Acceptance Criteria

Must Have (MVP)

• Component loads NIfTI volumes from Gradio file URLs
• Component displays background image (DWI)
• Component displays overlay mask (segmentation) with colormap
• Component works on HuggingFace Spaces
• No "Loading..." hang - failures are graceful
• All existing tests pass

Nice to Have (Future)

• Crosshair controls
• Slice orientation toggle (axial/coronal/sagittal)
• Opacity slider for overlay
• Pan/zoom/rotate controls
• Screenshot/export functionality
• Publish to PyPI for community use

---

Risk Assessment

Risk	Mitigation
Svelte/TypeScript learning curve	Follow gradio-litmodel3d example closely
NiiVue WebGL2 browser support	Explicit WebGL2 check in Svelte + graceful error UI
Build system complexity	Use gradio cc tooling, don't customize
HF Spaces static file serving	Component bundles NiiVue, no external deps
Build artifacts not in git	Phase 6a: Force-add compiled assets with git add -f
requirements.txt missing component	Phase 6b: Add -e ./packages/gradio-niivue-viewer

---

Alternatives Considered

Alternative 1: Keep Hacking gr.HTML

• Effort: Low
• Success probability: 0% (CONFIRMED FAILED)
• Why rejected: We tried 6 approaches over 2 days. ALL failed. This is not a viable path.

Alternative 2: Static HTML Space (No Gradio)

• Effort: High (rebuild entire UI)
• Success probability: 99%
• Why rejected: Lose Gradio's file upload, dropdowns, layout features. Too much work.

Alternative 3: Remove 3D Viewer (2D Only)

• Effort: Low
• Success probability: 100%
• Why rejected: Loses key feature. Static Report tab already works, but 3D is valuable.

---

Decision

Proceed with Gradio Custom Component approach.

This is the official Gradio-recommended solution. It's more work than hacking gr.HTML, but it's the architecturally correct approach with 90% success probability vs 30%.

---

Testing Matrix

Level 1: Local Build Verification

cd packages/gradio-niivue-viewer

# Build component
gradio cc build

# Install locally
pip install -e .

# Run demo app
python demo/app.py
# → Verify: App loads, no console errors, viewer renders

Pass criteria:

• gradio cc build completes without errors
• Demo app launches at localhost:7860
• No JavaScript console errors
• Canvas renders (black background visible)

Level 2: Volume Loading Test

# demo/app.py
import gradio as gr
from gradio_niivue_viewer import NiiVueViewer

def load_sample():
    # Use a known good NIfTI file
    return {
        "background_url": "/gradio_api/file=/path/to/sample.nii.gz",
        "overlay_url": None
    }

with gr.Blocks() as demo:
    viewer = NiiVueViewer()
    btn = gr.Button("Load Sample")
    btn.click(load_sample, outputs=viewer)

demo.launch()

Pass criteria:

• NIfTI file loads without errors
• Multiplanar view displays correctly
• Overlay mask renders with red colormap (when provided)

Level 3: HF Spaces Dry Run

Deploy to a private/throwaway Space before production:

# Create test space
huggingface-cli repo create test-niivue-viewer --type space --private

# Push and test
git push hf-test main

Pass criteria:

• Space shows "Running" (not stuck on "Loading...")
• Viewer initializes (no hydration deadlock)
• Volume loading works via Gradio file serving
• WebGL2 error shown gracefully if unsupported

Level 4: Integration Test

Replace gr.HTML in stroke-deepisles-demo:

# src/stroke_deepisles_demo/ui/components.py
from gradio_niivue_viewer import NiiVueViewer

def create_results_display():
    # ...
    niivue_viewer = NiiVueViewer(label="Interactive 3D Viewer")
    # ...

Pass criteria:

• Existing 136 tests still pass
• Segmentation pipeline works end-to-end
• Viewer displays DWI + mask overlay
• No "Loading..." hang on HF Spaces

---

Next Steps

1. Senior review of this spec (AUDIT_REPORT_2025_12_10.md)
2. Red team review - all gaps addressed (build artifacts, npm install, null handling)
3. Create packages/gradio-niivue-viewer/ subdirectory
4. Scaffold component with gradio cc create
5. Install NiiVue: cd frontend && npm install @niivue/niivue@0.65.0
6. Implement Svelte frontend (with WebGL2 checks + null value handling)
7. Implement Python backend
8. Level 1 test: Local build verification
9. Level 2 test: Volume loading
10. Level 3 test: HF Spaces dry run
11. Level 4 test: Integration
12. CRITICAL: Commit build artifacts to git
13. CRITICAL: Update requirements.txt with -e ./packages/gradio-niivue-viewer
14. (Optional) Publish to PyPI

---

Appendix: Why WebGL + gr.HTML Doesn't Work

From the ROOT_CAUSE_ANALYSIS.md and GRADIO_WEBGL_ANALYSIS.md research:

1. Gradio CAN do WebGL - proven by gradio-litmodel3d custom component
2. But NOT via gr.HTML - the js_on_load + import() pattern blocks Svelte hydration
3. Our A/B test proved it - disabling js_on_load makes the app load perfectly
4. Gradio closed NIfTI support (Issue #4511) - "Not planned for core"
5. Gradio closed WebGL canvas (Issue #7649) - "Not planned for core"
6. gr.HTML strips script tags - Security feature, can't bypass
7. HF Spaces CSP blocks external CDNs - Must vendor or bundle dependencies
8. Gradio maintainer recommendation: Custom Components

The pattern is clear: The gr.HTML + js_on_load + async import() pattern is fundamentally broken. The Custom Component is the officially supported path for WebGL content.