docs/specs/10-bug-niivue-viewer-black-screen.md

Bug #10: NiiVue 3D Viewer Renders Black Screen on HF Spaces

Status: PARTIALLY FIXED → See Bug #11

Date: 2025-12-09 Branch: fix/niivue-js-on-load (merged), now fix/niivue-js-rerun Discovered: After fixing Bug #9 (DeepISLES subprocess bridge)

Fix Applied (2025-12-09) - PARTIAL

Implemented js_on_load approach (Solution 1 from this spec):

1. viewer.py: Removed <script> tags, added NIIVUE_JS_ON_LOAD constant
2. components.py: Added js_on_load=NIIVUE_JS_ON_LOAD to gr.HTML
3. All 130 tests pass locally

The HTML now uses data-* attributes to pass volume URLs, and JavaScript executes via js_on_load instead of inline <script> tags.

Continued in Bug #11

After HF Spaces deployment, we discovered that js_on_load only runs once on component mount, not on value updates. This means the NiiVue viewer initializes correctly on page load, but when run_segmentation() updates the gr.HTML value with new data-* attributes, the JS doesn't re-execute.

See Bug #11 for the complete analysis and the verified fix using .then(fn=None, js=...).

---

TL;DR - ROOT CAUSE

Gradio's gr.HTML component does NOT execute <script> tags (including type="module").

Our code embeds NiiVue initialization JavaScript inside <script type="module"> tags within the HTML value. Gradio intentionally ignores these for security reasons. The canvas renders but NiiVue never initializes → black screen.

---

Symptom

After successful DeepISLES inference on HF Spaces, the NiiVue 3D viewer component (top-right panel) renders as a completely black rectangle. No brain scan or mask overlay is visible.

What IS working:

• DeepISLES inference completes successfully (~32 seconds)
• Slice Comparison (matplotlib 2D view) renders correctly
• Metrics JSON displays correctly
• Download button provides the prediction mask
• Ground truth overlay in Slice Comparison works

What is NOT working:

• NiiVue WebGL 3D viewer shows black screen
• No error message displayed in the viewer area
• No visible WebGL error fallback message

What SHOULD appear:

• Multi-planar view (axial/coronal/sagittal slices)
• Optional 3D volume rendering
• Interactive crosshairs for navigation
• DWI volume as grayscale background
• Prediction mask as semi-transparent red overlay

---

Root Cause Analysis

Evidence Chain

1. HuggingFace Forum:

   "You can't load scripts via gr.HTML"

2. Gradio Official Docs:

   "Only static HTML is rendered (e.g., no JavaScript). To render JavaScript, use the js or head parameters"

3. Gradio 6 Migration Guide:

   The js and head parameters moved from gr.Blocks() to launch() in Gradio 6

4. GitHub Issue #10250:

   Known issue with JavaScript in head param not executing reliably

Our Code (BROKEN)

# viewer.py:324-385 - Returns HTML with embedded script tags
def create_niivue_html(volume_url, mask_url, height=400) -> str:
    return f"""
    <div id="{container_id}" style="...">
        <canvas id="{canvas_id}" style="..."></canvas>
    </div>
    <script type="module">
        // THIS ENTIRE BLOCK IS IGNORED BY GRADIO!
        (async function() {{
            const niivueModule = await import('{NIIVUE_CDN_URL}');
            const Niivue = niivueModule.Niivue;
            const nv = new Niivue({{...}});
            await nv.attachToCanvas(document.getElementById('{canvas_id}'));
            await nv.loadVolumes([{{ url: {volume_url_js} }}]);
            // ... more initialization
        }})();
    </script>
    """

# components.py:42 - Basic HTML component without js_on_load
niivue_viewer = gr.HTML(label="Interactive 3D Viewer")  # No js_on_load!

Why It Fails

1. gr.HTML receives our HTML string as value
2. Gradio renders the <div> and <canvas> elements (static HTML)
3. Gradio strips or ignores the <script> tags for security
4. NiiVue JavaScript never executes
5. Canvas remains empty → black screen
6. Our try/catch error handling never runs (script doesn't execute at all)

---

Secondary Issues

Issue 1: Base64 Payload Size (~65MB)

Even if JavaScript executed, we're passing massive base64-encoded NIfTI data:

File	Raw Size	Base64 Size
DWI	30.1 MB	~40 MB
ADC	17.7 MB	~24 MB
Total	~48 MB	~65 MB

This could cause:

• Browser memory issues
• Gradio payload limits
• Slow/failed rendering

Issue 2: Gradio 6 Breaking Changes

Our code uses Gradio 5.x patterns. In Gradio 6.x:

• js, head, head_paths moved from gr.Blocks() to launch()
• padding default changed from True to False
• js_on_load is now the proper way for component-level JavaScript

Issue 3: No Error Visibility

Our JavaScript has try/catch that should display errors in the container, but since the script never executes, the error handling never runs. The canvas just stays black with no feedback to the user.

---

Code Locations

File	Lines	Description
src/stroke_deepisles_demo/ui/viewer.py	277-385	create_niivue_html() - generates broken HTML
src/stroke_deepisles_demo/ui/viewer.py	34-51	nifti_to_data_url() - base64 encoding
src/stroke_deepisles_demo/ui/app.py	101-117	NiiVue HTML generation in run_segmentation()
src/stroke_deepisles_demo/ui/components.py	41-42	gr.HTML component creation (missing js_on_load)

---

External Validation (2025-12-09)

An external agent review claimed js_on_load does not exist. This claim was REFUTED.

Verification Results

Claim	Status	Evidence
"gr.HTML does NOT have js_on_load parameter"	❌ REFUTED	Gradio Docs show js_on_load with default value
"js_on_load was added in PR #12098"	✅ Confirmed	Part of "gr.HTML custom components" feature
"Base64 payload (~65MB) is a risk"	✅ Confirmed	Valid concern, should use file URLs
"CSP headers may block CDN"	⚠️ Possible	HF Spaces typically allows unpkg.com, but worth testing

Validated js_on_load Signature

js_on_load: str | None = "element.addEventListener('click', function() { trigger('click') });"

Available in js_on_load context:

• element - The HTML DOM element
• trigger(event_name) - Fire Gradio events
• props - Access component props including props.value

Untested (needs verification):

• Async/await patterns
• Dynamic import() for CDN modules
• Error propagation to Gradio

---

Proposed Solutions (Ranked)

Solution 1: Use js_on_load Parameter (Recommended)

Gradio 6's gr.HTML supports js_on_load for component-level JavaScript (added in PR #12098):

def create_niivue_component(volume_url, mask_url, height=400):
    container_id = f"nv-{uuid.uuid4().hex[:8]}"

    html_content = f'<div id="{container_id}" style="height:{height}px;background:#000;"><canvas></canvas></div>'

    js_code = f"""
        (async () => {{
            try {{
                const {{ Niivue }} = await import('https://unpkg.com/@niivue/niivue@0.65.0/dist/index.js');
                const nv = new Niivue({{ logging: false, backColor: [0,0,0,1] }});
                await nv.attachToCanvas(element.querySelector('canvas'));
                await nv.loadVolumes([{{ url: {json.dumps(volume_url)} }}]);
                nv.setSliceType(nv.sliceTypeMultiplanar);
            }} catch (e) {{
                element.innerHTML = '<div style="color:#fff;padding:20px;">Error: ' + e.message + '</div>';
            }}
        }})();
    """

    return gr.HTML(
        value=html_content,
        js_on_load=js_code,
        label="Interactive 3D Viewer"
    )

Pros: Native Gradio 6 approach, component-scoped Cons: May have issues with dynamic import in js_on_load context

Solution 2: Use head Parameter in launch()

Load NiiVue globally via the head parameter:

# app.py
NIIVUE_HEAD = '''
<script type="module">
    import { Niivue } from 'https://unpkg.com/@niivue/niivue@0.65.0/dist/index.js';
    window.Niivue = Niivue;
</script>
'''

demo.launch(
    head=NIIVUE_HEAD,
    server_name="0.0.0.0",
    server_port=7860
)

Pros: Loads library once, available globally Cons: GitHub Issue #10250 reports unreliable execution

Solution 3: Server-Side File Serving

Instead of base64 data URLs, serve NIfTI files via Gradio's file system:

# Use Gradio's file URL instead of data URLs
from gradio import FileData
file_data = FileData(path=str(dwi_path))
# Pass file_data.url to NiiVue instead of base64

Pros: Avoids 65MB payload, better memory efficiency Cons: Requires refactoring data flow, CORS considerations

Solution 4: Custom Gradio Component

Build a proper gradio_niivue package:

gradio cc create NiiVue --template HTML
# Implement Svelte frontend with NiiVue
# Publish to PyPI

Pros: Most robust, reusable, proper architecture Cons: Significant development effort

Solution 5: Enhanced 2D Fallback (Simplest)

Remove NiiVue entirely, enhance matplotlib visualization:

def create_results_display():
    with gr.Group():
        # Remove: niivue_viewer = gr.HTML(...)

        # Enhanced 2D visualization
        slice_plot = gr.Plot(label="Multi-View Comparison")
        slice_slider = gr.Slider(label="Slice", minimum=0, maximum=100)

        # Add orthogonal views
        with gr.Row():
            axial_plot = gr.Plot(label="Axial")
            coronal_plot = gr.Plot(label="Coronal")
            sagittal_plot = gr.Plot(label="Sagittal")

Pros: Eliminates WebGL complexity, works reliably Cons: Loses 3D interactivity, less impressive demo

---

Investigation Steps

Step 0: Test Async/Await in js_on_load (CRITICAL)

Before implementing Solution 1, verify async works:

import gradio as gr

with gr.Blocks() as demo:
    html = gr.HTML(
        value="<div>Testing async...</div>",
        js_on_load="""
            (async () => {
                element.innerText = 'Async started...';
                await new Promise(r => setTimeout(r, 1000));
                element.innerText = 'Async works!';
                element.style.background = 'green';
            })();
        """
    )

demo.launch()

If this shows "Async works!" with green background after 1 second, async is supported.

Step 1: Verify js_on_load Works (Basic)

Create minimal test:

import gradio as gr

with gr.Blocks() as demo:
    html = gr.HTML(
        value="<div id='test'>Loading...</div>",
        js_on_load="element.style.background='green'; element.innerText='JS Works!';"
    )

demo.launch()

Step 2: Test Dynamic Import in js_on_load

js_on_load="""
    (async () => {
        const mod = await import('https://unpkg.com/@niivue/niivue@0.65.0/dist/index.js');
        console.log('NiiVue loaded:', mod);
        element.innerText = 'Import succeeded!';
    })();
"""

Step 3: Check Browser Console

1. Open HF Spaces demo
2. Open DevTools (F12) → Console
3. Look for errors related to:
   • Module loading failures
   • WebGL context issues
   • CORS errors
   • Memory errors

Step 4: Test with Smaller Files

Create downsampled test NIfTI (~1MB) to isolate size vs JS issues.

---

Related Issues

• Bug #9: DeepISLES modules not found (FIXED - subprocess bridge)
• Bug #8: HF Spaces streaming hang (FIXED)
• Technical Debt: NiiVue memory overhead (P2)
• Gradio #4511: 3D medical image support request (closed, not planned)
• Gradio #10250: JS in head param issues (open)

---

Priority Assessment

Severity: P2 (Medium)

• Core inference pipeline works correctly
• 2D visualization provides adequate fallback
• No data loss or security impact
• Demo is functional for evaluation purposes

Impact:

• Less impressive without 3D viewer
• Users can still evaluate predictions via 2D slices
• Download functionality unaffected

Recommendation:

1. First, validate inference accuracy across multiple cases
2. Then attempt Solution 1 (js_on_load) as quick fix
3. If that fails, implement Solution 5 (enhanced 2D) for reliability
4. Consider Solution 4 (custom component) for future enhancement

---

References

• Gradio HTML Docs
• Gradio Custom HTML Components Guide
• Gradio 6 Migration Guide
• HuggingFace Forum: JS doesn't work in gr.HTML
• GitHub Issue #10250: JS in head param
• GitHub Issue #4511: 3D Medical Images
• NiiVue GitHub
• ipyniivue (Jupyter Widget)
• Gradio 6 Announcement

---

Appendix: HF Spaces Logs

INFO: Running segmentation for sub-stroke0002
INFO: Case sub-stroke0002 ready: DWI=20.9MB, ADC=12.6MB
INFO: DeepISLES subprocess completed in 30.88s

Note: No JavaScript errors visible in server logs (client-side only).