design_docs/splat_rendering.md

# Design Doc: Splat-Aware Export with Per-Point Radii & View-Dependent Color

**Author:** Brian Clark  
**Last Updated:** 2025-11-07  
**Target Components:** Export pipeline (`predictions_to_glb`), downstream visualization stack  
**Goal:** Preserve the rich “Gaussian splat” appearance by exporting per-point radius/orientation/color information and using a splat-aware renderer instead of vanilla GLB point clouds.

---

## 1. Overview

The current GLB export stores XYZ + RGB points. Rendering them as fixed-size pixels loses the soft, blended look achieved by overlapping splats in the original predictions.  
This design introduces a splat representation (Gaussian or disk) with explicit radius and optional anisotropy, and changes the rendering path to honor those attributes.

---

## 2. Requirements

1. **Data export**
   - Include at least:
     - Center (`x,y,z`)
     - Radius (`r`) or covariance matrix (`Σ`)
     - Base color (RGB) and optional view-dependent coefficients (e.g., SH).
   - Allow fallback to disk splats (isotropic) if covariance unavailable.

2. **File format**
   - Options:
     - Extend GLB with custom vertex attributes + custom shader (three.js, deck.gl).
     - Use established Gaussian splat formats (e.g., `.splat`, `.ply` with extra attributes, or `gaussian-splatting` binary).
   - Include metadata describing attribute semantics for the viewer.

3. **Renderer**
   - Web: Three.js shader or deck.gl layer capable of drawing splats.
   - Native: Optionally integrate with existing Gaussian splatting viewers (Instant-NGP, Splatfacto, etc.).

4. **Performance**
   - Maintain reasonable splat counts (prefiltered via voxel reduction & density tests).
   - Provide LOD strategies (e.g., radius-aware culling, multi-resolution export).

---

## 3. Proposed Pipeline

### 3.1 Data preparation
1. Start with voxel-reduced points.
2. Estimate per-point covariance:
   - Option A: Use original support’s covariance (requires storing full neighbor set).
   - Option B: Compute isotropic radius from average neighbor distance (k-NN) or voxel size.
3. Store color as linear RGB; optionally store SH coefficients computed during inference (if available).

### 3.2 Export formats

| Format | Pros | Cons |
| ------ | ---- | ---- |
| **GLB + custom vertex attributes** | Simple integration with existing pipeline | Requires custom shader; not portable |
| **.splat binary (Gaussian Splatting)** | Compatibility with emerging viewers | New dependency; not widely standardized |
| **PLY/NPZ with attributes** | Readable, quick prototyping | Requires loader adaptation |

Initial recommendation: use GLB with custom attributes (e.g., `RADIUS`, `COV3x3`) and provide sample shaders for three.js.

### 3.3 Rendering changes
1. Publish a three.js script that:
   - Loads GLB.
   - Extracts attributes into buffers.
   - Renders via custom shader (screen-space splatting).
2. Provide deck.gl example using `PointCloudLayer` with `radiusPixels` or custom shader module.
3. Eventually support native Gaussian Splat format for plug-and-play compatibility.

---

## 4. API changes

Add new export options:
- `export_mode: {"glb", "splat_glb", "gaussian_binary"}`
- `splat_settings` (radius multiplier, anisotropy toggle, max anisotropy)
- `lod_strategy` (none, subsample, multi-file)

Ensure backward compatibility by keeping current GLB path as default.

---

## 5. Validation & Testing

1. Compare visual output in existing viewer vs. splat viewer (visual parity).
2. Measure load time, FPS for large scenes.
3. Unit tests for:
   - Attribute packing/unpacking.
   - Radius estimation.
   - Export file integrity.
4. Integration test: pipeline -> viewer round-trip (screenshot diff, metrics).

---

## 6. Risks & Mitigations

| Risk | Mitigation |
| ---- | ---------- |
| Custom GLB attributes unsupported by some tools | Provide fallback GLB path, document requirements |
| Increased file size due to extra attributes | Leverage quantization/compression, allow isotropic mode |
| Viewer complexity | Ship reference shader + deck.gl layer; adopt existing open-source splat renderer |
| Lack of covariance data | Start with isotropic radii derived from voxel size |

---

## 7. Timeline (rough)

1. Week 1: Data prep & attribute computation (radius/covariance).
2. Week 2: GLB exporter modifications + tests.
3. Week 3: Viewer shader integration, documentation.
4. Week 4: Optional Gaussian binary export + performance tuning.

---

## 8. Deliverables

1. Updated export pipeline producing splat-aware asset.
2. Viewer example repo (three.js + deck.gl).
3. Documentation covering format, tuning knobs, and integration steps.
4. Automated tests validating attribute correctness & rendering.

---