docs/formalization.md

# EndoGaussian-4D: Mathematical Formalization

## Physics-Informed Deformation Field for Real-Time 4D Gaussian Splatting in Endoscopic Surgery

*Week 1 Sprint — Phase 3 Deliverable*

---

## 1. Preliminary: 3D Gaussian Splatting

A scene is represented as N anisotropic 3D Gaussians:

$$\mathcal{G} = \{ G_i \}_{i=1}^{N}, \quad G_i = (\boldsymbol{\mu}_i, \mathbf{q}_i, \mathbf{s}_i, \alpha_i, \mathbf{c}_i)$$

where:
- **μᵢ ∈ ℝ³**: Mean (center position)
- **qᵢ ∈ ℝ⁴**: Unit quaternion (rotation)
- **sᵢ ∈ ℝ³**: Log-scale vector (anisotropic scaling)
- **αᵢ ∈ ℝ**: Logit-opacity
- **cᵢ ∈ ℝᴷˣ³**: Spherical harmonics coefficients (K = (ℓ+1)² for degree ℓ)

Covariance in world space:

$$\Sigma_i = R(\mathbf{q}_i) \, S(\mathbf{s}_i) \, S(\mathbf{s}_i)^\top \, R(\mathbf{q}_i)^\top$$

Pixel color via α-compositing (front-to-back):

$$C(\mathbf{p}) = \sum_{i \in \mathcal{N}} c_i \, \sigma_i \prod_{j=1}^{i-1} (1 - \sigma_j), \quad \sigma_i = \alpha_i \cdot g'_i(\mathbf{p})$$

---

## 2. Physics-Informed Deformation Field

### 2.1 Core Formulation

> **The key equation:**
>
> $$\boxed{G_i(t) = G_i^{(0)} + \boldsymbol{\Delta}_\theta(\boldsymbol{\mu}_i, t)}$$

where G₀ is the canonical (rest-state) Gaussian and Δ_θ is the learned deformation:

$$\boldsymbol{\Delta}_\theta(\boldsymbol{\mu}_i, t) = \begin{pmatrix} \Delta\boldsymbol{\mu}_i(t) \in \mathbb{R}^3 \\ \Delta\mathbf{q}_i(t) \in \mathbb{R}^4 \\ \Delta\mathbf{s}_i(t) \in \mathbb{R}^3 \\ \Delta\alpha_i(t) \in \mathbb{R} \end{pmatrix}$$

Deformed parameters:

| Parameter | Equation |
|-----------|----------|
| Position | μᵢ(t) = μᵢ⁰ + Δμᵢ(t) |
| Rotation | qᵢ(t) = normalize(qᵢ⁰ + Δqᵢ(t)) |
| Scale | sᵢ(t) = sᵢ⁰ + Δsᵢ(t) |
| Opacity | αᵢ(t) = αᵢ⁰ + Δαᵢ(t) |

### 2.2 HexPlane Spatio-Temporal Encoding

The 4D input (μᵢ, t) ∈ ℝ⁴ is factorized into 6 learnable 2D feature planes:

$$\mathcal{P} = \{(XY), (XZ), (YZ), (XT), (YT), (ZT)\}$$

Each plane F_l^(d₁,d₂) ∈ ℝ^{C × R_{d₁} × R_{d₂}} stores C-dimensional features.

The encoded feature:

$$E(\boldsymbol{\mu}_i, t) = \bigoplus_{l=1}^{L} \bigoplus_{(d_1, d_2) \in \mathcal{P}} \text{BilinearSample}\!\left(F_l^{(d_1, d_2)}, [\mu_i^{d_1}, \mu_i^{d_2}]\right)$$

**Memory complexity:** O(6LR²) vs O(R⁴) for dense 4D grid. With R=64 spatial, R_t=75 temporal, L=2 levels, C=32: **≈12 MB** (vs 20M entries dense).

### 2.3 Deformation Decoder

Shared MLP backbone → 4 output heads (all **zero-initialized**):

```
h₀ = E(μᵢ, t)                        # HexPlane features
hₖ = ReLU(Wₖ·hₖ₋₁ + bₖ)             # K shared layers
Δμ = W_μ · hₖ    (zero-init)          # Position displacement
Δq = W_q · hₖ    (zero-init)          # Rotation perturbation
Δs = W_s · hₖ    (zero-init)          # Scale adjustment
Δα = W_α · hₖ    (zero-init)          # Opacity adjustment
```

**Zero initialization is critical**: At startup, Δ_θ = 0, so the model begins from canonical Gaussians and gradually learns displacements. This prevents early-training instability.

---

## 3. Physics-Informed Priors

### 3.1 Tissue Biomechanics Motivation

Soft surgical tissue (liver, colon, uterus) has physical properties that constrain valid deformations:

1. **Locally smooth**: Tissue displacement fields are continuous and slowly varying in space
2. **Temporally coherent**: Tissue velocity changes smoothly over time
3. **Volume-preserving**: Biological tissue is nearly incompressible
4. **Bounded strain**: Tissue stretches/compresses within physiological limits

### 3.2 Temporal Smoothness Prior

$$\mathcal{L}_{\text{smooth}} = \frac{1}{N} \sum_{i=1}^{N} \left\| \boldsymbol{\Delta}_\theta(\boldsymbol{\mu}_i, t) - \boldsymbol{\Delta}_\theta(\boldsymbol{\mu}_i, t + \delta) \right\|_2^2$$

**Physical interpretation:** Approximates a penalty on tissue acceleration:

$$\mathcal{L}_{\text{smooth}} \approx \delta^2 \cdot \frac{1}{N} \sum_{i=1}^{N} \left\| \frac{\partial \boldsymbol{\Delta}_\theta}{\partial t}(\boldsymbol{\mu}_i, t) \right\|_2^2$$

This is Tikhonov regularization on the velocity field — standard in biomechanical simulation.

### 3.3 Total Variation on HexPlane Features

$$\mathcal{L}_{\text{TV}} = \frac{1}{6L} \sum_{l=1}^{L} \sum_{(d_1,d_2) \in \mathcal{P}} \left( \left\| \nabla_{h} F_l^{(d_1,d_2)} \right\|_1 + \left\| \nabla_{v} F_l^{(d_1,d_2)} \right\|_1 \right)$$

**Physical interpretation:** TV on temporal planes (XT, YT, ZT) enforces that nearby Gaussians undergo similar deformations — encoding that tissue is a continuum, not independent particles.

---

## 4. Complete Loss Function

### 4.1 Full Objective

$$\boxed{\mathcal{L} = \underbrace{(1-\lambda_1)\mathcal{L}_1 + \lambda_1 \mathcal{L}_{\text{D-SSIM}}}_{\text{appearance}} + \underbrace{\lambda_2 \mathcal{L}_{\text{depth}}}_{\text{geometry}} + \underbrace{\lambda_3 \mathcal{L}_{\text{smooth}} + \lambda_4 \mathcal{L}_{\text{TV}}}_{\text{regularization}}}$$

**Hyperparameters:** λ₁ = 0.2, λ₂ = 0.1, λ₃ = 0.01, λ₄ = 0.001

### 4.2 Appearance Loss Terms

**L1 Photometric (tool-masked):**
$$\mathcal{L}_1 = \frac{1}{|\mathcal{V}_t|} \sum_{\mathbf{p} \in \mathcal{V}_t} \left| \hat{I}(\mathbf{p}) - I^*(\mathbf{p}) \right|$$

**D-SSIM:**
$$\mathcal{L}_{\text{D-SSIM}} = \frac{1 - \text{SSIM}(\hat{I} \odot M_t, \; I^* \odot M_t)}{2}$$

### 4.3 Scale-Invariant Depth Loss

$$\mathcal{L}_{\text{depth}} = \frac{1}{|\mathcal{V}|} \sum_{\mathbf{p} \in \mathcal{V}} d_\mathbf{p}^2 - \frac{0.5}{|\mathcal{V}|^2} \left( \sum_{\mathbf{p} \in \mathcal{V}} d_\mathbf{p} \right)^2$$

where d_p = log D̂(p) - log D*(p). Scale-invariant formulation handles both known-scale (C3VD structured-light) and unknown-scale (Depth-Anything monocular) depth.

### 4.4 Tool Occlusion Handling

**Three-stage masking:**

1. **Initialization**: Tools excluded from HGI point cloud
   $$P = \bigcup_{t} K^{-1} T_t D_t (I_t \odot M_t)$$

2. **Loss**: All terms computed on tissue pixels only (V_t = {p : M_t(p) = 1})

3. **Densification**: Gradient signals from tool boundaries excluded from split/clone decisions

---

## 5. Holistic Gaussian Initialization (HGI)

$$\boxed{P = \text{Subsample}_{0.1\%}\!\left( \bigcup_{t=1}^{T} \Pi_t^{-1}(D_t, M_t) \right)}$$

Backprojection operator:
$$\Pi_t^{-1}(D_t, M_t) = \left\{ T_t \cdot K^{-1} \begin{pmatrix} u \\ v \\ 1 \end{pmatrix} D_t(u,v) \;\middle|\; M_t(u,v) = 1, \; D_t(u,v) > 0 \right\}$$

**Scale initialization from KNN:**
$$s_i^{(0)} = \log\!\left( \frac{1}{k} \sum_{j \in \text{kNN}(i)} \|\boldsymbol{\mu}_i - \boldsymbol{\mu}_j\| \right), \quad k=3$$

---

## 6. Training Schedule

| Phase | Iterations | What's Optimized | Density Control |
|-------|-----------|------------------|-----------------|
| Warmup | 0 → 1000 | Canonical Gaussians only | Active (500-1000) |
| Deformation | 1000 → 3000 | Gaussians + HexPlane + Decoder | Active (1000-2500) |

**Learning rates (Adam, exponential decay γ = 0.01^{1/3000}):**

| Parameter | lr₀ |
|-----------|-----|
| Means μ | 1.6×10⁻⁴ |
| Scales s | 5×10⁻³ |
| Quaternions q | 1×10⁻³ |
| Opacities α | 5×10⁻² |
| SH coefficients | 2.5×10⁻³ |
| Deformation θ | 1.6×10⁻³ |

---

## 7. Adaptive Density Control

**absgrad-based** (absolute gradient values, not norms):

$$\bar{g}_i = \frac{1}{|T_{\text{vis}}(i)|} \sum_{t \in T_{\text{vis}}(i)} \max_d \left| \frac{\partial \mathcal{L}}{\partial \mu'^{(d)}_i} \right|$$

| Operation | Condition | Action |
|-----------|-----------|--------|
| **Split** | ḡᵢ > 0.0002 AND max scale > 0.01 | Split into 2, shrink by log(1.6) |
| **Clone** | ḡᵢ > 0.0002 AND max scale ≤ 0.01 | Duplicate at same position |
| **Prune** | σ(αᵢ) < 0.005 OR max scale > 0.1 | Remove Gaussian |

---

## 8. Expected Performance

| Method | PSNR ↑ | SSIM ↑ | FPS | Train Time |
|--------|--------|--------|-----|------------|
| Static 3DGS | 30-32 | 0.90 | 200+ | 2 min |
| EndoNeRF | 35.8 | 0.96 | 0.2 | 12 hr |
| Endo-4DGS | 36.6 | 0.96 | 100 | 4-7 min |
| **EndoGaussian (target)** | **37.9** | **0.97** | **195** | **2 min** |

---

## References

1. Kerbl et al. "3D Gaussian Splatting for Real-Time Radiance Field Rendering." SIGGRAPH 2023.
2. Liu et al. "EndoGaussian: Real-time Gaussian Splatting for Dynamic Endoscopic Scene Reconstruction." arXiv:2401.12561, 2024.
3. Huang et al. "Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting." MICCAI 2024.
4. Cao & Johnson. "HexPlane: A Fast Representation for Dynamic Scenes." CVPR 2023.
5. Eigen et al. "Depth Map Prediction from a Single Image using a Multi-Scale Deep Network." NeurIPS 2014.
6. Wang et al. "Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery." MICCAI 2022.
7. Ye et al. "gsplat: An Open-Source Library for Gaussian Splatting." arXiv:2409.06765, 2024.
8. Yang et al. "Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data." CVPR 2024.