| # EndoGaussian-4D: Mathematical Formalization |
|
|
| ## Physics-Informed Deformation Field for Real-Time 4D Gaussian Splatting in Endoscopic Surgery |
|
|
| *Week 1 Sprint — Phase 3 Deliverable* |
|
|
| --- |
|
|
| ## 1. Preliminary: 3D Gaussian Splatting |
|
|
| A scene is represented as N anisotropic 3D Gaussians: |
|
|
| $$\mathcal{G} = \{ G_i \}_{i=1}^{N}, \quad G_i = (\boldsymbol{\mu}_i, \mathbf{q}_i, \mathbf{s}_i, \alpha_i, \mathbf{c}_i)$$ |
|
|
| where: |
| - **μᵢ ∈ ℝ³**: Mean (center position) |
| - **qᵢ ∈ ℝ⁴**: Unit quaternion (rotation) |
| - **sᵢ ∈ ℝ³**: Log-scale vector (anisotropic scaling) |
| - **αᵢ ∈ ℝ**: Logit-opacity |
| - **cᵢ ∈ ℝᴷˣ³**: Spherical harmonics coefficients (K = (ℓ+1)² for degree ℓ) |
|
|
| Covariance in world space: |
|
|
| $$\Sigma_i = R(\mathbf{q}_i) \, S(\mathbf{s}_i) \, S(\mathbf{s}_i)^\top \, R(\mathbf{q}_i)^\top$$ |
| |
| Pixel color via α-compositing (front-to-back): |
| |
| $$C(\mathbf{p}) = \sum_{i \in \mathcal{N}} c_i \, \sigma_i \prod_{j=1}^{i-1} (1 - \sigma_j), \quad \sigma_i = \alpha_i \cdot g'_i(\mathbf{p})$$ |
| |
| --- |
| |
| ## 2. Physics-Informed Deformation Field |
| |
| ### 2.1 Core Formulation |
| |
| > **The key equation:** |
| > |
| > $$\boxed{G_i(t) = G_i^{(0)} + \boldsymbol{\Delta}_\theta(\boldsymbol{\mu}_i, t)}$$ |
| |
| where G₀ is the canonical (rest-state) Gaussian and Δ_θ is the learned deformation: |
|
|
| $$\boldsymbol{\Delta}_\theta(\boldsymbol{\mu}_i, t) = \begin{pmatrix} \Delta\boldsymbol{\mu}_i(t) \in \mathbb{R}^3 \\ \Delta\mathbf{q}_i(t) \in \mathbb{R}^4 \\ \Delta\mathbf{s}_i(t) \in \mathbb{R}^3 \\ \Delta\alpha_i(t) \in \mathbb{R} \end{pmatrix}$$ |
|
|
| Deformed parameters: |
|
|
| | Parameter | Equation | |
| |-----------|----------| |
| | Position | μᵢ(t) = μᵢ⁰ + Δμᵢ(t) | |
| | Rotation | qᵢ(t) = normalize(qᵢ⁰ + Δqᵢ(t)) | |
| | Scale | sᵢ(t) = sᵢ⁰ + Δsᵢ(t) | |
| | Opacity | αᵢ(t) = αᵢ⁰ + Δαᵢ(t) | |
|
|
| ### 2.2 HexPlane Spatio-Temporal Encoding |
|
|
| The 4D input (μᵢ, t) ∈ ℝ⁴ is factorized into 6 learnable 2D feature planes: |
|
|
| $$\mathcal{P} = \{(XY), (XZ), (YZ), (XT), (YT), (ZT)\}$$ |
|
|
| Each plane F_l^(d₁,d₂) ∈ ℝ^{C × R_{d₁} × R_{d₂}} stores C-dimensional features. |
| |
| The encoded feature: |
| |
| $$E(\boldsymbol{\mu}_i, t) = \bigoplus_{l=1}^{L} \bigoplus_{(d_1, d_2) \in \mathcal{P}} \text{BilinearSample}\!\left(F_l^{(d_1, d_2)}, [\mu_i^{d_1}, \mu_i^{d_2}]\right)$$ |
| |
| **Memory complexity:** O(6LR²) vs O(R⁴) for dense 4D grid. With R=64 spatial, R_t=75 temporal, L=2 levels, C=32: **≈12 MB** (vs 20M entries dense). |
|
|
| ### 2.3 Deformation Decoder |
|
|
| Shared MLP backbone → 4 output heads (all **zero-initialized**): |
|
|
| ``` |
| h₀ = E(μᵢ, t) # HexPlane features |
| hₖ = ReLU(Wₖ·hₖ₋₁ + bₖ) # K shared layers |
| Δμ = W_μ · hₖ (zero-init) # Position displacement |
| Δq = W_q · hₖ (zero-init) # Rotation perturbation |
| Δs = W_s · hₖ (zero-init) # Scale adjustment |
| Δα = W_α · hₖ (zero-init) # Opacity adjustment |
| ``` |
|
|
| **Zero initialization is critical**: At startup, Δ_θ = 0, so the model begins from canonical Gaussians and gradually learns displacements. This prevents early-training instability. |
| |
| --- |
| |
| ## 3. Physics-Informed Priors |
| |
| ### 3.1 Tissue Biomechanics Motivation |
| |
| Soft surgical tissue (liver, colon, uterus) has physical properties that constrain valid deformations: |
| |
| 1. **Locally smooth**: Tissue displacement fields are continuous and slowly varying in space |
| 2. **Temporally coherent**: Tissue velocity changes smoothly over time |
| 3. **Volume-preserving**: Biological tissue is nearly incompressible |
| 4. **Bounded strain**: Tissue stretches/compresses within physiological limits |
| |
| ### 3.2 Temporal Smoothness Prior |
| |
| $$\mathcal{L}_{\text{smooth}} = \frac{1}{N} \sum_{i=1}^{N} \left\| \boldsymbol{\Delta}_\theta(\boldsymbol{\mu}_i, t) - \boldsymbol{\Delta}_\theta(\boldsymbol{\mu}_i, t + \delta) \right\|_2^2$$ |
|
|
| **Physical interpretation:** Approximates a penalty on tissue acceleration: |
|
|
| $$\mathcal{L}_{\text{smooth}} \approx \delta^2 \cdot \frac{1}{N} \sum_{i=1}^{N} \left\| \frac{\partial \boldsymbol{\Delta}_\theta}{\partial t}(\boldsymbol{\mu}_i, t) \right\|_2^2$$ |
| |
| This is Tikhonov regularization on the velocity field — standard in biomechanical simulation. |
| |
| ### 3.3 Total Variation on HexPlane Features |
| |
| $$\mathcal{L}_{\text{TV}} = \frac{1}{6L} \sum_{l=1}^{L} \sum_{(d_1,d_2) \in \mathcal{P}} \left( \left\| \nabla_{h} F_l^{(d_1,d_2)} \right\|_1 + \left\| \nabla_{v} F_l^{(d_1,d_2)} \right\|_1 \right)$$ |
|
|
| **Physical interpretation:** TV on temporal planes (XT, YT, ZT) enforces that nearby Gaussians undergo similar deformations — encoding that tissue is a continuum, not independent particles. |
|
|
| --- |
|
|
| ## 4. Complete Loss Function |
|
|
| ### 4.1 Full Objective |
|
|
| $$\boxed{\mathcal{L} = \underbrace{(1-\lambda_1)\mathcal{L}_1 + \lambda_1 \mathcal{L}_{\text{D-SSIM}}}_{\text{appearance}} + \underbrace{\lambda_2 \mathcal{L}_{\text{depth}}}_{\text{geometry}} + \underbrace{\lambda_3 \mathcal{L}_{\text{smooth}} + \lambda_4 \mathcal{L}_{\text{TV}}}_{\text{regularization}}}$$ |
| |
| **Hyperparameters:** λ₁ = 0.2, λ₂ = 0.1, λ₃ = 0.01, λ₄ = 0.001 |
| |
| ### 4.2 Appearance Loss Terms |
| |
| **L1 Photometric (tool-masked):** |
| $$\mathcal{L}_1 = \frac{1}{|\mathcal{V}_t|} \sum_{\mathbf{p} \in \mathcal{V}_t} \left| \hat{I}(\mathbf{p}) - I^*(\mathbf{p}) \right|$$ |
| |
| **D-SSIM:** |
| $$\mathcal{L}_{\text{D-SSIM}} = \frac{1 - \text{SSIM}(\hat{I} \odot M_t, \; I^* \odot M_t)}{2}$$ |
|
|
| ### 4.3 Scale-Invariant Depth Loss |
|
|
| $$\mathcal{L}_{\text{depth}} = \frac{1}{|\mathcal{V}|} \sum_{\mathbf{p} \in \mathcal{V}} d_\mathbf{p}^2 - \frac{0.5}{|\mathcal{V}|^2} \left( \sum_{\mathbf{p} \in \mathcal{V}} d_\mathbf{p} \right)^2$$ |
| |
| where d_p = log D̂(p) - log D*(p). Scale-invariant formulation handles both known-scale (C3VD structured-light) and unknown-scale (Depth-Anything monocular) depth. |
| |
| ### 4.4 Tool Occlusion Handling |
| |
| **Three-stage masking:** |
| |
| 1. **Initialization**: Tools excluded from HGI point cloud |
| $$P = \bigcup_{t} K^{-1} T_t D_t (I_t \odot M_t)$$ |
| |
| 2. **Loss**: All terms computed on tissue pixels only (V_t = {p : M_t(p) = 1}) |
| |
| 3. **Densification**: Gradient signals from tool boundaries excluded from split/clone decisions |
| |
| --- |
| |
| ## 5. Holistic Gaussian Initialization (HGI) |
| |
| $$\boxed{P = \text{Subsample}_{0.1\%}\!\left( \bigcup_{t=1}^{T} \Pi_t^{-1}(D_t, M_t) \right)}$$ |
| |
| Backprojection operator: |
| $$\Pi_t^{-1}(D_t, M_t) = \left\{ T_t \cdot K^{-1} \begin{pmatrix} u \\ v \\ 1 \end{pmatrix} D_t(u,v) \;\middle|\; M_t(u,v) = 1, \; D_t(u,v) > 0 \right\}$$ |
| |
| **Scale initialization from KNN:** |
| $$s_i^{(0)} = \log\!\left( \frac{1}{k} \sum_{j \in \text{kNN}(i)} \|\boldsymbol{\mu}_i - \boldsymbol{\mu}_j\| \right), \quad k=3$$ |
| |
| --- |
| |
| ## 6. Training Schedule |
| |
| | Phase | Iterations | What's Optimized | Density Control | |
| |-------|-----------|------------------|-----------------| |
| | Warmup | 0 → 1000 | Canonical Gaussians only | Active (500-1000) | |
| | Deformation | 1000 → 3000 | Gaussians + HexPlane + Decoder | Active (1000-2500) | |
| |
| **Learning rates (Adam, exponential decay γ = 0.01^{1/3000}):** |
| |
| | Parameter | lr₀ | |
| |-----------|-----| |
| | Means μ | 1.6×10⁻⁴ | |
| | Scales s | 5×10⁻³ | |
| | Quaternions q | 1×10⁻³ | |
| | Opacities α | 5×10⁻² | |
| | SH coefficients | 2.5×10⁻³ | |
| | Deformation θ | 1.6×10⁻³ | |
| |
| --- |
| |
| ## 7. Adaptive Density Control |
| |
| **absgrad-based** (absolute gradient values, not norms): |
| |
| $$\bar{g}_i = \frac{1}{|T_{\text{vis}}(i)|} \sum_{t \in T_{\text{vis}}(i)} \max_d \left| \frac{\partial \mathcal{L}}{\partial \mu'^{(d)}_i} \right|$$ |
| |
| | Operation | Condition | Action | |
| |-----------|-----------|--------| |
| | **Split** | ḡᵢ > 0.0002 AND max scale > 0.01 | Split into 2, shrink by log(1.6) | |
| | **Clone** | ḡᵢ > 0.0002 AND max scale ≤ 0.01 | Duplicate at same position | |
| | **Prune** | σ(αᵢ) < 0.005 OR max scale > 0.1 | Remove Gaussian | |
| |
| --- |
| |
| ## 8. Expected Performance |
| |
| | Method | PSNR ↑ | SSIM ↑ | FPS | Train Time | |
| |--------|--------|--------|-----|------------| |
| | Static 3DGS | 30-32 | 0.90 | 200+ | 2 min | |
| | EndoNeRF | 35.8 | 0.96 | 0.2 | 12 hr | |
| | Endo-4DGS | 36.6 | 0.96 | 100 | 4-7 min | |
| | **EndoGaussian (target)** | **37.9** | **0.97** | **195** | **2 min** | |
| |
| --- |
| |
| ## References |
| |
| 1. Kerbl et al. "3D Gaussian Splatting for Real-Time Radiance Field Rendering." SIGGRAPH 2023. |
| 2. Liu et al. "EndoGaussian: Real-time Gaussian Splatting for Dynamic Endoscopic Scene Reconstruction." arXiv:2401.12561, 2024. |
| 3. Huang et al. "Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting." MICCAI 2024. |
| 4. Cao & Johnson. "HexPlane: A Fast Representation for Dynamic Scenes." CVPR 2023. |
| 5. Eigen et al. "Depth Map Prediction from a Single Image using a Multi-Scale Deep Network." NeurIPS 2014. |
| 6. Wang et al. "Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery." MICCAI 2022. |
| 7. Ye et al. "gsplat: An Open-Source Library for Gaussian Splatting." arXiv:2409.06765, 2024. |
| 8. Yang et al. "Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data." CVPR 2024. |
| |
