docs/formalization.md

EndoGaussian-4D: Mathematical Formalization

Physics-Informed Deformation Field for Real-Time 4D Gaussian Splatting in Endoscopic Surgery

Week 1 Sprint — Phase 3 Deliverable

---

1. Preliminary: 3D Gaussian Splatting

A scene is represented as N anisotropic 3D Gaussians:

$$\mathcal{G}=\{G_i{\}}_{i=1}^N,G_i=({\mathit{\mu }}_i,{\mathbf{q}}_i,{\mathbf{s}}_i,{\alpha }_i,{\mathbf{c}}_i)\backslash mathcal\{G\}\; =\; \backslash \{\; G\_i\; \backslash \}\_\{i=1\}^\{N\},\; \backslash quad\; G\_i\; =\; (\backslash boldsymbol\{\backslash mu\}\_i,\; \backslash mathbf\{q\}\_i,\; \backslash mathbf\{s\}\_i,\; \backslash alpha\_i,\; \backslash mathbf\{c\}\_i)$$

where:

• μᵢ ∈ ℝ³: Mean (center position)
• qᵢ ∈ ℝ⁴: Unit quaternion (rotation)
• sᵢ ∈ ℝ³: Log-scale vector (anisotropic scaling)
• αᵢ ∈ ℝ: Logit-opacity
• cᵢ ∈ ℝᴷˣ³: Spherical harmonics coefficients (K = (ℓ+1)² for degree ℓ)

Covariance in world space:

$${\mathrm{\Sigma }}_i=R({\mathbf{q}}_i)S({\mathbf{s}}_i)S({\mathbf{s}}_i{)}^{\mathrm{\top }}R({\mathbf{q}}_i{)}^{\mathrm{\top }}\backslash Sigma\_i\; =\; R(\backslash mathbf\{q\}\_i)\; \backslash ,\; S(\backslash mathbf\{s\}\_i)\; \backslash ,\; S(\backslash mathbf\{s\}\_i)^\backslash top\; \backslash ,\; R(\backslash mathbf\{q\}\_i)^\backslash top$$

Pixel color via α-compositing (front-to-back):

$$C(\mathbf{p})=\sum _{i\in \mathcal{N}}{c}_i{\sigma }_i\prod _{j=1}^{i-1}(1-{\sigma }_j),{\sigma }_i={\alpha }_i\cdot g_i^{\mathrm{\prime }}(\mathbf{p})C(\backslash mathbf\{p\})\; =\; \backslash sum\_\{i\; \backslash in\; \backslash mathcal\{N\}\}\; c\_i\; \backslash ,\; \backslash sigma\_i\; \backslash prod\_\{j=1\}^\{i-1\}\; (1\; -\; \backslash sigma\_j),\; \backslash quad\; \backslash sigma\_i\; =\; \backslash alpha\_i\; \backslash cdot\; g\text{'}\_i(\backslash mathbf\{p\})$$

---

2. Physics-Informed Deformation Field

2.1 Core Formulation

The key equation:

$$\boxed{G_i(t) = G_i^{(0)} + \boldsymbol{\Delta}_\theta(\boldsymbol{\mu}_i, t)}$$

where G₀ is the canonical (rest-state) Gaussian and Δ_θ is the learned deformation:

$${\mathbf{\Delta }}_{\theta }({\mathit{\mu }}_i,t)=\left(\begin{array}{c}\mathrm{\Delta }{\mathit{\mu }}_i(t)\in {\mathbb{R}}^3\\ \mathrm{\Delta }{\mathbf{q}}_i(t)\in {\mathbb{R}}^4\\ \mathrm{\Delta }{\mathbf{s}}_i(t)\in {\mathbb{R}}^3\\ \mathrm{\Delta }{\alpha }_i(t)\in \mathbb{R}\end{array}\right)\backslash boldsymbol\{\backslash Delta\}\_\backslash theta(\backslash boldsymbol\{\backslash mu\}\_i,\; t)\; =\; \backslash begin\{pmatrix\}\; \backslash Delta\backslash boldsymbol\{\backslash mu\}\_i(t)\; \backslash in\; \backslash mathbb\{R\}^3\; \backslash \backslash \; \backslash Delta\backslash mathbf\{q\}\_i(t)\; \backslash in\; \backslash mathbb\{R\}^4\; \backslash \backslash \; \backslash Delta\backslash mathbf\{s\}\_i(t)\; \backslash in\; \backslash mathbb\{R\}^3\; \backslash \backslash \; \backslash Delta\backslash alpha\_i(t)\; \backslash in\; \backslash mathbb\{R\}\; \backslash end\{pmatrix\}$$

Deformed parameters:

Parameter	Equation
Position	μᵢ(t) = μᵢ⁰ + Δμᵢ(t)
Rotation	qᵢ(t) = normalize(qᵢ⁰ + Δqᵢ(t))
Scale	sᵢ(t) = sᵢ⁰ + Δsᵢ(t)
Opacity	αᵢ(t) = αᵢ⁰ + Δαᵢ(t)

2.2 HexPlane Spatio-Temporal Encoding

The 4D input (μᵢ, t) ∈ ℝ⁴ is factorized into 6 learnable 2D feature planes:

$$\mathcal{P}=\{(XY),(XZ),(YZ),(XT),(YT),(ZT)\}\backslash mathcal\{P\}\; =\; \backslash \{(XY),\; (XZ),\; (YZ),\; (XT),\; (YT),\; (ZT)\backslash \}$$

Each plane F_l^(d₁,d₂) ∈ ℝ^{C × R_{d₁} × R_{d₂}} stores C-dimensional features.

The encoded feature:

$$E({\mathit{\mu }}_i,t)=\underset{l=1}{\overset{L}{⨁}}\underset{(d_1,d_2)\in \mathcal{P}}{⨁}\text{BilinearSample}\text{\hspace{0.17em}⁣}(F_l^{(d_1,d_2)},[{\mu }_i^{d_1},{\mu }_i^{d_2}])E(\backslash boldsymbol\{\backslash mu\}\_i,\; t)\; =\; \backslash bigoplus\_\{l=1\}^\{L\}\; \backslash bigoplus\_\{(d\_1,\; d\_2)\; \backslash in\; \backslash mathcal\{P\}\}\; \backslash text\{BilinearSample\}\backslash !\backslash left(F\_l^\{(d\_1,\; d\_2)\},\; [\backslash mu\_i^\{d\_1\},\; \backslash mu\_i^\{d\_2\}]\backslash right)$$

Memory complexity: O(6LR²) vs O(R⁴) for dense 4D grid. With R=64 spatial, R_t=75 temporal, L=2 levels, C=32: ≈12 MB (vs 20M entries dense).

2.3 Deformation Decoder

Shared MLP backbone → 4 output heads (all zero-initialized):

h₀ = E(μᵢ, t)                        # HexPlane features
hₖ = ReLU(Wₖ·hₖ₋₁ + bₖ)             # K shared layers
Δμ = W_μ · hₖ    (zero-init)          # Position displacement
Δq = W_q · hₖ    (zero-init)          # Rotation perturbation
Δs = W_s · hₖ    (zero-init)          # Scale adjustment
Δα = W_α · hₖ    (zero-init)          # Opacity adjustment

Zero initialization is critical: At startup, Δ_θ = 0, so the model begins from canonical Gaussians and gradually learns displacements. This prevents early-training instability.

---

3. Physics-Informed Priors

3.1 Tissue Biomechanics Motivation

Soft surgical tissue (liver, colon, uterus) has physical properties that constrain valid deformations:

1. Locally smooth: Tissue displacement fields are continuous and slowly varying in space
2. Temporally coherent: Tissue velocity changes smoothly over time
3. Volume-preserving: Biological tissue is nearly incompressible
4. Bounded strain: Tissue stretches/compresses within physiological limits

3.2 Temporal Smoothness Prior

$${\mathcal{L}}_{\text{smooth}}=\frac{1}{N}\sum _{i=1}^N{\parallel {\mathbf{\Delta }}_{\theta }({\mathit{\mu }}_i,t)-{\mathbf{\Delta }}_{\theta }({\mathit{\mu }}_i,t+\delta )\parallel }_2^2\backslash mathcal\{L\}\_\{\backslash text\{smooth\}\}\; =\; \backslash frac\{1\}\{N\}\; \backslash sum\_\{i=1\}^\{N\}\; \backslash left\backslash |\; \backslash boldsymbol\{\backslash Delta\}\_\backslash theta(\backslash boldsymbol\{\backslash mu\}\_i,\; t)\; -\; \backslash boldsymbol\{\backslash Delta\}\_\backslash theta(\backslash boldsymbol\{\backslash mu\}\_i,\; t\; +\; \backslash delta)\; \backslash right\backslash |\_2^2$$

Physical interpretation: Approximates a penalty on tissue acceleration:

$${\mathcal{L}}_{\text{smooth}}\approx {\delta }^2\cdot \frac{1}{N}\sum _{i=1}^N{\parallel \frac{\mathrm{\partial }{\mathbf{\Delta }}_{\theta }}{\mathrm{\partial }t}({\mathit{\mu }}_i,t)\parallel }_2^2\backslash mathcal\{L\}\_\{\backslash text\{smooth\}\}\; \backslash approx\; \backslash delta^2\; \backslash cdot\; \backslash frac\{1\}\{N\}\; \backslash sum\_\{i=1\}^\{N\}\; \backslash left\backslash |\; \backslash frac\{\backslash partial\; \backslash boldsymbol\{\backslash Delta\}\_\backslash theta\}\{\backslash partial\; t\}(\backslash boldsymbol\{\backslash mu\}\_i,\; t)\; \backslash right\backslash |\_2^2$$

This is Tikhonov regularization on the velocity field — standard in biomechanical simulation.

3.3 Total Variation on HexPlane Features

$${\mathcal{L}}_{\text{TV}}=\frac{1}{6L}\sum _{l=1}^L\sum _{(d_1,d_2)\in \mathcal{P}}({\parallel {\mathrm{\nabla }}_h{F}_l^{(d_1,d_2)}\parallel }_1+{\parallel {\mathrm{\nabla }}_v{F}_l^{(d_1,d_2)}\parallel }_1)\backslash mathcal\{L\}\_\{\backslash text\{TV\}\}\; =\; \backslash frac\{1\}\{6L\}\; \backslash sum\_\{l=1\}^\{L\}\; \backslash sum\_\{(d\_1,d\_2)\; \backslash in\; \backslash mathcal\{P\}\}\; \backslash left(\; \backslash left\backslash |\; \backslash nabla\_\{h\}\; F\_l^\{(d\_1,d\_2)\}\; \backslash right\backslash |\_1\; +\; \backslash left\backslash |\; \backslash nabla\_\{v\}\; F\_l^\{(d\_1,d\_2)\}\; \backslash right\backslash |\_1\; \backslash right)$$

Physical interpretation: TV on temporal planes (XT, YT, ZT) enforces that nearby Gaussians undergo similar deformations — encoding that tissue is a continuum, not independent particles.

---

4. Complete Loss Function

4.1 Full Objective

$$\boxed{{\displaystyle \mathcal{L}=\underset{\text{appearance}}{\underbrace{(1-{\lambda }_1){\mathcal{L}}_1+{\lambda }_1{\mathcal{L}}_{\text{D-SSIM}}}}+\underset{\text{geometry}}{\underbrace{{\lambda }_2{\mathcal{L}}_{\text{depth}}}}+\underset{\text{regularization}}{\underbrace{{\lambda }_3{\mathcal{L}}_{\text{smooth}}+{\lambda }_4{\mathcal{L}}_{\text{TV}}}}}}\backslash boxed\{\backslash mathcal\{L\}\; =\; \backslash underbrace\{(1-\backslash lambda\_1)\backslash mathcal\{L\}\_1\; +\; \backslash lambda\_1\; \backslash mathcal\{L\}\_\{\backslash text\{D-SSIM\}\}\}\_\{\backslash text\{appearance\}\}\; +\; \backslash underbrace\{\backslash lambda\_2\; \backslash mathcal\{L\}\_\{\backslash text\{depth\}\}\}\_\{\backslash text\{geometry\}\}\; +\; \backslash underbrace\{\backslash lambda\_3\; \backslash mathcal\{L\}\_\{\backslash text\{smooth\}\}\; +\; \backslash lambda\_4\; \backslash mathcal\{L\}\_\{\backslash text\{TV\}\}\}\_\{\backslash text\{regularization\}\}\}$$

Hyperparameters: λ₁ = 0.2, λ₂ = 0.1, λ₃ = 0.01, λ₄ = 0.001

4.2 Appearance Loss Terms

L1 Photometric (tool-masked): $${\mathcal{L}}_1=\frac{1}{\mathrm{\mid }{\mathcal{V}}_t\mathrm{\mid }}\sum _{\mathbf{p}\in {\mathcal{V}}_t}\mid \widehat{I}(\mathbf{p})-I^{\ast }(\mathbf{p})\mid \backslash mathcal\{L\}\_1\; =\; \backslash frac\{1\}\{|\backslash mathcal\{V\}\_t|\}\; \backslash sum\_\{\backslash mathbf\{p\}\; \backslash in\; \backslash mathcal\{V\}\_t\}\; \backslash left|\; \backslash hat\{I\}(\backslash mathbf\{p\})\; -\; I^*(\backslash mathbf\{p\})\; \backslash right|$$

D-SSIM: $${\mathcal{L}}_{\text{D-SSIM}}=\frac{1-\text{SSIM}(\widehat{I}\odot M_t,I^{\ast }\odot M_t)}{2}\backslash mathcal\{L\}\_\{\backslash text\{D-SSIM\}\}\; =\; \backslash frac\{1\; -\; \backslash text\{SSIM\}(\backslash hat\{I\}\; \backslash odot\; M\_t,\; \backslash ;\; I^*\; \backslash odot\; M\_t)\}\{2\}$$

4.3 Scale-Invariant Depth Loss

$${\mathcal{L}}_{\text{depth}}=\frac{1}{\mathrm{\mid }\mathcal{V}\mathrm{\mid }}\sum _{\mathbf{p}\in \mathcal{V}}{d}_{\mathbf{p}}^2-\frac{0.5}{\mathrm{\mid }\mathcal{V}{\mathrm{\mid }}^2}{\left(\sum _{\mathbf{p}\in \mathcal{V}}{d}_{\mathbf{p}}\right)}^2\backslash mathcal\{L\}\_\{\backslash text\{depth\}\}\; =\; \backslash frac\{1\}\{|\backslash mathcal\{V\}|\}\; \backslash sum\_\{\backslash mathbf\{p\}\; \backslash in\; \backslash mathcal\{V\}\}\; d\_\backslash mathbf\{p\}^2\; -\; \backslash frac\{0.5\}\{|\backslash mathcal\{V\}|^2\}\; \backslash left(\; \backslash sum\_\{\backslash mathbf\{p\}\; \backslash in\; \backslash mathcal\{V\}\}\; d\_\backslash mathbf\{p\}\; \backslash right)^2$$

where d_p = log D̂(p) - log D*(p). Scale-invariant formulation handles both known-scale (C3VD structured-light) and unknown-scale (Depth-Anything monocular) depth.

4.4 Tool Occlusion Handling

Three-stage masking:

1. Initialization: Tools excluded from HGI point cloud $$P = \bigcup_{t} K^{-1} T_t D_t (I_t \odot M_t)$$

2. Loss: All terms computed on tissue pixels only (V_t = {p : M_t(p) = 1})

3. Densification: Gradient signals from tool boundaries excluded from split/clone decisions

---

5. Holistic Gaussian Initialization (HGI)

$$\boxed{{\displaystyle P={\text{Subsample}}_{0.1\mathrm{\%}}\text{\hspace{0.17em}⁣}(\bigcup _{t=1}^T{\mathrm{\Pi }}_t^{-1}(D_t,M_t))}}\backslash boxed\{P\; =\; \backslash text\{Subsample\}\_\{0.1\backslash \%\}\backslash !\backslash left(\; \backslash bigcup\_\{t=1\}^\{T\}\; \backslash Pi\_t^\{-1\}(D\_t,\; M\_t)\; \backslash right)\}$$

Backprojection operator: $${\mathrm{\Pi }}_t^{-1}(D_t,M_t)=\{T_t\cdot K^{-1}\left(\begin{array}{c}u\\ v\\ 1\end{array}\right)D_t(u,v)|M_t(u,v)=1,D_t(u,v)>0\}\backslash Pi\_t^\{-1\}(D\_t,\; M\_t)\; =\; \backslash left\backslash \{\; T\_t\; \backslash cdot\; K^\{-1\}\; \backslash begin\{pmatrix\}\; u\; \backslash \backslash \; v\; \backslash \backslash \; 1\; \backslash end\{pmatrix\}\; D\_t(u,v)\; \backslash ;\backslash middle|\backslash ;\; M\_t(u,v)\; =\; 1,\; \backslash ;\; D\_t(u,v)\; >\; 0\; \backslash right\backslash \}$$

Scale initialization from KNN: $$s_i^{(0)}=\log \text{\hspace{0.17em}⁣}(\frac{1}{k}\sum _{j\in \text{kNN}(i)}\mathrm{\parallel }{\mathit{\mu }}_i-{\mathit{\mu }}_j\mathrm{\parallel }),k=3s\_i^\{(0)\}\; =\; \backslash log\backslash !\backslash left(\; \backslash frac\{1\}\{k\}\; \backslash sum\_\{j\; \backslash in\; \backslash text\{kNN\}(i)\}\; \backslash |\backslash boldsymbol\{\backslash mu\}\_i\; -\; \backslash boldsymbol\{\backslash mu\}\_j\backslash |\; \backslash right),\; \backslash quad\; k=3$$

---

6. Training Schedule

Phase	Iterations	What's Optimized	Density Control
Warmup	0 → 1000	Canonical Gaussians only	Active (500-1000)
Deformation	1000 → 3000	Gaussians + HexPlane + Decoder	Active (1000-2500)

Learning rates (Adam, exponential decay γ = 0.01^{1/3000}):

Parameter	lr₀
Means μ	1.6×10⁻⁴
Scales s	5×10⁻³
Quaternions q	1×10⁻³
Opacities α	5×10⁻²
SH coefficients	2.5×10⁻³
Deformation θ	1.6×10⁻³

---

7. Adaptive Density Control

absgrad-based (absolute gradient values, not norms):

$${\bar{g}}_i=\frac{1}{\mathrm{\mid }{T}_{\text{vis}}(i)\mathrm{\mid }}\sum _{t\in T_{\text{vis}}(i)}\underset{d}{\max }\mid \frac{\mathrm{\partial }\mathcal{L}}{\mathrm{\partial }{\mu }_i^{\mathrm{\prime }(d)}}\mid \backslash bar\{g\}\_i\; =\; \backslash frac\{1\}\{|T\_\{\backslash text\{vis\}\}(i)|\}\; \backslash sum\_\{t\; \backslash in\; T\_\{\backslash text\{vis\}\}(i)\}\; \backslash max\_d\; \backslash left|\; \backslash frac\{\backslash partial\; \backslash mathcal\{L\}\}\{\backslash partial\; \backslash mu\text{'}^\{(d)\}\_i\}\; \backslash right|$$

Operation	Condition	Action
Split	ḡᵢ > 0.0002 AND max scale > 0.01	Split into 2, shrink by log(1.6)
Clone	ḡᵢ > 0.0002 AND max scale ≤ 0.01	Duplicate at same position
Prune	σ(αᵢ) < 0.005 OR max scale > 0.1	Remove Gaussian

---

8. Expected Performance

Method	PSNR ↑	SSIM ↑	FPS	Train Time
Static 3DGS	30-32	0.90	200+	2 min
EndoNeRF	35.8	0.96	0.2	12 hr
Endo-4DGS	36.6	0.96	100	4-7 min
EndoGaussian (target)	37.9	0.97	195	2 min

---

References

1. Kerbl et al. "3D Gaussian Splatting for Real-Time Radiance Field Rendering." SIGGRAPH 2023.
2. Liu et al. "EndoGaussian: Real-time Gaussian Splatting for Dynamic Endoscopic Scene Reconstruction." arXiv:2401.12561, 2024.
3. Huang et al. "Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting." MICCAI 2024.
4. Cao & Johnson. "HexPlane: A Fast Representation for Dynamic Scenes." CVPR 2023.
5. Eigen et al. "Depth Map Prediction from a Single Image using a Multi-Scale Deep Network." NeurIPS 2014.
6. Wang et al. "Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery." MICCAI 2022.
7. Ye et al. "gsplat: An Open-Source Library for Gaussian Splatting." arXiv:2409.06765, 2024.
8. Yang et al. "Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data." CVPR 2024.