# Open Problems, Missing Pieces, and Research Directions

---

## 1. Open Design Questions

### 1.1 Observation Models Are Completely Unspecified

The representation requires a differentiable sensor model (renderer) or learned likelihood. This is the encoder side of the system; without concrete observation models, the system cannot be implemented end to end.

**What's needed for each modality:**

**Vision (RGB):**
The observation model p(y_image | S_t) requires a differentiable renderer. Options:
- Volume rendering (NeRF-style): integrate color along rays using SDF-to-density conversion. Mature, well-understood, but slow for planning (must render full image).
- Rasterization-based: convert SDF to mesh, rasterize. Faster but loses differentiability through mesh extraction.
- Patch-based: only render small patches around task-relevant regions. Fastest for planning but loses global context.

**Recommended approach:** Volume rendering for training (maximum gradient signal), patch-based for planning (speed). The SDF field gives density σ(x) = sigmoid(-d(x)/β) where β controls sharpness.

**Depth:**
p(y_depth | S_t) = raytrace SDF along pixel rays. The predicted depth at pixel (u,v) is the first zero-crossing of d(x) along the corresponding ray. Straightforward given the SDF field.

**Tactile (e.g., GelSight):**
p(y_tactile | S_t) requires modeling the deformation of the tactile sensor pad against the object surface. Given:
- Object SDF d(x) at contact region
- Material stiffness E of both sensor and object
- Applied force F from the robot

Predict: deformation field → simulated GelSight image. This is a contact mechanics problem. Simplified version: use the SDF gradient (surface normal) + penetration depth as features, pass through a learned decoder to predicted tactile image.

**Audio:**
p(y_audio | S_t) for impact sounds:
- Material parameters (density, stiffness, acoustic damping γ) determine resonant frequencies and decay
- Contact velocity determines amplitude
- Object geometry (from SDF) determines mode shapes

Simplified version: predict a spectral envelope (frequency spectrum) from material + geometry features, not raw waveform. Compare to observed spectrum.

**Thermal:**
p(y_thermal | S_t) is a rendering problem: given temperature field T(x,t) in the scene, integrate thermal emission along rays. Simpler than RGB rendering because there's no complex light transport — just emission.

### 1.2 The ELBO is Not Derived

The model needs an explicit ELBO. A candidate form is:

```
log p(y_{1:T} | a_{1:T-1}) ≥ Σ_t [ E_{q(S_t)} [Σ_m log p(y_t^(m) | S_t)]        ... reconstruction
                                    - KL(q(S_t | y_≤t, a<t) || p(S_t | S_{t-1}, a_{t-1}))  ... dynamics prior
                                  ]

where:
  q(S_t | y_≤t, a<t) = q(Φ_t, G_t | y_≤t, a<t)
                      = q(z_{1:N}(t), T_{1:N}(t), e_{1:N}(t), c_edges(t) | y_≤t, a<t)
```

The KL term factorizes over objects if we assume conditional independence given the graph:

```
KL(q || p) ≈ Σ_i KL(q(z_i(t)) || p(z_i(t) | z_i(t-1), a_{t-1}, messages_i))
           + Σ_i KL(q(T_i(t)) || p(T_i(t) | T_i(t-1), a_{t-1}, messages_i))
           + Σ_{(i,j)} KL(q(c_ij(t)) || p(c_ij(t) | S_{t-1}))
```

The field Φ is not a separate random variable — it's deterministically parameterized by {z_i, T_i} through the mixture. So the KL over the field is captured by the KLs over object latents.

### 1.3 How the Encoder Actually Works (Amortized Inference Architecture)

The encoder q_ψ(S_t | y_≤t, a<t) must:
1. Process multimodal observations → feature maps
2. Detect/track objects → update graph topology
3. Estimate per-object latent z_i and pose T_i
4. Maintain temporal state (filtering)

**Concrete architecture:**

```
Per timestep:
1. Vision backbone (e.g., ResNet-50 or ViT) → feature map F_t ∈ R^{H×W×C}
2. Object detection/segmentation:
   - Use SAM-style segmentation or learned slot attention
   - Match detected segments to existing graph nodes (Hungarian matching on appearance + position)
   - Handle birth/death via existence probabilities
3. Per-object crop + RoI feature extraction:
   - For each object i, extract features from its image region
   - Fuse with tactile/audio if available for that object
4. Per-object posterior update:
   - GRU or transformer-based temporal model:
     h_i(t) = GRU(h_i(t-1), f_i(t), a_{t-1})
   - Output: q(z_i(t)) = Normal(μ_z(h_i(t)), σ_z(h_i(t)))
            q(T_i(t)) = SE(3) distribution from predicted translation + rotation
5. Graph structure update:
   - Compute pairwise contact probabilities from proximity + features
   - Update edge features

This is essentially a structured variational autoencoder with GNN + GRU backbone.
```

---

## 2. Deeper Open Problems (Research-Level)

### 2.1 Identifiability of Material Parameters from Vision Alone

**Problem:** Some material parameters are genuinely unidentifiable from RGB video. A perfect RGB rendering of a steel ball and an identically-painted hollow aluminum ball are indistinguishable. Mass, density, and internal structure are invisible.

**What's identifiable from vision:**
- Deformation → stiffness (soft things deform visibly)
- Sliding behavior → friction (sliding objects decelerate observably)
- Bouncing → restitution (bounce height is visible)
- Sloshing → viscosity (flow speed is visible for liquids)

**What's NOT identifiable from vision alone:**
- Mass/density (unless interaction with gravity is observed — but two objects can have different mass and identical gravitational behavior if they have the same density)
- Internal structure (solid vs hollow)
- Thermal properties (invisible without thermal camera)
- Fine friction coefficients (need to observe sliding at different speeds/forces)

**What this means for the paper:**
- Be explicit about the identifiability boundary. Don't claim the field can estimate arbitrary material properties from video.
- The multimodal aspect is not optional luxury — tactile and audio observations provide information that is mathematically unrecoverable from vision. This is an argument FOR the multimodal representation, not a limitation.
- Internet video pre-training gives you coarse physics priors (stiffness ranking, friction ranking). Robot interaction gives you calibrated values.

### 2.2 Scalability to Complex Scenes

**Problem:** The mixture-of-experts scales linearly with number of objects: each query evaluates N local fields + gating. At N=100 objects, this is 100x slower than a single-field model.

**Potential solutions:**
- **Spatial hashing**: Only evaluate objects whose bounding boxes contain the query point. Reduces effective N per query to ~2-3 for typical scenes.
- **Hierarchical decomposition**: Group objects into regions, only expand relevant region.
- **Shared field backbone**: Instead of independent MLPs per object, use a shared backbone with object-specific conditioning (saves parameters, enables batching).

This is an engineering problem, not a scientific one, but it needs to be addressed to be taken seriously.

### 2.3 Consistency Between Graph Relations and Field Predictions

**Problem:** The graph might say objects i and j are in contact (c_ij ≈ 1) while the SDF field says they're 2cm apart (d_ij > 0). Or the graph might not have a support edge while the field clearly shows object i resting on j.

**Solution:** Bidirectional consistency losses.

```
L_consistency = λ_1 · ||c_ij - σ(-min_x d_ij(x) / τ)||²    ... contact prob should match SDF proximity
              + λ_2 · Σ_{(i,j)} c_ij · max(0, min_x d_ij(x))  ... contacting objects should have SDF ≈ 0
```

This forces the graph and field to agree. The graph provides the high-level claim ("these are in contact"), the field provides the geometric evidence. Disagreement generates gradient that corrects both.

### 2.4 Handling Articulated and Deformable Objects

**Problem:** The current model assumes each object has a single rigid pose T_i ∈ SE(3). This breaks for:
- Articulated objects (drawer, door, robot arm itself)
- Deformable objects (cloth, rope, dough)

**Solution for articulated objects:**
Add joint state to graph edges. If edge (i,j) has type "hinge":
```
Edge stores: joint angle θ_ij, joint axis a_ij, joint limits [θ_min, θ_max]
Object j's pose is constrained: T_j = T_i · Transform(a_ij, θ_ij)
```

This is standard in URDF/articulation models. The graph naturally represents kinematic chains.

**Solution for deformable objects:**
Replace the single pose T_i with a deformation field. The object-local field ϕ_i already captures arbitrary shapes through z_i — if z_i has enough capacity, it can represent deformed configurations. The key change:

```
Instead of: x_i = T_i^{-1} x  (rigid transform)
Use:        x_i = T_i^{-1} x + δ_i(T_i^{-1} x, t)  (rigid transform + local deformation)
```

where δ_i is a small deformation network conditioned on z_i. This adds minimal parameters but allows local shape changes.

### 2.5 Real-Time Filtering on Physical Robot

**Problem:** The encoder must run in real-time (10+ Hz) with real sensor data. Real images have noise, motion blur, partial occlusion, and calibration errors that simulation doesn't have.

**Key challenges:**
- Domain gap between simulated training data and real sensors
- Latency: encoder must be fast enough for control loop
- Robustness: single bad observation shouldn't corrupt the belief state

**Mitigation strategies:**
- **Domain randomization** during simulation training (vary textures, lighting, noise, blur)
- **Real-data fine-tuning** with self-supervised losses (no ground truth needed — use prediction error)
- **Temporal smoothing**: The GRU-based temporal model acts as a low-pass filter on the state estimate. A single bad frame produces a small update; consecutive bad frames are detectable as anomalies.
- **Fallback to prior**: If observation quality is low (detected via reconstruction error), weight the dynamics prior more heavily and the observation less. This is standard in Kalman filtering.


---

## 3. What You Should Ask For Help With (Concrete Tasks)

Based on everything above, here are the concrete tasks where you'd benefit from working together:

### Immediate (next session):
1. **Architecture diagram**: Create a comprehensive figure showing all modules, data flow, and loss functions. This is the single most impactful thing for the paper.
2. **Concrete pseudocode for the full training loop**: Sim pre-training → internet video → robot fine-tuning. Publishable algorithm box.
3. **Related work positioning**: Who has done pieces of this? Position against NeRF-style models, scene graphs, world models (Dreamer/IRIS/etc), video prediction models, and physics-informed neural fields.

### Medium-term:
4. **Implementation scaffolding**: Set up the PyTorch codebase structure, key data structures, and training script skeleton.
5. **Simulation environment design**: Choose specific objects, scenes, and tasks in Isaac Gym. Define evaluation protocol.
6. **Internet video training pipeline**: Script for downloading, preprocessing, and training on Something-Something V2 or similar.

### Strategic:
7. **Venue targeting**: Is this an RSS paper? CoRL? ICRA? NeurIPS? The positioning changes based on venue.
8. **Figure design**: The figures make or break a world-model paper. Need the architecture diagram, the data flow diagram, the "why hybrid" comparison figure, and result figures.
9. **Writing the introduction**: The internet-scale-data argument needs to hit hard in the first paragraph.