Open Problems, Missing Pieces, and Research Directions
1. Open Design Questions
1.1 Observation Models Are Completely Unspecified
The representation requires a differentiable sensor model (renderer) or learned likelihood. This is the encoder side of the system; without concrete observation models, the system cannot be implemented end to end.
What's needed for each modality:
Vision (RGB): The observation model p(y_image | S_t) requires a differentiable renderer. Options:
- Volume rendering (NeRF-style): integrate color along rays using SDF-to-density conversion. Mature, well-understood, but slow for planning (must render full image).
- Rasterization-based: convert SDF to mesh, rasterize. Faster but loses differentiability through mesh extraction.
- Patch-based: only render small patches around task-relevant regions. Fastest for planning but loses global context.
Recommended approach: Volume rendering for training (maximum gradient signal), patch-based for planning (speed). The SDF field gives density σ(x) = sigmoid(-d(x)/β) where β controls sharpness.
Depth: p(y_depth | S_t) = raytrace SDF along pixel rays. The predicted depth at pixel (u,v) is the first zero-crossing of d(x) along the corresponding ray. Straightforward given the SDF field.
Tactile (e.g., GelSight): p(y_tactile | S_t) requires modeling the deformation of the tactile sensor pad against the object surface. Given:
- Object SDF d(x) at contact region
- Material stiffness E of both sensor and object
- Applied force F from the robot
Predict: deformation field → simulated GelSight image. This is a contact mechanics problem. Simplified version: use the SDF gradient (surface normal) + penetration depth as features, pass through a learned decoder to predicted tactile image.
Audio: p(y_audio | S_t) for impact sounds:
- Material parameters (density, stiffness, acoustic damping γ) determine resonant frequencies and decay
- Contact velocity determines amplitude
- Object geometry (from SDF) determines mode shapes
Simplified version: predict a spectral envelope (frequency spectrum) from material + geometry features, not raw waveform. Compare to observed spectrum.
Thermal: p(y_thermal | S_t) is a rendering problem: given temperature field T(x,t) in the scene, integrate thermal emission along rays. Simpler than RGB rendering because there's no complex light transport — just emission.
1.2 The ELBO is Not Derived
The model needs an explicit ELBO. A candidate form is:
log p(y_{1:T} | a_{1:T-1}) ≥ Σ_t [ E_{q(S_t)} [Σ_m log p(y_t^(m) | S_t)] ... reconstruction
- KL(q(S_t | y_≤t, a<t) || p(S_t | S_{t-1}, a_{t-1})) ... dynamics prior
]
where:
q(S_t | y_≤t, a<t) = q(Φ_t, G_t | y_≤t, a<t)
= q(z_{1:N}(t), T_{1:N}(t), e_{1:N}(t), c_edges(t) | y_≤t, a<t)
The KL term factorizes over objects if we assume conditional independence given the graph:
KL(q || p) ≈ Σ_i KL(q(z_i(t)) || p(z_i(t) | z_i(t-1), a_{t-1}, messages_i))
+ Σ_i KL(q(T_i(t)) || p(T_i(t) | T_i(t-1), a_{t-1}, messages_i))
+ Σ_{(i,j)} KL(q(c_ij(t)) || p(c_ij(t) | S_{t-1}))
The field Φ is not a separate random variable — it's deterministically parameterized by {z_i, T_i} through the mixture. So the KL over the field is captured by the KLs over object latents.
1.3 How the Encoder Actually Works (Amortized Inference Architecture)
The encoder q_ψ(S_t | y_≤t, a<t) must:
- Process multimodal observations → feature maps
- Detect/track objects → update graph topology
- Estimate per-object latent z_i and pose T_i
- Maintain temporal state (filtering)
Concrete architecture:
Per timestep:
1. Vision backbone (e.g., ResNet-50 or ViT) → feature map F_t ∈ R^{H×W×C}
2. Object detection/segmentation:
- Use SAM-style segmentation or learned slot attention
- Match detected segments to existing graph nodes (Hungarian matching on appearance + position)
- Handle birth/death via existence probabilities
3. Per-object crop + RoI feature extraction:
- For each object i, extract features from its image region
- Fuse with tactile/audio if available for that object
4. Per-object posterior update:
- GRU or transformer-based temporal model:
h_i(t) = GRU(h_i(t-1), f_i(t), a_{t-1})
- Output: q(z_i(t)) = Normal(μ_z(h_i(t)), σ_z(h_i(t)))
q(T_i(t)) = SE(3) distribution from predicted translation + rotation
5. Graph structure update:
- Compute pairwise contact probabilities from proximity + features
- Update edge features
This is essentially a structured variational autoencoder with GNN + GRU backbone.
2. Deeper Open Problems (Research-Level)
2.1 Identifiability of Material Parameters from Vision Alone
Problem: Some material parameters are genuinely unidentifiable from RGB video. A perfect RGB rendering of a steel ball and an identically-painted hollow aluminum ball are indistinguishable. Mass, density, and internal structure are invisible.
What's identifiable from vision:
- Deformation → stiffness (soft things deform visibly)
- Sliding behavior → friction (sliding objects decelerate observably)
- Bouncing → restitution (bounce height is visible)
- Sloshing → viscosity (flow speed is visible for liquids)
What's NOT identifiable from vision alone:
- Mass/density (unless interaction with gravity is observed — but two objects can have different mass and identical gravitational behavior if they have the same density)
- Internal structure (solid vs hollow)
- Thermal properties (invisible without thermal camera)
- Fine friction coefficients (need to observe sliding at different speeds/forces)
What this means for the paper:
- Be explicit about the identifiability boundary. Don't claim the field can estimate arbitrary material properties from video.
- The multimodal aspect is not optional luxury — tactile and audio observations provide information that is mathematically unrecoverable from vision. This is an argument FOR the multimodal representation, not a limitation.
- Internet video pre-training gives you coarse physics priors (stiffness ranking, friction ranking). Robot interaction gives you calibrated values.
2.2 Scalability to Complex Scenes
Problem: The mixture-of-experts scales linearly with number of objects: each query evaluates N local fields + gating. At N=100 objects, this is 100x slower than a single-field model.
Potential solutions:
- Spatial hashing: Only evaluate objects whose bounding boxes contain the query point. Reduces effective N per query to ~2-3 for typical scenes.
- Hierarchical decomposition: Group objects into regions, only expand relevant region.
- Shared field backbone: Instead of independent MLPs per object, use a shared backbone with object-specific conditioning (saves parameters, enables batching).
This is an engineering problem, not a scientific one, but it needs to be addressed to be taken seriously.
2.3 Consistency Between Graph Relations and Field Predictions
Problem: The graph might say objects i and j are in contact (c_ij ≈ 1) while the SDF field says they're 2cm apart (d_ij > 0). Or the graph might not have a support edge while the field clearly shows object i resting on j.
Solution: Bidirectional consistency losses.
L_consistency = λ_1 · ||c_ij - σ(-min_x d_ij(x) / τ)||² ... contact prob should match SDF proximity
+ λ_2 · Σ_{(i,j)} c_ij · max(0, min_x d_ij(x)) ... contacting objects should have SDF ≈ 0
This forces the graph and field to agree. The graph provides the high-level claim ("these are in contact"), the field provides the geometric evidence. Disagreement generates gradient that corrects both.
2.4 Handling Articulated and Deformable Objects
Problem: The current model assumes each object has a single rigid pose T_i ∈ SE(3). This breaks for:
- Articulated objects (drawer, door, robot arm itself)
- Deformable objects (cloth, rope, dough)
Solution for articulated objects: Add joint state to graph edges. If edge (i,j) has type "hinge":
Edge stores: joint angle θ_ij, joint axis a_ij, joint limits [θ_min, θ_max]
Object j's pose is constrained: T_j = T_i · Transform(a_ij, θ_ij)
This is standard in URDF/articulation models. The graph naturally represents kinematic chains.
Solution for deformable objects: Replace the single pose T_i with a deformation field. The object-local field ϕ_i already captures arbitrary shapes through z_i — if z_i has enough capacity, it can represent deformed configurations. The key change:
Instead of: x_i = T_i^{-1} x (rigid transform)
Use: x_i = T_i^{-1} x + δ_i(T_i^{-1} x, t) (rigid transform + local deformation)
where δ_i is a small deformation network conditioned on z_i. This adds minimal parameters but allows local shape changes.
2.5 Real-Time Filtering on Physical Robot
Problem: The encoder must run in real-time (10+ Hz) with real sensor data. Real images have noise, motion blur, partial occlusion, and calibration errors that simulation doesn't have.
Key challenges:
- Domain gap between simulated training data and real sensors
- Latency: encoder must be fast enough for control loop
- Robustness: single bad observation shouldn't corrupt the belief state
Mitigation strategies:
- Domain randomization during simulation training (vary textures, lighting, noise, blur)
- Real-data fine-tuning with self-supervised losses (no ground truth needed — use prediction error)
- Temporal smoothing: The GRU-based temporal model acts as a low-pass filter on the state estimate. A single bad frame produces a small update; consecutive bad frames are detectable as anomalies.
- Fallback to prior: If observation quality is low (detected via reconstruction error), weight the dynamics prior more heavily and the observation less. This is standard in Kalman filtering.
3. What You Should Ask For Help With (Concrete Tasks)
Based on everything above, here are the concrete tasks where you'd benefit from working together:
Immediate (next session):
- Architecture diagram: Create a comprehensive figure showing all modules, data flow, and loss functions. This is the single most impactful thing for the paper.
- Concrete pseudocode for the full training loop: Sim pre-training → internet video → robot fine-tuning. Publishable algorithm box.
- Related work positioning: Who has done pieces of this? Position against NeRF-style models, scene graphs, world models (Dreamer/IRIS/etc), video prediction models, and physics-informed neural fields.
Medium-term:
- Implementation scaffolding: Set up the PyTorch codebase structure, key data structures, and training script skeleton.
- Simulation environment design: Choose specific objects, scenes, and tasks in Isaac Gym. Define evaluation protocol.
- Internet video training pipeline: Script for downloading, preprocessing, and training on Something-Something V2 or similar.
Strategic:
- Venue targeting: Is this an RSS paper? CoRL? ICRA? NeurIPS? The positioning changes based on venue.
- Figure design: The figures make or break a world-model paper. Need the architecture diagram, the data flow diagram, the "why hybrid" comparison figure, and result figures.
- Writing the introduction: The internet-scale-data argument needs to hit hard in the first paragraph.