Engram-Guided Structural Memory for Single-View 3D Reconstruction in Humanoid Robotics

Author: GrooveJ (Danny Lee)
Affiliation: Independent Researcher (With the assistance of Many LLMs)

Conceptual paper (no experimental results).
This article presents a complete mathematical specification with intuitive explanations of all operators and symbols, designed for future implementation and empirical validation.

---

Abstract

Single-view 3D reconstruction under occlusion is a critical bottleneck in humanoid robotics: functional structures—handles, joints, supports—are frequently hidden, causing grasp failures and unsafe actions. While Large Reconstruction Models (LRMs) produce visually plausible shapes, they lack mechanisms for reusing structural knowledge across objects. Inspired by the principle that decoupling memory from computation yields significant reasoning gains, we propose Engram-Guided 2D-to-3D (EG-3D): a framework that integrates structure-only 3D memory with LLM-conditioned perception and Vision-Language-Action (VLA) control.

Given an RGB image and a task prompt, an LLM-conditioned encoder extracts part-level structural keys via Slot Attention, anchored to functional parts through a prioritized supervision hierarchy (contact signals → geometric priors → relational consistency). These keys retrieve Modular Engrams—decomposed into geometric, relational, symmetry, and affordance subcomponents—each injected through dedicated pathways. We enforce structural constraints via kinematic consistency loss ensuring relational tokens are not ignored by the generator. For single-view pose estimation without ground truth, we leverage gravity-aligned priors (from robot IMU) combined with multi-view consistency. We also introduce a staged memory construction protocol that enables deployment with minimal initial annotation (geometry + symmetry only), progressively adding relations and affordances.

This paper provides: (1) a complete mathematical specification with detailed explanations for every equation, (2) a minimal "EG-3D Lite" configuration for practical deployment, (3) explicit mechanisms preventing generators from ignoring structural constraints, and (4) clear differentiation from prior retrieval-based methods.

---

1. Introduction

1.1 The Problem: Occlusion-Induced Failures in Manipulation

Humanoid robots increasingly rely on monocular head-mounted cameras. However, real-world manipulation involves inherent underspecification: crucial functional elements—door handles, hinges, support legs, tool tips, internal joints—are frequently occluded by the object itself or by the robot's body. When these structures are misreconstructed or not inferred at all, robots select incorrect grasp points, apply unsafe forces, collide unexpectedly, or fail tasks entirely.

1.2 Industry Context: Why Now?

The strategic importance of single-view 3D reconstruction for robotics has become increasingly apparent. Recent acquisitions and investments around single-image-to-3D asset generation highlight the urgency of robust 2D→3D pipelines for embodied systems. Major technology companies are actively acquiring startups in this space, recognizing that perception-to-action pipelines are critical bottlenecks for deploying robots at scale.

1.3 The Gap: Missing Structural Memory

Recent LRMs (TripoSR, LRM, Wonder3D) scale transformers and diffusion to generate plausible geometry from single images. Yet they exhibit systematic structural errors: wrong part counts, symmetry collapse, hallucinated connections, or missing functional subparts. We argue that a central missing ingredient is reusable structural memory.

1.4 Inspiration: Conditional Memory as a Modeling Primitive

The Engram-Guided Structural Memory concept demonstrates that decoupling memory from computation yields significant reasoning gains. We extend this principle to 3D perception.

1.5 Key Technical Contributions

1. Modular Engram Architecture with explicit composition operators and kinematic consistency loss ensuring relational constraints are enforced

2. Prioritized Slot-Part Alignment: contact/interaction signals (strongest) → geometric priors → relational consistency (weakest)

3. Gravity-Aligned Pose Estimation: leveraging robot IMU for coarse alignment without ground-truth poses

4. Staged Memory Construction: minimal viable memory (geo + sym) → progressive enrichment (rel → aff)

5. Clear Differentiation from retrieval-based priors: EG-3D is a structural assembly system, not a retrieval prior

---

2. Related Work and Differentiation

2.1 Retrieval-Based 3D Priors vs. Structural Assembly Memory

A key question is: How does EG-3D differ from existing retrieval-based 3D priors?

Our key insight: EG-3D is not "retrieval + generation" but "structural assembly + generation". We retrieve parts, align them via SE(3), enforce kinematic constraints, and compose them with explicit operators.

2.2 Memory in Vision-Language-Action Models

Recent VLA models incorporate memory:

• MemoryVLA: Temporal memory for long-horizon tasks
• IVE: Episodic memory for exploration

Our work addresses structural memory (reusable 3D motifs across objects), complementary to temporal/episodic memory.

---

3. Notation and Preliminaries

3.1 Inputs and Embeddings

3.2 Part-Level Queries (Slots)

3.3 Modular Engram Structure

Each Engram ${\mathbf{E}}_m\backslash mathbf\{E\}\_m$ is a modular tuple:

$${\mathbf{E}}_m=(e_m^{\text{geo}},e_m^{\text{rel}},e_m^{\text{sym}},e_m^{\text{aff}},T_m^c)\backslash mathbf\{E\}\_m\; =\; (e\_m^\{\backslash text\{geo\}\},\; e\_m^\{\backslash text\{rel\}\},\; e\_m^\{\backslash text\{sym\}\},\; e\_m^\{\backslash text\{aff\}\},\; T\_m^c)$$

Symbol-by-symbol explanation:

• $mm$: Index of the Engram in memory (\(m = 1, \ldots, M\))
• $e_m^{\text{geo}}\in {\mathbb{R}}^{d_g}e\_m^\{\backslash text\{geo\}\}\; \backslash in\; \backslash mathbb\{R\}^\{d\_g\}$: Geometric component — latent vector encoding 3D shape
• $e_m^{\text{rel}}\in {\mathbb{R}}^{N_r\times d_r}e\_m^\{\backslash text\{rel\}\}\; \backslash in\; \backslash mathbb\{R\}^\{N\_r\; \backslash times\; d\_r\}$: Relational component — $N_{r}N\_r$ tokens encoding part connectivity
• $e_m^{\text{sym}}\in {\mathbb{R}}^{d_s}e\_m^\{\backslash text\{sym\}\}\; \backslash in\; \backslash mathbb\{R\}^\{d\_s\}$: Symmetry component — descriptor of symmetry type and axes
• $e_m^{\text{aff}}\in {\mathbb{R}}^{N_a\times d_a}e\_m^\{\backslash text\{aff\}\}\; \backslash in\; \backslash mathbb\{R\}^\{N\_a\; \backslash times\; d\_a\}$: Affordance component — $N_{a}N\_a$ markers for interaction points
• $T_m^c\in SE(3)T\_m^c\; \backslash in\; SE(3)$: Canonical frame — reference pose for spatial alignment

3.4 Memory Structure

3.5 3D Generator and Action

---

4. Core Framework

4.1 Pipeline Overview

The complete pipeline consists of five stages:

---

Equation (1): Visual-Language Encoding

$$\{z_t{\}}_{t=1}^{N+T}=E_{\text{VL}}(P,\mathcal{I}),z=\text{Pool}(\{z_t\})\backslash \{z\_t\backslash \}\_\{t=1\}^\{N+T\}\; =\; E\_\{\backslash text\{VL\}\}(P,\; \backslash mathcal\{I\}),\; \backslash quad\; z\; =\; \backslash text\{Pool\}(\backslash \{z\_t\backslash \})$$

Symbol-by-symbol:

• $E_{\text{VL}}E\_\{\backslash text\{VL\}\}$: Visual-Language encoder (any VLM backbone)
• $PP$: Input text prompt
• $\mathcal{I}\backslash mathcal\{I\}$: Input RGB image
• $\{z_t{\}}_{t=1}^{N+T}\backslash \{z\_t\backslash \}\_\{t=1\}^\{N+T\}$: Output token sequence (\(N\) visual + $TT$ text tokens)
• $\text{Pool}(\cdot )\backslash text\{Pool\}(\backslash cdot)$: Pooling operation (mean or attention pooling)
• $zz$: Global embedding summarizing the entire input

Intuitive Explanation:
This is the robot's "perception + understanding" step. The encoder looks at the image and reads the instruction, producing a sequence of tokens that represent both visual content and linguistic intent. The pooled vector $zz$ captures the overall context.

Why this matters:
Joint visual-language encoding allows the system to focus on task-relevant parts. For "grab the handle," the encoder emphasizes handle-related visual regions.

---

Equation (2): Structure-Only Key Extraction

$$\{k_j{\}}_{j=1}^J=\mathcal{K}(\{z_t\})\backslash \{k\_j\backslash \}\_\{j=1\}^\{J\}\; =\; \backslash mathcal\{K\}(\backslash \{z\_t\backslash \})$$

Symbol-by-symbol:

• $\mathcal{K}\backslash mathcal\{K\}$: Key extraction function (implemented via Slot Attention)
• $\{z_t\}\backslash \{z\_t\backslash \}$: Input token sequence from encoder
• $k_j\in {\mathbb{R}}^{D}k\_j\; \backslash in\; \backslash mathbb\{R\}^D$: Output key for slot $jj$
• $JJ$: Number of slots (part hypotheses)

Expanded form (Slot Attention):

$$k_j=\sum _{t=1}^{N+T}{a}_{j,t}\cdot z_{t}k\_j\; =\; \backslash sum\_\{t=1\}^\{N+T\}\; a\_\{j,t\}\; \backslash cdot\; z\_t$$

where attention weights are computed as:

$$a_{j,t}=\frac{\exp (q_j^{\mathrm{\top }}{k}_t\mathrm{/}\sqrt{D})}{\sum _{j^{\mathrm{\prime }}=1}^J\exp (q_{j^{\mathrm{\prime }}}^{\mathrm{\top }}{k}_t\mathrm{/}\sqrt{D})}a\_\{j,t\}\; =\; \backslash frac\{\backslash exp(q\_j^\backslash top\; k\_t\; /\; \backslash sqrt\{D\})\}\{\backslash sum\_\{j\text{'}=1\}^\{J\}\; \backslash exp(q\_\{j\text{'}\}^\backslash top\; k\_t\; /\; \backslash sqrt\{D\})\}$$

Critical constraint (competition across slots):

$$\sum _{j=1}^J{a}_{j,t}=1\text{for each token }t\backslash sum\_\{j=1\}^\{J\}\; a\_\{j,t\}\; =\; 1\; \backslash quad\; \backslash text\{for\; each\; token\; \}\; t$$

Intuitive Explanation:
Slot Attention splits the scene into parts. Each slot "competes" to explain different tokens—one slot might capture "the handle," another "the door body," another "the hinge." The key $k_{j}k\_j$ summarizes what slot $jj$ has captured.

Why the competition constraint?
Without it, all slots might attend to everything equally, producing identical keys. Competition forces specialization.

---

Equation (3): Modular Engram Retrieval with SE(3) Alignment

$${\mathbf{Z}}_e={\mathcal{R}}_{\text{SE(3)}}(\{k_j\};\mathcal{M})\backslash mathbf\{Z\}\_e\; =\; \backslash mathcal\{R\}\_\{\backslash text\{SE(3)\}\}(\backslash \{k\_j\backslash \};\; \backslash mathcal\{M\})$$

Symbol-by-symbol:

• ${\mathcal{R}}_{\text{SE(3)}}\backslash mathcal\{R\}\_\{\backslash text\{SE(3)\}\}$: Retrieval function with SE(3) alignment
• $\{k_j\}\backslash \{k\_j\backslash \}$: Query keys from slots
• $\mathcal{M}\backslash mathcal\{M\}$: Memory bank
• ${\mathbf{Z}}_e\backslash mathbf\{Z\}\_e$: Retrieved and composed Engram components

Expanded form (retrieval weights):

$$s_{j,m}=\frac{k_j^{\mathrm{\top }}{k}_m}{\mathrm{\parallel }{k}_j{\mathrm{\parallel }}_2\mathrm{\parallel }{k}_m{\mathrm{\parallel }}_2}s\_\{j,m\}\; =\; \backslash frac\{k\_j^\backslash top\; k\_m\}\{\backslash |k\_j\backslash |\_2\; \backslash |k\_m\backslash |\_2\}$$

$${\alpha }_{j,m}=\frac{\exp (s_{j,m}\mathrm{/}\tau )}{\sum _{l=1}^M\exp (s_{j,l}\mathrm{/}\tau )}\backslash alpha\_\{j,m\}\; =\; \backslash frac\{\backslash exp(s\_\{j,m\}/\backslash tau)\}\{\backslash sum\_\{l=1\}^\{M\}\backslash exp(s\_\{j,l\}/\backslash tau)\}$$

Symbol-by-symbol:

• $s_{j,m}\in [-1,1]s\_\{j,m\}\; \backslash in\; [-1,\; 1]$: Cosine similarity between query $k_{j}k\_j$ and memory key $k_{m}k\_m$
• $\tau >0\backslash tau\; >\; 0$: Temperature (lower = sharper retrieval, higher = softer)
• ${\alpha }_{j,m}\in [0,1]\backslash alpha\_\{j,m\}\; \backslash in\; [0,\; 1]$: Normalized retrieval weight (sums to 1 over $mm$)

Intuitive Explanation:
For each slot's key, we find the most similar Engrams in memory. The softmax converts similarities into probability-like weights. Temperature controls sharpness: $\tau \to 0\backslash tau\; \backslash to\; 0$ gives winner-take-all, $\tau \to \mathrm{\infty }\backslash tau\; \backslash to\; \backslash infty$ gives uniform weighting.

---

Equation (4): Multi-Pathway 3D Generation with Kinematic Constraints

Symbol-by-symbol:

• $G_{\theta }G\_\backslash theta$: 3D generator network with parameters $\theta \backslash theta$
• $\mathcal{I}\backslash mathcal\{I\}$: Input image (provides appearance guidance)
• ${\mathbf{Z}}_e\backslash mathbf\{Z\}\_e$: Composed Engram components (structural guidance)
• ${\mathcal{L}}_{\text{kin}}\backslash mathcal\{L\}\_\{\backslash text\{kin\}\}$: Kinematic consistency loss
• $ϵ\backslash epsilon$: Constraint tolerance

Intuitive Explanation:
The generator produces 3D geometry guided by both the image (for appearance) and retrieved Engrams (for structure). The kinematic constraint ensures the output respects relational information (e.g., joint axes, connection points).

---

Equation (5): Action Inference

$$A={\pi }_{\text{VLA}}(\text{Render}(G),P)A\; =\; \backslash pi\_\{\backslash text\{VLA\}\}(\backslash text\{Render\}(G),\; P)$$

Symbol-by-symbol:

• ${\pi }_{\text{VLA}}\backslash pi\_\{\backslash text\{VLA\}\}$: Vision-Language-Action policy
• $\text{Render}(G)\backslash text\{Render\}(G)$: Rendered images from 3D reconstruction
• $PP$: Original task prompt
• $A\in {\mathbb{R}}^{K}A\; \backslash in\; \backslash mathbb\{R\}^K$: Output action vector

Intuitive Explanation:
Given the reconstructed 3D (rendered as images) and the task instruction, the VLA policy outputs robot actions. The explicit 3D structure helps the policy reason about occluded parts.

---

4.2 Slot-Part Alignment with Prioritized Supervision

Critical Challenge: Slot Attention often produces "semantic blobs" rather than functional parts.

Our Solution: Prioritized Supervision Hierarchy

---

Combined Slot Alignment Loss:

$${\mathcal{L}}_{\text{slot-align}}=w_1{\mathcal{L}}_{\text{contact}}+w_2{\mathcal{L}}_{\text{geo-prior}}+w_3{\mathcal{L}}_{\text{rel-consist}}+w_4{\mathcal{L}}_{\text{diversity}}\backslash mathcal\{L\}\_\{\backslash text\{slot-align\}\}\; =\; w\_1\; \backslash mathcal\{L\}\_\{\backslash text\{contact\}\}\; +\; w\_2\; \backslash mathcal\{L\}\_\{\backslash text\{geo-prior\}\}\; +\; w\_3\; \backslash mathcal\{L\}\_\{\backslash text\{rel-consist\}\}\; +\; w\_4\; \backslash mathcal\{L\}\_\{\backslash text\{diversity\}\}$$

Symbol-by-symbol:

• $w_1,w_2,w_3,w_{4}w\_1,\; w\_2,\; w\_3,\; w\_4$: Priority weights (\(w_1 > w_2 > w_3 > w_4\))
• ${\mathcal{L}}_{\text{contact}}\backslash mathcal\{L\}\_\{\backslash text\{contact\}\}$: Contact signal loss
• ${\mathcal{L}}_{\text{geo-prior}}\backslash mathcal\{L\}\_\{\backslash text\{geo-prior\}\}$: Geometric affordance prior loss
• ${\mathcal{L}}_{\text{rel-consist}}\backslash mathcal\{L\}\_\{\backslash text\{rel-consist\}\}$: Relational consistency loss
• ${\mathcal{L}}_{\text{diversity}}\backslash mathcal\{L\}\_\{\backslash text\{diversity\}\}$: Slot diversity regularization

Intuitive Explanation:
We stack multiple supervision signals by priority. Contact signals (where people actually grab objects) are strongest because they directly indicate functional parts. Lower-priority signals fill in when higher-priority signals are unavailable.

---

Priority 1: Contact Signal Loss

$${\mathcal{L}}_{\text{contact}}=\sum _{j=1}^J\text{BCE}({\text{SlotMask}}_j,\text{ContactMask})\backslash mathcal\{L\}\_\{\backslash text\{contact\}\}\; =\; \backslash sum\_\{j=1\}^\{J\}\; \backslash text\{BCE\}\backslash big(\backslash text\{SlotMask\}\_j,\; \backslash text\{ContactMask\}\backslash big)$$

Symbol-by-symbol:

• ${\text{SlotMask}}_j\in [0,1{]}^{H\times W}\backslash text\{SlotMask\}\_j\; \backslash in\; [0,1]^\{H\; \backslash times\; W\}$: Attention mask for slot $jj$ (where it attends in the image)
• $\text{ContactMask}\in \{0,1{\}}^{H\times W}\backslash text\{ContactMask\}\; \backslash in\; \backslash \{0,1\backslash \}^\{H\; \backslash times\; W\}$: Ground-truth contact regions (from HOI data)
• $\text{BCE}\backslash text\{BCE\}$: Binary Cross-Entropy loss

Expanded BCE:

$$\text{BCE}(p,q)=-\frac{1}{HW}\sum _{i,j}[q_{ij}\log p_{ij}+(1-q_{ij})\log (1-p_{ij})]\backslash text\{BCE\}(p,\; q)\; =\; -\backslash frac\{1\}\{HW\}\backslash sum\_\{i,j\}\backslash big[q\_\{ij\}\backslash log\; p\_\{ij\}\; +\; (1-q\_\{ij\})\backslash log(1-p\_\{ij\})\backslash big]$$

Intuitive Explanation:
If we know where humans touch objects (from videos of people using objects), we train slots to attend to those regions. A slot attending to a handle should align with hand-contact regions on handles.

Why highest priority?
Contact directly indicates functional relevance. A handle is a handle because people grab it.

---

Priority 2: Geometric Affordance Prior Loss

$${\mathcal{L}}_{\text{geo-prior}}=-\sum _{j=1}^J\sum _{a\in \mathcal{A}}\mathbb{1}[{\text{SlotMask}}_j\cap {\text{GeoPrior}}_a\mathrm{\ne }\mathrm{\varnothing }]\cdot \log p(a\mathrm{\mid }{k}_j)\backslash mathcal\{L\}\_\{\backslash text\{geo-prior\}\}\; =\; -\backslash sum\_\{j=1\}^\{J\}\; \backslash sum\_\{a\; \backslash in\; \backslash mathcal\{A\}\}\; \backslash mathbb\{1\}[\backslash text\{SlotMask\}\_j\; \backslash cap\; \backslash text\{GeoPrior\}\_a\; \backslash neq\; \backslash emptyset]\; \backslash cdot\; \backslash log\; p(a\; |\; k\_j)$$

Symbol-by-symbol:

• $\mathcal{A}\backslash mathcal\{A\}$: Set of affordance types {grasp, support, hinge, button, ...}
• ${\text{GeoPrior}}_a\backslash text\{GeoPrior\}\_a$: Regions satisfying geometric prior for affordance $aa$
• $\mathbb{1}[\cdot ]\backslash mathbb\{1\}[\backslash cdot]$: Indicator function (1 if condition true, 0 otherwise)
• $p(a\mathrm{\mid }{k}_j)p(a\; |\; k\_j)$: Predicted probability of affordance $aa$ given slot key $k_{j}k\_j$

Geometric priors (examples):

• Grasp (handle): Elongated cylindrical regions where ${\lambda }_1\gg {\lambda }_2\approx {\lambda }_3\backslash lambda\_1\; \backslash gg\; \backslash lambda\_2\; \backslash approx\; \backslash lambda\_3$ (PCA eigenvalues)
• Support: Horizontal surfaces with upward normal: $\mathbf{n}\cdot \widehat{z}>0.9\backslash mathbf\{n\}\; \backslash cdot\; \backslash hat\{z\}\; >\; 0.9$
• Hinge: Concave junctions between planar surfaces
• Button: Small convex protrusions on flat surfaces

Intuitive Explanation:
If a slot attends to a cylindrical region, it should predict "grasp" affordance. This loss encourages geometric consistency between attention patterns and affordance predictions.

---

Priority 3: Relational Consistency Loss

$${\mathcal{L}}_{\text{rel-consist}}=\sum _{(i,j)\in {\mathcal{E}}_{\text{rel}}}\mathrm{\parallel }\text{Centroid}({\text{SlotMask}}_i)-\text{Centroid}({\text{SlotMask}}_j)-\mathrm{\Delta }{t}_{ij}^{\text{rel}}{\mathrm{\parallel }}_2^2\backslash mathcal\{L\}\_\{\backslash text\{rel-consist\}\}\; =\; \backslash sum\_\{(i,j)\; \backslash in\; \backslash mathcal\{E\}\_\{\backslash text\{rel\}\}\}\; \backslash |\; \backslash text\{Centroid\}(\backslash text\{SlotMask\}\_i)\; -\; \backslash text\{Centroid\}(\backslash text\{SlotMask\}\_j)\; -\; \backslash Delta\; t\_\{ij\}^\{\backslash text\{rel\}\}\; \backslash |\_2^2$$

Symbol-by-symbol:

• ${\mathcal{E}}_{\text{rel}}\backslash mathcal\{E\}\_\{\backslash text\{rel\}\}$: Set of slot pairs connected by relational tokens
• $\text{Centroid}({\text{SlotMask}}_i)\in {\mathbb{R}}^2\backslash text\{Centroid\}(\backslash text\{SlotMask\}\_i)\; \backslash in\; \backslash mathbb\{R\}^2$: 2D centroid of slot $ii$'s attention mask
• $\mathrm{\Delta }{t}_{ij}^{\text{rel}}\in {\mathbb{R}}^2\backslash Delta\; t\_\{ij\}^\{\backslash text\{rel\}\}\; \backslash in\; \backslash mathbb\{R\}^2$: Expected relative position from relational token (projected to 2D)

Intuitive Explanation:
If relational tokens say "slot $ii$ connects to slot $jj$ with offset $\mathrm{\Delta }t\backslash Delta\; t$," then their attention centroids should reflect this spatial relationship. A handle connected to a door should attend to adjacent regions.

---

Priority 4: Slot Diversity Regularization

$${\mathcal{L}}_{\text{diversity}}=\underset{\text{entropy maximization}}{\underbrace{-\sum _{j=1}^J\mathcal{H}(a_j)}}+\underset{\text{repulsion}}{\underbrace{{\lambda }_{\text{rep}}\sum _{i\mathrm{\ne }j}\max (0,\cos (k_i,k_j)-\delta )}}\backslash mathcal\{L\}\_\{\backslash text\{diversity\}\}\; =\; \backslash underbrace\{-\backslash sum\_\{j=1\}^\{J\}\; \backslash mathcal\{H\}(a\_j)\}\_\{\backslash text\{entropy\; maximization\}\}\; +\; \backslash underbrace\{\backslash lambda\_\{\backslash text\{rep\}\}\; \backslash sum\_\{i\; \backslash neq\; j\}\; \backslash max(0,\; \backslash cos(k\_i,\; k\_j)\; -\; \backslash delta)\}\_\{\backslash text\{repulsion\}\}$$

Symbol-by-symbol:

• $\mathcal{H}(a_j)=-{\sum }_t{a}_{j,t}\log a_{j,t}\backslash mathcal\{H\}(a\_j)\; =\; -\backslash sum\_t\; a\_\{j,t\}\; \backslash log\; a\_\{j,t\}$: Entropy of slot $jj$'s attention distribution
• $\cos (k_i,k_j)\backslash cos(k\_i,\; k\_j)$: Cosine similarity between slot keys
• $\delta \backslash delta$: Similarity threshold (typical: 0.5)
• ${\lambda }_{\text{rep}}\backslash lambda\_\{\backslash text\{rep\}\}$: Repulsion strength

Intuitive Explanation:
The entropy term encourages each slot to attend broadly (not collapse to a single token). The repulsion term pushes slot keys apart, preventing multiple slots from representing the same part.

---

4.3 Structure-Only Key Learning via Invariance

Problem: Keys might encode texture/color, causing retrieval to match appearance rather than structure.

Solution: Invariance regularization forces keys to ignore appearance.

---

Combined Invariance Loss:

$${\mathcal{L}}_{\text{inv}}={\mathcal{L}}_{\text{inv}}^{\text{tex}}+{\lambda }_p{\mathcal{L}}_{\text{inv}}^{\text{prompt}}+{\lambda }_c{\mathcal{L}}_{\text{inv}}^{\text{cross}}\backslash mathcal\{L\}\_\{\backslash text\{inv\}\}\; =\; \backslash mathcal\{L\}\_\{\backslash text\{inv\}\}^\{\backslash text\{tex\}\}\; +\; \backslash lambda\_p\; \backslash mathcal\{L\}\_\{\backslash text\{inv\}\}^\{\backslash text\{prompt\}\}\; +\; \backslash lambda\_c\; \backslash mathcal\{L\}\_\{\backslash text\{inv\}\}^\{\backslash text\{cross\}\}$$

---

Texture Augmentation Invariance:

$${\mathcal{L}}_{\text{inv}}^{\text{tex}}=\sum _{j=1}^J\mathrm{\parallel }{k}_j(\mathcal{I})-k_j({\text{Aug}}_{\text{tex}}(\mathcal{I})){\mathrm{\parallel }}_2^2\backslash mathcal\{L\}\_\{\backslash text\{inv\}\}^\{\backslash text\{tex\}\}\; =\; \backslash sum\_\{j=1\}^\{J\}\; \backslash |\; k\_j(\backslash mathcal\{I\})\; -\; k\_j(\backslash text\{Aug\}\_\{\backslash text\{tex\}\}(\backslash mathcal\{I\}))\; \backslash |\_2^2$$

Symbol-by-symbol:

• $k_j(\mathcal{I})k\_j(\backslash mathcal\{I\})$: Key from slot $jj$ given original image
• ${\text{Aug}}_{\text{tex}}(\mathcal{I})\backslash text\{Aug\}\_\{\backslash text\{tex\}\}(\backslash mathcal\{I\})$: Texture-augmented image (hue shift, saturation change, style transfer)
• $\mathrm{\parallel }\cdot {\mathrm{\parallel }}_2^2\backslash |\backslash cdot\backslash |\_2^2$: Squared L2 norm

Intuitive Explanation:
The same object with different colors/textures should produce identical keys. We augment textures and penalize key differences.

---

Prompt Invariance (Selective):

$${\mathcal{L}}_{\text{inv}}^{\text{prompt}}=\sum _{j=1}^J\mathrm{\parallel }{k}_j(P_1,\mathcal{I})-k_j(P_2,\mathcal{I}){\mathrm{\parallel }}_2^2\cdot \mathbb{1}[\text{SameStructure}(P_1,P_2)]\backslash mathcal\{L\}\_\{\backslash text\{inv\}\}^\{\backslash text\{prompt\}\}\; =\; \backslash sum\_\{j=1\}^\{J\}\; \backslash |\; k\_j(P\_1,\; \backslash mathcal\{I\})\; -\; k\_j(P\_2,\; \backslash mathcal\{I\})\; \backslash |\_2^2\; \backslash cdot\; \backslash mathbb\{1\}[\backslash text\{SameStructure\}(P\_1,\; P\_2)]$$

Symbol-by-symbol:

• $P_1,P_{2}P\_1,\; P\_2$: Two different prompts
• $\text{SameStructure}(P_1,P_2)\backslash text\{SameStructure\}(P\_1,\; P\_2)$: True if prompts describe same structure with different appearance

Examples:

• $P_{1}P\_1$ = "red wooden mug" vs $P_{2}P\_2$ = "blue ceramic mug" → Same structure, apply invariance
• $P_{1}P\_1$ = "mug with handle" vs $P_{2}P\_2$ = "mug without handle" → Different structure, skip invariance

Intuitive Explanation:
"Red wooden chair" and "blue metal chair" should retrieve the same structural Engrams. But "chair with armrests" and "chair without armrests" are structurally different.

---

Cross-Instance Structural Alignment:

$${\mathcal{L}}_{\text{inv}}^{\text{cross}}=\underset{\text{attract same-structure pairs}}{\underbrace{-\sum _{(i,j)\in {\mathcal{S}}_{\text{same}}}\cos (k_i,k_j)}}+\underset{\text{repel different-structure pairs}}{\underbrace{\sum _{(i,j)\in {\mathcal{S}}_{\text{diff}}}\max (0,\cos (k_i,k_j)-\delta )}}\backslash mathcal\{L\}\_\{\backslash text\{inv\}\}^\{\backslash text\{cross\}\}\; =\; \backslash underbrace\{-\backslash sum\_\{(i,j)\; \backslash in\; \backslash mathcal\{S\}\_\{\backslash text\{same\}\}\}\; \backslash cos(k\_i,\; k\_j)\}\_\{\backslash text\{attract\; same-structure\; pairs\}\}\; +\; \backslash underbrace\{\backslash sum\_\{(i,j)\; \backslash in\; \backslash mathcal\{S\}\_\{\backslash text\{diff\}\}\}\; \backslash max(0,\; \backslash cos(k\_i,\; k\_j)\; -\; \backslash delta)\}\_\{\backslash text\{repel\; different-structure\; pairs\}\}$$

Symbol-by-symbol:

• ${\mathcal{S}}_{\text{same}}\backslash mathcal\{S\}\_\{\backslash text\{same\}\}$: Pairs of keys from same structure (different instances/textures)
• ${\mathcal{S}}_{\text{diff}}\backslash mathcal\{S\}\_\{\backslash text\{diff\}\}$: Pairs of keys from different structures
• $\delta \backslash delta$: Margin threshold

Intuitive Explanation:
This is contrastive learning for structure. Keys from structurally similar objects should be close; keys from different structures should be far.

---

4.4 Memory Key Generation

Strategy A (Recommended): Shared Encoder Pipeline

$$k_m=\frac{1}{\mathrm{\mid }\mathcal{V}\mathrm{\mid }\cdot \mathrm{\mid }\mathcal{T}\mathrm{\mid }}\sum _{v\in \mathcal{V}}\sum _{\text{tex}\in \mathcal{T}}\mathcal{K}(E_{\text{VL}}(P_{\text{neutral}},{\text{Render}}_v(S_m,\text{tex})))k\_m\; =\; \backslash frac\{1\}\{|\backslash mathcal\{V\}|\; \backslash cdot\; |\backslash mathcal\{T\}|\}\; \backslash sum\_\{v\; \backslash in\; \backslash mathcal\{V\}\}\; \backslash sum\_\{\backslash text\{tex\}\; \backslash in\; \backslash mathcal\{T\}\}\; \backslash mathcal\{K\}\backslash big(E\_\{\backslash text\{VL\}\}(P\_\{\backslash text\{neutral\}\},\; \backslash text\{Render\}\_\{v\}(S\_m,\; \backslash text\{tex\}))\backslash big)$$

Symbol-by-symbol:

• $S_{m}S\_m$: Source 3D shape for Engram $mm$
• $\mathcal{V}\backslash mathcal\{V\}$: Set of viewpoints (e.g., 8 canonical views)
• $\mathcal{T}\backslash mathcal\{T\}$: Set of texture variations
• ${\text{Render}}_v(S_m,\text{tex})\backslash text\{Render\}\_\{v\}(S\_m,\; \backslash text\{tex\})$: Rendered image from viewpoint $vv$ with texture $\text{tex}\backslash text\{tex\}$
• $P_{\text{neutral}}P\_\{\backslash text\{neutral\}\}$: Structure-focused prompt (e.g., "a 3D object")

Intuitive Explanation:
Memory keys are generated using the same pipeline as query keys, ensuring they live in the same space. Averaging over viewpoints and textures ensures the key captures structure, not appearance or viewpoint.

---

4.5 Modular Engram Components (Detailed Specifications)

4.5.1 Relational Token Format

Storage Format (23-dimensional raw vector):

Model Input Format (projected to $d_{r}d\_r$ dimensions):

$$e_m^{\text{rel, proj}}=\text{MLP}([\text{Embed}(\text{part\_i});\text{Embed}(\text{part\_j});\text{continuous\_fields}])e\_m^\{\backslash text\{rel,\; proj\}\}\; =\; \backslash text\{MLP\}\backslash big([\backslash text\{Embed\}(\backslash text\{part\backslash \_i\});\; \backslash text\{Embed\}(\backslash text\{part\backslash \_j\});\; \backslash text\{continuous\backslash \_fields\}]\backslash big)$$

Symbol-by-symbol:

• $\text{Embed}(\cdot )\backslash text\{Embed\}(\backslash cdot)$: Learned embedding table for integer indices
• $\text{continuous\_fields}\backslash text\{continuous\backslash \_fields\}$: The 20 continuous dimensions (joint_type through valid_mask)
• $\text{MLP}\backslash text\{MLP\}$: Multi-layer perceptron projecting to $d_{r}d\_r$ dimensions

---

4.5.2 Symmetry Component Format

---

4.5.3 Affordance Component Format

---

4.6 Modular Engram Composition

Geometric Component (weighted average):

$$Z_e^{\text{geo}}=\sum _{j=1}^J\sum _{m=1}^M{\alpha }_{j,m}\cdot \text{SE3Transform}(e_m^{\text{geo}},\mathrm{\Delta }{T}_{j,m})Z\_e^\{\backslash text\{geo\}\}\; =\; \backslash sum\_\{j=1\}^\{J\}\; \backslash sum\_\{m=1\}^\{M\}\; \backslash alpha\_\{j,m\}\; \backslash cdot\; \backslash text\{SE3Transform\}(e\_m^\{\backslash text\{geo\}\},\; \backslash Delta\; T\_\{j,m\})$$

Symbol-by-symbol:

• ${\alpha }_{j,m}\backslash alpha\_\{j,m\}$: Retrieval weight (slot $jj$, Engram $mm$)
• $e_m^{\text{geo}}e\_m^\{\backslash text\{geo\}\}$: Geometric latent of Engram $mm$
• $\mathrm{\Delta }{T}_{j,m}=T_j\cdot (T_m^c{)}^{-1}\backslash Delta\; T\_\{j,m\}\; =\; T\_j\; \backslash cdot\; (T\_m^c)^\{-1\}$: Relative transformation from canonical to target pose
• $\text{SE3Transform}\backslash text\{SE3Transform\}$: Transformation operator (MLP approximation in latent space)

Intuitive Explanation:
Each retrieved geometric Engram is transformed to its target pose, then all are blended by retrieval weights.

---

Relational Component (concatenation with threshold):

$$Z_e^{\text{rel}}={\text{Concat}}_{j,m:{\alpha }_{j,m}>{\theta }_{\text{rel}}}\left[e_{j,m}^{\text{rel, proj}}\right]Z\_e^\{\backslash text\{rel\}\}\; =\; \backslash text\{Concat\}\_\{j,m:\; \backslash alpha\_\{j,m\}\; >\; \backslash theta\_\{\backslash text\{rel\}\}\}\; \backslash big[\; e\_\{j,m\}^\{\backslash text\{rel,\; proj\}\}\; \backslash big]$$

Intuitive Explanation:
We concatenate relational tokens from high-confidence retrievals. Thresholding prevents noisy low-weight retrievals from polluting the relation set.

---

Symmetry Component (weighted voting):

$$Z_e^{\text{sym}}=\arg \underset{\text{type}}{\max }\sum _{j,m}{\alpha }_{j,m}\cdot \mathbb{1}[e_m^{\text{sym}}\mathrm{.}\text{type}=\text{type}]Z\_e^\{\backslash text\{sym\}\}\; =\; \backslash arg\backslash max\_\{\backslash text\{type\}\}\; \backslash sum\_\{j,m\}\; \backslash alpha\_\{j,m\}\; \backslash cdot\; \backslash mathbb\{1\}[e\_m^\{\backslash text\{sym\}\}.\backslash text\{type\}\; =\; \backslash text\{type\}]$$

Intuitive Explanation:
Symmetry is discrete (an object is either bilaterally symmetric or not). We take a weighted vote over retrieved symmetry types.

---

Affordance Component (transform and merge):

$$Z_e^{\text{aff}}=\bigcup _{j,m:{\alpha }_{j,m}>{\theta }_{\text{aff}}}\text{Transform}(e_m^{\text{aff}},\mathrm{\Delta }{T}_{j,m})Z\_e^\{\backslash text\{aff\}\}\; =\; \backslash bigcup\_\{j,m:\; \backslash alpha\_\{j,m\}\; >\; \backslash theta\_\{\backslash text\{aff\}\}\}\; \backslash text\{Transform\}(e\_m^\{\backslash text\{aff\}\},\; \backslash Delta\; T\_\{j,m\})$$

Intuitive Explanation:
Affordance markers (grasp points, etc.) are transformed to target poses and merged. The union collects all relevant interaction points.

---

4.7 Enforcing Relational Constraints: Kinematic Consistency Loss

Problem: Cross-attention alone allows the generator to "politely ignore" relational tokens.

Solution: Explicit loss penalizing kinematic violations.

---

Combined Kinematic Loss:

$${\mathcal{L}}_{\text{kin}}={\mathcal{L}}_{\text{kin}}^{\text{joint}}+{\lambda }_{\text{axis}}{\mathcal{L}}_{\text{kin}}^{\text{axis}}+{\lambda }_{\text{limit}}{\mathcal{L}}_{\text{kin}}^{\text{limit}}\backslash mathcal\{L\}\_\{\backslash text\{kin\}\}\; =\; \backslash mathcal\{L\}\_\{\backslash text\{kin\}\}^\{\backslash text\{joint\}\}\; +\; \backslash lambda\_\{\backslash text\{axis\}\}\; \backslash mathcal\{L\}\_\{\backslash text\{kin\}\}^\{\backslash text\{axis\}\}\; +\; \backslash lambda\_\{\backslash text\{limit\}\}\; \backslash mathcal\{L\}\_\{\backslash text\{kin\}\}^\{\backslash text\{limit\}\}$$

---

Joint Position Consistency:

$${\mathcal{L}}_{\text{kin}}^{\text{joint}}=\sum _{(i,j)\in \mathcal{E}}\mathrm{\parallel }(T_i^{-1}\cdot T_j)-T_{ij}^{\text{rel}}{\mathrm{\parallel }}_{\text{geo}}^2\backslash mathcal\{L\}\_\{\backslash text\{kin\}\}^\{\backslash text\{joint\}\}\; =\; \backslash sum\_\{(i,j)\; \backslash in\; \backslash mathcal\{E\}\}\; \backslash |\; (T\_i^\{-1\}\; \backslash cdot\; T\_j)\; -\; T\_\{ij\}^\{\backslash text\{rel\}\}\; \backslash |\_\{\backslash text\{geo\}\}^2$$

Symbol-by-symbol:

• $\mathcal{E}\backslash mathcal\{E\}$: Set of connected part pairs from relational tokens
• $T_i,T_j\in SE(3)T\_i,\; T\_j\; \backslash in\; SE(3)$: Estimated poses of slots $ii$ and $jj$
• $T_i^{-1}\cdot T_{j}T\_i^\{-1\}\; \backslash cdot\; T\_j$: Relative pose from $ii$ to $jj$
• $T_{ij}^{\text{rel}}T\_\{ij\}^\{\backslash text\{rel\}\}$: Expected relative pose from relational token
• $\mathrm{\parallel }\cdot {\mathrm{\parallel }}_{\text{geo}}\backslash |\backslash cdot\backslash |\_\{\backslash text\{geo\}\}$: Geodesic distance on SE(3)

Geodesic distance on SE(3):

$$\mathrm{\parallel }{T}_1-T_2{\mathrm{\parallel }}_{\text{geo}}=\mathrm{\parallel }\log (R_1^{\mathrm{\top }}{R}_2){\mathrm{\parallel }}_F+{\lambda }_t\mathrm{\parallel }{t}_1-t_2{\mathrm{\parallel }}_2\backslash |T\_1\; -\; T\_2\backslash |\_\{\backslash text\{geo\}\}\; =\; \backslash |\backslash log(R\_1^\backslash top\; R\_2)\backslash |\_F\; +\; \backslash lambda\_t\; \backslash |t\_1\; -\; t\_2\backslash |\_2$$

where $\log (\cdot )\backslash log(\backslash cdot)$ is the matrix logarithm mapping rotation to axis-angle.

Intuitive Explanation:
If a relational token says "the handle is attached to the door with relative pose $T_{ij}^{\text{rel}}T\_\{ij\}^\{\backslash text\{rel\}\}$," the generated geometry must satisfy this. Deviation is penalized.

---

Joint Axis Alignment:

$${\mathcal{L}}_{\text{kin}}^{\text{axis}}=\sum _{(i,j):\text{revolute/prismatic}}(1-\mathrm{\mid }{\text{axis}}_{ij}^{\mathrm{\top }}\cdot {\text{PCA}}_1(G_{\text{joint}}^{ij})\mathrm{\mid })\backslash mathcal\{L\}\_\{\backslash text\{kin\}\}^\{\backslash text\{axis\}\}\; =\; \backslash sum\_\{(i,j):\; \backslash text\{revolute/prismatic\}\}\; \backslash big(1\; -\; |\backslash text\{axis\}\_\{ij\}^\backslash top\; \backslash cdot\; \backslash text\{PCA\}\_1(G\_\{\backslash text\{joint\}\}^\{ij\})|\backslash big)$$

Symbol-by-symbol:

• ${\text{axis}}_{ij}\backslash text\{axis\}\_\{ij\}$: Joint axis from relational token (unit vector)
• $G_{\text{joint}}^{ij}G\_\{\backslash text\{joint\}\}^\{ij\}$: Geometry near the joint between parts $ii$ and $jj$
• ${\text{PCA}}_1(\cdot )\backslash text\{PCA\}\_1(\backslash cdot)$: First principal component (dominant direction)

Intuitive Explanation:
For a revolute joint (like a door hinge), the local geometry should be aligned with the joint axis. A poorly-aligned hinge would have high loss.

---

Joint Limit Enforcement:

$${\mathcal{L}}_{\text{kin}}^{\text{limit}}=\sum _{(i,j)}[\max (0,{\theta }_{ij}-{\theta }_{\max })+\max (0,{\theta }_{\min }-{\theta }_{ij})]\backslash mathcal\{L\}\_\{\backslash text\{kin\}\}^\{\backslash text\{limit\}\}\; =\; \backslash sum\_\{(i,j)\}\; \backslash big[\backslash max(0,\; \backslash theta\_\{ij\}\; -\; \backslash theta\_\{\backslash max\})\; +\; \backslash max(0,\; \backslash theta\_\{\backslash min\}\; -\; \backslash theta\_\{ij\})\backslash big]$$

Symbol-by-symbol:

• ${\theta }_{ij}\backslash theta\_\{ij\}$: Current joint angle between parts $ii$ and $jj$
• ${\theta }_{\min },{\theta }_{\max }\backslash theta\_\{\backslash min\},\; \backslash theta\_\{\backslash max\}$: Joint limits from relational token
• $\max (0,\cdot )\backslash max(0,\; \backslash cdot)$: ReLU (only penalize violations)

Intuitive Explanation:
A door can only open so far. If the generated geometry implies the door is open beyond its limit, this is penalized.

---

Relational Tokens also Feed Pose Refinement:

$$T_j^{\text{refined}}=\text{PoseRefineHead}(T_j^{\text{coarse}},k_j,Z_e^{\text{rel}})T\_j^\{\backslash text\{refined\}\}\; =\; \backslash text\{PoseRefineHead\}(T\_j^\{\backslash text\{coarse\}\},\; k\_j,\; Z\_e^\{\backslash text\{rel\}\})$$

Intuitive Explanation:
Beyond cross-attention in the generator, relational tokens also directly inform pose estimation. This creates multiple pathways for relational information, making it harder to ignore.

---

4.8 Gravity-Aligned Pose Estimation (No Ground Truth)

Challenge: Single-view pose estimation usually needs GT supervision.

Solution: Use robot IMU + multi-view consistency.

---

Gravity Prior Loss:

$${\mathcal{L}}_{\text{gravity}}=\sum _{j=1}^J(1-\mathrm{\mid }{\mathbf{g}}^{\mathrm{\top }}\cdot R_j\cdot \widehat{z}\mathrm{\mid })\cdot \mathbb{1}[\text{GravityAligned}(j)]\backslash mathcal\{L\}\_\{\backslash text\{gravity\}\}\; =\; \backslash sum\_\{j=1\}^\{J\}\; \backslash big(1\; -\; |\backslash mathbf\{g\}^\backslash top\; \backslash cdot\; R\_j\; \backslash cdot\; \backslash hat\{z\}|\backslash big)\; \backslash cdot\; \backslash mathbb\{1\}[\backslash text\{GravityAligned\}(j)]$$

Symbol-by-symbol:

• $\mathbf{g}\in {\mathbb{R}}^3\backslash mathbf\{g\}\; \backslash in\; \backslash mathbb\{R\}^3$: Gravity direction from IMU (unit vector)
• $R_j\in SO(3)R\_j\; \backslash in\; SO(3)$: Estimated rotation for slot $jj$
• $\widehat{z}=[0,0,1{]}^{\mathrm{\top }}\backslash hat\{z\}\; =\; [0,\; 0,\; 1]^\backslash top$: Canonical up-vector
• $\mathbb{1}[\text{GravityAligned}(j)]\backslash mathbb\{1\}[\backslash text\{GravityAligned\}(j)]$: 1 if slot $jj$ should be gravity-aligned (supports, tables, floors)

Intuitive Explanation:
A table surface should be horizontal (aligned with gravity). We use the robot's IMU to know which way is "up" and penalize misalignment.

---

Multi-View Consistency Loss:

$${\mathcal{L}}_{\text{MV}}=\sum _{v_1\mathrm{\ne }{v}_2}\mathrm{\parallel }{\text{Render}}_{v_1}(G)-{\text{Warp}}_{v_1\leftarrow v_2}({\text{Render}}_{v_2}(G)){\mathrm{\parallel }}_1\backslash mathcal\{L\}\_\{\backslash text\{MV\}\}\; =\; \backslash sum\_\{v\_1\; \backslash neq\; v\_2\}\; \backslash |\; \backslash text\{Render\}\_\{v\_1\}(G)\; -\; \backslash text\{Warp\}\_\{v\_1\; \backslash leftarrow\; v\_2\}(\backslash text\{Render\}\_\{v\_2\}(G))\; \backslash |\_1$$

Symbol-by-symbol:

• ${\text{Render}}_v(G)\backslash text\{Render\}\_\{v\}(G)$: Rendering from viewpoint $vv$
• ${\text{Warp}}_{v_1\leftarrow v_2}\backslash text\{Warp\}\_\{v\_1\; \backslash leftarrow\; v\_2\}$: Image warping from $v_{2}v\_2$ to $v_{1}v\_1$ using estimated depth/pose

Intuitive Explanation:
If the 3D reconstruction is correct, rendering from different viewpoints and warping should produce consistent images. Inconsistencies indicate pose or geometry errors.

---

Combined Pose Loss (No GT):

$${\mathcal{L}}_{\text{pose}}^{\text{no-GT}}={\lambda }_g{\mathcal{L}}_{\text{gravity}}+{\lambda }_{\text{MV}}{\mathcal{L}}_{\text{MV}}+{\lambda }_{\text{kin}}{\mathcal{L}}_{\text{kin}}\backslash mathcal\{L\}\_\{\backslash text\{pose\}\}^\{\backslash text\{no-GT\}\}\; =\; \backslash lambda\_g\; \backslash mathcal\{L\}\_\{\backslash text\{gravity\}\}\; +\; \backslash lambda\_\{\backslash text\{MV\}\}\; \backslash mathcal\{L\}\_\{\backslash text\{MV\}\}\; +\; \backslash lambda\_\{\backslash text\{kin\}\}\; \backslash mathcal\{L\}\_\{\backslash text\{kin\}\}$$

Intuitive Explanation:
Without ground-truth poses, we combine: (1) gravity alignment (from IMU), (2) multi-view consistency (self-supervised), (3) kinematic constraints (from relational tokens). Together, these provide sufficient signal to learn reasonable poses.

---

4.9 Multi-Pathway Generator Conditioning

Pathway A: AdaLN for Geometry

$$h_l^{\mathrm{\prime }}={\gamma }_l(Z_e^{\text{geo}})\odot \text{LN}(h_l)+{\beta }_l(Z_e^{\text{geo}})h\text{'}\_l\; =\; \backslash gamma\_l(Z\_e^\{\backslash text\{geo\}\})\; \backslash odot\; \backslash text\{LN\}(h\_l)\; +\; \backslash beta\_l(Z\_e^\{\backslash text\{geo\}\})$$

Symbol-by-symbol:

• $h_l\in {\mathbb{R}}^{B\times C}h\_l\; \backslash in\; \backslash mathbb\{R\}^\{B\; \backslash times\; C\}$: Activation at layer $ll$ (batch $BB$, channels $CC$)
• $\text{LN}(h_l)=\frac{h_l-{\mu }_l}{{\sigma }_l}\backslash text\{LN\}(h\_l)\; =\; \backslash frac\{h\_l\; -\; \backslash mu\_l\}\{\backslash sigma\_l\}$: Layer normalization
• ${\mu }_l,{\sigma }_l\backslash mu\_l,\; \backslash sigma\_l$: Mean and std (per-sample, across channels for MLPs; per-token for Transformers)
• ${\gamma }_l(Z_e^{\text{geo}})=W_{\gamma }^l{Z}_e^{\text{geo}}+b_{\gamma }^l\backslash gamma\_l(Z\_e^\{\backslash text\{geo\}\})\; =\; W\_\backslash gamma^l\; Z\_e^\{\backslash text\{geo\}\}\; +\; b\_\backslash gamma^l$: Scale from geometric prior
• ${\beta }_l(Z_e^{\text{geo}})=W_{\beta }^l{Z}_e^{\text{geo}}+b_{\beta }^l\backslash beta\_l(Z\_e^\{\backslash text\{geo\}\})\; =\; W\_\backslash beta^l\; Z\_e^\{\backslash text\{geo\}\}\; +\; b\_\backslash beta^l$: Shift from geometric prior
• $\odot \backslash odot$: Element-wise multiplication

Intuitive Explanation:
AdaLN injects the geometric prior by controlling the scale and shift of activations at every layer. This is a strong form of conditioning—the prior directly modulates the "communication channels" in the network.

---

Pathway B: Cross-Attention for Relations

$$h_l^{\mathrm{\prime }}=h_l+\text{CrossAttn}(Q=h_l,K=Z_e^{\text{rel}},V=Z_e^{\text{rel}},M=M_{\text{valid}})h\text{'}\_l\; =\; h\_l\; +\; \backslash text\{CrossAttn\}(Q=h\_l,\; K=Z\_e^\{\backslash text\{rel\}\},\; V=Z\_e^\{\backslash text\{rel\}\},\; M=M\_\{\backslash text\{valid\}\})$$

Expanded Cross-Attention:

$$\text{CrossAttn}(Q,K,V,M)=\text{softmax}(\frac{Q{K}^{\mathrm{\top }}}{\sqrt{d}}+M)V\backslash text\{CrossAttn\}(Q,\; K,\; V,\; M)\; =\; \backslash text\{softmax\}\backslash bigg(\backslash frac\{QK^\backslash top\}\{\backslash sqrt\{d\}\}\; +\; M\backslash bigg)\; V$$

where $M_{ij}=-\mathrm{\infty }M\_\{ij\}\; =\; -\backslash infty$ if token $jj$ is invalid (padding).

Intuitive Explanation:
The generator can "ask questions" about relational structure. "Is there a joint here? What type? What axis?" The cross-attention mechanism retrieves relevant relational information.

---

Pathway C: Per-Slot Symmetry

Symbol-by-symbol:

• $G_j(x)G\_j(x)$: Final field value for slot $jj$ at point $xx$
• $G_{\text{base}}^{j}G\_\{\backslash text\{base\}\}^j$: Base generator output (before symmetry)
• $n_{j}n\_j$: Symmetry count (e.g., 4 for chair legs)
• $R_{\text{axis}}^{\theta }R\_\{\backslash text\{axis\}\}^\{\backslash theta\}$: Rotation by $\theta \backslash theta$ around symmetry axis
• $x_{\text{local}}=(T_j^c{)}^{-1}\cdot xx\_\{\backslash text\{local\}\}\; =\; (T\_j^c)^\{-1\}\; \backslash cdot\; x$: Point transformed to slot's local frame
• $\text{Reflect}(\cdot )\backslash text\{Reflect\}(\backslash cdot)$: Reflection across symmetry plane

Intuitive Explanation:
Symmetry is enforced by averaging the field over symmetric transformations. For 4-way radial symmetry (like chair legs), we evaluate the base field at 4 rotated positions and average. This guarantees perfect symmetry.

---

Pathway D: Affordance Head

$$A_{\text{pred}}=\text{AffordanceHead}(G_{\text{features}},Z_e^{\text{aff}})A\_\{\backslash text\{pred\}\}\; =\; \backslash text\{AffordanceHead\}(G\_\{\backslash text\{features\}\},\; Z\_e^\{\backslash text\{aff\}\})$$

Intuitive Explanation:
A dedicated head predicts affordance heatmaps (grasp points, support surfaces) using both generator features and retrieved affordance markers. This provides explicit manipulation-relevant outputs.

---

4.10 Composition Operators (Generator-Specific)

SDF Composition (Smooth Minimum):

$$f(x)={\text{SoftMin}}_j(f_j(x))=-\frac{1}{\beta }\log \sum _{j=1}^J\exp (-\beta \cdot f_j(x))f(x)\; =\; \backslash text\{SoftMin\}\_j\backslash big(f\_j(x)\backslash big)\; =\; -\backslash frac\{1\}\{\backslash beta\}\backslash log\backslash sum\_\{j=1\}^\{J\}\; \backslash exp(-\backslash beta\; \backslash cdot\; f\_j(x))$$

Symbol-by-symbol:

• $f_j(x)f\_j(x)$: SDF value from slot $jj$ at point $xx$ (negative inside, positive outside)
• $\beta >0\backslash beta\; >\; 0$: Sharpness parameter (larger = closer to hard min)

Limit behavior:

• $\beta \to \mathrm{\infty }\backslash beta\; \backslash to\; \backslash infty$: $\text{SoftMin}\to \min \backslash text\{SoftMin\}\; \backslash to\; \backslash min$ (hard union)
• $\beta \to 0\backslash beta\; \backslash to\; 0$: $\text{SoftMin}\to \text{mean}\backslash text\{SoftMin\}\; \backslash to\; \backslash text\{mean\}$ (smooth blend)

Intuitive Explanation:
SDFs represent shapes as distance fields. To combine shapes (union), we take the minimum. Soft-min provides smooth transitions at part boundaries.

---

NeRF Composition (Density Mixture):

$$\sigma (x)=\sum _{j=1}^J{w}_j(x)\cdot {\sigma }_j(x)\backslash sigma(x)\; =\; \backslash sum\_\{j=1\}^\{J\}\; w\_j(x)\; \backslash cdot\; \backslash sigma\_j(x)$$

$$c(x)=\frac{\sum _{j=1}^J{w}_j(x)\cdot {\sigma }_j(x)\cdot c_j(x)}{\sum _{j=1}^J{w}_j(x)\cdot {\sigma }_j(x)+ϵ}c(x)\; =\; \backslash frac\{\backslash sum\_\{j=1\}^\{J\}\; w\_j(x)\; \backslash cdot\; \backslash sigma\_j(x)\; \backslash cdot\; c\_j(x)\}\{\backslash sum\_\{j=1\}^\{J\}\; w\_j(x)\; \backslash cdot\; \backslash sigma\_j(x)\; +\; \backslash epsilon\}$$

Symbol-by-symbol:

• ${\sigma }_j(x)\backslash sigma\_j(x)$: Density from slot $jj$ at point $xx$
• $c_j(x)c\_j(x)$: Color from slot $jj$
• $w_j(x)w\_j(x)$: Spatial weight for slot $jj$ (from lifted attention mask)
• $ϵ\backslash epsilon$: Small constant for numerical stability

Intuitive Explanation:
NeRF represents scenes as density and color fields. We combine slot contributions using spatial weights, with colors weighted by density (denser regions contribute more to final color).

---

3DGS Composition (Concatenation + Pruning):

$${\mathcal{G}}_{\text{final}}=\text{Prune}(\bigcup _{j=1}^J{\mathcal{G}}_j,{\theta }_{\text{opacity}})\backslash mathcal\{G\}\_\{\backslash text\{final\}\}\; =\; \backslash text\{Prune\}\backslash bigg(\backslash bigcup\_\{j=1\}^\{J\}\; \backslash mathcal\{G\}\_j,\; \backslash ;\backslash theta\_\{\backslash text\{opacity\}\}\backslash bigg)$$

Symbol-by-symbol:

• ${\mathcal{G}}_j\backslash mathcal\{G\}\_j$: Set of Gaussians from slot $jj$
• $\bigcup \backslash bigcup$: Set union (concatenation)
• $\text{Prune}(\cdot ,\theta )\backslash text\{Prune\}(\backslash cdot,\; \backslash theta)$: Remove Gaussians with opacity below $\theta \backslash theta$

Intuitive Explanation:
3D Gaussian Splatting represents scenes as collections of Gaussian primitives. We simply concatenate Gaussians from all slots, then prune low-opacity ones to remove redundancy.

---

5. Staged Memory Construction Protocol

Key Insight: Full Engram annotation is expensive. We propose staged construction for practical deployment.

5.1 Stage 1: Minimal Viable Memory (EG-3D Lite)

Required: $e^{\text{geo}}e^\{\backslash text\{geo\}\}$, $e^{\text{sym}}e^\{\backslash text\{sym\}\}$, $T^{c}T^c$ (coarse)

Annotation effort: Low (automatic from 3D scans)

• Geometry: PointNet++ encoder on mesh/point cloud
• Symmetry: PCA + RANSAC detection (fully automatic)
• Canonical frame: Gravity + principal axis alignment (automatic)

Capabilities: Basic shape retrieval, symmetry enforcement, rigid object reconstruction

---

5.2 Stage 2: Add Relational Structure

Added: $e^{\text{rel}}e^\{\backslash text\{rel\}\}$

Annotation effort: Medium (semi-automatic)

• Source: PartNet-Mobility dataset, or learned from articulation videos

Capabilities: Articulated object reconstruction, kinematic constraint enforcement

---

5.3 Stage 3: Add Affordance

Added: $e^{\text{aff}}e^\{\backslash text\{aff\}\}$

Annotation effort: Medium-High (from interaction data)

• Source: HOI videos, robot demonstrations, geometric priors

Capabilities: Full affordance-grounded reconstruction, direct grasp point prediction

---

5.4 Staged Training Protocol

Phase 1: Train with geo + sym only
         → Learn basic retrieval and composition

Phase 2: Freeze geo/sym modules, add rel
         → Learn relational cross-attention
         → Enable kinematic consistency loss

Phase 3: Freeze geo/sym/rel, add aff
         → Learn affordance head
         → Fine-tune with prioritized slot alignment

---

6. EG-3D Lite: Minimal Configuration

EG-3D Lite Loss:

$${\mathcal{L}}_{\text{Lite}}={\mathcal{L}}_{\text{rec}}+{\lambda }_{\text{inv}}{\mathcal{L}}_{\text{inv}}+{\lambda }_{\text{sym}}{\mathcal{L}}_{\text{sym}}+{\lambda }_{\text{gravity}}{\mathcal{L}}_{\text{gravity}}\backslash mathcal\{L\}\_\{\backslash text\{Lite\}\}\; =\; \backslash mathcal\{L\}\_\{\backslash text\{rec\}\}\; +\; \backslash lambda\_\{\backslash text\{inv\}\}\; \backslash mathcal\{L\}\_\{\backslash text\{inv\}\}\; +\; \backslash lambda\_\{\backslash text\{sym\}\}\; \backslash mathcal\{L\}\_\{\backslash text\{sym\}\}\; +\; \backslash lambda\_\{\backslash text\{gravity\}\}\; \backslash mathcal\{L\}\_\{\backslash text\{gravity\}\}$$

---

7. Full Training Objective

$$\mathcal{L}={\mathcal{L}}_{\text{rec}}+{\lambda }_{\text{inv}}{\mathcal{L}}_{\text{inv}}+{\lambda }_{\text{slot}}{\mathcal{L}}_{\text{slot-align}}+{\lambda }_{\text{kin}}{\mathcal{L}}_{\text{kin}}+{\lambda }_{\text{pose}}{\mathcal{L}}_{\text{pose}}+{\lambda }_{\text{aff}}{\mathcal{L}}_{\text{aff}}\backslash mathcal\{L\}\; =\; \backslash mathcal\{L\}\_\{\backslash text\{rec\}\}\; +\; \backslash lambda\_\{\backslash text\{inv\}\}\; \backslash mathcal\{L\}\_\{\backslash text\{inv\}\}\; +\; \backslash lambda\_\{\backslash text\{slot\}\}\; \backslash mathcal\{L\}\_\{\backslash text\{slot-align\}\}\; +\; \backslash lambda\_\{\backslash text\{kin\}\}\; \backslash mathcal\{L\}\_\{\backslash text\{kin\}\}\; +\; \backslash lambda\_\{\backslash text\{pose\}\}\; \backslash mathcal\{L\}\_\{\backslash text\{pose\}\}\; +\; \backslash lambda\_\{\backslash text\{aff\}\}\; \backslash mathcal\{L\}\_\{\backslash text\{aff\}\}$$

---

Reconstruction Loss:

$${\mathcal{L}}_{\text{rec}}=\underset{\text{image reconstruction}}{\underbrace{\mathrm{\parallel }R-R^{\ast }{\mathrm{\parallel }}_1}}+{\lambda }_{\text{CD}}\underset{\text{geometry reconstruction}}{\underbrace{\text{CD}(G,G^{\ast })}}\backslash mathcal\{L\}\_\{\backslash text\{rec\}\}\; =\; \backslash underbrace\{\backslash |R\; -\; R^*\backslash |\_1\}\_\{\backslash text\{image\; reconstruction\}\}\; +\; \backslash lambda\_\{\backslash text\{CD\}\}\; \backslash underbrace\{\backslash text\{CD\}(G,\; G^*)\}\_\{\backslash text\{geometry\; reconstruction\}\}$$

Chamfer Distance:

$$\text{CD}(G,G^{\ast })=\frac{1}{\mathrm{\mid }G\mathrm{\mid }}\sum _{x\in G}\underset{y\in G^{\ast }}{\min }\mathrm{\parallel }x-y{\mathrm{\parallel }}_2^2+\frac{1}{\mathrm{\mid }{G}^{\ast }\mathrm{\mid }}\sum _{y\in G^{\ast }}\underset{x\in G}{\min }\mathrm{\parallel }x-y{\mathrm{\parallel }}_2^2\backslash text\{CD\}(G,\; G^*)\; =\; \backslash frac\{1\}\{|G|\}\backslash sum\_\{x\; \backslash in\; G\}\; \backslash min\_\{y\; \backslash in\; G^*\}\; \backslash |x\; -\; y\backslash |\_2^2\; +\; \backslash frac\{1\}\{|G^*|\}\backslash sum\_\{y\; \backslash in\; G^*\}\; \backslash min\_\{x\; \backslash in\; G\}\; \backslash |x\; -\; y\backslash |\_2^2$$

Intuitive Explanation:
Reconstruction loss has two parts: (1) rendered images should match target images (L1 loss), (2) reconstructed geometry should match ground-truth geometry (Chamfer distance measures point cloud similarity).

---

Surface Regularization (Eikonal, for SDF):

$${\mathcal{L}}_{\text{surf}}={\mathbb{E}}_{x\sim U(\mathrm{\Omega })}[(\mathrm{\parallel }{\mathrm{\nabla }}_{x}f(x){\mathrm{\parallel }}_2-1{)}^2]\backslash mathcal\{L\}\_\{\backslash text\{surf\}\}\; =\; \backslash mathbb\{E\}\_\{x\; \backslash sim\; U(\backslash Omega)\}\; \backslash big[\; (\backslash |\backslash nabla\_x\; f(x)\backslash |\_2\; -\; 1)^2\; \backslash big]$$

Intuitive Explanation:
A valid SDF should have unit gradient magnitude (\(|\nabla f| = 1\)) almost everywhere. This regularization prevents degenerate solutions.

---

8. Computational Considerations

---

9. Failure Modes and Mitigations

9.1 Slot Collapse to Semantic Blobs

Mitigations (by priority):

1. Contact signal supervision (highest priority)
2. Geometric affordance priors
3. Relational consistency
4. Diversity regularization

9.2 Generator Ignoring Relational Tokens

Mitigations:

1. Kinematic consistency loss (explicit constraint)
2. Feed relational tokens to pose refinement head
3. Monitor ${\mathrm{\nabla }}_{\text{rel}}\mathcal{L}\backslash nabla\_\{\backslash text\{rel\}\}\; \backslash mathcal\{L\}$; alert if too small

9.3 Pose Estimation Without GT

Mitigations:

1. IMU gravity prior
2. Multi-view consistency
3. Kinematic loss provides indirect supervision

---

10. Hypotheses and Future Work

10.1 Hypotheses (To Be Validated)

1. Prioritized supervision → more functionally meaningful slots
2. Kinematic consistency loss → improved articulated object reconstruction
3. EG-3D Lite → meaningful gains even without rel/aff
4. Staged construction → practical deployment with minimal effort

10.2 Proposed Experiments

---

11. Conclusion

We presented Engram-Guided 2D-to-3D (EG-3D), a structural assembly framework for single-view 3D reconstruction. Key contributions:

1. Modular Engram Architecture with explicit composition operators
2. Prioritized Slot-Part Alignment: contact → geometric → relational → diversity
3. Kinematic Consistency Loss enforcing relational constraints
4. Gravity-Aligned Pose Estimation for GT-free settings
5. Staged Memory Construction enabling incremental deployment
6. Detailed mathematical specifications with intuitive explanations for every equation

This paper provides a complete foundation for implementation and empirical validation.

---

12. Engram-Guided Training Cycle

# Algorithm 1: Engram-Guided Training Cycle
# Input: EngramBank M, Dataset D, Stages S = {Lite, +Rel, +Aff}
# Output: Trained EG-3D model and VLA policy

# =======================================================
# STAGE-WISE TRAINING OF EG-3D
# =======================================================
for stage_s in S:                             # Iterate over Lite → +Rel → +Aff stages
    freeze_unused_components(stage_s)        # Freeze modules not used in the current stage

    for (I, P, G_star) in D:                 # Sample a minibatch (image, prompt, GT 3D)

        # -------------------------------------------------------
        # Visual-language encoding and part-level slot extraction
        # -------------------------------------------------------
        Z_tokens = E_VL(I, P)                # Fused visual-language tokens {z_t}
        K_slots  = SlotAttention(Z_tokens)  # Extract part-level structural keys {k_j}

        # -------------------------------------------------------
        # Engram retrieval and modular structural composition
        # -------------------------------------------------------
        Z_e = RetrieveAndCompose(            # Retrieve and assemble Engram components
                K_slots, M, stage_s)         # Z_e contains geo/sym/(rel)/(aff) by stage

        # -------------------------------------------------------
        # 3D generation and coarse pose estimation
        # -------------------------------------------------------
        G, T_coarse = G_theta(I, Z_e)        # Predict 3D geometry G and coarse slot poses {T_j^coarse}

        # -------------------------------------------------------
        # Pose refinement using relational tokens (if enabled)
        # -------------------------------------------------------
        if stage_s >= "+Rel":                # Only refine poses when relational structure is available
            T_ref = PoseRefine(              # Refine poses with relational and slot information
                        T_coarse, K_slots, Z_e.rel)   # Output refined poses {T_j^ref}
        else:
            T_ref = T_coarse                # Use coarse poses in Lite stage

        # -------------------------------------------------------
        # Compute stage-dependent training losses
        # -------------------------------------------------------
        L = L_rec(G, G_star)                # Reconstruction loss on geometry and rendering
            + λ_inv * L_inv(K_slots)        # Invariance loss to enforce structure-only keys
            + λ_sym * L_sym(G, Z_e.sym)     # Symmetry regularization loss

        if stage_s >= "+Rel":               # Additional losses for relational stages
            L += λ_slot * L_slot_align(K_slots)   # Slot-part alignment loss
               + λ_kin  * L_kin(G, T_ref, Z_e.rel) # Kinematic consistency loss
               + λ_pose * L_pose(T_ref, I)        # Pose loss (gravity + multi-view + kinematic)

        if stage_s == "+Aff":               # Additional loss for affordance stage
            L += λ_aff * L_aff(G, Z_e.aff)  # Affordance supervision loss

        # -------------------------------------------------------
        # Update only the parameters active in the current stage
        # -------------------------------------------------------
        update(params_of(stage_s), loss=L)  # Stage-wise parameter update


# =======================================================
# TRAINING VLA POLICY WITH FROZEN EG-3D
# =======================================================
freeze(EG3D)                                # Freeze all EG-3D modules during policy learning

for episode in RL_episodes:                # Iterate over reinforcement learning episodes
    I_t, P = observe()                     # Observe current image and task prompt

    # -------------------------------------------------------
    # Forward pass through frozen EG-3D for structure-aware state
    # -------------------------------------------------------
    G_t, Z_e_t = EG3D(I_t, P)               # Obtain reconstructed 3D and retrieved structure tokens
                                             # Z_e_t.rel / Z_e_t.aff may be empty depending on stage

    # -------------------------------------------------------
    # Vision-Language-Action policy inference and execution
    # -------------------------------------------------------
    A_t = π_VLA(Render(G_t), P)             # Predict action from rendered 3D and prompt
    execute(A_t)                           # Execute action in simulator or robot

    # -------------------------------------------------------
    # Compute reward with structural shaping and safety penalties
    # -------------------------------------------------------
    r_t = r_success()                     # Task success reward
          + α * r_struct(                # Structural shaping reward using rel/aff tokens
                Z_e_t.rel, Z_e_t.aff, A_t)
          - β * r_collision()             # Collision penalty
          - γ * r_torque()                # Excessive torque penalty

    # -------------------------------------------------------
    # Update VLA policy using PPO or SAC
    # -------------------------------------------------------
    update(π_VLA, algorithm="PPO/SAC", reward=r_t)

---

References

[1] D. Tochilkin et al. "TripoSR." arXiv:2403.02151, 2024. [2] Y. Hong et al. "LRM." arXiv:2311.04400, 2023. [3] X. Long et al. "Wonder3D." arXiv:2311.00005, 2023. [4] M. J. Kim et al. "OpenVLA." arXiv:2406.09246, 2024. [5] H. Shi et al. "MemoryVLA." arXiv:2508.19236, 2025. [6] F. Locatello et al. "Slot Attention." NeurIPS, 2020. [7] K. Mo et al. "PartNet." CVPR, 2019. [8] Y. Zhou et al. "Continuity of Rotation Representations." CVPR, 2019. [9] B. Mildenhall et al. "NeRF." ECCV, 2020. [10] J. J. Park et al. "DeepSDF." CVPR, 2019. [11] B. Kerbl et al. "3D Gaussian Splatting." ACM TOG, 2023. [12] W. Peebles and S. Xie. "DiT." CVPR, 2023. [13] K. Grauman et al. "Ego4D." CVPR, 2022. [14] S. Brahmbhatt et al. "ContactDB." CVPR, 2019.

---