| # InteriorFusion Architecture Design |
|
|
| ## Design Philosophy |
|
|
| InteriorFusion is built on a critical insight: **interior scenes are fundamentally different from single objects**. Current SOTA models (TRELLIS, Hunyuan3D-2, TripoSR, SF3D) are trained on object-centric datasets (Objaverse) and produce unit-cube-scaled assets. They have no concept of: |
|
|
| - Room topology (walls, floors, ceilings) |
| - Spatial relationships (table NEAR sofa, lamp ON nightstand) |
| - Real-world scale (meters, not arbitrary units) |
| - Multi-object coherence (furniture doesn't float) |
| - Semantic room understanding (kitchen vs bedroom vs office) |
|
|
| InteriorFusion addresses all of these through a **5-phase hybrid pipeline**. |
|
|
| --- |
|
|
| ## Phase 1: Scene Understanding |
|
|
| ### 1.1 Metric Depth Estimation |
| **Model**: `depth-anything/Depth-Anything-V2-Metric-Indoor-Large-hf` |
|
|
| Why metric indoor variant? It predicts depth in **real-world meters** (trained on Hypersim), essential for correct furniture scaling. Non-metric depth estimators produce relative depth that breaks room reconstruction. |
|
|
| ### 1.2 Room Layout Estimation |
| **Model**: `manycore-research/SpatialLM-Llama-1B` (or Qwen-0.5B for Apache 2.0) |
|
|
| SpatialLM processes point clouds from depth + camera intrinsics to produce structured scene scripts: |
| ```python |
| @dataclass |
| class RoomLayout: |
| walls: List[Plane] # Wall planes with normals |
| floor: Plane # Floor plane |
| ceiling: Plane # Ceiling plane |
| doors: List[Doorway] # Doorway locations |
| windows: List[Window] # Window locations |
| objects: List[ObjectBBox] # Furniture bounding boxes |
| ``` |
|
|
| ### 1.3 Semantic Segmentation |
| **Model**: Mask2Former / OneFormer with indoor-trained heads |
|
|
| Segments the input image into: |
| - Wall regions (with material type: paint, wallpaper, brick) |
| - Floor regions (wood, tile, carpet) |
| - Ceiling region |
| - Per-furniture instances (sofa, table, lamp, etc.) |
| - Decorative elements (plants, paintings, curtains) |
|
|
| ### 1.4 Multi-Object Detection & Isolation |
| Using SAM (Segment Anything Model) with indoor priors: |
| - Segment each furniture piece |
| - Extract per-object crops with alpha masks |
| - Remove background context for clean object generation |
|
|
| --- |
|
|
| ## Phase 2: Multi-View Generation |
|
|
| ### 2.1 Per-Object Multi-View Diffusion |
| **Model**: `stabilityai/stable-zero123` or Zero123++ community pipeline |
|
|
| For each segmented furniture object: |
| - Generate 6 consistent orthographic views (0°, 60°, 120°, 180°, 240°, 300° azimuth) |
| - Condition on the original crop + depth edge map |
| - Use depth-conditioned ControlNet for geometric consistency |
|
|
| ### 2.2 Room Shell Multi-View |
| For walls, floor, ceiling: |
| - Generate panoramic-style extended views from the single image |
| - Use depth-guided inpainting for occluded regions |
| - Produce ceiling, floor, and wall texture atlases |
|
|
| ### 2.3 Depth-Conditioned View Synthesis |
| Condition all multi-view generation on the metric depth map: |
| - Depth acts as a geometric prior preventing shape hallucination |
| - Cross-view depth consistency enforced via depth-normal consistency loss |
|
|
| --- |
|
|
| ## Phase 3: 3D Reconstruction |
|
|
| ### 3.1 Room Shell Reconstruction |
| Walls, floor, ceiling are reconstructed as **planar meshes** with UV atlases: |
| - Walls: Extruded from detected wall planes + depth boundaries |
| - Floor: Planar mesh with UV-mapped texture |
| - Ceiling: Planar mesh with texture from inpainted ceiling view |
|
|
| ### 3.2 Per-Object 3D Generation |
| Each furniture object is reconstructed using a **hybrid approach**: |
|
|
| **Small objects** (lamps, vases, decor): TRELLIS.2-4B → mesh with PBR |
| **Medium objects** (chairs, tables): TRELLIS.2-4B or InteriorFusion-L native |
| **Large objects** (sofas, beds, wardrobes): InteriorFusion-L with spatial constraints |
|
|
| The key innovation: **Spatial Constraint Injection** |
| - Object position is constrained by the room layout from Phase 1 |
| - Object scale is constrained by metric depth |
| - Object orientation is constrained by floor plane normal |
|
|
| ### 3.3 Gaussian Splatting Layer |
| For the entire scene, we maintain a parallel **3D Gaussian Splatting representation**: |
| - Fast novel view synthesis for interactive preview |
| - Per-object Gaussian subsets for editing |
| - Global scene Gaussians for background/room shell |
|
|
| --- |
|
|
| ## Phase 4: Scene Assembly |
|
|
| ### 4.1 Layout Optimization |
| Using SpatialLM's scene graph + learned layout prior: |
| - Place objects at detected positions from Phase 1 |
| - Resolve collisions using physics-based relaxation |
| - Ensure objects rest on floor (gravity constraint) |
| - Ensure objects don't intersect walls |
|
|
| ### 4.2 Scale Normalization |
| All objects normalized to metric scale: |
| - Use known furniture dimensions (e.g., standard chair height ~45cm) |
| - Use depth consistency to resolve ambiguous scales |
| - Human-scale reference from detected people/artifacts |
|
|
| ### 4.3 Scene Graph Construction |
| ```python |
| @dataclass |
| class SceneGraph: |
| nodes: Dict[str, SceneNode] # Objects + room shell |
| edges: List[SpatialRelation] # "on", "next to", "in front of", etc. |
| room_type: str # "modern_living_room", "scandinavian_kitchen" |
| style: str # "modern", "scandinavian", "luxury", "indian" |
| ``` |
|
|
| --- |
|
|
| ## Phase 5: Material & Texture |
|
|
| ### 5.1 PBR Material Generation |
| For each surface: |
| - Base color/albedo (diffuse) |
| - Metallic map |
| - Roughness map |
| - Normal map (bump) |
| - Ambient occlusion (optional) |
|
|
| **Model**: Custom material diffusion network fine-tuned on Hypersim + InteriorNet |
|
|
| ### 5.2 Texture Baking |
| - Project multi-view generated textures onto UV atlases |
| - Visibility-aware blending (occlusion handling) |
| - Seamless tiling for large surfaces (walls, floors) |
|
|
| ### 5.3 Lighting Estimation |
| Estimate scene lighting from the input image: |
| - HDR environment map extraction |
| - Key light / fill light / ambient light decomposition |
| - IBL (Image-Based Lighting) setup for game engines |
|
|
| --- |
|
|
| ## Core Model: InteriorFusion-L (4B Parameters) |
|
|
| ### Encoder |
| - **Image encoder**: DINOv3-L (frozen, feature extraction) |
| - **Depth encoder**: Custom CNN processing metric depth map |
| - **Layout encoder**: Transformer processing SpatialLM scene graph tokens |
| - **Semantic encoder**: Mask2Former feature pyramid |
|
|
| ### Latent Representation: SLAT-Interior |
| Extension of TRELLIS SLAT optimized for indoor scenes: |
| - Sparse 3D voxel grid, resolution 1024³ |
| - Active voxels only on surfaces (wall, furniture) |
| - Per-voxel features: shape + material + semantic class |
| - Room-shell voxels flagged separately from object voxels |
|
|
| ### Decoder |
| Three parallel decoders: |
| 1. **Mesh decoder**: Produces watertight or arbitrary-topology meshes (from O-Voxel) |
| 2. **Gaussian decoder**: Produces per-voxel Gaussian parameters |
| 3. **Material decoder**: Produces PBR material parameters per surface |
|
|
| ### Generation Pipeline |
| Two-stage rectified flow (following TRELLIS pattern): |
| 1. **Structure generation**: Dense occupancy grid → sparse structure |
| 2. **Latent generation**: Per-active-voxel features → shape + material |
|
|
| Conditioned on: DINOv3 image features + depth map + room layout tokens + semantic segmentation tokens |
|
|
| --- |
|
|
| ## Training Strategy |
|
|
| ### Stage 1: VAE Pre-training (1 week, 8×A100) |
| - Train SLAT-Interior VAE on 3D-FRONT + Structured3D rooms |
| - Multi-resolution: 256³ → 512³ → 1024³ curriculum |
| - Loss: MSE reconstruction + KL divergence + depth consistency + normal consistency |
|
|
| ### Stage 2: Flow-Matching DiT (2 weeks, 32×A100) |
| - Train rectified flow transformer for structure generation |
| - Curriculum: 256³ → 512³ → 1024³ |
| - Conditioning: image + depth + layout |
|
|
| ### Stage 3: Material DiT (1 week, 16×A100) |
| - Train material generation DiT conditioned on geometry + input image |
| - PBR material prediction: albedo, metallic, roughness, normal |
|
|
| ### Stage 4: Fine-tuning (3 days, 8×A100) |
| - LoRA fine-tuning on real interior photos (ScanNet + HM3D) |
| - Domain adaptation from synthetic to real |
| - Reinforcement learning for geometry consistency (GRPO-style) |
|
|
| ### Total Training: ~4 weeks on 32×A100 |
|
|
| --- |
|
|
| ## Inference Optimization |
|
|
| ### RTX 4090 (24GB VRAM) |
| - Model quantization: INT8 via GPTQ |
| - Gradient checkpointing disabled (inference only) |
| - Gaussian splatting for real-time preview |
| - Full mesh generation: ~15 seconds |
|
|
| ### A100 (80GB VRAM) |
| - FP16 inference |
| - Batch generation for multiple objects |
| - Full pipeline: ~8 seconds |
|
|
| ### H100 (80GB VRAM) |
| - BF16 inference |
| - ~5 seconds full generation |
|
|
| ### Edge / Mobile |
| - Core depth + layout estimation only (~2 seconds) |
| - Cloud-based 3D generation with streaming |
| - Reduced mesh quality (decimated, lower texture resolution) |
|
|
| --- |
|
|
| ## Export Formats |
|
|
| | Format | Use Case | Features | |
| |--------|----------|----------| |
| | **GLB** | Web, AR, Unity, Godot | PBR materials, animations, all data | |
| | **FBX** | Unreal Engine, Maya, 3ds Max | Full rigging support, PBR | |
| | **OBJ** | Legacy compatibility | Basic materials (MTL) | |
| | **USDZ** | iOS AR (ARKit) | Apple's native format | |
| | **3DGS (.ply)** | Real-time viewing | Gaussian splatting render | |
| | **BLEND** | Blender native | Full editability, nodes | |
|
|
