--- license: other pipeline_tag: image-to-3d tags: - 3d-scene-reconstruction - rgb-d - single-image - flow-matching - textured-mesh --- # FIRE3D: Feed-forward Interactive 3D Scene Reconstruction Within A Minute [Paper](https://arxiv.org/pdf/2609.08848) | [Code](https://github.com/xiahongchi/Fire3D) | [Inference data](https://huggingface.co/datasets/hongchi/Fire3D) Fire3D is a unified feed-forward framework that transforms a single RGB image or casual RGB video into simulation-ready 3D scene assets. It predicts a compositional scene representation with object-level 6-DoF pose, bounding box, mesh geometry, and texture, without test-time optimization. This release provides the inference code, model checkpoints, processed example inputs, and frozen protocols needed to reproduce Fire3D results on iTHOR, Imaginarium, ScanNet++, and single-image scenes. The inference code and installation instructions are maintained at [xiahongchi/Fire3D](https://github.com/xiahongchi/Fire3D). Processed release inputs are hosted at [datasets/hongchi/Fire3D](https://huggingface.co/datasets/hongchi/Fire3D). ## Overview ![Fire3D reconstructs compositional, simulation-ready 3D scenes from RGB-D observations.](assets/teaser.png) **Teaser.** Fire3D reconstructs a compositional 3D scene from a single RGB image or casual RGB video. The output contains object-level pose, geometry, and material assets that can be rendered, edited, and used in simulation. [![Fire3D end-to-end perception and reconstruction method.](assets/method_overview.png)](assets/method_overview.pdf) **Method overview.** Fire3D lifts RGB-D observations into a shared 3D scene representation, predicts object instances and 6-DoF poses, and reconstructs the sparse structure, shape, and PBR fields with three cascaded flow-matching models. Batched decoders and mesh post-processing produce the final textured scene assets. [![Fire3D Hierarchical Compression VAE architecture.](assets/hcvae.png)](assets/hcvae.pdf) **HC-VAE.** The Hierarchical Compression VAE reduces each sparse SC-VAE shape or material field from a `32^3 x 32` representation to an `8^3 x 64` latent. This 32x reduction makes scene-level batched flow sampling practical while retaining the frozen TRELLIS.2 decoding path. ## Model Bundle The repository contains the scene perception checkpoint, the three flow-matching checkpoints, sparse VAE encoders and decoders, latent normalization statistics, and the TRELLIS.2 shape/PBR decoders used by the frozen release protocols. `manifest.json` records the byte size and SHA-256 digest of every inference file; `checksums.sha256` provides the same values in standard form. Fire3D's Hierarchical Compression VAE (HC-VAE) compresses each sparse SC-VAE shape and material field from a `32^3` representation with 32 channels into an `8^3` representation with 64 channels. This 32x reduction enables batched flow sampling and VAE decoding across many scene instances while retaining the frozen TRELLIS.2 SC-VAE decoding path. | Component | Bundle path | |---|---| | Scene perception | `perception/model.pt` | | Sparse-structure flow | `reconstruction/flows/ss/model.pt` | | Shape flow | `reconstruction/flows/shape/model.pt` | | PBR flow | `reconstruction/flows/pbr/model.pt` | | Sparse-structure VAE | `reconstruction/vae/ss/ckpts/encoder.pt`, `decoder.pt` | | Shape HC-VAE | `reconstruction/vae/shape/ckpts/encoder.pt`, `decoder.pt` | | PBR HC-VAE | `reconstruction/vae/pbr/ckpts/encoder.pt`, `decoder.pt` | | DINOv3 encoder | `external/dinov3_vitl16_pretrain_lvd1689m-8aa4cbdd.pth` | | TRELLIS.2 shape decoder | `external/trellis2/shape_dec_next_dc_f16c32_fp16.safetensors` | | TRELLIS.2 PBR decoder | `external/trellis2/tex_dec_next_dc_f16c32_fp16.safetensors` | Public filenames intentionally do not encode private training iteration numbers. The sparse-structure VAE encoder and decoder operate on binary `8^3` object supports through an `8 x 2 x 2 x 2` latent. The normal joint-training recipe uses BCE-with-logits plus KL and is available in the [code release](https://github.com/xiahongchi/Fire3D/blob/main/training/README.md#sparse-structure-vae). When training a new SS flow, regenerate its latent cache and statistics with a matched VAE encoder. The released pretrained flow remains pinned to its own latent artifact contract. The root `config.json` is the model repository's Hugging Face query file. A normal `snapshot_download` includes it in the model snapshot, allowing the Hub to maintain the standard repository download count without client-side telemetry. See the [Hugging Face download-statistics documentation](https://huggingface.co/docs/hub/models-download-stats). The default protocols use classifier-free guidance 3, flow and VAE batches of 16, and the batch CuMesh postprocessor. The iTHOR, Imaginarium, and ScanNet++ protocols enable the validated background room-box fit before background reconstruction. The released HC-VAE recipe uses one process on one GPU; its configured per-GPU batch size is therefore the effective global batch size. ## Training Data Fire3D does not redistribute its training corpora. All source datasets are publicly available and retain their original licenses and access terms. For object training data, use [TRELLIS-500K](https://huggingface.co/datasets/JeffreyXiang/TRELLIS-500K), which provides the metadata and preparation tools for the ObjaverseXL, ABO, 3D-FUTURE, and HSSD sources used by the object pipeline. Official scene-data links and Fire3D preprocessing entry points are listed in the [code release](https://github.com/xiahongchi/Fire3D/tree/main/data_processing). ## Use ```bash git clone https://github.com/xiahongchi/Fire3D.git cd Fire3D bash scripts/install.sh conda activate fire3d bash scripts/install_blender.sh fire3d download --models fire3d download --data --dataset ithor --scene-id iTHOR_FloorPlan312_physics bash scripts/run_ithor_example.sh ``` The downloader places this repository under `checkpoints/Fire3D/` and verifies the release manifest before inference. ## Scope The released checkpoints are intended for inference through the frozen Fire3D protocols. Performance can degrade for scenes with inaccurate camera poses or depth, severe occlusion, unusual scale, non-room backgrounds, or objects far outside the training distribution. The generated meshes should be reviewed before use in safety-critical or physically deployed applications. ## Licenses Original Fire3D code is MIT licensed. This model bundle includes DINOv3 and selected TRELLIS.2 files, which retain their upstream licenses. Review the included `licenses/` directory and the repository's `THIRD_PARTY_NOTICES.md` before use or redistribution. ## Citation ```bibtex @article{xia2026fire3d, title={{FIRE3D}: Feed-forward Interactive 3D Scene Reconstruction Within A Minute}, author={Xia, Hongchi and Cheng, Tianhang and Ma, Wei-Chiu and Wang, Shenlong}, journal={arXiv preprint arXiv:2609.08848}, year={2026}, url={https://arxiv.org/pdf/2609.08848} } ```