ReconSplat: Generalizable 3D Scene Reconstruction Beyond Observed Views

Giuseppe Stracquadanio, Kevin Raj, Julia Grabinski, Stefan Roth  ·  ECCV 2026

Paper · Project page · Code

ReconSplat reconstructs a scene from a handful of posed images, synthesizing novel views and depth inside and outside the observed frustums. It pairs a feed-forward Gaussian reconstruction with a multi-view latent diffusion model that jointly generates appearance and geometry, so unobserved regions are completed rather than left empty.

This repository holds the released weights. Two stages are trained, and running the model needs one checkpoint from each:

• Stage 1, multi-view latent field. From sparse input views with known cameras, we predict variational 3D Gaussians, enconding latent appeareance and geometry, that can be rasterized to any view.
• Stage 2, multi-view latent diffusion (MV-LDM). Conditioned on input views and on rasterized features for target views, a multi-view diffusion models renders novel views and depth.

Checkpoint hierarchy

Below we describe our checkpoint hierarchy. The "Init. from" column denotes the weights we use for initializing the corresponding checkpoint. N is the number of context (input) views, M the number of target views, f the VAE downsampling factor relating image to latent resolution, and B_eff the effective batch size. All files live under stage2-diffusion/.

Checkpoint	Init. from	Latent res.	N	M	Beff	Steps	Trainable
RE10K, 256²
re10k-base.ckpt  ·  BRE10K	SD 2.1	32²	2	4	80	200K	all
↳ re10k-upsample-ft.ckpt  ·  + f = 8 → 4	BRE10K	64²	2	4	24	50K	all
↳ re10k-video-ft.ckpt  ·  + Video	BRE10K	32²	2	10	32	100K	3D convs. only
DL3DV, 256 × 448
dl3dv-base.ckpt  ·  BDL3DV	BRE10K	32 × 56	4	4	32	140K	all
↳ dl3dv-upsample-ft.ckpt  ·  + f = 8 → 4	BDL3DV	64 × 112	4	4	8	50K	all
↳ dl3dv-video-ft.ckpt  ·  + Video	BDL3DV	32 × 56	4	10	24	100K	3D convs. only

• -upsample-ft halves the effective downsampling factor (f = 8 → 4) by bilinearly upsampling the input images 2×, so the denoiser runs on a 4× larger token grid and recovers finer detail. Predictions are mapped back to the native resolution.
• -video-ft fine-tunes on 10-frame sequences for temporally consistent dense camera trajectories. Only the "temporal" residual blocks (i.e., 3D convolutions) are trained; every other parameter stays frozen.
• DL3DV inputs are center-cropped from 256 × 480 to 256 × 448 so both spatial dimensions are multiples of the patch and downsampling factors.

-video-ft: Use these for rendering videos and for point-cloud extraction (recommended).

The table covers only what differs between checkpoints. For the shared training setup, including objective, optimizer and schedule, noise schedule, guidance, and Stage-1 VAE recipe, see the paper.

Files

File	Size	Pairs with
stage1-vae/re10k.ckpt	0.7 GB	any re10k-* Stage-2 checkpoint
stage1-vae/dl3dv.ckpt	0.7 GB	any dl3dv-* Stage-2 checkpoint
stage2-diffusion/re10k-base.ckpt	8.2 GB	stage1-vae/re10k.ckpt
stage2-diffusion/re10k-upsample-ft.ckpt	8.2 GB	stage1-vae/re10k.ckpt
stage2-diffusion/re10k-video-ft.ckpt	8.9 GB	stage1-vae/re10k.ckpt
stage2-diffusion/dl3dv-base.ckpt	8.2 GB	stage1-vae/dl3dv.ckpt
stage2-diffusion/dl3dv-upsample-ft.ckpt	8.2 GB	stage1-vae/dl3dv.ckpt
stage2-diffusion/dl3dv-video-ft.ckpt	8.9 GB	stage1-vae/dl3dv.ckpt

Both Stage 1 VAEs were trained for 100K steps.

Usage

Clone the code and follow its setup instructions (this includes building a custom CUDA rasterization kernel), then download the pair you need:

git clone https://github.com/visinf/reconsplat.git reconsplat
cd reconsplat

pip install -U "huggingface_hub[cli]"

# e.g., to download the base RE10K model.
hf download gpstracquadanio/reconsplat \
    stage1-vae/re10k.ckpt stage2-diffusion/re10k-base.ckpt \
    --local-dir checkpoints

Each checkpoint has a matching experiment config. Pass the checkpoint you want to evaluate explicitly: the configs' own load_first_stage/load_second_stage record the paths used for training (load_second_stage being what training initialized from), not the released weights. Alternatively, checkpointing.load_first_stage=auto infers the Stage 1 checkpoint from the Stage 2 path, so only one path has to be given.

python -m src.main +experiment=re10k_diffusion_release_low_res_latent \
    mode=test \
    checkpointing.load_first_stage=auto \
    checkpointing.load_second_stage=checkpoints/stage2-diffusion/re10k-base.ckpt \
    dataset/view_sampler=evaluation \
    dataset.view_sampler.index_path=assets/re10k_evaluation/<index>.json \
    test.compute_scores=true \
    wandb.mode=disabled

Checkpoint	+experiment=
re10k-base	re10k_diffusion_release_low_res_latent
re10k-upsample-ft	re10k_diffusion_release_upsample_ft
re10k-video-ft	re10k_diffusion_release_video_ft
dl3dv-base	dl3dv_diffusion_release_ft
dl3dv-upsample-ft	dl3dv_diffusion_release_upsample_ft
dl3dv-video-ft	dl3dv_diffusion_release_video_ft

The config must match the checkpoint. The -upsample-ft models change the denoiser's geometry (upsample_latents, downsample_preds, and a halved cross-view attention scale), and the -video-ft models add temporal blocks — checkpoint loading is non-strict, so a mismatched config loads quietly and then produces bad samples rather than raising an error.

To render videos instead of scoring individual views, add test.save_video=true test.save_image=false test.compute_scores=false and point index_path at a video evaluation index.

Stage 2 checkpoints contain both the EMA and the raw denoiser weights. Inference uses the EMA weights by default (load_ema_weights always resolves to true in test mode); you can use the raw weights for further fine-tuning.

Training data

• RealEstate10K — real-estate video tours, mainly indoor.
• DL3DV-10K — (a split with ~2K scenes) real-world scene captures, mainly outdoor.

Depth supervision comes from pseudo-labels mined per scene with generate_pseudo_labels.py (configured via config/labels.yaml): VGGT predicts per-pixel depth and camera poses feed-forward, optionally rescaled to match scale-consistent dense depth from Video Depth Anything (VDA).

License

Apache 2.0. The codebase also incorporates third-party MIT-licensed components, whose copyright notices are retained in the files concerned. The Stage-2 models are derived from Stable Diffusion 2.1 and inherit the use restrictions of its CreativeML Open RAIL++-M license. Dataset terms remain with their original providers.

Citation

@article{stracquadanio2026reconsplat,
    title   = {ReconSplat: Generalizable 3D scene reconstruction beyond observed views},
    author  = {Stracquadanio, Giuseppe and Raj, Kevin and Grabinski, Julia and Roth, Stefan},
    journal = {{ECCV}},
    year    = {2026},
}