license: apache-2.0
pipeline_tag: image-to-image
Model Card for Model ID
Joint RGB and Depth Representation Autoencoder (RAE-d) and Conditional Flow Matching (CFM) model for underwater benthic environment generation.
Model Details
Model Description
BenthicFlow is a unified generative pipeline for benthic (seafloor) environments. A single conditional flow matching model jointly synthesizes aligned RGB and depth, and a MultiDiffusion-inspired windowed sampler extends generation to scenes of unbounded spatial extent — no separate inpainting or stitching network. Generated RGB-D mosaics are lifted into continuous 3D scenes with surface-aligned Gaussian surfels.
- Developed by: Joaquín Figueira Chacon, Camile C. Lendering, Giacomo D'Amicantontion, Erkut Akdag and Egor Bondarev
- Funded by: ITEA Advisor and Xecs Marisens projects winthin the Eindhoven University of Technology.
- Model type: Generative model
- License: Apache 2.0
- Adapted from model: facebook/dinov2-base
Model Sources [optional]
- Repository: https://github.com/jacomof/BenthicFlow
- Paper: BenthicFlow: Generating Extensible Underwater Environments via Flow Matching
Uses
Use to create RGBD mosaics of benthic environments.
How to Get Started with the Model
Download the Github repository and follow the provided instructions.
Training Details
Training Data
Data from the Squidle+ benthic survey framework is used for training. Images are downsampled to 518 resolution, and 224 crops of the the downsampled images are used for training. Further details on the precise subset used are present in the Github repository and paper.
Training Procedure
- RAE: A representation autoencoder (using a DINOv2-B encoder and a convolutional decoder) is trained to reconstruct images following a 3 stage regime for 16 epochs with a batch size of 16 crops. Following standard RAE practices, phase 1 applies an L1 reconstruction loss on RGB and depth. From epoch 6, phase 2 adds a Learned Perceptual Image Patch Similarity (LPIPS) term on RGB. From epoch 8, phase 3 adds a hinge adversarial term on RGB, with a discriminator formed from a frozen DINO-S/8 backbone and a small trainable convolutional head, and DiffAugment applied identically to real and reconstructed inputs.
- CFM: A DIBOv2 average pooled token of produced by encoding a reference image is computed and appended to the timestep encoder of a CFM network. The CFM network is implemente using a UNet architecture.
Training Hyperparameters
- Training regime: fp16 mixed precision.
- Batch size: 16 (d-RAE) and 256 (CFM).
- Learning rate: 2e-4 (d-RAE) and 1e-4 (CFM).
- Exponential Moving Average: 0.9999 decay.
Evaluation
Please refer to the paper when published.
Environmental Impact
Estimated as 34560.0 g eq. CO2 using the Machine Learning Impact calculator.
- Hardware Type: H100 GPUs
- Hours used: Approx. 200 H100 GPU hours.
- Carbon Emitted: 34560.0 g eq. CO2
Technical Specifications [optional]
Model Architecture and Objective
- The RAE-d architecture consists of a frozen DINOv2-B RGB encoder, a depth encoder consisting of 8 attention blocks with embedding dimension 256 and 8 attention heads. The decoder consists of a convolutional network with two pre-norm GroupNorm/SiLU residual up-convolution blocks (inspired by the SD-VAE decoder) with upsampling factors 2 and 7 and hidden dimensions 256 and 128.
- The CFM's UNet consists of 2 downsampling residual blocks, 1 middle residual block and 2 up-sampling residual blocks. Adaptive Group Normalization is used in all the blocks.
Compute Infrastructure
Snellius: the National Supercomputer of the Netherlands. It uses a SLURM management system.
Hardware
4 H100 GPUs, 64 CPU cores, 512 GB of RAM are used for training.
Citation
To be released soon in Arxiv and ECCV 2026 proceedings.
Model Card Contact
- email: j.figueira@tue.nl