| --- |
| license: apache-2.0 |
| pipeline_tag: image-to-image |
| --- |
| |
| # Model Card for Model ID |
|
|
| <!-- Provide a quick summary of what the model is/does. --> |
|
|
| Joint RGB and Depth Representation Autoencoder (RAE-d) and Conditional Flow Matching (CFM) model for underwater benthic environment generation. |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| <!-- Provide a longer summary of what this model is. --> |
|
|
| BenthicFlow is a unified generative pipeline for benthic (seafloor) environments. |
| A **single conditional flow matching model** jointly synthesizes aligned RGB and |
| depth, and a MultiDiffusion-inspired windowed sampler extends generation to |
| scenes of unbounded spatial extent — no separate inpainting or stitching network. |
| Generated RGB-D mosaics are lifted into continuous 3D scenes with |
| **surface-aligned Gaussian surfels**. |
|
|
|
|
| - **Developed by:** Joaquín Figueira Chacon, Camile C. Lendering, Giacomo D'Amicantontion, Erkut Akdag and Egor Bondarev |
| - **Funded by:** ITEA Advisor and Xecs Marisens projects winthin the Eindhoven University of Technology. |
| - **Model type:** Generative model |
| - **License:** Apache 2.0 |
| - **Adapted from model:** facebook/dinov2-base |
|
|
| ### Model Sources [optional] |
|
|
| <!-- Provide the basic links for the model. --> |
|
|
| - **Repository:** https://github.com/jacomof/BenthicFlow |
| - **Paper:** [BenthicFlow: Generating Extensible Underwater Environments via Flow Matching](https://huggingface.co/papers/2608.23173) |
|
|
| ## Uses |
|
|
| Use to create RGBD mosaics of benthic environments. |
|
|
| ## How to Get Started with the Model |
|
|
| Download the Github repository and follow the provided instructions. |
|
|
| ## Training Details |
|
|
| ### Training Data |
|
|
| Data from the Squidle+ benthic survey framework is used for training. Images are downsampled to 518 resolution, and 224 crops of the the downsampled images are used for |
| training. Further details on the precise subset used are present in the Github repository and paper. |
|
|
| ### Training Procedure |
|
|
| - RAE: A representation autoencoder (using a DINOv2-B encoder and a convolutional decoder) is trained to reconstruct images following a 3 stage regime for 16 epochs with a batch size of 16 |
| crops. |
| Following standard RAE practices, phase 1 applies an L1 |
| reconstruction loss on RGB and depth. From epoch 6, phase 2 adds a Learned Perceptual Image Patch Similarity (LPIPS) term on RGB. |
| From epoch 8, phase 3 adds a hinge adversarial term on RGB, with a |
| discriminator formed from a frozen DINO-S/8 backbone and a small trainable |
| convolutional head, and DiffAugment applied identically to real and reconstructed |
| inputs. |
| - CFM: A DIBOv2 average pooled token of produced by encoding a reference image is computed and appended to the timestep encoder of a CFM network. The CFM network is implemente |
| using a UNet architecture. |
|
|
|
|
| #### Training Hyperparameters |
|
|
| - **Training regime:** fp16 mixed precision. |
| - **Batch size:** 16 (d-RAE) and 256 (CFM). |
| - **Learning rate:** 2e-4 (d-RAE) and 1e-4 (CFM). |
| - **Exponential Moving Average:** 0.9999 decay. |
|
|
| ## Evaluation |
|
|
| Please refer to the paper when published. |
|
|
| ## Environmental Impact |
|
|
| <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly --> |
|
|
| Estimated as 34560.0 g eq. CO2 using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute). |
|
|
| - **Hardware Type:** H100 GPUs |
| - **Hours used:** Approx. 200 H100 GPU hours. |
| - **Carbon Emitted:** 34560.0 g eq. CO2 |
|
|
| ## Technical Specifications [optional] |
|
|
| ### Model Architecture and Objective |
|
|
| - The RAE-d architecture consists of a frozen DINOv2-B RGB encoder, a depth encoder consisting of 8 attention blocks with embedding dimension 256 and 8 attention heads. |
| The decoder consists of a convolutional network with two pre-norm GroupNorm/SiLU residual up-convolution blocks (inspired by the SD-VAE decoder) with upsampling factors |
| 2 and 7 and hidden dimensions 256 and 128. |
| - The CFM's UNet consists of 2 downsampling residual blocks, 1 middle residual block and 2 up-sampling residual blocks. Adaptive Group Normalization is used in all the |
| blocks. |
|
|
| ### Compute Infrastructure |
|
|
| [Snellius: the National Supercomputer of the Netherlands](https://www.surf.nl/en/services/compute/snellius-the-national-supercomputer). It uses a SLURM management system. |
|
|
| #### Hardware |
|
|
| 4 H100 GPUs, 64 CPU cores, 512 GB of RAM are used for training. |
|
|
| ## Citation |
|
|
| To be released soon in Arxiv and ECCV 2026 proceedings. |
|
|
| ## Model Card Contact |
|
|
| - email: j.figueira@tue.nl |
