File size: 7,631 Bytes

# WorldGen Vulkan validation

## Environment

```text
Windows 11
AMD Radeon RX 6900 XT 16GB
Vulkan 1.4 (AMD proprietary driver)
No CUDA
```

## Pipeline

```text
t2s:
  prompt
   -> FLUX.1-dev Q4_K_S + WorldGen text2scene LoRA
      (stable-diffusion.cpp, SD_VULKAN=ON)
   -> 1024x512 equirectangular panorama
   -> DA-2 depth (ncnn Vulkan, fp32)
   -> worldgen-vk (Rust): RGBD -> 3DGS PLY + mesh PLY

i2s:
  image
   -> 90-degree full-height band -> cond/mask
   -> FLUX.1-Fill-dev Q4_K (official safetensors converted with sd-cli)
      + WorldGen img2scene LoRA
   -> Fill pano, composite original image region back
   -> DA-2 depth (ncnn Vulkan)
   -> worldgen-vk (Rust): RGBD -> 3DGS PLY + mesh PLY
```

## Verified runs

### t2s

```text
prompt: a cozy wooden bedroom with warm lamps
1024x512, 20 steps
output: output/t2s-ncnn-clean/
  524,288 splats
  1,046,528 triangles
```

### i2s

```text
input: assets/rainforest/photo.jpg
1024x512, 20 steps
output: output/i2s-ncnn-clean/
  524,288 splats
  1,046,528 triangles
```

### i2s seam blending

The Fill output and the original input band are composited with
`cv2.seamlessClone` (NORMAL_CLONE). Without OpenCV, a 24 px feathered alpha
blend is used. This removes the visible vertical band boundary; it does not
remove Fill object repetition.

### 1600x800

```text
t2s, 1600x800, 10 steps: 739 s
output: 1,280,000 splats, 1600x800 pano

DA-2 native 1600x800 trace:
  graph      da2_trace_1600.ncnn.param/.bin
  time       6.29 s (Vulkan f32)
  max abs    0.144
  mean abs   0.0041
  cosine     0.99954

full 50 steps: roughly 1 hour on this GPU
```

### Open3D / TSDF mesh

Default mesh is now a **voxel signed-distance field + marching cubes**
(`worldgen_mesh.py --method voxel`):

```text
voxel-res 0.25, voxel-stride 4, median 5, trim-poles 0.02, voxel-smooth 2.5
50k verts / 98k tris
~3.3 s at 1024x512
i2s end-to-end 20 steps: ~70 s (vs 61.5 s shell)
```

Median filtering, pole trimming and a 2.5-voxel Gaussian smoothing remove the
top tab and reduce the left fold. The left crescent is still visible, so this
is a large improvement over the needle shell but not a perfect reconstruction.

Poisson/Ball/Alpha are available but were not better:

```text
Poisson: 537,145 verts / 1,067,522 tris, 6.8 s, crescent fold + ripples
Ball:    985,990 tris, shell-equivalent
Alpha:   noisy surface with holes
```

The voxel TSDF still shows a fold at the left depth discontinuity and a small
pole tab, but it removes the needle-like shell stretching. `--mesh-shell`
keeps the old Rust mesh (`mesh_shell.ply`).

### GPU 3DGS renderer

`gpu-splat/` is a wgpu/Vulkan splat renderer. The camera was fixed to
inside-out and the projection now matches the CPU reference; increasing the
splat radius 2x and raising opacity removes the earlier wash/blue-dome.

```text
gpu-splat.exe gaussians14.ply 0 out.png 0.2
```

It renders the official painting scene on the RX 6900 XT; it is not identical
to the official demo camera (the GPU view is closer/lower), but it is a real
Vulkan GPU 3DGS image.

### 3DGS render / official comparison

`render_splat_points.py --equirect` projects all 524,288 gaussians back to a
1024x512 equirect image and reconstructs the panorama (colors/geometry intact).
An inside-out 90-degree perspective point render also produces a coherent
scene:

```text
ply: 524,288 gaussians
inside 90 deg: 39,976 projected points
```

![3DGS comparison](assets/splat_compare.png)

`render_splat_gaussian.py` adds 3D covariance + alpha compositing for an
equirect image. `render_splat_perspective.py` projects the same covariance
through a 90-degree pinhole camera; at 832x480 it produces a scene close to the
official demo viewpoint. `--scale-factor 0.7` is the current best balance
between blur (1.0) and holes (0.3).

The existing `vulkan-splat` viewer still frames a full panorama badly (washed
or tiny orbit view); this is a viewer camera issue, not a PLY/numeric issue.
The 3DGS center/scale/quaternion formula is validated at ~1e-7 against the
WorldGen Python reference.

### Fill foreground cleaning

By default `worldgen_i2s_prep.py` writes `cond_clean.png`: GrabCut foreground
is removed from the Fill condition with OpenCV inpaint, while the original
band (`cond.png`) is composited back after Fill. This stops Fill from repeating
the foreground object on both sides. Use `--keep-fg` to disable.

The official painting demo now produces a clean panorama with the house once
(no duplicated foreground) and no visible band edge.

### inpaint_bg (simplified)

`worldgen_i2s.py --inpaint-bg`:

```text
GrabCut foreground mask
-> OpenCV Telea background inpaint
-> DA-2 depth on background
-> worldgen-vk splat
-> merge foreground splats + background splats in the foreground mask
```

Test: 1024x512, 5 steps, 524,288 base splats + 20,529 background splats
= 544,817 merged vertices. This is not the official OneFormer + FLUX Fill
branch; it is a CPU/OpenCV approximation.

### Clean builds from source

Tested with local source checkouts (same scripts also clone when `-Source` is
omitted):

```text
build_sd.ps1 -Source <stable-diffusion.cpp>   -> build-sd/bin/Release/sd-cli.exe
build_da2.ps1 -Source <ncnn>                  -> tools/da2-ncnn/da2ncnn.exe + ncnn.dll
build_rust.ps1                                -> tools/worldgen-vk/worldgen-vk.exe
```

`sd-cli.exe --list-devices` reports `Vulkan0  AMD Radeon RX 6900 XT`; the
freshly built DA-2 runner reproduces the 1024x512 output on the same GPU.

### DA-2 numeric and performance

ncnn Vulkan vs PyTorch, 1024x512:

```text
max abs   0.0161
mean abs  0.0027
cosine    0.9999994
time      3.05 s (f32 Vulkan)
```

The ncnn graph has **0 custom layers and 0 CPU-fallback layers**: the final
`Squeeze` was replaced by a `Reshape` (`0=1024 1=512`, or `1600 800` for the
1600 trace). The runner must be compiled with `-DNCNN_VULKAN` and call
`net.set_vulkan_device(0)`; otherwise ncnn silently falls back to CPU.

### Latency (1024x512, 20 steps, RX 6900 XT)

```text
t2s total          56.7 s
i2s total          61.5 s
FLUX sampling      ~100% GPU 3D utilization
VAE decode         9.0 s untiled -> 1.35 s with --vae-tiling --vae-tile-size 512x512
DA-2               3.05 s
Rust RGBD->3DGS    0.29 s
```

### RGBD -> 3DGS math

Compared with the WorldGen Python formula on synthetic panorama/depth:

```text
center      ~1e-7
scale       ~1e-7
quaternion  ~1e-7
```

### FLUX

- stable-diffusion.cpp built with `SD_VULKAN=ON`
- `Vulkan0 = AMD Radeon RX 6900 XT`
- FLUX.1-dev Q4_K_S SHA256 `962d536a...` matches
- FLUX.1-Fill-dev official safetensors SHA256 `03e289f5...` matches
- converted q4_K: 780 tensors
- text2scene LoRA: 312/312 applied
- img2scene LoRA: 456/684 applied (228 incompatible keys skipped)

## No PyTorch at inference

`worldgen_t2s.py`, `worldgen_i2s.py`, `worldgen_i2s_prep.py` and
`worldgen_da2.py` do not import torch. Model inference is:

```text
FLUX/Fill -> sd-cli (C++/Vulkan)
DA-2      -> ncnn (C++/Vulkan)
3DGS/mesh -> worldgen-vk (Rust, CPU geometry + PLY)
```

## Known gaps

- Not byte/quality-equivalent to official WorldGen:
  - Fill repeats objects and has a visible band boundary.
  - Mesh has some stretching from depth discontinuities.
- Optional `use_sharp=True` and `inpaint_bg=True` branches are not wired.
- Official WorldGen visual/numeric comparison of the full FLUX stage is not done;
  only component-level validation (DA-2, 3DGS math, weights, pipeline runs).
- Default validation resolution is 1024x512; 1600x800 is supported by
  resampling but is much slower (~1 h per 50-step panorama).
- ncnn/pnnx toolchain is C++, not Rust; Rust covers RGBD->3DGS/mesh.