| # WorldGen Vulkan validation |
|
|
| ## Environment |
|
|
| ```text |
| Windows 11 |
| AMD Radeon RX 6900 XT 16GB |
| Vulkan 1.4 (AMD proprietary driver) |
| No CUDA |
| ``` |
|
|
| ## Pipeline |
|
|
| ```text |
| t2s: |
| prompt |
| -> FLUX.1-dev Q4_K_S + WorldGen text2scene LoRA |
| (stable-diffusion.cpp, SD_VULKAN=ON) |
| -> 1024x512 equirectangular panorama |
| -> DA-2 depth (ncnn Vulkan, fp32) |
| -> worldgen-vk (Rust): RGBD -> 3DGS PLY + mesh PLY |
| |
| i2s: |
| image |
| -> 90-degree full-height band -> cond/mask |
| -> FLUX.1-Fill-dev Q4_K (official safetensors converted with sd-cli) |
| + WorldGen img2scene LoRA |
| -> Fill pano, composite original image region back |
| -> DA-2 depth (ncnn Vulkan) |
| -> worldgen-vk (Rust): RGBD -> 3DGS PLY + mesh PLY |
| ``` |
|
|
| ## Verified runs |
|
|
| ### t2s |
|
|
| ```text |
| prompt: a cozy wooden bedroom with warm lamps |
| 1024x512, 20 steps |
| output: output/t2s-ncnn-clean/ |
| 524,288 splats |
| 1,046,528 triangles |
| ``` |
|
|
| ### i2s |
|
|
| ```text |
| input: assets/rainforest/photo.jpg |
| 1024x512, 20 steps |
| output: output/i2s-ncnn-clean/ |
| 524,288 splats |
| 1,046,528 triangles |
| ``` |
|
|
| ### i2s seam blending |
|
|
| The Fill output and the original input band are composited with |
| `cv2.seamlessClone` (NORMAL_CLONE). Without OpenCV, a 24 px feathered alpha |
| blend is used. This removes the visible vertical band boundary; it does not |
| remove Fill object repetition. |
| |
| ### 1600x800 |
| |
| ```text |
| t2s, 1600x800, 10 steps: 739 s |
| output: 1,280,000 splats, 1600x800 pano |
| |
| DA-2 native 1600x800 trace: |
| graph da2_trace_1600.ncnn.param/.bin |
| time 6.29 s (Vulkan f32) |
| max abs 0.144 |
| mean abs 0.0041 |
| cosine 0.99954 |
| |
| full 50 steps: roughly 1 hour on this GPU |
| ``` |
| |
| ### Open3D / TSDF mesh |
| |
| Default mesh is now a **voxel signed-distance field + marching cubes** |
| (`worldgen_mesh.py --method voxel`): |
|
|
| ```text |
| voxel-res 0.25, voxel-stride 4, median 5, trim-poles 0.02, voxel-smooth 2.5 |
| 50k verts / 98k tris |
| ~3.3 s at 1024x512 |
| i2s end-to-end 20 steps: ~70 s (vs 61.5 s shell) |
| ``` |
|
|
| Median filtering, pole trimming and a 2.5-voxel Gaussian smoothing remove the |
| top tab and reduce the left fold. The left crescent is still visible, so this |
| is a large improvement over the needle shell but not a perfect reconstruction. |
|
|
| Poisson/Ball/Alpha are available but were not better: |
|
|
| ```text |
| Poisson: 537,145 verts / 1,067,522 tris, 6.8 s, crescent fold + ripples |
| Ball: 985,990 tris, shell-equivalent |
| Alpha: noisy surface with holes |
| ``` |
|
|
| The voxel TSDF still shows a fold at the left depth discontinuity and a small |
| pole tab, but it removes the needle-like shell stretching. `--mesh-shell` |
| keeps the old Rust mesh (`mesh_shell.ply`). |
|
|
| ### GPU 3DGS renderer |
|
|
| `gpu-splat/` is a wgpu/Vulkan splat renderer. The camera was fixed to |
| inside-out and the projection now matches the CPU reference; increasing the |
| splat radius 2x and raising opacity removes the earlier wash/blue-dome. |
|
|
| ```text |
| gpu-splat.exe gaussians14.ply 0 out.png 0.2 |
| ``` |
|
|
| It renders the official painting scene on the RX 6900 XT; it is not identical |
| to the official demo camera (the GPU view is closer/lower), but it is a real |
| Vulkan GPU 3DGS image. |
|
|
| ### 3DGS render / official comparison |
|
|
| `render_splat_points.py --equirect` projects all 524,288 gaussians back to a |
| 1024x512 equirect image and reconstructs the panorama (colors/geometry intact). |
| An inside-out 90-degree perspective point render also produces a coherent |
| scene: |
|
|
| ```text |
| ply: 524,288 gaussians |
| inside 90 deg: 39,976 projected points |
| ``` |
|
|
|  |
|
|
| `render_splat_gaussian.py` adds 3D covariance + alpha compositing for an |
| equirect image. `render_splat_perspective.py` projects the same covariance |
| through a 90-degree pinhole camera; at 832x480 it produces a scene close to the |
| official demo viewpoint. `--scale-factor 0.7` is the current best balance |
| between blur (1.0) and holes (0.3). |
|
|
| The existing `vulkan-splat` viewer still frames a full panorama badly (washed |
| or tiny orbit view); this is a viewer camera issue, not a PLY/numeric issue. |
| The 3DGS center/scale/quaternion formula is validated at ~1e-7 against the |
| WorldGen Python reference. |
|
|
| ### Fill foreground cleaning |
|
|
| By default `worldgen_i2s_prep.py` writes `cond_clean.png`: GrabCut foreground |
| is removed from the Fill condition with OpenCV inpaint, while the original |
| band (`cond.png`) is composited back after Fill. This stops Fill from repeating |
| the foreground object on both sides. Use `--keep-fg` to disable. |
|
|
| The official painting demo now produces a clean panorama with the house once |
| (no duplicated foreground) and no visible band edge. |
|
|
| ### inpaint_bg (simplified) |
| |
| `worldgen_i2s.py --inpaint-bg`: |
|
|
| ```text |
| GrabCut foreground mask |
| -> OpenCV Telea background inpaint |
| -> DA-2 depth on background |
| -> worldgen-vk splat |
| -> merge foreground splats + background splats in the foreground mask |
| ``` |
|
|
| Test: 1024x512, 5 steps, 524,288 base splats + 20,529 background splats |
| = 544,817 merged vertices. This is not the official OneFormer + FLUX Fill |
| branch; it is a CPU/OpenCV approximation. |
|
|
| ### Clean builds from source |
|
|
| Tested with local source checkouts (same scripts also clone when `-Source` is |
| omitted): |
|
|
| ```text |
| build_sd.ps1 -Source <stable-diffusion.cpp> -> build-sd/bin/Release/sd-cli.exe |
| build_da2.ps1 -Source <ncnn> -> tools/da2-ncnn/da2ncnn.exe + ncnn.dll |
| build_rust.ps1 -> tools/worldgen-vk/worldgen-vk.exe |
| ``` |
|
|
| `sd-cli.exe --list-devices` reports `Vulkan0 AMD Radeon RX 6900 XT`; the |
| freshly built DA-2 runner reproduces the 1024x512 output on the same GPU. |
|
|
| ### DA-2 numeric and performance |
|
|
| ncnn Vulkan vs PyTorch, 1024x512: |
|
|
| ```text |
| max abs 0.0161 |
| mean abs 0.0027 |
| cosine 0.9999994 |
| time 3.05 s (f32 Vulkan) |
| ``` |
|
|
| The ncnn graph has **0 custom layers and 0 CPU-fallback layers**: the final |
| `Squeeze` was replaced by a `Reshape` (`0=1024 1=512`, or `1600 800` for the |
| 1600 trace). The runner must be compiled with `-DNCNN_VULKAN` and call |
| `net.set_vulkan_device(0)`; otherwise ncnn silently falls back to CPU. |
|
|
| ### Latency (1024x512, 20 steps, RX 6900 XT) |
|
|
| ```text |
| t2s total 56.7 s |
| i2s total 61.5 s |
| FLUX sampling ~100% GPU 3D utilization |
| VAE decode 9.0 s untiled -> 1.35 s with --vae-tiling --vae-tile-size 512x512 |
| DA-2 3.05 s |
| Rust RGBD->3DGS 0.29 s |
| ``` |
|
|
| ### RGBD -> 3DGS math |
|
|
| Compared with the WorldGen Python formula on synthetic panorama/depth: |
|
|
| ```text |
| center ~1e-7 |
| scale ~1e-7 |
| quaternion ~1e-7 |
| ``` |
|
|
| ### FLUX |
|
|
| - stable-diffusion.cpp built with `SD_VULKAN=ON` |
| - `Vulkan0 = AMD Radeon RX 6900 XT` |
| - FLUX.1-dev Q4_K_S SHA256 `962d536a...` matches |
| - FLUX.1-Fill-dev official safetensors SHA256 `03e289f5...` matches |
| - converted q4_K: 780 tensors |
| - text2scene LoRA: 312/312 applied |
| - img2scene LoRA: 456/684 applied (228 incompatible keys skipped) |
| |
| ## No PyTorch at inference |
| |
| `worldgen_t2s.py`, `worldgen_i2s.py`, `worldgen_i2s_prep.py` and |
| `worldgen_da2.py` do not import torch. Model inference is: |
|
|
| ```text |
| FLUX/Fill -> sd-cli (C++/Vulkan) |
| DA-2 -> ncnn (C++/Vulkan) |
| 3DGS/mesh -> worldgen-vk (Rust, CPU geometry + PLY) |
| ``` |
|
|
| ## Known gaps |
|
|
| - Not byte/quality-equivalent to official WorldGen: |
| - Fill repeats objects and has a visible band boundary. |
| - Mesh has some stretching from depth discontinuities. |
| - Optional `use_sharp=True` and `inpaint_bg=True` branches are not wired. |
| - Official WorldGen visual/numeric comparison of the full FLUX stage is not done; |
| only component-level validation (DA-2, 3DGS math, weights, pipeline runs). |
| - Default validation resolution is 1024x512; 1600x800 is supported by |
| resampling but is much slower (~1 h per 50-step panorama). |
| - ncnn/pnnx toolchain is C++, not Rust; Rust covers RGBD->3DGS/mesh. |
|
|
