WorldGen: 0 CPU fallback DA-2 params

ae92738 verified 1 day ago

7.63 kB

WorldGen Vulkan validation

Environment

Windows 11
AMD Radeon RX 6900 XT 16GB
Vulkan 1.4 (AMD proprietary driver)
No CUDA

Pipeline

t2s:
  prompt
   -> FLUX.1-dev Q4_K_S + WorldGen text2scene LoRA
      (stable-diffusion.cpp, SD_VULKAN=ON)
   -> 1024x512 equirectangular panorama
   -> DA-2 depth (ncnn Vulkan, fp32)
   -> worldgen-vk (Rust): RGBD -> 3DGS PLY + mesh PLY

i2s:
  image
   -> 90-degree full-height band -> cond/mask
   -> FLUX.1-Fill-dev Q4_K (official safetensors converted with sd-cli)
      + WorldGen img2scene LoRA
   -> Fill pano, composite original image region back
   -> DA-2 depth (ncnn Vulkan)
   -> worldgen-vk (Rust): RGBD -> 3DGS PLY + mesh PLY

Verified runs

t2s

prompt: a cozy wooden bedroom with warm lamps
1024x512, 20 steps
output: output/t2s-ncnn-clean/
  524,288 splats
  1,046,528 triangles

i2s

input: assets/rainforest/photo.jpg
1024x512, 20 steps
output: output/i2s-ncnn-clean/
  524,288 splats
  1,046,528 triangles

i2s seam blending

The Fill output and the original input band are composited with cv2.seamlessClone (NORMAL_CLONE). Without OpenCV, a 24 px feathered alpha blend is used. This removes the visible vertical band boundary; it does not remove Fill object repetition.

1600x800

t2s, 1600x800, 10 steps: 739 s
output: 1,280,000 splats, 1600x800 pano

DA-2 native 1600x800 trace:
  graph      da2_trace_1600.ncnn.param/.bin
  time       6.29 s (Vulkan f32)
  max abs    0.144
  mean abs   0.0041
  cosine     0.99954

full 50 steps: roughly 1 hour on this GPU

Open3D / TSDF mesh

Default mesh is now a voxel signed-distance field + marching cubes (worldgen_mesh.py --method voxel):

voxel-res 0.25, voxel-stride 4, median 5, trim-poles 0.02, voxel-smooth 2.5
50k verts / 98k tris
~3.3 s at 1024x512
i2s end-to-end 20 steps: ~70 s (vs 61.5 s shell)

Median filtering, pole trimming and a 2.5-voxel Gaussian smoothing remove the top tab and reduce the left fold. The left crescent is still visible, so this is a large improvement over the needle shell but not a perfect reconstruction.

Poisson/Ball/Alpha are available but were not better:

Poisson: 537,145 verts / 1,067,522 tris, 6.8 s, crescent fold + ripples
Ball:    985,990 tris, shell-equivalent
Alpha:   noisy surface with holes

The voxel TSDF still shows a fold at the left depth discontinuity and a small pole tab, but it removes the needle-like shell stretching. --mesh-shell keeps the old Rust mesh (mesh_shell.ply).

GPU 3DGS renderer

gpu-splat/ is a wgpu/Vulkan splat renderer. The camera was fixed to inside-out and the projection now matches the CPU reference; increasing the splat radius 2x and raising opacity removes the earlier wash/blue-dome.

gpu-splat.exe gaussians14.ply 0 out.png 0.2

It renders the official painting scene on the RX 6900 XT; it is not identical to the official demo camera (the GPU view is closer/lower), but it is a real Vulkan GPU 3DGS image.

3DGS render / official comparison

render_splat_points.py --equirect projects all 524,288 gaussians back to a 1024x512 equirect image and reconstructs the panorama (colors/geometry intact). An inside-out 90-degree perspective point render also produces a coherent scene:

ply: 524,288 gaussians
inside 90 deg: 39,976 projected points

render_splat_gaussian.py adds 3D covariance + alpha compositing for an equirect image. render_splat_perspective.py projects the same covariance through a 90-degree pinhole camera; at 832x480 it produces a scene close to the official demo viewpoint. --scale-factor 0.7 is the current best balance between blur (1.0) and holes (0.3).

The existing vulkan-splat viewer still frames a full panorama badly (washed or tiny orbit view); this is a viewer camera issue, not a PLY/numeric issue. The 3DGS center/scale/quaternion formula is validated at ~1e-7 against the WorldGen Python reference.

Fill foreground cleaning

By default worldgen_i2s_prep.py writes cond_clean.png: GrabCut foreground is removed from the Fill condition with OpenCV inpaint, while the original band (cond.png) is composited back after Fill. This stops Fill from repeating the foreground object on both sides. Use --keep-fg to disable.

The official painting demo now produces a clean panorama with the house once (no duplicated foreground) and no visible band edge.

inpaint_bg (simplified)

worldgen_i2s.py --inpaint-bg:

GrabCut foreground mask
-> OpenCV Telea background inpaint
-> DA-2 depth on background
-> worldgen-vk splat
-> merge foreground splats + background splats in the foreground mask

Test: 1024x512, 5 steps, 524,288 base splats + 20,529 background splats = 544,817 merged vertices. This is not the official OneFormer + FLUX Fill branch; it is a CPU/OpenCV approximation.

Clean builds from source

Tested with local source checkouts (same scripts also clone when -Source is omitted):

build_sd.ps1 -Source <stable-diffusion.cpp>   -> build-sd/bin/Release/sd-cli.exe
build_da2.ps1 -Source <ncnn>                  -> tools/da2-ncnn/da2ncnn.exe + ncnn.dll
build_rust.ps1                                -> tools/worldgen-vk/worldgen-vk.exe

sd-cli.exe --list-devices reports Vulkan0 AMD Radeon RX 6900 XT; the freshly built DA-2 runner reproduces the 1024x512 output on the same GPU.

DA-2 numeric and performance

ncnn Vulkan vs PyTorch, 1024x512:

max abs   0.0161
mean abs  0.0027
cosine    0.9999994
time      3.05 s (f32 Vulkan)

The ncnn graph has 0 custom layers and 0 CPU-fallback layers: the final Squeeze was replaced by a Reshape (0=1024 1=512, or 1600 800 for the 1600 trace). The runner must be compiled with -DNCNN_VULKAN and call net.set_vulkan_device(0); otherwise ncnn silently falls back to CPU.

Latency (1024x512, 20 steps, RX 6900 XT)

t2s total          56.7 s
i2s total          61.5 s
FLUX sampling      ~100% GPU 3D utilization
VAE decode         9.0 s untiled -> 1.35 s with --vae-tiling --vae-tile-size 512x512
DA-2               3.05 s
Rust RGBD->3DGS    0.29 s

RGBD -> 3DGS math

Compared with the WorldGen Python formula on synthetic panorama/depth:

center      ~1e-7
scale       ~1e-7
quaternion  ~1e-7

FLUX

stable-diffusion.cpp built with SD_VULKAN=ON
Vulkan0 = AMD Radeon RX 6900 XT
FLUX.1-dev Q4_K_S SHA256 962d536a... matches
FLUX.1-Fill-dev official safetensors SHA256 03e289f5... matches
converted q4_K: 780 tensors
text2scene LoRA: 312/312 applied
img2scene LoRA: 456/684 applied (228 incompatible keys skipped)

No PyTorch at inference

worldgen_t2s.py, worldgen_i2s.py, worldgen_i2s_prep.py and worldgen_da2.py do not import torch. Model inference is:

FLUX/Fill -> sd-cli (C++/Vulkan)
DA-2      -> ncnn (C++/Vulkan)
3DGS/mesh -> worldgen-vk (Rust, CPU geometry + PLY)

Known gaps

Not byte/quality-equivalent to official WorldGen:
- Fill repeats objects and has a visible band boundary.
- Mesh has some stretching from depth discontinuities.
Optional use_sharp=True and inpaint_bg=True branches are not wired.
Official WorldGen visual/numeric comparison of the full FLUX stage is not done; only component-level validation (DA-2, 3DGS math, weights, pipeline runs).
Default validation resolution is 1024x512; 1600x800 is supported by resampling but is much slower (~1 h per 50-step panorama).
ncnn/pnnx toolchain is C++, not Rust; Rust covers RGBD->3DGS/mesh.