zhaoshiwen/blur-slam-bpn-code

Blur-SLAM BPN: per-frame blur-kernel learning for 3D Gaussian / Triangle Splatting

This repo contains the code used in an ongoing research project on reconstructing sharp 3D scenes (TUM RGB-D, ScanNet) from blurry monocular video. The pipeline:

1. EVSSM deblurs every raw RGB frame (single-image deblur, used to seed COLMAP and as a per-frame "recon target").
2. COLMAP (CPU) registers cameras and produces a sparse point cloud from the (deblurred) frames; a sparse-depth map is generated per view.
3. A 3D Gaussian / Triangle-Splatting model is trained on the COLMAP scene, with an additional BPN (Blur-Producing Network): a small per-frame MLP that learns to re-blur the current render back toward each frame's original raw blurry image. This lets the 3DGS/triangle representation converge to a sharp, blur-free scene while the BPN absorbs each view's blur — only frames that are NOT already sharp are passed through the BPN loss (sharp-frame skip, selected via Laplacian variance + NIMA-koniq).

A companion dataset repo with the pipeline-intermediate data and a sample trained result (TUM fr1_desk, full-frame COLMAP reconstruction, 613/613 frames registered) is at zhaoshiwen/blur-slam-bpn-data.

Repo contents

scripts/              orchestration scripts accumulated over the project
                       (dataset building, training launchers, sharp-frame
                       selection, evaluation, visualization — ~260 scripts,
                       the full experiment history; see "Reproducing" below
                       for the specific ones used for the fr1_desk pilot)
triangle-splatting/   fork of trianglesplatting/triangle-splatting
                       + train_bpn.py (BPN-augmented training entrypoint)
                       + local edits to arguments/__init__.py, render.py,
                         scene/__init__.py (BPN args, raw-blurry-frame cache,
                         sharp-skip set, sparse-depth loading)
BAGS/                  fork of snldmt/BAGS (Mip-Splatting + BPN)
                       + local edits to arguments/__init__.py and train.py
                         (same BPN / sharp-skip / sparse-depth additions)
EVSSM/                 fork of kkkls/EVSSM (single-image event-based deblur),
                       used as the deblur stage in build_*_scene.sh

Upstream attribution & licenses

• triangle-splatting/ — fork of https://github.com/trianglesplatting/triangle-splatting (Apache-2.0, LICENSE.md; the bundled Gaussian-Splatting rasterizer submodule is under the original Inria/MPII non-commercial research license, LICENSE_GS.md — review this before any commercial use).
• BAGS/ — fork of https://github.com/snldmt/BAGS (Apache-2.0, LICENSE).
• EVSSM/ — fork of https://github.com/kkkls/EVSSM (no explicit license file upstream as of the forked commit — check the upstream repo for current terms before reuse).

Local modifications in all three forks are limited to: BPN MLP + loss wiring, raw-blurry-frame caching (--raw_blurry_glob/--raw_blurry_stride), sharp-frame skip (--bpn_skip_sharp_json), and sparse-depth loss support. Submodules (diff-gaussian-rasterization, simple-knn, etc.) are unchanged from upstream.

Environments

Three separate conda environments (different torch/CUDA/python combos required by each upstream repo):

env	python	torch	used for
trigsplat	3.11	2.4.0+cu121	triangle-splatting training/render
bags	3.8	2.3.0+cu121	BAGS (Mip-Splatting) training/render
evssm	3.10	2.1.0+cu121	EVSSM deblur preprocessing

All three additionally need opencv-python; trigsplat also needs pyiqa (for NIMA-koniq sharp-frame scoring). Set each env up per the upstream repo's own install instructions (each fork's submodules need pip install -e . for the CUDA rasterizer/kNN extensions), then pip install pyiqa into trigsplat.

Also required: a CPU build of COLMAP on $PATH (used with --SiftExtraction.use_gpu 0 --SiftMatching.use_gpu 0 so it doesn't compete with training jobs for GPU memory).

Path setup (read this first)

These scripts were written for one machine's directory layout and use hardcoded absolute paths at the top of each file, typically:

BASE=/home/szha0669/storage/blur_slam_exp        # data + outputs root
REPO_TRI=/srv2/szha0669/blur_slam_exp/repos/triangle-splatting
REPO_BAGS=/srv2/szha0669/blur_slam_exp/repos/BAGS

(and the equivalent BASE = "..." constants at the top of the .py scripts). To reproduce on a different machine, edit these path variables at the top of whichever script you run to point at:

• where you cloned this code repo (for REPO_TRI/REPO_BAGS/EVSSM paths), and
• where you put the dataset repo's contents (for BASE/DATA — see the dataset repo's README for its expected layout, which matches BASE/data/... and BASE/outputs/... here).

Reproducing the fr1_desk full-frame pilot

This is the pipeline that produced the tum_fr1_desk_full data/results in the companion dataset repo (TUM fr1_desk, all 613 raw frames, 613/613 COLMAP-registered):

1. scripts/build_tum_fr1desk_full_scene.sh — EVSSM-deblur all 613 raw frames, run COLMAP (feature_extractor → sequential_matcher → mapper → model_converter, all CPU) on the full sequence, build the i2slam_trigsplat/tum_fr1_desk_full_abl1 training scene (images + sparse/0 + 8 held-out test views), and generate sparse depth maps.
2. scripts/select_sharp_tum_fr1desk_full.py — score each registered frame's raw image (Laplacian variance + NIMA-koniq), select the sharpest 15% as tum_fr1desk_full_sharp_frames.json (these are skipped by the BPN loss).
3. scripts/run_tum_fr1desk_full_50k.sh — train both:
   • train_bpn.py (triangle-splatting, trigsplat env) with --bpn_lr_kernel 4.5e-5 --bpn_lr_mask 4.5e-4 --raw_blurry_stride 1 --bpn_skip_sharp_json <sharp json>, 50K iterations, -r 2, --use_depth_loss --depth_loss_alpha 0.1.
   • train.py (BAGS, bags env) with the equivalent BPN/depth/mask flags and --bpn_lr_kernel 4.5e-6.
   • then render.py at iterations 10K/20K/30K/40K/50K for both.

For other scenes (TUM fr2_xyz/fr3_office, ScanNet), see the analogous build_*_scene.{sh,py} / run_*.sh scripts in scripts/ — the same EVSSM → COLMAP → BPN-training pattern applies, with per-scene path/config edits.