docs/whu_reference/README.md

WHU Reference Set — Real-Data Pipeline Test (NOT Delhi)

Purpose: validate the detection harness against real satellite imagery and real building footprints, while the actual Delhi evaluation set (docs/delhi_eval/) is still being curated. This is not a Delhi substitute and is not used for any Delhi F1 target — see Accuracy_Improvement_Plan.xlsx ("Non-negotiable first step: Build a Delhi evaluation set... LEVIR-CD tiles do not represent Delhi imagery"). This dataset has the same domain-mismatch problem relative to Delhi that LEVIR-CD does.

Source

WHU Building Dataset, "Satellite dataset I (global cities)" — https://gpcv.whu.edu.cn/data/building_dataset.html. 204 image tiles (512x512, multiple satellite sensors, 0.3–2.5m GSD) with hand-delineated building-footprint labels. Downloaded to data/whu_reference/raw/ (gitignored, not committed — see .gitignore).

We originally attempted the WHU Building Change Detection Dataset (genuine 2012→2016 Christchurch, NZ bi-temporal pairs, 5.43GB) but the source server was too unreliable from this network (stalled repeatedly, one stall lasted ~2.7h; ~50% downloaded in ~4h before we gave up). Switched to the 113MB single-time dataset instead.

How the pairs were built

scripts/build_whu_reference_pairs.py turns single-time tiles into semi-synthetic before/after pairs:

• after = the original real tile (buildings present)
• before = the same tile with the building-mask region inpainted away (cv2.inpaint, Telea)
• gt = the real building-footprint label (== the synthetic "change" region)

This is not a genuine bi-temporal pair — no real second acquisition, no real illumination/season/registration differences. It's useful for pipeline plumbing and rough sensitivity checks, not for accuracy claims.

Regenerate with:

python scripts/build_whu_reference_pairs.py --count 5
python scripts/compare_methods.py --manifest data/whu_reference/pairs/manifest.json \
    --methods "AI-Based Deep Learning,Feature-Based,Hybrid Approach" --sensitivities 0.5 \
    --out runs/whu_reference_test

Result (2026-07-13, sensitivity=0.5, 5 tiles x 3 methods)

Mean IoU=0.042, F1=0.078 — high precision (0.46-1.0), very low recall (0.03-0.14). The pipeline ran correctly end-to-end (no crashes, masks aligned, metrics computed), but under-detects real building-shaped change at default sensitivity. Directionally consistent with the plan's Day 4 calibration step being needed — not a substitute for it.

Full per-pair numbers: runs/whu_reference_test/manifest_report.json (gitignored).