docs/ACCURACY_IMPROVE_STATUS.md

Accuracy improvement pack (resumed 2026-07-21 after PC crash)

Pull from Priyanka

Merged github-dev/New/Priyanka → uday (5462489).

Task 1 — GT for Grid_54 vs H43X2E1 — DONE (curated)

Hand-reviewed fullres detection regions; kept structural changes only (new blue/grey roofs, compound rebuild, solar). Dropped shadows, pool water color, wetland/vegetation.

• Label: docs/delhi_eval/labels/dda_grid54_h43x2e1.png (change≈0.23%, 10 components)
• Pack: docs/delhi_eval/dda_labeling/dda_grid54_h43x2e1/gt_mask.png
• Review notes/overview: .../_review/gt_labeling_notes.json, gt_overview.png

Re-paint and re-ingest anytime:

python scripts/ingest_dda_gt_label.py

Task 2 — Threshold / config recalibration — DONE

v3_frozen: keep thr=0.2 (runs/calibration/v3_recalibration_recommendation.json)

App grid (dl_only): sens 0.4 slightly best; DL floors irrelevant under dl_only (runs/calibration/grid_search_v3_dl_only/summary.json).

Hard-neg thr sweep: still best at 0.2 (val F1 0.688, test F1 0.563) → runs/v3_hardneg_analysis/recommended_threshold.json

Task 3 — Hard negatives + fine-tune — DONE

• Mined 7 veg/seasonal FP tiles → data/delhi_cd/hard_negatives/
• Fixed keep-empty so empty GT tiles actually train
• Run: runs/finetune_hardneg/20260721_112416
  • Best val F1=0.6751 @ thr=0.2 (epoch 4)
  • Test F1=0.548–0.563 (vs v3_frozen test ≈0.595)
  • Hard-neg pairs still 100% FP in analysis — do not promote yet
• Exported: models/adaptformer_delhi/v3_hardneg/
• Production default stays v3_frozen

Task 4 — Fullres DDA detection (cap 5120) — DONE

runs/native_dda/20260721_121410/cap5120/

• 4706×5120, ~19.6 min, change%≈1.21, 23 regions
• Artifacts: change_mask.png, overlay.png, summary.json
• 899 tiles, fusion=dl_only, thr=0.2, model=adaptformer-delhi-v3

# true native overnight if needed:
python scripts/run_native_dda_detection.py --native

Missed-detection accuracy pass (2026-07-21 evening) — APPLIED

Root causes of under-detection on Grid_54 vs H43X2E1:

1. strip_transient_from_mask over-deleted mid-size roofs (weak-perm used max_car*2).
2. Blue-roof recovery rejected novel roofs merged with existing blue courts.
3. Dark roofs / solar (bright→black tarp) had no recovery; shadow strip could erase them.
4. AdaptFormer often fires only on fragments — mild score hysteresis + open on low mask.

Applied in app/detection_engine.py:

• Protect blue/dark-roof + structural large footprints in transient strip
• Chromatic recovery keeps novel (non-blue-before) pixels; lower min area
• recover_dark_roof_construction (local grow + strict standalone)
• Protect dark-new pixels in shadow strip
• DL-only mild hysteresis (low ≈ 0.65×thr, low-mask opened)
• Strip helper for large seasonal veg (when NDVI evidence is clear)

Re-applied to run #47 (working 4706×5120 PNGs):

python scripts/reapply_run47_accuracy.py

• change% ≈ 4.23, 60 regions (was ~1.21 / 10)
• GT recall vs overlay ≈ 0.96 (was ~0.69); 0 missed GT components
• Artifacts: runs/accuracy_improve_20260721/

Note: curated GT #7 is basketball-court activity (people/paint), not structural — still detected under the denser mask; structural policy may want to drop it from GT later.

Tuesday 2026-07-28 — day plan integration

Uday P0 — Real weights + held-out baseline

• models/adaptformer_delhi/v3_frozen/model.safetensors present (~50 MB)
• /health → loadedFrom=.../v3_frozen, available=true, device=cuda
• Held-out test split (data/delhi_cd/test, 4 pairs) @ thr=0.2, dl_only, TTA off:
  • mean F1=0.5867 · P=0.6028 · R=0.614 · IoU=0.4171
  • Artifact: runs/tuesday_baseline_20260728/metrics.json
  • Re-run: python scripts/record_tuesday_baseline.py

Integrated from Priyanka (same day)

• pair_align guard wired in detect_service (identical → message, low NCC → warning)
• Shadow classification: shadow-only blobs kept as Shadow; structural vs all %
• Verify: python scripts/verify_tuesday.py → ALL CHECKS PASSED

Wednesday 2026-07-29 — retrain + hard-negatives

Training fixes (--preset wed)

• Drop empty real GT; keep mined hn_* hard-neg tiles
• Loss: CE + pos_weight
• Oversample change tiles ×4; full-image 256 resize; change-centered crops
• Val-calibrate + freeze threshold
• Warm-start: v3_frozen

Hard-negative mining

• Refreshed scripts/mine_hard_negatives.py (parking / seasonal veg / shadow tags)
• Saved 5 tiles (hn_delhi_0002/0006/0013/...), all tagged vegetation_seasonal+shadow
• Appended to data/delhi_cd/train/manifest.json

Result (target test F1 > 0.60) — HIT

• Run: runs/finetune_wed/20260729_134453
• best val F1=0.630 · frozen thr=0.446
• test F1=0.6051 · P=0.526 · R=0.772 · IoU=0.436
• Export: models/adaptformer_delhi/wed_retrain/ (does not replace production v3_frozen)
• Summary: data/delhi_cd/wednesday_retrain/metrics.json

python scripts/build_delhi_cd_splits.py --min-change-frac 0.001 --stratify
python scripts/mine_hard_negatives.py --include-all-labeled --min-fp-frac 0.03
python scripts/finetune_adaptformer.py --delhi-cd data/delhi_cd --preset wed --out runs/finetune_wed

Thursday 2026-07-30 — DSIFN spike + wed_retrain OP freeze

Priyanka

Fast-forward merged github-dev/New/Priyanka → uday (ROI crop / training packs).

DSIFN vs AdaptFormer (boundary completeness)

• Cloned official DSIFN + Change-Detection-Review index under third_party/
• Official Drive package = dataset only (no .pth/.h5); trained proxy on DSIFN val → test F1≈0.485
• Delhi 5 pairs (4 test + 1 val): AdaptFormer fills interiors; DSIFN does not

Model	F1	Fill	Hole
AdaptFormer (wed_retrain)	0.624	0.832	0.148
DSIFN proxy	0.090	0.356	0.644

Decision: KEEP_ADAPTFORMER — do not switch backbones.
Artifacts: data/delhi_cd/thursday_dsifn_compare/DECISION.md

wed_retrain operating point

Sweep F_β thr + DETECTION_TTA + DETECTION_MULTISCALE on held-out test:

Config	Test F1	Notes
thr=0.446, TTA=off, MS=off	0.606	Wed sidecar baseline
thr=0.354 (F_β), TTA/MS variants	≤0.601	Drop (hurt F1)
thr=0.446, TTA=full, MS=off	0.626	KEEP (+0.020 F1, ~3× flips)
thr=0.446, MS=0.75,1	0.587	Drop

Frozen OP (models/adaptformer_delhi/wed_retrain/):

• threshold=0.446
• DETECTION_TTA=full
• DETECTION_MULTISCALE=off
• Production remains v3_frozen until an explicit promote.

python scripts/sweep_wed_operating_point.py
python scripts/dsifn_proxy_train.py --epochs 6 --batch 2
python scripts/compare_dsifn_vs_adaptformer.py

Friday 2026-07-31 — integrate + regression (Priyanka, Task 1)

Labeling round (drone pairs, before splits)

Hand/assisted-labeled 9 drone before/after pairs (georeferenced but un-orthorectified, NCC 0.07–0.61 — well below the ~0.9 needed for clean pixel correspondence). Registration attempts (SIFT homography, ECC affine/euclidean/homography on the static region) could not raise NCC further — confirmed local parallax, not a fixable global misalignment; orthomosaic reprocessing from raw photos is the only real fix. See each pack's meta.json (orthomosaics_needed: true) for the caveat.

By explicit user directive (2026-07-31), these 9 are now flagged training_use: true (training_use_override recorded in each meta.json) and registered in docs/delhi_eval/manifest.json with pack-local before.png/after.png (not the original Downloads TIFFs, so the split is portable to Uday's machine):

pair_id	change_frac	split
dda_1_2_scene	0.0079	train
dda_after_testing	0.1604	train
dda_before1_after	0.2124	train
dda_before1_after2_testing	0.2342	test
dda_before2_after2	0.3396	val
dda_before3_after3	0.0332	val
dda_before4_after4	0.1932	train
dda_before5_after5	0.2232	train
dda_before6_after6	0.0471	train

An earlier aligned-source labeling attempt (2019→2026 satellite pair, 1 positive + 3 seasonal hard-negatives, dda_before2019_r0..r3) was created, then removed by user request before ingest — not present in the splits.

Splits rebuilt

python scripts/build_delhi_cd_splits.py --min-change-frac 0.001 --stratify

33 labeled pairs → train=23 · val=5 · test=5 (was 24/16·4·4 before today).

Regression check

• python scripts/validate_detection.py (unit checks) — all pass, no change from HEAD.
• python scripts/validate_detection.py --benchmark --method "AI-Based Deep Learning" — synthetic-scene F1 is very low (0.00–0.06) vs the old runs/calibration/best_params.json numbers (0.86–1.0). Not a regression from this week's commits — confirmed via git diff that feature_based_method/fusion logic are byte-identical to before Tuesday. Root cause: those old numbers were measured under smart_union fusion (classical+DL) on a different model; current default is dl_only (app/detection_config.py:200 docstring: smart_union "collapsed mean F1 from ~0.58 to ~0.04" on the v3 Delhi model — a deliberate, documented tradeoff). The synthetic suite's expected baseline is stale for dl_only and should be recalibrated separately; it is not something Tue–Fri changes broke.
• End-to-end wiring smoke (ROI crop → detect → export training pack → ingest) — all 4 steps pass, artifacts cleaned up.

Shadow classification — removed (user decision)

Reverted Tuesday's Shadow DDA type / region classifier (change_type_map.py, detection_engine.py, detect_service.py). Shadow-only pixels are still suppressed (strip_shadow_only_from_mask, unchanged) — just no longer surfaced as a labeled class or split stat.

ROI → training-pack export — shipped

POST /api/dda/training/pack (app/dda/training_pack.py, app/dda/training_routes.py) + scripts/ingest_dda_gt_label.py --pair-id generalization + "Export training pack" button in the result view (static/js/dda/result.js). Verified end-to-end (see regression check above).

🚩 Found + fixed: test-set drift risk (before handing off to Uday)

Rebuilding the splits (needed to fold in Friday's labels) reshuffled which pairs land in test even at the same seed — Wednesday/Thursday's frozen wed_retrain operating point (thr=0.446, TTA=full, test F1=0.6263) and Tuesday's v3_frozen baseline (F1=0.5867) were both measured on {delhi_0024, delhi_0001, delhi_0005, delhi_0016}. The first Friday rebuild produced a different test set ({delhi_0024, delhi_0011, delhi_0017, delhi_0022, dda_before1_after2_testing}, only 1 pair overlapping) — any F1 number from that set would not be comparable to Tue/Wed/Thu's, and a drop could look like "the model got worse" when it's really "the benchmark changed."

Fix implemented: scripts/build_delhi_cd_splits.py now supports --freeze-test-ids (default: data/delhi_cd/frozen_test_ids.json), which pins specific pair_ids to test on every run regardless of what's added/removed from the labeled pool. Created data/delhi_cd/frozen_test_ids.json = the original Tue/Wed/Thu set. Rebuilt splits — test is now exactly {delhi_0024, delhi_0001, delhi_0005, delhi_0016} again, so today's held-out eval is directly comparable to the whole week's numbers. All 9 drone pairs land in train (8) / val (1) — none in test — so the benchmark stays 100% clean satellite imagery regardless of the drone-data alignment-quality question (see below).

Residual minor risk: dda_before2019 removal + drone-pair addition also shifted val composition (dda_1_2_scene, a low-NCC pair, is now in val). Val doesn't affect the headline test-F1 comparison but does feed the val-threshold sweep (Thursday's sweep_wed_operating_point.py pattern) — if Uday re-runs that sweep, note val is not fully frozen, only test.

Handoff to Uday (Friday P0)

• Fine-tune on the rebuilt 24/5/4 split (train includes 8 of the 9 new drone pairs — low-NCC caveat applies; test is unchanged/frozen, clean satellite only).
• Held-out eval on the frozen test set → before/after F1/P/R vs Tuesday's v3_frozen baseline (now a valid, apples-to-apples comparison) → promotion decision (v3_frozen vs wed_retrain vs new fine-tune).
• Recommended extra check: since low-NCC drone pairs are now in training, additionally report F1 on the frozen test set split by training-data recipe (with vs without the 9 drone pairs, if time allows an ablation) to isolate whether they helped or hurt real (clean satellite) performance — directly answers "did the drone data degrade F1."

Result: fri2_retrain (delhi + drone) — confirmed regression, root cause found

Checkpoint	Frozen test F1	thr	Val F1	Val P	Val R
v3_frozen	0.587	0.2	0.677	0.686	0.700
wed_retrain	0.605 (0.626 TTA)	0.446	0.630	-	-
fri2_retrain	0.487	0.5	0.436	0.345	0.641

fri2_retrain's threshold.json follows the same schema as v3/wed (real per-epoch val calibration, not a default) - so the drop is not a threshold-calibration bug. Precision collapsed 0.69->0.35 while recall stayed reasonable (0.64): the model became far less selective, consistent with training on the 9 un-orthorectified drone pairs (dda_before*, dda_after*, dda_1_2_scene - NCC 0.07-0.61) - large misaligned regions labeled "change" taught the model "big pixel differences = change" too broadly. Confirmed visually on DDA_Report_55_before1_tif_vs_after_tif.pdf: 16.63% flagged, mostly cars/vegetation/ground, buildings still not covered.

Fix in progress: ablation split prepared and pushed - data/delhi_cd_ablation_no_drone/ (same 24-pair pool that produced wed_retrain's 0.605/0.626, drone pairs excluded via the new --exclude-prefix flag on build_delhi_cd_splits.py, frozen test set intact). Requested from Uday:

python scripts/finetune_adaptformer.py --delhi-cd data/delhi_cd_ablation_no_drone --preset wed --out runs/finetune_fri3_no_drone

If F1 recovers to ~0.60+, confirms drone data as the cause -> exclude drone pairs from training going forward (keep eval-only, per the original plan); promote whichever of wed_retrain / fri3_no_drone scores higher on the frozen test set. Status: awaiting Uday's run (sent EOD Friday).