docs/PROGRESS_REPORT_2026-08.md

DDA Change Detection — Progress Report

Period: 21 Jul 2026 – 13 Aug 2026
App: Change Detection Webapp (DDA mode)
Live model (13 Aug 2026): AdaptFormer wed_retrain · threshold 0.446 · TTA full · fusion dl_only · CUDA

This report documents data, methods tried, accuracy (previous vs current), per-area / per-class results, live before/after detections, challenges, and current status.

---

1. Headline accuracy (frozen satellite test)

Same 4 held-out Sentinel-2 pairs every time: delhi_0024, delhi_0001, delhi_0005, delhi_0016.
Metric is pixel F1 for change vs no-change (not DDA class F1).

Checkpoint	Date	Thr	TTA	F1	Precision	Recall	IoU	Decision
v3_frozen (previous production)	28 Jul	0.20	off	58.7%	60.3%	61.4%	41.7%	Rollback baseline
wed_retrain	29 Jul	0.446	off	60.5%	52.6%	77.3%	43.7%	Promoted OP
wed_retrain + TTA full (current live)	30 Jul	0.446	full	62.6%	53.3%	83.1%	46.0%	Live
fri3_no_drone	31 Jul	0.50	—	58.9%	48.8%	82.1%	—	Keep wed
sat_v2_retrain	1 Aug	0.50	—	57.6%	49.5%	79.7%	—	Do not promote
fri2_retrain (drone in train)	31 Jul	0.50	—	48.7%	36.1%	84.9%	—	Rejected
DSIFN proxy (not live)	30 Jul	—	—	9.0%	—	—	—	Rejected

Previous → current (same test set): 58.7% F1 → 60.5% F1 (62.6% with TTA)
Gain: +1.8 pp without TTA, +3.9 pp with TTA full. Recall rose sharply (61% → 83%); precision fell (60% → 53%).

Sources: data/delhi_cd/tuesday_baseline/metrics.json, models/adaptformer_delhi/wed_retrain/metrics.json, data/delhi_cd/thursday_op_sweep/metrics.json, checkpoint summary.json files.

---

2. Accuracy per frozen-test pair (current model)

AdaptFormer wed_retrain @ 0.446 on the frozen test + 1 val pair (from the DSIFN compare, same weights).

Pair	Zone / type	F1	Precision	Recall	Fill (interior)	Hole rate
delhi_0024	Sentinel-2 Delhi · mixed_gsd + vegetation	54.8%	40.3%	85.4%	81.4%	15.8%
delhi_0001	Sentinel-2 Delhi · mixed_gsd + vegetation	62.8%	46.2%	97.8%	93.7%	6.8%
delhi_0005	Sentinel-2 Delhi · mixed_gsd + vegetation	55.3%	61.8%	50.1%	66.5%	31.5%
delhi_0016	Sentinel-2 Delhi · mixed_gsd + vegetation	69.6%	56.4%	91.0%	84.1%	15.1%
delhi_0021 (val, not in headline)	Sentinel-2 Delhi · mixed_gsd + vegetation	69.4%	55.2%	93.7%	90.1%	5.0%
Mean (5 Delhi pairs)		62.4%			83.2%	14.8%

Tuesday v3_frozen per-pair F1 (older split, for reference only — pair IDs differ except delhi_0005):
delhi_0028 66.5% · delhi_0023 59.1% · delhi_0005 51.9% · delhi_0012 57.2% · mean 58.7%.

---

3. Areas and classes tested

3.1 Pixel-change classes in the labeled eval set

Class / coverage tag	What it means	How tested	Pixel F1 available?
mixed_gsd	Coarse 10 m Sentinel-2 (2019 vs 2026, season-matched)	Frozen test (4 pairs) + train/val	Yes — this is the 58.7% / 62.6% number
vegetation	Phenology / land-cover on same Sentinel-2 tiles	Same pairs (tagged together with mixed_gsd)	Not split out from building; joint pixel F1 only
building / road	Individual roofs, roads	DDA GeoTIFF + drone packs	No class-wise F1 on frozen test (10 m S2 cannot resolve buildings)
Structural roofs / solar (Grid_54 vs H43X2E1)	Hand GT: new blue/grey roofs, rebuild, solar	Curated mask + run #47 re-apply	GT component recall ≈ 96% after 21 Jul accuracy pass (was ~69%)
Hard-negatives	Parking, seasonal veg, shadow FP tiles	5 mined hn_* tiles in train	Used for training, not a test score

DDA report classes (classifier on detected blobs, not GT F1):

DDA class	Internal engine label	Notes
New Construction	New Construction/Building	Roofs, solar, structure
Demolition	demolition / clearing / debris	Rare on recent drone reports
Extension	expansion / renovation	Keyword map; seldom fired
Vegetation Change	vegetation / tree / crop	Often over-fires on misaligned drone
Other	Unclassified Ground Change, bare soil	Dominant on poorly aligned pairs

There is no held-out F1 per DDA class. Class columns below are counts from live reports, not accuracy vs labeled class GT.

3.2 Drone / high-res pairs vs hand GT (pixel F1)

Un-orthorectified drone mosaics. NCC 0.07–0.61. Ceiling is low until true orthomosaics exist.

Pair	GT change %	Pred % (best drone-fix pass)	F1	NCC
dda_before1_after	21.24	11.93	8.2%	0.07
dda_before3_after3	3.32	12.29	33.3%	0.61
dda_before4_after4	19.32	48.95	30.0%	0.41
dda_before5_after5	22.32	19.46	31.8%	0.46
dda_before6_after6	4.71	8.24	14.5%	0.51
Mean			23.6%

Baseline on the same packs (v3_frozen @ 0.5): mean F1 22.3%.
After wed_retrain + skip-reg auto: 23.6% (+1.3 pp). Success bar was +15 pp or ≥45% — not met.

Aligned v2 drone packs (dda_before5_after5_v2 NCC 0.74, dda_before6_after6_v2 NCC 0.75) were added to train in sat_v2_retrain and did not beat wed_retrain on frozen satellite test (57.6% vs 60.5%).

---

4. Live DDA reports (before / after / overlay)

Images are the app’s stored working copies (registered/resized grid), not the original GeoTIFF bytes.

4.1 before6 vs after6 — main debug pair (Central Delhi drone)

Report	Date	Change %	Regions	New Construction	Other	Notes
#61 (reference)	1 Aug	11.5%	31	8	23	Stronger recall; sat_v2 / older post-process
#71–72	5 Aug	1.13%	5	—	—	Shadow strip wiped true dark roofs
#73	6 Aug	7.3%	24	—	—	Soft strip
#74–75	6 Aug	10.2%	29	—	—	Recovery flood / edge ribbons
#77	6 Aug	9.0%	24	—	—	Still blob/edge heavy
#78	6 Aug	3.3%	10	—	—	Over-suppressed
#79 (after OP switch)	8 Aug	3.4%	17	1	16	wed_retrain + weak-align skip-recovery — under-detect

Hand GT for this scene is ~4.7% change. Report 61 over-fired vs GT; report 79 under-fired on construction class (16/17 blobs labeled Other; one New Construction of 0.3 m²).

Report 61 (reference) — before / after / detection overlay

Report 79 (current OP) — before / after / detection overlay

4.2 Other live pairs (latest completed run per title)

Pair	Run	Change %	Regions	New Construction	Vegetation	Other	Overlay
before5 vs after5	80	7.73%	13	3	2	8	overlays/7_79fbb9f50ef845b8ac9fbce8fe770632.png
before7 vs after7	70	6.40%	6	2	0	4	overlays/7_6adbd9998af64f24abc6980eddfbc764.png
before1 vs after	68	17.53%	17	3	1	13	overlays/7_9a026711e2684a4294b1cbd7665dd707.png
before3 vs after3	66	18.64%	37	7	0	30	overlays/7_a3f6c74870f544b0b2ec25e1ff9974c7.png
before4 vs after4	56	7.97%	25	6	2	17	overlays/7_a7d5ff6c1d3b4efdb3014da94b565326.png
Grid_54 vs H43X2E1	51	3.25%	60	9	2	49	overlays/6_7a53ee025d81485b80d0e6b3f0de215c.png
TEST-1 vs TEST-2 (8k JPEG)	81	13.13%	60	2	51	7	overlays/7_6c78393faec548d9b24afa9b3b24ef13.png

before5 (run 80) — before / after / overlay

TEST-1 vs TEST-2 (run 81) — before / after / overlay
First attempt (job 72) stuck 2.3 h at 46% on 8192×4320 with TTA full. Re-run completed (~3 h) at 13.1% change, mostly Vegetation Change (51/60).

---

5. Data used

Source	Role	Notes
Sentinel-2 L2A (MGRS 43RFM), 2019-06-29 vs 2026-06-17	Train / val / frozen test	10 m GSD; vegetation + mixed_gsd; cannot score individual buildings
DDA GeoTIFFs (Grid_54, H43X2E1, GRID 54_2025/2026, 0304, etc.)	Live detection + Grid_54 hand GT	Some files corrupt (GRID 54_2025/2026 TIFF directory unreadable)
Drone before/after TIFFs (before1–before7)	Live QA + attempted train	Un-orthorectified; NCC often 0.07–0.61
Aligned v2 drone packs (before5/6 v2)	Train experiment (sat_v2)	NCC ~0.74–0.75; did not lift frozen F1
Hard-neg tiles hn_delhi_*	Train	Parking / seasonal veg / shadow FPs
Frozen test pin	data/delhi_cd/frozen_test_ids.json	Prevents split reshuffle from faking regressions

---

6. Process and approaches tried

1. AdaptFormer Delhi fine-tune (v3_frozen) — thr 0.2, dl_only. Held-out F1 58.7%. Production rollback.
2. Hard-neg fine-tune (v3_hardneg) — val F1 0.675 but test 54.8–56.3%. Not promoted (FP tiles still 100% FP in analysis).
3. Post-process accuracy pass (21 Jul) — protect roofs in transient/shadow strip; chromatic + dark-roof recovery; mild DL hysteresis. Grid_54 GT recall 0.69 → 0.96; change% 1.21 → 4.23.
4. Wednesday retrain (wed_retrain) — CE + pos_weight, drop empty GT, keep hard-negs, oversample change ×4. Test F1 60.5%. Hit the >60% target.
5. Operating-point sweep — F_β thr 0.354 dropped F1. TTA full kept: 62.6%. Multiscale dropped.
6. DSIFN backbone spike — Delhi F1 9% vs AdaptFormer 62%. Decision: keep AdaptFormer. BIT-CD ensemble coded but weights missing.
7. Drone-in-train (fri2) — frozen F1 48.7% (precision collapse). Drone packs excluded from training thereafter.
8. No-drone ablation (fri3) — F1 58.9% < wed. Promote wed.
9. Aligned v2 drone train (sat_v2) — F1 57.6%. Do not promote. Briefly used live @ thr 0.5; caused poor UI quality.
10. Live OP restore (8 Aug) — switch .env to wed_retrain @ 0.446; NCC-gate GeoTIFF registration; skip roof recovery on weak-align pairs; unify ECC gate at 0.55.
11. Polygon overlays — clip fill to change mask; tighter contour extraction (earlier in the branch).

---

7. Challenges

Issue	Impact	Status
Un-orthorectified drone (local parallax)	Pixel GT F1 stuck ~24%; NCC 0.07–0.61	Blocker without new orthomosaics
Training on those drone packs	Frozen satellite F1 60.5% → 48.7%	Fixed: exclude from train
10 m Sentinel-2 test set	Headline F1 does not measure building/road skill	Still the only frozen apples-to-apples number
Weak-align post-process tug-of-war	before6 reports swung 1% ↔ 11.5%	Soft-strip + skip recovery; over-suppressed (#79 = 3.4%)
Shadow strip vs dark new roofs	True construction deleted	Soft path when registration_ok=False
8k×4k TTA-full inference	Job 72 hung 2.3 h at 46%	Cancelled; job 73 finished in ~3 h
Corrupt GRID 54_2025/2026 TIFFs	Library skip	Unrelated to model
No per-class GT F1	Cannot quote “building accuracy = X%”	Only blob counts + pixel change F1
BIT-CD / extra NN	Ensemble off; DSIFN rejected	Stay on AdaptFormer

---

8. Achievements

• Held-out satellite F1 58.7% → 60.5% (62.6% TTA) on a frozen 4-pair test.
• Hit the Wednesday target test F1 > 60%.
• Proved drone-in-train hurts clean satellite F1; froze test IDs so later labels cannot fake a drop.
• Rejected a weaker backbone (DSIFN) with measured Delhi F1 9% vs 62%.
• Grid_54 structural GT recall ~96% after recovery/strip fixes (offline re-apply).
• Live stack documented: AdaptFormer neural net already in production (AI-Based Deep Learning).
• DDA product path: jobs, reports, polygon overlays, ROI → training-pack export.

---

9. Current status (13 Aug 2026)

Item	Value
Live weights	models/adaptformer_delhi/wed_retrain
Calibrated threshold	0.446
Fusion / TTA	dl_only / full
Headline accuracy	62.6% F1 (TTA) / 60.5% F1 (no TTA) vs previous 58.7%
Best drone-pack F1	~24% mean (alignment-limited)
Latest before6 report	#79 · 3.4% · 17 regions (worse visually than #61 11.5%)
Latest large JPEG	TEST-1/2 13.1%, 60 regions, 51 vegetation
Open gap	Weak-align recall vs FP; need orthomosaics or a pair-specific recall path
Not recommended	New backbone, drone-in-train, sat_v2 @ 0.5

Recommended next work: restore selective dark-roof recovery on weak pairs (or a slightly lower thr for low-NCC GeoTIFFs) so before6 construction returns without re-flooding; keep wed_retrain as the satellite OP; treat drone F1 as a geometry problem, not a model-swap problem.