Leaves.PH canopy classifier: RESULTS.md

RESULTS.md

@@ -0,0 +1,95 @@
+# Manual high-resolution labeling — reality-anchored accuracy (Claim 3 gold truth)
+
+Done 2026-05-29. The first accuracy number for Leaves.PH measured against
+**human visual labels on high-resolution imagery**, not against another satellite
+product. Replaces "agreement with ESA WorldCover" as the headline accuracy.
+
+## What was done
+
+- **Sample.** 42 30m pixels inside the 17-LGU mask, drawn (seed 42) across **6
+  disjoint strata** that partition the 904,715 valid pixels, so the sample is
+  population-weightable (Horvitz-Thompson) and both predicted classes are present:
+
+  | Stratum | Definition | Pop pixels | Pop % | Sampled |
+  |---|---|---|---|---|
+  | D clear-canopy | NDVI > 0.65 | 77,845 | 8.6% | 10 |
+  | C boundary | NDVI in [0.55, 0.65] | 37,953 | 4.2% | 10 |
+  | A dense-urban | ESA built-up & NDVI < 0.55 | 475,429 | 52.6% | 6 |
+  | B reclaimed | ESA bare & NDVI < 0.55 | 6,209 | 0.7% | 6 |
+  | E green-fringe | ESA tree & NDVI < 0.55 | 32,660 | 3.6% | 6 |
+  | F other-low | other ESA & NDVI < 0.55 | 274,619 | 30.4% | 4 |
+
+- **Reference imagery.** Esri World Imagery (~0.5–1m, no-auth ArcGIS export) per
+  chip, bbox = target 30m cell ± 3.5 cells (~210m), with a red box drawn on the
+  **exact** 30m target cell. Sentinel-2 RGB crop (`s2_rgb_2021.tif`) as a second
+  view. Ambiguous cells re-inspected at 2.3× and 3.4× zoom.
+- **Labeling.** Each chip labeled by Claude via visual inspection (read the
+  annotated chip, decide "is ≥25% of the marked 30m cell woody **tree** canopy?").
+  Labels + one-line reasons in `my_labels.csv`; chips in `chips/`, zooms in `zoom/`,
+  ultra-zooms in `uz/`, contact sheet `contact_sheet.png`.
+
+## Headline result — NDVI > 0.62 mask vs manual labels
+
+Confusion (n=42): **TP 10, FP 3, FN 5, TN 24.**
+
+| Metric | Pooled (stratified sample) | Population-weighted (H-T) |
+|---|---|---|
+| Precision | 0.77 (95% CI 0.50–0.92) | **0.78** (95% CI 0.54–1.0) |
+| Recall | 0.67 (95% CI 0.42–0.85) | **0.73** (95% CI 0.61–0.88) |
+| F1 | 0.71 | **0.76** |
+| IoU | 0.56 | **0.61** |
+| Accuracy | 0.81 | 0.95 |
+
+- **Implied true canopy fraction = 10.5%**, against the mask's 9.86% — the
+  reality-anchored canopy area lands within ~0.7pp of the published 9.79% estimate.
+- Dropping the 5 cells Claude flagged ambiguous (n=37): precision 0.86, recall 0.83,
+  IoU 0.73 (population-weighted precision 0.86, recall 0.83).
+
+### Where the errors live (matches the prior ESA-gap analysis)
+
+- **3 false positives, all high-NDVI non-tree vegetation:** riparian scrub on a
+  gravel bar (#0), a dense grass/low-veg slope (#6), a dry-grass field with a green
+  edge (#12). Exactly the "we over-call dense grass/scrub" failure the threshold
+  analysis predicted.
+- **5 false negatives, all real canopy the strict 0.62 cut or 30m mixing misses:**
+  a tall-canopy cell at NDVI 0.619 just under the cut (#13, Meta 11m), and four
+  ESA-tree green-fringe cells (#32/33/35/37) where sub-5m or sparse street/yard trees
+  dilute the 30m NDVI below threshold.
+- Per-stratum: dense-urban (A), reclaimed (B) and water/other (F) are 100% correctly
+  negative; all recall loss is concentrated in the green-fringe (E) stratum.
+
+## Detection-model ceiling — Meta height ≥ 5m vs manual labels
+
+The detection model (CLIP + gradient-boosted regression) is trained to **reproduce
+Meta's 1m canopy fraction**, so the Meta target is its accuracy ceiling. Meta height
+≥ 5m as a classifier vs the same manual labels:
+
+| Metric | Pooled | Population-weighted |
+|---|---|---|
+| Precision | 1.00 (95% CI 0.68–1.0) | 1.00 |
+| Recall | 0.53 (95% CI 0.30–0.75) | 0.59 (95% CI 0.34–0.81) |
+| F1 / IoU | 0.70 / 0.53 | 0.74 / 0.59 |
+
+Meta never false-positives in this sample (every Meta ≥ 5m cell is real canopy by
+eye) but recovers only ~59% of canopy and implies just 6.2% canopy vs the 10.5%
+truth — it misses sub-5m and sparse urban trees. The NDVI mask trades some precision
+(0.78 vs 1.00) for much higher recall (0.73 vs 0.59); the two are complementary, and
+the model, reproducing Meta at R² 0.83–0.86, inherits Meta's high-precision /
+moderate-recall profile.
+
+## Honest caveats
+
+- n=42 manual labels → wide CIs; this is a defensible first reality-anchored number,
+  not a definitive accuracy. Single labeler (Claude) — no second-rater κ.
+- "Canopy" = ≥25% of the 30m cell is woody tree canopy by eye on ~0.6m imagery dated
+  near (not exactly) 2021; canopy moves slowly so ±1–2yr basemap drift is minor.
+- Population-weighting leans on small per-stratum n (A=6, F=4 carry large weights);
+  those strata are unambiguous (roofs / bare / water), so their TN weight is robust,
+  but the weighted recall depends on the 6 green-fringe chips.
+
+## Files
+
+`sample_metadata.csv` · `my_labels.csv` · `accuracy_results.json` ·
+`strata_pop.json` · `contact_sheet.png` · `chips/` `zoom/` `uz/` `s2crops/` ·
+scripts `sample_chips.py` `build_composites.py` `zoom.py` `uz.py` `uz2.py`
+`compute_accuracy.py` `build_contact_sheet.py`.