RESULTS.md

# Manual high-resolution labeling — reality-anchored accuracy (Claim 3 gold truth)

Done 2026-05-29. The first accuracy number for Leaves.PH measured against
**human visual labels on high-resolution imagery**, not against another satellite
product. Replaces "agreement with ESA WorldCover" as the headline accuracy.

## What was done

- **Sample.** 42 30m pixels inside the 17-LGU mask, drawn (seed 42) across **6
  disjoint strata** that partition the 904,715 valid pixels, so the sample is
  population-weightable (Horvitz-Thompson) and both predicted classes are present:

  | Stratum | Definition | Pop pixels | Pop % | Sampled |
  |---|---|---|---|---|
  | D clear-canopy | NDVI > 0.65 | 77,845 | 8.6% | 10 |
  | C boundary | NDVI in [0.55, 0.65] | 37,953 | 4.2% | 10 |
  | A dense-urban | ESA built-up & NDVI < 0.55 | 475,429 | 52.6% | 6 |
  | B reclaimed | ESA bare & NDVI < 0.55 | 6,209 | 0.7% | 6 |
  | E green-fringe | ESA tree & NDVI < 0.55 | 32,660 | 3.6% | 6 |
  | F other-low | other ESA & NDVI < 0.55 | 274,619 | 30.4% | 4 |

- **Reference imagery.** Esri World Imagery (~0.5–1m, no-auth ArcGIS export) per
  chip, bbox = target 30m cell ± 3.5 cells (~210m), with a red box drawn on the
  **exact** 30m target cell. Sentinel-2 RGB crop (`s2_rgb_2021.tif`) as a second
  view. Ambiguous cells re-inspected at 2.3× and 3.4× zoom.
- **Labeling.** Each chip labeled by Claude via visual inspection (read the
  annotated chip, decide "is ≥25% of the marked 30m cell woody **tree** canopy?").
  Labels + one-line reasons in `my_labels.csv`; chips in `chips/`, zooms in `zoom/`,
  ultra-zooms in `uz/`, contact sheet `contact_sheet.png`.

## Headline result — NDVI > 0.62 mask vs manual labels

Confusion (n=42): **TP 10, FP 3, FN 5, TN 24.**

| Metric | Pooled (stratified sample) | Population-weighted (H-T) |
|---|---|---|
| Precision | 0.77 (95% CI 0.50–0.92) | **0.78** (95% CI 0.54–1.0) |
| Recall | 0.67 (95% CI 0.42–0.85) | **0.73** (95% CI 0.61–0.88) |
| F1 | 0.71 | **0.76** |
| IoU | 0.56 | **0.61** |
| Accuracy | 0.81 | 0.95 |

- **Implied true canopy fraction = 10.5%**, against the mask's 9.86% — the
  reality-anchored canopy area lands within ~0.7pp of the published 9.79% estimate.
- Dropping the 5 cells Claude flagged ambiguous (n=37): precision 0.86, recall 0.83,
  IoU 0.73 (population-weighted precision 0.86, recall 0.83).

### Where the errors live (matches the prior ESA-gap analysis)

- **3 false positives, all high-NDVI non-tree vegetation:** riparian scrub on a
  gravel bar (#0), a dense grass/low-veg slope (#6), a dry-grass field with a green
  edge (#12). Exactly the "we over-call dense grass/scrub" failure the threshold
  analysis predicted.
- **5 false negatives, all real canopy the strict 0.62 cut or 30m mixing misses:**
  a tall-canopy cell at NDVI 0.619 just under the cut (#13, Meta 11m), and four
  ESA-tree green-fringe cells (#32/33/35/37) where sub-5m or sparse street/yard trees
  dilute the 30m NDVI below threshold.
- Per-stratum: dense-urban (A), reclaimed (B) and water/other (F) are 100% correctly
  negative; all recall loss is concentrated in the green-fringe (E) stratum.

## Detection-model ceiling — Meta height ≥ 5m vs manual labels

The detection model (CLIP + gradient-boosted regression) is trained to **reproduce
Meta's 1m canopy fraction**, so the Meta target is its accuracy ceiling. Meta height
≥ 5m as a classifier vs the same manual labels:

| Metric | Pooled | Population-weighted |
|---|---|---|
| Precision | 1.00 (95% CI 0.68–1.0) | 1.00 |
| Recall | 0.53 (95% CI 0.30–0.75) | 0.59 (95% CI 0.34–0.81) |
| F1 / IoU | 0.70 / 0.53 | 0.74 / 0.59 |

Meta never false-positives in this sample (every Meta ≥ 5m cell is real canopy by
eye) but recovers only ~59% of canopy and implies just 6.2% canopy vs the 10.5%
truth — it misses sub-5m and sparse urban trees. The NDVI mask trades some precision
(0.78 vs 1.00) for much higher recall (0.73 vs 0.59); the two are complementary, and
the model, reproducing Meta at R² 0.83–0.86, inherits Meta's high-precision /
moderate-recall profile.

## Honest caveats

- n=42 manual labels → wide CIs; this is a defensible first reality-anchored number,
  not a definitive accuracy. Single labeler (Claude) — no second-rater κ.
- "Canopy" = ≥25% of the 30m cell is woody tree canopy by eye on ~0.6m imagery dated
  near (not exactly) 2021; canopy moves slowly so ±1–2yr basemap drift is minor.
- Population-weighting leans on small per-stratum n (A=6, F=4 carry large weights);
  those strata are unambiguous (roofs / bare / water), so their TN weight is robust,
  but the weighted recall depends on the 6 green-fringe chips.

## Files

`sample_metadata.csv` · `my_labels.csv` · `accuracy_results.json` ·
`strata_pop.json` · `contact_sheet.png` · `chips/` `zoom/` `uz/` `s2crops/` ·
scripts `sample_chips.py` `build_composites.py` `zoom.py` `uz.py` `uz2.py`
`compute_accuracy.py` `build_contact_sheet.py`.