docs/accuracy-experiments.md

Accuracy Experiments

The estimation is accurate enough to ship (and now drives pricing), but the partner's visual review surfaced a recurring over-count that inflates prices, plus a seasonal-imagery weakness. This brief scopes three experiments to close that gap. Read CLAUDE.md (architecture) and ROADMAP.md first.

Why this matters

Our lawn estimate (lawn_sqft) is the pricing basis (Phase 6, live). The partner reviewed production (Google-imagery) measurements against his own job sheet and confirmed they're solid overall but over-count in three recurring ways, each of which pushes the price up:

• Sidewalk in the right-of-way counted as lawn.
• Backyard concrete (patios/pads) counted as lawn.
• Roof edges bleeding into the lawn mask.

He also flagged a season issue: current Google imagery is sometimes leaf-off (bare trees) with dormant, dry grass, which the green-keyed color baseline and the segmentation model read poorly.

Reconciliation context (see the lawn-estimator-pricing memory): our measured lawn runs a median ~1.27× his recorded sqft, so pricing off it is ~+22% vs his historical charges. He accepts that (he under-measured), but tightening the over-count directly improves price accuracy — so accuracy is the current highest-value work.

Hard constraints — read before touching anything

HISTORICAL NOTE (2026-07): the "SAM is LOCAL ONLY" constraint below was written while SAM was an experiment. It is superseded — Experiment 2 shipped: SAM (facebook/sam-vit-base) is now a lazy-loaded prod dependency, pre-baked into the Docker image, behind the SAM_RESTRICT flag (ON in prod). It is imported by restrict.py and reached from strategies.py. Prod segmentation still runs on mfaytin/mask2former-satellite; SAM is a not-lawn restriction, not the segmenter. The rest of this brief is preserved as-written for provenance.

• Do not perturb production or the byte-identical Douglas gate. The estimation path is guarded by data/regression/douglas_qa.csv: run python -m lawn_estimator.cli --csv data/regression/douglas_qa.csv --imagery naip and diff the numeric columns against data/regression/phase35_douglas.csv. A change that intentionally shifts numbers (Experiment 1 will) is a deliberate re-baseline — gate it behind a config flag and review, don't silently move prod output.
• SAM work is LOCAL and EXPERIMENTAL only. Do NOT import SAM into pipeline.py / segmentation.py or the prod dependency set / Docker image. Use notebooks/ or a scratch script and a separate optional dependency. Prod stays on mfaytin/mask2former-satellite.
• Branch/PR flow: work on a feature branch → PR to dev (auto-deploys the staging Space) → PR dev → main (auto-deploys prod). CI (ruff + pytest) must be green. Never push to the space remote by hand. Details in the lawn-estimator-cicd memory.
• Env: always "c:/Users/sergi/miniconda3/Scripts/conda.exe" run -p ./conda-env --no-capture-output python … (invoking conda-env/python.exe directly crashes numpy's LAPACK with no traceback). conda run cannot execute python -c with newlines — write a temp .py file.
• NAIP ≠ prod. The QA baseline uses NAIP; production uses Google (imagery=auto + billed key), which reads ~10–15% lower. Evaluate accuracy on Google imagery, never the NAIP QA CSV.

Evaluation — the owner judges, visually

The partner's square footage is a loose reference, NOT ground truth. He under-measures (reconciliation showed our lawn ~1.27× his recorded sqft), so do not tune to match data/lawn_care_schedule.csv — treat it only as a rough sanity check.

The real check is visual, and the OWNER makes the call. Generate the 4-panel visualization for each test address (panel 4 = the measured lawn, cyan outline) and surface the images to the owner so he can verify accuracy himself — the same "View" experience as the live app.

• Produce the PNGs by running the pipeline: python -m lawn_estimator.cli --csv <addresses> --imagery google (or --address "…") → PNGs land in data/outputs/google/; panel 4 is the measured lawn.
• Show them where the owner can actually see them. He is often away / on mobile, so the editor side panel isn't enough — publish a shareable page that embeds the PNGs (data-URI <img>s, openable on a phone) or point him to the staging/live app to run the addresses and hit View. For a geometry change (Experiment 1) show before/after of the same properties so the effect is obvious.
• data/regression/phase35_douglas.csv is the NAIP byte-identical gate (regression only, not an accuracy reference).

Experiment 1 — Carve the sidewalk out of the right-of-way

Idea: public sidewalks sit at a fairly standard position/width inside the ROW (commonly a ~4–5 ft walk set back a foot or two from the property line). The ROW extension currently adds the full ROW_BUFFER_FT (12 ft) strip as candidate lawn on street-facing edges; a standard sidewalk band within that strip is hardscape, not turf.

Where: geometry.py — parallel_offset_zone builds the outward quads on street-facing edges (within STREET_MAX_DIST_M); build_estimation_geometry assembles the estimation geometry + the per-frontage extensions. The sidewalk band is a parallel sub-strip to subtract from the ROW quad (or to exclude from the lawn mask within the ROW zone).

Notes / risks: offsets/widths vary; a fixed assumption is an approximation — validate against the viz. The RGB mask already excludes some pavement, so measure the incremental effect and don't double-subtract. This touches estimation geometry, so it WILL move Douglas QA numbers → treat as a deliberate, reviewed re-baseline, ideally behind a config flag (off by default) until proven. Highest-value, most tractable, pure geometry — start here.

Experiment 2 — SAM vs mask2former

Idea: Segment Anything (SAM/SAM2) is class-agnostic and may give tighter turf/hardscape edges than mfaytin/mask2former-satellite. The roadmap frames it as boundary refinement (prompt SAM with our existing lawn mask), not a wholesale replacement.

Where (reference only — do NOT modify for prod): segmentation.py (run_lawn_area_model, _run_segmentation, MODEL_LAWN_CLASS_IDS).

How: local notebook/script; install SAM in a separate/optional env (likely needs GPU). Run SAM on the same Google tiles for the partner addresses; compare resulting lawn area/edges to mask2former, against the viz and his sqft. Decide whether SAM as a refinement pass earns its weight. If it graduates, it returns as a proposal — never a silent import into prod.

Findings (2026-07 — tested locally on 10 Douglas lots through the real LiDAR+RGB math):

• SAM runs on CPU with no new dependency (transformers ships SamModel). Naive point-prompt refinement expands the mask (wrong direction). "Segment-and-classify" (keep SAM pieces overlapping the mask2former lawn) drops real turf — 82% of what pure-SAM removes is grass across the sample.
• The only safe design is the hybrid: mask2former for coverage, SAM subtracts only its confirmed hardscape pieces. Effect is small (~−2%) because the LiDAR path already ignores roof bleed (roofs return no ground points) and the sidewalk carve (Exp 1) handles the ROW.
• A SAM-first per-segment classifier (transparent feature vote per piece) agrees with mask2former within ±6% on confident pieces; the only disagreement is shadowed/ambiguous strips, which no signal we have resolves — LiDAR intensity is a coin flip (AUC 0.51).
• Net: no free accuracy win — the billed number is LiDAR-dominated and every segmentation swap agrees or worsens. See the lawn-estimator-accuracy-exp2 memory + the review pages.

The promising direction — SAM as a hardscape restriction mask (not a lawn finder): SAM segments house and concrete beautifully (crisp boundaries) even without a single "building" class, and both LiDAR and mask2former sometimes eat into the roof. Use SAM's precise house/driveway/patio segments to restrict the whole pipeline — exclude any LiDAR ground point (and mask pixel) that falls inside them — to stop that roof/edge bleed. This is the near-term lever, and it also makes SAM the natural auto-labeler for Experiment 4.

Experiment 3 — Season / imagery vintage

Idea: current Google imagery can be leaf-off with dormant/dry grass. The color baseline run_vegetation_color_threshold (segmentation.py) keys on green (hue + excess-green), so dry turf fails it, and the model may weaken on dormant grass.

Where: imagery adapters sources/imagery.py (NAIP, county orthos) + sources/google.py; _fetch_analysis_image in pipeline.py selects the source; the model + color baseline in segmentation.py. Roadmap "Later" already flags "Winter/leaf-off imagery."

How (measurement-only unless a change proves out): compare sources/seasons for the same addresses — Google (current, maybe dormant) vs NAIP (leaf-on ~2022) — and measure how lawn_sqft and the masks shift. Quantify what dry-grass season costs and whether a source preference or threshold/model tweak helps.

Finding (2026-07): premise NOT reproducible — CLOSED. Current Google analysis imagery is leaf-ON; the dormant-looking tiles were the county-ortho DISPLAY layer (never analyzed). An initial "signal" was an artifact of that conflation plus a hardscape-confounded greenness proxy. NAIP is out (resolution too low). No code. Revisit only if Google actually serves leaf-off imagery. See lawn-estimator-accuracy-exp3.

Experiment 4 — Train our own segmentation model (LATER — the real ceiling-raise)

Why: every off-the-shelf swap (mask2former variants, SAM, per-segment) either agrees with the current number or worsens it — the billed number is LiDAR-dominated and the disputed area is small

• genuinely ambiguous (shadowed strips, flat hardscape). The only way to move the ceiling is a model trained on our own metro imagery with our classes (house / driveway / sidewalk / deck-patio / lawn / canopy). This is the roadmap's "Fine-tuned segmentation" item, promoted to a first-class experiment.

The bottleneck is labeling, not architecture — so this is about AUTO-LABELING / data-engine techniques, not hand-labeling:

• SAM as the auto-annotator (Meta's Segment Anything data engine — how SA-1B was built: model-assisted masks, human-in-the-loop on the hard cases, progressing toward fully automatic). We've shown SAM segments house/concrete crisply → it proposes object masks we auto-classify (mask2former vote + color + LiDAR ground/roof) into training labels.
• Karpathy-style "data engine" flywheel (Tesla Autopilot): model proposes labels → focus human review on the uncertain / failure cases (our shadowed strips) → retrain → repeat. Active learning, not bulk labeling.
• Free supervision we already have: LiDAR classification (building=6 vs ground=2) auto-labels roof vs at-grade; county building footprints (Microsoft/Google Open Buildings — a roadmap item) auto-label the house; parcel − footprint − SAM-hardscape ≈ lawn candidate. Combine these weak/auto labels to bootstrap a training set at near-zero labeling cost.

Model + validation: fine-tune mask2former or train a DeepLabV3+/UNet on the auto-labeled metro set; validate against LiDAR + owner review before it ever replaces mfaytin/mask2former-satellite.

Constraints: training needs a GPU (this box is CPU-only) → cloud GPU, out of band. Strictly experimental; prod stays on the current model until a validated replacement earns it. Method references to evaluate: Karpathy's auto-labeling / data-engine ("auto-research") and Meta's AutoData / SAM data-engine work.

Experiment 5 — Green reclaim (turf the model mislabels as water/grass)

Trigger (2026-07-16, owner report): 8571 Young St, Omaha 68122 — a mostly-lawn new-construction lot measured 1,215 sqft (16.4% of the estimation area) while the color baseline alone saw 4,399 sqft (59.5%). Obvious under-count, the opposite of the over-counts above.

Root cause (debug_hardscape.py + pinned-revision check — identical either way): the model splits large uniform flat turf into classes we don't count. On the Young St tile, class 6 (published "water") covered 44% of the tile — the entire contiguous backyard lawn — and 98.7% of those pixels pass the green color threshold. The empirical [1, 4] lawn mapping was derived on established, tree-heavy lots where class 1 dominates; on treeless uniform turf (new construction, big open backyards) the model reads the texture as water/grass, the mask excludes it, the LiDAR ground points there are dropped, and the estimate comes out several× low.

Tile survey (23 addresses: 5 QA + 18 schedule, Google imagery):

• Class 6 is absent (0%) on 20/23 tiles. Where present it is either turf (Young St 44%, 657 J E George Blvd 3.3% @ 98.8% green, 17531 Madison St 2.4% @ 97% green) or actual water that fails the green gate (708 Kountze Memorial Dr, Bellevue lakefront: class 6 only 3.6% green → correctly not reclaimed).
• Class 2 (published "grass") is a mixed street-edge band on every tile (7–32%) but only 0–8% green — blanket-adding class 2 would wreck good tiles; the green gate keeps its reclaim to +0.2–1.8% of a tile.

Fix (SHIPPED to branch feat/green-reclaim, flag GREEN_RECLAIM, off by default = byte-identical): reclaim_green_turf in segmentation.py — pixels the model assigned to GREEN_RECLAIM_CLASS_IDS ([2, 6]) that also pass run_vegetation_color_threshold are added back to the lawn mask before it gates the LiDAR points. Reuses the veg mask the pipeline already computes (no extra pass). Zero-network unit tests in tests/test_segmentation_reclaim.py.

Limitation: the green gate can't recover dormant/dry mislabeled turf (brown grass fails the color threshold — Young St's unestablished front strips stay uncounted). That remaining gap is Experiment 4 territory.

Status / order (2026-07)

1. Experiment 1 (sidewalk) — RETIRED (FAILED, owner verdict 2026-07). Built behind SIDEWALK_CARVE on feat/sidewalk-carve (−4%, byte-identical off) but the fixed-geometry ROW sub-band carve did not hold up and will NOT ship. Superseded by the SAM/labeler sidewalk-road line-projection direction and the leaf-on/leaf-off fusion carve (empirical, not a fixed band).
2. Experiment 3 (season) — CLOSED: premise not reproducible (analysis imagery is leaf-on).
3. Experiment 2 (SAM) — EXPLORED: no free accuracy win (LiDAR-dominated). Safe use is the hybrid / hardscape-restriction mask (~−2%); real value is interpretability + auto-labeling for #4.
4. Experiment 4 (train our own model) — IN PROGRESS: Phase 1 bake-off + the shipped cascade (LAWN_CASCADE) live in docs/model-upgrade-plan.md (the plan of record for Phases 2–3, incl. per-step licensing); research facts in docs/phase2-findings.md. Exp 5 (green reclaim, below) shipped as v0.2.