feat(gate): tune wrong-crop thresholds (MARGIN 0.12 / OTHER_MIN 0.80)

From a 57-image cross-crop sweep: false-reject 5.6%->2.8%, catch ~78% held.
Adds sweep tool, crop-ID training scaffold, and production plan.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs/CROP_ID_GATE.md

@@ -0,0 +1,86 @@
+# Wrong-crop gate & field-change: production hardening
+
+This documents the current heuristic, what a 57-image cross-crop sweep showed,
+and the two model-based upgrades that take both features to production grade.
+
+## 1. Current state (shipped)
+
+**Wrong-crop gate** (`ml/serve/inference_app.py::_cross_crop_check`): when the
+selected crop's result is unsure, the image is scored against the other in-memory
+crop models and the request is blocked (with a suggested crop) if a different
+crop fits clearly better. Confident, in-catalog results skip the pass.
+
+Thresholds were tuned from `scripts/cross_crop_sweep.py` (57 images, all 5 crops,
+each scored by all 5 models):
+
+| Thresholds (STRONG/MARGIN/OTHER_MIN) | False-reject | Catch | False-accept |
+|---|---|---|---|
+| 0.85 / 0.15 / 0.75 (initial) | 5.6% | 78.5% | 31 |
+| **0.85 / 0.12 / 0.80 (deployed)** | **2.8%** | **78.5%** | 31 |
+
+**Field-change comparison** (`lib/healthComparison.ts`): continuous expected
+health over the model's full probability distribution (replaced 4 fixed
+severity buckets). Detects healthy↔diseased shifts and uncertain/borderline
+moves; cannot measure spread of a confidently-identical disease.
+
+## 2. What the sweep proved about the ceiling
+
+The heuristic compares five **disease** classifiers' confidences. That caps out
+at ~78% catch / ~3% false-reject because:
+
+- The lone false-reject is a rice leaf the **rice model itself** scores 50% on
+  (already "no clear match") while corn scores 85%.
+- False-accepts concentrate on out-of-crop overconfidence, worst among the
+  grasses and the weak rice model: rice→corn 5/7 slip, wheat→soybean 4/8,
+  rice→{others} ~3/7.
+
+Disease models are simply not calibrated for "is this even my crop?".
+
+## 3. Upgrade A — dedicated crop-ID classifier (the real gate)
+
+A single 5-class "which crop is this leaf?" model. Crops are far more visually
+separable than diseases within a crop, so expect >>95% accuracy from a small
+model. It replaces "compare five disease models" with one reliable signal, which
+should eliminate the rice false-rejects and most cross-crop false-accepts.
+
+**Data:** already on disk — every `ml/data/<crop>/**` image is implicitly
+labeled by its crop. No new collection needed. Hold out by source/folder to
+avoid the framing shortcut documented in the soybean/rice notes.
+
+**Train (needs a TF environment — not runnable in the Py3.14 repo venv):**
+```
+python scripts/train_crop_id.py --epochs 8 --out ml/models/crop_id
+```
+Scaffold provided in `scripts/train_crop_id.py` (EfficientNetB0, in-model
+rescaling, TFLite export — mirrors the per-crop pipeline).
+
+**Serve:** load `crop_id` alongside the disease models; in `/predict`, run it
+first. If `argmax(crop_id) != selected_crop` with margin ≥ τ, block and suggest
+`argmax`. Keep the current heuristic as a fallback when the classifier is
+absent. Re-run `scripts/cross_crop_sweep.py` to set τ and confirm
+false-reject ≈ 0.
+
+## 4. Upgrade B — severity / leaf-coverage model (real spread detection)
+
+The field-change comparison can't quantify worsening of a confirmed disease
+because the classifier outputs identity, not severity. A pixel-color
+"% leaf affected" heuristic was prototyped and rejected — too crop-dependent
+(healthy soybean read 53% "damaged").
+
+**Right approach:** a severity regressor/segmenter (e.g. lesion-area
+segmentation, or an ordinal severity head) trained on **severity-labeled** data
+(PlantVillage severity subsets, or in-house annotation). Output a 0–100
+affected-area per check; the comparison then trends the delta of a real
+measurement instead of a label. This is a genuine data+training project, not a
+config change.
+
+## 5. Reproduce / re-tune
+
+```
+# against the live (rate-limited) space:
+python scripts/cross_crop_sweep.py --url https://jaithrap-cropintel.hf.space --pace 3.3 --out sweep.json
+# against a local service:
+python scripts/cross_crop_sweep.py --url http://127.0.0.1:8000 --out sweep.json
+# re-analyze only:
+python scripts/cross_crop_sweep.py --analyze sweep.json
+```

ml/serve/inference_app.py

@@ -88,9 +88,15 @@ _log_lock = threading.Lock()
 #  - Otherwise block only when another crop is itself confident AND beats the
 #    selected crop by a clear margin — so genuinely close calls stay accepted.
 # All confidences below are fractions in [0, 1].
+# Defaults tuned from a 57-image cross-crop sweep against the live models
+# (scripts/cross_crop_sweep.py): MARGIN 0.12 + OTHER_MIN 0.80 minimized
+# false-rejects (valid leaves wrongly blocked: 5.6% -> 2.8%) with no loss of
+# catch rate (~78%). The residual false-rejects/accepts trace to the disease
+# models being overconfident on out-of-crop leaves (esp. the rice model); the
+# durable fix is a dedicated crop-ID classifier — see docs/CROP_ID_GATE.md.
 CROP_STRONG_CONF = float(os.environ.get("CROPINTEL_CROP_STRONG_CONF", "0.85"))
-CROP_MISMATCH_MARGIN = float(os.environ.get("CROPINTEL_CROP_MISMATCH_MARGIN", "0.15"))
-CROP_OTHER_MIN_CONF = float(os.environ.get("CROPINTEL_CROP_OTHER_MIN_CONF", "0.75"))
+CROP_MISMATCH_MARGIN = float(os.environ.get("CROPINTEL_CROP_MISMATCH_MARGIN", "0.12"))
+CROP_OTHER_MIN_CONF = float(os.environ.get("CROPINTEL_CROP_OTHER_MIN_CONF", "0.80"))
 
 
 def _cross_crop_check(pil_image, selected_crop: str, selected_conf: float,

scripts/cross_crop_sweep.py

@@ -0,0 +1,178 @@
+"""
+Cross-crop gate characterization + threshold tuning.
+
+Sends sample leaf images through every crop model (via the running inference
+service or a deployed URL) to capture, per image, each crop's top-1 confidence
+and not-in-catalog flag. Then simulates the wrong-crop gate over a grid of
+thresholds to report false-reject (valid leaf wrongly blocked) and catch
+(wrong-crop correctly blocked) rates, and recommends thresholds.
+
+This is how the defaults in ml/serve/inference_app.py were tuned. Re-run it
+after retraining any crop model or adding a crop-ID classifier.
+
+Usage:
+  python scripts/cross_crop_sweep.py --url http://127.0.0.1:8000 \
+      --per-crop 9 --out /tmp/sweep.json
+  python scripts/cross_crop_sweep.py --analyze /tmp/sweep.json
+"""
+import argparse
+import io
+import json
+import os
+import time
+from itertools import product
+
+import numpy as np
+from PIL import Image
+
+CROPS = ["corn", "soybean", "wheat", "rice", "tomato"]
+DISEASE_KW = [
+    "blight", "rust", "spot", "mold", "blast", "virus", "mildew", "septoria",
+    "smut", "fusarium", "mosaic", "pustule", "death", "bacterial",
+]
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def quality_ok(fp: str) -> bool:
+    """Mirror inference_app's image-quality gate so we only probe usable photos."""
+    try:
+        im = Image.open(fp).convert("RGB")
+        w, h = im.size
+        if w < 200 or h < 200:
+            return False
+        a = np.asarray(im, dtype=np.float32)
+        r, g, b = a[:, :, 0], a[:, :, 1], a[:, :, 2]
+        if float(np.mean((g > 40) & (g > r * 1.05) & (g > b * 1.05))) < 0.03:
+            return False
+        gray = 0.299 * r + 0.587 * g + 0.114 * b
+        sv = float(np.var(np.concatenate(
+            [np.diff(gray, axis=1).ravel(), np.diff(gray, axis=0).ravel()])))
+        return sv >= 25.0
+    except Exception:
+        return False
+
+
+def pick_images(crop: str, n_healthy: int, n_diseased: int):
+    h, d = [], []
+    base = os.path.join(ROOT, "ml", "data", crop)
+    for root, _, fs in os.walk(base):
+        low = root.lower()
+        for f in sorted(fs):
+            if not f.lower().endswith((".jpg", ".jpeg", ".png")):
+                continue
+            fp = os.path.join(root, f)
+            if not quality_ok(fp):
+                continue
+            if "healthy" in low and len(h) < n_healthy:
+                h.append(fp)
+            elif "healthy" not in low and any(k in low for k in DISEASE_KW) and len(d) < n_diseased:
+                d.append(fp)
+        if len(h) >= n_healthy and len(d) >= n_diseased:
+            break
+    return [("healthy", x) for x in h] + [("diseased", x) for x in d]
+
+
+def predict(url: str, fp: str, crop: str) -> dict:
+    import requests  # local import so --analyze works without requests
+    with open(fp, "rb") as fh:
+        r = requests.post(f"{url}/predict",
+                          files={"image": fh}, data={"crop": crop}, timeout=45)
+    try:
+        d = r.json()
+    except Exception:
+        return {"err": "noparse"}
+    if "disease" not in d:
+        return {"err": str(d.get("error", "?"))[:40]}
+    return {"disease": d["disease"], "conf": d["confidence"],
+            "nic": bool(d.get("not_in_catalog")), "mismatch": bool(d.get("crop_mismatch"))}
+
+
+def run_sweep(url: str, per_crop: int, out: str, pace: float):
+    n_h = max(1, per_crop // 2)
+    results = []
+    for tc in CROPS:
+        for kind, img in pick_images(tc, n_h, per_crop - n_h):
+            vec = {sc: predict(url, img, sc) for sc in CROPS
+                   for _ in [time.sleep(pace)]}
+            results.append({"true_crop": tc, "kind": kind,
+                            "img": os.path.basename(img), "vec": vec})
+            json.dump(results, open(out, "w"), indent=1)
+            print(f"{tc:8} {kind:8} self={vec[tc]}")
+    print(f"DONE -> {out}")
+
+
+def _frac(v):
+    return None if v.get("conf") is None else v["conf"] / 100.0
+
+
+def _gate(sc, snic, others, strong, margin, other_min):
+    if sc is None:
+        return None
+    if sc >= strong and not snic:
+        return False
+    best = max([o for o in others if o is not None], default=0.0)
+    return best >= other_min and (best - sc) >= margin
+
+
+def _evaluate(data, strong, margin, other_min):
+    fr = fr_tot = catch = cross_tot = fa = 0
+    for r in data:
+        T, v = r["true_crop"], r["vec"]
+        if "err" in v.get(T, {}):
+            continue
+        others = [_frac(v[c]) for c in CROPS if c != T and "err" not in v.get(c, {})]
+        g = _gate(_frac(v[T]), v[T].get("nic", False), others, strong, margin, other_min)
+        if g is not None:
+            fr_tot += 1
+            fr += int(g)
+        for sx in CROPS:
+            if sx == T or "err" in v.get(sx, {}):
+                continue
+            oth = [_frac(v[c]) for c in CROPS if c != sx and "err" not in v.get(c, {})]
+            gg = _gate(_frac(v[sx]), v[sx].get("nic", False), oth, strong, margin, other_min)
+            if gg is None:
+                continue
+            cross_tot += 1
+            catch += int(gg)
+            fa += int(not gg)
+    return dict(fr=fr, fr_tot=fr_tot, fr_rate=fr / fr_tot if fr_tot else 0,
+                catch=catch, cross_tot=cross_tot,
+                catch_rate=catch / cross_tot if cross_tot else 0, fa=fa)
+
+
+def analyze(path: str):
+    data = json.load(open(path))
+    print(f"records: {len(data)}")
+    cur = _evaluate(data, 0.85, 0.12, 0.80)
+    print(f"\nDEPLOYED (0.85/0.12/0.80): "
+          f"false-reject {cur['fr']}/{cur['fr_tot']} ({cur['fr_rate']*100:.1f}%), "
+          f"catch {cur['catch']}/{cur['cross_tot']} ({cur['catch_rate']*100:.1f}%), "
+          f"false-accept {cur['fa']}")
+    best = None
+    for strong, margin, other_min in product(
+            [0.80, 0.82, 0.85, 0.88, 0.90], [0.10, 0.12, 0.15, 0.18, 0.22, 0.25],
+            [0.70, 0.75, 0.80, 0.85]):
+        m = _evaluate(data, strong, margin, other_min)
+        key = (m["fr_rate"], -m["catch_rate"])
+        if best is None or key < best[0]:
+            best = (key, (strong, margin, other_min), m)
+    (_, p, m) = best
+    print(f"\nGRID BEST (min false-reject, then max catch): "
+          f"STRONG={p[0]} MARGIN={p[1]} OTHER_MIN={p[2]} -> "
+          f"false-reject {m['fr_rate']*100:.1f}%, catch {m['catch_rate']*100:.1f}%")
+
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--url", default="http://127.0.0.1:8000")
+    ap.add_argument("--per-crop", type=int, default=9)
+    ap.add_argument("--out", default="/tmp/cross_crop_sweep.json")
+    ap.add_argument("--pace", type=float, default=0.2,
+                    help="seconds between calls (raise to ~3.3 against a rate-limited host)")
+    ap.add_argument("--analyze", help="analyze an existing results JSON and exit")
+    a = ap.parse_args()
+    if a.analyze:
+        analyze(a.analyze)
+    else:
+        run_sweep(a.url, a.per_crop, a.out, a.pace)
+        analyze(a.out)

scripts/train_crop_id.py

@@ -0,0 +1,97 @@
+"""
+Train the crop-ID classifier: "which of the 5 crops is this leaf?"
+
+This is the durable replacement for the heuristic wrong-crop gate (see
+docs/CROP_ID_GATE.md). Crops are far more visually separable than diseases
+within a crop, so a small transfer-learned model should reach very high
+accuracy and remove the cross-crop false-accepts/rejects the heuristic can't.
+
+Data: reuses the existing per-crop image folders — every image under
+ml/data/<crop>/** is implicitly labeled by <crop>. No new data needed.
+
+NOTE: requires the ML training environment (TensorFlow). It will NOT run in the
+repo's Py3.14 inference venv. Run it where the per-crop trainer runs.
+
+Usage:
+  python scripts/train_crop_id.py --epochs 8 --out ml/models/crop_id
+"""
+import argparse
+import json
+from datetime import datetime
+from pathlib import Path
+
+import tensorflow as tf
+
+from ml.config import CROPS, DATA_DIR, MODEL_CONFIG
+from ml.utils.model_builder import build_model, unfreeze_model
+from ml.utils.tflite_converter import convert_to_tflite
+
+CROP_NAMES = list(CROPS.keys())  # stable label order: corn, soybean, wheat, rice, tomato
+IMG_SIZE = MODEL_CONFIG["input_shape"][:2]
+
+
+def make_datasets(val_split: float, batch: int, seed: int):
+    """Label = crop folder name. image_dataset_from_directory recurses into each
+    crop's nested disease subfolders, so every leaf image is labeled by its crop.
+
+    IMPORTANT: deliver [0, 1] floats and DO NOT use brightness augmentation —
+    ImageDataGenerator(brightness_range) zeroes [0,1] float images (see
+    feedback_imagedatagen_brightness_bug). The model carries its own rescaling.
+    """
+    common = dict(directory=str(DATA_DIR), labels="inferred", label_mode="int",
+                  class_names=CROP_NAMES, image_size=IMG_SIZE, batch_size=batch, seed=seed)
+    train = tf.keras.utils.image_dataset_from_directory(
+        validation_split=val_split, subset="training", **common)
+    val = tf.keras.utils.image_dataset_from_directory(
+        validation_split=val_split, subset="validation", **common)
+    norm = tf.keras.layers.Rescaling(1.0 / 255)
+    aug = tf.keras.Sequential([
+        tf.keras.layers.RandomFlip("horizontal"),
+        tf.keras.layers.RandomRotation(0.1),
+        tf.keras.layers.RandomZoom(0.1),
+    ])
+    AUTOTUNE = tf.data.AUTOTUNE
+    train = train.map(lambda x, y: (aug(norm(x), training=True), y), AUTOTUNE).prefetch(AUTOTUNE)
+    val = val.map(lambda x, y: (norm(x), y), AUTOTUNE).prefetch(AUTOTUNE)
+    return train, val
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--epochs", type=int, default=8)
+    ap.add_argument("--fine-tune-epochs", type=int, default=4)
+    ap.add_argument("--batch", type=int, default=32)
+    ap.add_argument("--val-split", type=float, default=0.2)
+    ap.add_argument("--seed", type=int, default=1337)
+    ap.add_argument("--out", default="ml/models/crop_id")
+    a = ap.parse_args()
+
+    train, val = make_datasets(a.val_split, a.batch, a.seed)
+
+    model = build_model(num_classes=len(CROP_NAMES), crop="crop_id")
+    model.compile(optimizer="adam",
+                  loss=tf.keras.losses.SparseCategoricalCrossentropy(),
+                  metrics=["accuracy"])
+    model.fit(train, validation_data=val, epochs=a.epochs)
+
+    # Fine-tune the backbone for a few more epochs.
+    unfreeze_model(model)
+    model.fit(train, validation_data=val, epochs=a.fine_tune_epochs)
+
+    version = "v1_" + datetime.now().strftime("%Y%m%d_%H%M%S")
+    out = Path(a.out) / version
+    out.mkdir(parents=True, exist_ok=True)
+    model.save(out / "checkpoint.keras")
+    convert_to_tflite(model, out / "model.tflite")
+    json.dump({"class_names": CROP_NAMES, "input_shape": list(MODEL_CONFIG["input_shape"])},
+              open(out / "metadata.json", "w"), indent=2)
+    # production pointer
+    json.dump({"version": version}, open(Path(a.out) / "production.json", "w"), indent=2)
+    val_acc = model.evaluate(val, return_dict=True).get("accuracy")
+    print(f"crop-ID model saved to {out}  val_accuracy={val_acc}")
+    print("Next: load it in inference_app and gate /predict on argmax != selected_crop; "
+          "then re-tune with scripts/cross_crop_sweep.py.")
+
+
+if __name__ == "__main__":
+    main()