thenukegun10x/PlantDetect-V2-Preview

PlantDetect V2 Preview

Built with DINOv3. This is the first preview of the completed V2 mean-pooling model for 2,991 plant species from Western Australia. Supply one photo, or 2–4 distinct photos from the same plant observation. This release uses the epoch-12 EMA weights from the expanded-data BF16 training run. The CLI uses self-contained TorchScript graphs (encoder.ts and classifier.ts) and imports no project modules. FP32 model.safetensors is also included for research use. Optimizer state and training data are excluded.

Measured accuracy

Evaluation	Photos / observations	Top-1	Top-5
Internal validation, individual photos	42,944 photos	86.25%	93.64%
Internal validation, one-view fused	18,848 observations	90.15%	96.16%
Internal validation, multi-photo fused	10,491 observations	93.24%	98.16%
Separate WA single-photo benchmark	1,822 observations, 271 species	86.33%	93.47%

The external benchmark was selected before predictions and audited for known training overlap. It samples species across three training-count bands and covers 271 of 2,991 classes. It is not a natural-use-frequency estimate or an all-class accuracy measurement. Its single-photo metric is directly comparable with the internal individual photos row, not the one-view fused cohort. Photos mostly come through the GBIF/iNaturalist source family; community labels and incomplete historical upstream observation IDs limit the audit. See evaluation.json for the benchmark manifest checksum and breakdown. These rows were measured from the source EMA checkpoint with GPU BF16 autocast. The FP32 export contains exactly the same weights; CPU FP32 predictions may differ on close calls.

Harness measurements

Every figure here was measured through the artifacts in this repository on the external WA single-photo benchmark (1,822 photos, 271 species), and the centre-crop baseline reproduces the released 86.33% exactly, so the deltas are attributable to the harness.

Calibration (on by default, no extra compute). Raw softmax was over-confident: mean top-1 confidence 0.962 at 0.863 accuracy, ECE 0.101. Dividing the logits by a fitted temperature of 1.90 brings mean confidence to 0.84 and cuts ECE to 0.040 in sample and 0.062 held out (grouped 5-fold cross-validation by species chose 1.904 +/- 0.014, so the scalar is not overfit). Temperature is a strictly monotone rescale, so top-1, top-5 and every argmax-based check are provably unchanged; this was checked explicitly across T in 0.25-12 with zero top-1 flips.

Test-time augmentation (--tta, off by default). Training used random-resized-crop at scale 0.5-1.0 from a 576px cache, but inference only ever saw the single centre 448 crop, so the model is crop-robust in a way the default harness did not exploit. --tta fuses 5, 10 or 12 crops of one photo. Fusion is in probability space; averaging the embeddings of crops of one photo measured worse than the plain centre crop (86.22% vs 86.33%), because crops of a single photo do not lie on the multi-view manifold the pooling was trained for.

single-photo mode	Top-1	Top-5	ECE
centre crop (default)	86.33%	93.47%	0.101
--tta 5	86.39%	94.18%	0.072
--tta 10	86.88%	94.57%	0.072
--tta 12	87.05%	94.62%	0.064

Paired McNemar against the baseline at --tta 12: 26 photos fixed, 13 broken, net +13, p = 0.053. The top-5 gain is the more reliable of the two, and top-5 is also what feeds the candidate list and the taxonomic corroboration. TTA applies to a single photo only: with 2-4 real photos the multi-view path already pools better than any TTA of one photo, so the CLI rejects the combination rather than quietly ignoring it.

Clutter check (with --tta, advisory). A mixed frame - your plant standing in a bush of other species - is where the model is most likely to return a neighbour. Measured on 600 benchmark photos tiled with three other species, the number of distinct genera among one photo's crops is 1.14 for a lone plant, 1.10 for four photos of the same species, and 3.06 for a mixed frame (AUROC 0.947). At the shipped threshold of 3 it nudges 3.0% of single-plant photos and 1.7% of same-species frames, while catching 67.7% of the mixed frames the model gets wrong. It is advisory: it prints a warning and never vetoes an identification. Do not read it as a guarantee - on hand-picked cases it both false-alarmed on a same-genus mix (the model's own sub-crop errors invent spurious genera) and missed a four-genus frame that was still named at 0.82 confidence. The reliable fix is the one the warning asks for: crop to a single subject. It needs --tta > 1, since it reuses those crops; on the default path the JSON reports "checked": false.

TTA is opt-in because it costs one encoder pass per crop. On the machine used for these measurements a CPU 448x448 encode takes ~2.45 s at 4 threads, so one photo costs ~2.5 s by default and ~29 s at --tta 12; the _gpu graphs are far faster. The --tta 12 fused distribution is already well calibrated, so its scores are not scaled again.

Run locally

Requires Python 3.11+, PyTorch, Pillow, and NumPy. The CLI works on Windows and Linux; it does not require timm, safetensors, or a checkout of this project.

pip install -r requirements.txt
python plant_v2_cli.py identify leaf.jpg flower.jpg --device cpu --topk 5

The CLI also accepts --images leaf.jpg flower.jpg, and info prints model metadata. Inputs are RGB, bilinear resized to 576 x 576, centre cropped to 448 x 448, normalized with ImageNet mean/std, and EXIF orientation is applied (phone photos routinely carry an orientation tag; pass --no-exif to disable). The encoder ships as bf16 for both CPU and GPU (encoder.ts / encoder_gpu.ts); bf16 is ~35x faster than fp32 for this model on CPUs with AVX512-BF16 (oneDNN: 1124 vs 32 GFLOP/s) and matches the bf16 autocast the accuracy numbers above were measured with.

Output is a short human-readable report by default; --format json emits the full structured result (strict JSON, so non-finite values become null). Exit codes make it scriptable: 0 identified, 1 not confidently known, 2 usage or input error (missing or corrupt image, checksum mismatch, unavailable device). Scores are calibrated: the softmax is divided by a fitted temperature (1.90), cross-validated on the external benchmark, which cuts expected calibration error from 0.101 to 0.062. --no-calibration gives raw softmax. On a single photo, --tta 5|10|12 trades encoder time for accuracy and adds an advisory clutter warning (see Harness measurements); it is rejected with 2-4 photos, which already pool better.

Unknown species

The classifier can only ever return one of the known taxa, so for a plant outside the taxonomy the honest answer is "not confidently a known species". The CLI requires several independent signals before it names a species, and otherwise reports rejected_unknown with the failing checks - it never silently snaps a guess to the nearest class.

• Taxonomic agreement (always available). The species head's pick must sit in the same genus and family that the genus and family heads predict, and with 2-4 photos every photo's genus must match the fused pick. This is the strongest signal and needs no extra download.
• Nearest-centroid distance (optional). Each known species has an embedding centroid built from training photos; the nearest-centroid cosine must exceed a threshold calibrated on held-out photos of known species, so the false-rejection rate on knowns is a chosen quantile rather than a guess.

The centroid bundle (species_centroids.npz + centroid_report.json, ~5.4 MB) is optional. Without it the CLI still runs and still applies the taxonomic checks, and says so in the output. It is never downloaded implicitly: when it is missing and you are at a terminal the CLI offers to fetch it, or you can run it explicitly.

python plant_v2_cli.py fetch-centroids          # cache it, sha256-verified
python plant_v2_cli.py fetch-centroids --force  # re-download

The cache defaults to tempfile.gettempdir()/plantdetect; override with --cache-dir or PLANTDETECT_CACHE_DIR. --yes skips the prompt, --no-download never offers.

--relaxed keeps only the calibrated distance gate (the taxonomic checks become advisory); --no-reject disables rejection entirely (unsafe - a species is always named); --min-centroid-cosine moves the threshold.

Measured both ways with a grouped 5-fold cross-validation (threshold chosen on 4 folds, scored on the held-out fold), against 1,822 known photos (external WA benchmark, 271 species) and 6,318 photos of 819 species the model was never trained on (image_expansion rows absent from the taxonomy):

threshold	known accepted	unknown rejected
0.60 (first release)	89.0%	52.1%
0.84 (default, max Youden J)	73.0%	83.6%
0.90 (more protective)	62.7%	89.8%
0.95 (most protective)	46.3%	95.5%

Cross-validated: 72.6% +/- 4.8 known accepted and 83.8% +/- 0.8 unknown rejected at threshold 0.842 +/- 0.004 - essentially the same as in-sample, so the operating point is not overfit.

The distance gate cannot be both protective and usable: reaching ~95% unknown rejection costs ~54% of known species. Two earlier figures were wrong and are corrected here - a knowns-only calibration put the threshold at 0.26 (52% unknown rejection), and a 4-species unknown set (103 of 110 photos one seagrass) suggested 93.6% at 0.60. Neither survives contact with 819 real species. centroid_report.json records the full protection curve and the selection rule; --min-centroid-cosine moves the line. The always-on taxonomic checks reject only ~31% of unknowns on their own, so they corroborate rather than carry.

Native ExecuTorch PTE and FP8 weights

executorch/PlantDetect-WA-V2-INT8.pte is a W8A8 export of this same mean-pooling checkpoint for CPU inference without PyTorch. It uses custom plant_quant::* kernels (INT8 weights with per-output-channel FP32 scales, FP16 activations, FP32 accumulation) and ships for the Arm I8MM, SDOT and plain FP16-FML paths. On a Galaxy S24 (Exynos 2400) one 448x448 encode runs in roughly 3 seconds at 5 threads; on an Allwinner A733 (2x Cortex-A76 + 6x Cortex-A55) it runs in roughly 25 seconds at 2 threads. The W8A8 path measures 86.11% top-1 on the separate WA single-photo benchmark above, 0.22 points below the FP32 export. The PTE is single-photo (one image per encode); multi-photo fusion uses classifier.ts with one embedding per photo.

fp8/model.safetensors is the AdaRound E4M3FN per-channel weight-only export of the same checkpoint. Adaptive rounding improved 107 of 108 layer reconstructions (0.875% aggregate MSE reduction over nearest rounding); see fp8/adaround_report.json.

Running the PTE. It does not load on a stock ExecuTorch runtime — it requires the four custom plant_quant::* operators. The operator sources, build instructions and a prebuilt Windows x64 runner are included:

executorch/windows-x64/plant_runner.exe executorch/PlantDetect-WA-V2-INT8.pte encode image.bin embedding.bin
executorch/windows-x64/plant_runner.exe executorch/PlantDetect-WA-V2-INT8.pte classify slots.bin out 4

image.bin is float32 ImageNet-normalised RGB 448x448 (3*448*448 floats); slots.bin is 4*512 floats. Set OMP_NUM_THREADS (5 on an Exynos 2400, 2 on an Allwinner A733). Required ExecuTorch version and commit, ISA tiers (PLANTDETECT_MARCH), activation modes (PLANTDETECT_ACT) and profiling switches are documented in extension/moe/README.md.

Attribution and terms

Built with DINOv3 (vit_base_patch16_dinov3.lvd1689m). DINO-derived weights remain subject to the included DINOv3 agreement. Project-specific code is Apache-2.0. No training or benchmark photos are redistributed. See NOTICE.md.