# Datasheet — SpiceNet-Bench

A datasheet (Gebru et al., 2021) for the SpiceNet-Bench cross-source spice
recognition benchmark. Accompanies the finding paper
(`paper/paper_draft.md`). All counts are reproducible from
`outputs/unified_benchmark.json` and the audit scripts named below.

## Motivation

SpiceNet-Bench was built to measure **cross-acquisition-source generalization**
in fine-grained spice recognition — a property invisible to the single-source
studio benchmarks that dominate the published literature. It unifies one
"in-the-wild" source and one "studio" source so that a model trained on one can
be tested on the other.

## Composition

| Source | Style | Classes | Images | Resolution |
|---|---|---:|---:|---|
| SpiceSpectrum (SS) | in-the-wild (mixed lighting/background, market & vendor shots, powder + whole) | 11 | 11,000 | 512×512 |
| Mendeley Indian Spice (Indian) | studio (uniform white background, controlled lighting) | 19 | 10,991 | mixed |
| **Unified** | both | **22** | **21,991** | — |

- **Overlap classes (8):** `black_pepper, cinnamon, cloves, coriander, cumin,
  ginger, green_cardamom, nutmeg`. These drive the controlled cross-source 2×2.
- Source of each image is recoverable from its path (`spice_spectrum/` vs
  `indian_spices/`); `audit_dedup.py:source_of`.

## Splits (deterministic, seed 42)

| Manifest | Classes | Train | Val | Test |
|---|---:|---:|---:|---:|
| `unified_benchmark.json` | 22 | 15,382 | 3,291 | 3,318 |
| `manifest_overlap_ss.json` | 8 | 5,551 | 1,231 | 1,218 |
| `manifest_overlap_indian.json` | 8 | 2,688 | 533 | 557 |

## Preprocessing / cleaning — duplication & leakage audit

Method (`audit_dedup.py`): 64-bit dHash candidate gate (Hamming ≤ 5) followed by
**pixel-level verification** (32×32 grayscale, normalized RMSE ≤ 0.05). dHash
alone over-reports on uniform studio backgrounds, so every candidate is
pixel-confirmed.

**(1) Cross-source duplication — CLEAN.** Across the 8 overlap classes there are
**zero** pixel-verified cross-source duplicates (0 even at the loose RMSE ≤ 0.10;
0 exact-hash cross-source groups). The studio↔wild asymmetric finding is **not**
a cross-source duplication artifact.

**(2) Within-source train/test leakage — present in source data, corrected.**
The studio source contains burst-capture sequences (consecutive near-identical
frames, e.g. `BayLeaf711.jpg`/`BayLeaf712.jpg`, RMSE = 0.0) that the random split
scatters across train and test. Pixel-verified leaked **test** images:

| Manifest | Leaked test imgs | Test before → after |
|---|---:|---:|
| `manifest_overlap_indian.json` | 123 (22.1%) | 557 → 434 |
| `manifest_overlap_ss.json` | 74 (6.1%) | 1,218 → 1,144 |
| `unified_benchmark.json` | 2,554 verified pairs | (see `outputs/dedup_audit.json`) |

We release **leakage-free splits** (`*_dedup.json`, via
`make_dedup_manifests.py`) that remove leaked images from the **test** set only
(train/val untouched, so existing checkpoints remain valid for re-evaluation).

**(3) Effect on the headline — negligible (finding is leakage-robust).**
Re-evaluating the same checkpoints on the leakage-free test sets:

| Direction | Original | Leakage-corrected |
|---|---:|---:|
| SS-trained → Indian (easy) tax | +0.48 pp | +0.70 pp |
| Indian-trained → SS (broken) tax | +37.77 pp | +38.08 pp |

The asymmetric shortcut tax is unchanged (slightly larger) after removing all
leakage — it is a genuine representation-level effect, not memorized duplicates.
Raw: `outputs/shortcut_test_matrix.json`, `outputs/shortcut_test_matrix_dedup.json`,
`outputs/dedup_audit.json`, `outputs/dedup_overlap_{indian,ss}.json`.

## Uses

Intended: benchmarking cross-source / domain-generalization robustness for
fine-grained granular-material recognition. **Not** intended as a
production food-safety / adulteration classifier without further validation.

## Distribution / License

- **Mendeley Indian Spice Dataset** — Thite, Godse, Patil, Chumchu, Nyandoro,
  "Facilitating spice recognition and classification: An image dataset of Indian
  spices," *Data in Brief* 57:110936, 2024 — **CC BY 4.0** (doi:10.1016/j.dib.2024.110936;
  Mendeley Data `vg77y9rtjb`).
- **SpiceSpectrum** — Ramim, Islam, Towkir, Fuad, Arnob, "SpiceSpectrum:
  Class-balanced dataset of commercially valuable spice cultivars," *Data in Brief*
  63:112097, 2025 (doi:10.1016/j.dib.2025.112097). The image deposit (Mendeley Data
  doi:10.17632/5v7w2hx8n5.2) is **CC BY-ND 4.0**. This paper's corresponding author
  is the first author of SpiceSpectrum.
- The two sources carry **different terms**: the studio source is CC BY 4.0, but the
  in-the-wild source is CC BY-ND 4.0, whose NoDerivatives clause forbids
  redistributing modified or merged images. We therefore distribute SpiceNet strictly
  as **manifests (path + label) + deterministic splits + audit/eval scripts**, never
  as a re-hosted or resized image archive. A user reconstructs the benchmark by
  downloading each source from its original DOI and applying our splits, so no
  derivative of the ND source is redistributed and each source's attribution is
  preserved at its original DOI.

## Maintenance / reproduction

```bash
python audit_dedup.py --base outputs/unified_benchmark.json      # full audit
python make_dedup_manifests.py --manifest outputs/manifest_overlap_indian.json
python make_dedup_manifests.py --manifest outputs/manifest_overlap_ss.json
python eval_shortcut_test.py --ss_ckpt outputs/checkpoints/overlap_ss/p1_best.pth \
  --in_ckpt outputs/checkpoints/overlap_indian/p1_best.pth \
  --ss_manifest outputs/manifest_overlap_ss_dedup.json \
  --in_manifest outputs/manifest_overlap_indian_dedup.json --out_suffix dedup
```