| # Datasheet — SpiceNet-Bench | |
| A datasheet (Gebru et al., 2021) for the SpiceNet-Bench cross-source spice | |
| recognition benchmark. Accompanies the finding paper | |
| (`paper/paper_draft.md`). All counts are reproducible from | |
| `outputs/unified_benchmark.json` and the audit scripts named below. | |
| ## Motivation | |
| SpiceNet-Bench was built to measure **cross-acquisition-source generalization** | |
| in fine-grained spice recognition — a property invisible to the single-source | |
| studio benchmarks that dominate the published literature. It unifies one | |
| "in-the-wild" source and one "studio" source so that a model trained on one can | |
| be tested on the other. | |
| ## Composition | |
| | Source | Style | Classes | Images | Resolution | | |
| |---|---|---:|---:|---| | |
| | SpiceSpectrum (SS) | in-the-wild (mixed lighting/background, market & vendor shots, powder + whole) | 11 | 11,000 | 512×512 | | |
| | Mendeley Indian Spice (Indian) | studio (uniform white background, controlled lighting) | 19 | 10,991 | mixed | | |
| | **Unified** | both | **22** | **21,991** | — | | |
| - **Overlap classes (8):** `black_pepper, cinnamon, cloves, coriander, cumin, | |
| ginger, green_cardamom, nutmeg`. These drive the controlled cross-source 2×2. | |
| - Source of each image is recoverable from its path (`spice_spectrum/` vs | |
| `indian_spices/`); `audit_dedup.py:source_of`. | |
| ## Splits (deterministic, seed 42) | |
| | Manifest | Classes | Train | Val | Test | | |
| |---|---:|---:|---:|---:| | |
| | `unified_benchmark.json` | 22 | 15,382 | 3,291 | 3,318 | | |
| | `manifest_overlap_ss.json` | 8 | 5,551 | 1,231 | 1,218 | | |
| | `manifest_overlap_indian.json` | 8 | 2,688 | 533 | 557 | | |
| ## Preprocessing / cleaning — duplication & leakage audit | |
| Method (`audit_dedup.py`): 64-bit dHash candidate gate (Hamming ≤ 5) followed by | |
| **pixel-level verification** (32×32 grayscale, normalized RMSE ≤ 0.05). dHash | |
| alone over-reports on uniform studio backgrounds, so every candidate is | |
| pixel-confirmed. | |
| **(1) Cross-source duplication — CLEAN.** Across the 8 overlap classes there are | |
| **zero** pixel-verified cross-source duplicates (0 even at the loose RMSE ≤ 0.10; | |
| 0 exact-hash cross-source groups). The studio↔wild asymmetric finding is **not** | |
| a cross-source duplication artifact. | |
| **(2) Within-source train/test leakage — present in source data, corrected.** | |
| The studio source contains burst-capture sequences (consecutive near-identical | |
| frames, e.g. `BayLeaf711.jpg`/`BayLeaf712.jpg`, RMSE = 0.0) that the random split | |
| scatters across train and test. Pixel-verified leaked **test** images: | |
| | Manifest | Leaked test imgs | Test before → after | | |
| |---|---:|---:| | |
| | `manifest_overlap_indian.json` | 123 (22.1%) | 557 → 434 | | |
| | `manifest_overlap_ss.json` | 74 (6.1%) | 1,218 → 1,144 | | |
| | `unified_benchmark.json` | 2,554 verified pairs | (see `outputs/dedup_audit.json`) | | |
| We release **leakage-free splits** (`*_dedup.json`, via | |
| `make_dedup_manifests.py`) that remove leaked images from the **test** set only | |
| (train/val untouched, so existing checkpoints remain valid for re-evaluation). | |
| **(3) Effect on the headline — negligible (finding is leakage-robust).** | |
| Re-evaluating the same checkpoints on the leakage-free test sets: | |
| | Direction | Original | Leakage-corrected | | |
| |---|---:|---:| | |
| | SS-trained → Indian (easy) tax | +0.48 pp | +0.70 pp | | |
| | Indian-trained → SS (broken) tax | +37.77 pp | +38.08 pp | | |
| The asymmetric shortcut tax is unchanged (slightly larger) after removing all | |
| leakage — it is a genuine representation-level effect, not memorized duplicates. | |
| Raw: `outputs/shortcut_test_matrix.json`, `outputs/shortcut_test_matrix_dedup.json`, | |
| `outputs/dedup_audit.json`, `outputs/dedup_overlap_{indian,ss}.json`. | |
| ## Uses | |
| Intended: benchmarking cross-source / domain-generalization robustness for | |
| fine-grained granular-material recognition. **Not** intended as a | |
| production food-safety / adulteration classifier without further validation. | |
| ## Distribution / License | |
| - **Mendeley Indian Spice Dataset** — Thite, Godse, Patil, Chumchu, Nyandoro, | |
| "Facilitating spice recognition and classification: An image dataset of Indian | |
| spices," *Data in Brief* 57:110936, 2024 — **CC BY 4.0** (doi:10.1016/j.dib.2024.110936; | |
| Mendeley Data `vg77y9rtjb`). | |
| - **SpiceSpectrum** — Ramim, Islam, Towkir, Fuad, Arnob, "SpiceSpectrum: | |
| Class-balanced dataset of commercially valuable spice cultivars," *Data in Brief* | |
| 63:112097, 2025 (doi:10.1016/j.dib.2025.112097). The image deposit (Mendeley Data | |
| doi:10.17632/5v7w2hx8n5.2) is **CC BY-ND 4.0**. This paper's corresponding author | |
| is the first author of SpiceSpectrum. | |
| - The two sources carry **different terms**: the studio source is CC BY 4.0, but the | |
| in-the-wild source is CC BY-ND 4.0, whose NoDerivatives clause forbids | |
| redistributing modified or merged images. We therefore distribute SpiceNet strictly | |
| as **manifests (path + label) + deterministic splits + audit/eval scripts**, never | |
| as a re-hosted or resized image archive. A user reconstructs the benchmark by | |
| downloading each source from its original DOI and applying our splits, so no | |
| derivative of the ND source is redistributed and each source's attribution is | |
| preserved at its original DOI. | |
| ## Maintenance / reproduction | |
| ```bash | |
| python audit_dedup.py --base outputs/unified_benchmark.json # full audit | |
| python make_dedup_manifests.py --manifest outputs/manifest_overlap_indian.json | |
| python make_dedup_manifests.py --manifest outputs/manifest_overlap_ss.json | |
| python eval_shortcut_test.py --ss_ckpt outputs/checkpoints/overlap_ss/p1_best.pth \ | |
| --in_ckpt outputs/checkpoints/overlap_indian/p1_best.pth \ | |
| --ss_manifest outputs/manifest_overlap_ss_dedup.json \ | |
| --in_manifest outputs/manifest_overlap_indian_dedup.json --out_suffix dedup | |
| ``` | |