Datasheet — SpiceNet-Bench
A datasheet (Gebru et al., 2021) for the SpiceNet-Bench cross-source spice
recognition benchmark. Accompanies the finding paper
(paper/paper_draft.md). All counts are reproducible from
outputs/unified_benchmark.json and the audit scripts named below.
Motivation
SpiceNet-Bench was built to measure cross-acquisition-source generalization in fine-grained spice recognition — a property invisible to the single-source studio benchmarks that dominate the published literature. It unifies one "in-the-wild" source and one "studio" source so that a model trained on one can be tested on the other.
Composition
| Source | Style | Classes | Images | Resolution |
|---|---|---|---|---|
| SpiceSpectrum (SS) | in-the-wild (mixed lighting/background, market & vendor shots, powder + whole) | 11 | 11,000 | 512×512 |
| Mendeley Indian Spice (Indian) | studio (uniform white background, controlled lighting) | 19 | 10,991 | mixed |
| Unified | both | 22 | 21,991 | — |
- Overlap classes (8):
black_pepper, cinnamon, cloves, coriander, cumin, ginger, green_cardamom, nutmeg. These drive the controlled cross-source 2×2. - Source of each image is recoverable from its path (
spice_spectrum/vsindian_spices/);audit_dedup.py:source_of.
Splits (deterministic, seed 42)
| Manifest | Classes | Train | Val | Test |
|---|---|---|---|---|
unified_benchmark.json |
22 | 15,382 | 3,291 | 3,318 |
manifest_overlap_ss.json |
8 | 5,551 | 1,231 | 1,218 |
manifest_overlap_indian.json |
8 | 2,688 | 533 | 557 |
Preprocessing / cleaning — duplication & leakage audit
Method (audit_dedup.py): 64-bit dHash candidate gate (Hamming ≤ 5) followed by
pixel-level verification (32×32 grayscale, normalized RMSE ≤ 0.05). dHash
alone over-reports on uniform studio backgrounds, so every candidate is
pixel-confirmed.
(1) Cross-source duplication — CLEAN. Across the 8 overlap classes there are zero pixel-verified cross-source duplicates (0 even at the loose RMSE ≤ 0.10; 0 exact-hash cross-source groups). The studio↔wild asymmetric finding is not a cross-source duplication artifact.
(2) Within-source train/test leakage — present in source data, corrected.
The studio source contains burst-capture sequences (consecutive near-identical
frames, e.g. BayLeaf711.jpg/BayLeaf712.jpg, RMSE = 0.0) that the random split
scatters across train and test. Pixel-verified leaked test images:
| Manifest | Leaked test imgs | Test before → after |
|---|---|---|
manifest_overlap_indian.json |
123 (22.1%) | 557 → 434 |
manifest_overlap_ss.json |
74 (6.1%) | 1,218 → 1,144 |
unified_benchmark.json |
2,554 verified pairs | (see outputs/dedup_audit.json) |
We release leakage-free splits (*_dedup.json, via
make_dedup_manifests.py) that remove leaked images from the test set only
(train/val untouched, so existing checkpoints remain valid for re-evaluation).
(3) Effect on the headline — negligible (finding is leakage-robust). Re-evaluating the same checkpoints on the leakage-free test sets:
| Direction | Original | Leakage-corrected |
|---|---|---|
| SS-trained → Indian (easy) tax | +0.48 pp | +0.70 pp |
| Indian-trained → SS (broken) tax | +37.77 pp | +38.08 pp |
The asymmetric shortcut tax is unchanged (slightly larger) after removing all
leakage — it is a genuine representation-level effect, not memorized duplicates.
Raw: outputs/shortcut_test_matrix.json, outputs/shortcut_test_matrix_dedup.json,
outputs/dedup_audit.json, outputs/dedup_overlap_{indian,ss}.json.
Uses
Intended: benchmarking cross-source / domain-generalization robustness for fine-grained granular-material recognition. Not intended as a production food-safety / adulteration classifier without further validation.
Distribution / License
- Mendeley Indian Spice Dataset — Thite, Godse, Patil, Chumchu, Nyandoro,
"Facilitating spice recognition and classification: An image dataset of Indian
spices," Data in Brief 57:110936, 2024 — CC BY 4.0 (doi:10.1016/j.dib.2024.110936;
Mendeley Data
vg77y9rtjb). - SpiceSpectrum — Ramim, Islam, Towkir, Fuad, Arnob, "SpiceSpectrum: Class-balanced dataset of commercially valuable spice cultivars," Data in Brief 63:112097, 2025 (doi:10.1016/j.dib.2025.112097). The image deposit (Mendeley Data doi:10.17632/5v7w2hx8n5.2) is CC BY-ND 4.0. This paper's corresponding author is the first author of SpiceSpectrum.
- The two sources carry different terms: the studio source is CC BY 4.0, but the in-the-wild source is CC BY-ND 4.0, whose NoDerivatives clause forbids redistributing modified or merged images. We therefore distribute SpiceNet strictly as manifests (path + label) + deterministic splits + audit/eval scripts, never as a re-hosted or resized image archive. A user reconstructs the benchmark by downloading each source from its original DOI and applying our splits, so no derivative of the ND source is redistributed and each source's attribution is preserved at its original DOI.
Maintenance / reproduction
python audit_dedup.py --base outputs/unified_benchmark.json # full audit
python make_dedup_manifests.py --manifest outputs/manifest_overlap_indian.json
python make_dedup_manifests.py --manifest outputs/manifest_overlap_ss.json
python eval_shortcut_test.py --ss_ckpt outputs/checkpoints/overlap_ss/p1_best.pth \
--in_ckpt outputs/checkpoints/overlap_indian/p1_best.pth \
--ss_manifest outputs/manifest_overlap_ss_dedup.json \
--in_manifest outputs/manifest_overlap_indian_dedup.json --out_suffix dedup