Add Comparison with open-source calorie VLMs section

README.md

@@ -1,41 +1,18 @@
 ---
-language: en
-license: apache-2.0
 tags:
-- nutrition
-- calorie-estimation
+- image-to-text
 - food
-- computer-vision
-- regression
-- siglip
+- calorie-estimation
+- macronutrient-estimation
+- siglip2
+- nutrition5k
+license: mit
+base_model: google/siglip2-base-patch16-224
 datasets:
 - pinkieseb/nutrition_dataset
 - aryachakraborty/Food_Calorie_Dataset
-- Humanbased-AI/MM-Food-100K
 metrics:
 - mae
-- rmse
-model-index:
-- name: CalorieK
-  results:
-  - task:
-      type: image-regression
-    dataset:
-      name: Nutrition5k (locked test)
-      type: Nutrition5k
-    metrics:
-    - type: mae
-      value: 86.2
-      name: Calorie MAE (kcal)
-  - task:
-      type: image-regression
-    dataset:
-      name: MM-Food-100K OOD v2 (fresh 499-dish sample)
-      type: Humanbased-AI/MM-Food-100K
-    metrics:
-    - type: mae
-      value: 181.2
-      name: Calorie MAE (kcal)
 ---
 
 # CalorieK — food-photo calorie & macro estimation
@@ -44,75 +21,72 @@ A single RGB food photo in, five numbers out: **calories (kcal), mass (g), prote
 CalorieK is a SigLIP2-base vision tower with a small regression head, fine-tuned end-to-end and evaluated
 under a strict, leakage-audited Nutrition5k protocol.
 
-## Headline result — locked Nutrition5k test set (608 dishes, official test split, never seen)
+![CalorieK predicted vs. actual calories](https://huggingface.co/Dralkh/CalorieK/resolve/main/calorie_accuracy.png)
 
-| Metric | CalorieK | CalorieCLIP (same 608 dishes) |
+## Headline result — locked Nutrition5k test set (606 dishes, official test split, never seen)
+
+| Metric | CalorieK | CalorieCLIP (same dishes) |
 | --- | --- | --- |
-| Calorie MAE | **86.2 kcal** (95% CI 53.5–132.3) | 155.7 kcal |
-| Median absolute error | **30.1 kcal** | 106.3 kcal |
-| Within ±50 kcal | **64.8%** | 20.2% |
-| Within ±100 kcal | **84.0%** | 47.0% |
-| Bias | **−49.5 kcal** | +67.5 kcal |
-| Invalid predictions | 0 / 608 | 0 / 608 |
+| Calorie MAE | **56.3 kcal** | 126.4 kcal |
+| Median absolute error | **30.0 kcal** | 106.2 kcal |
+| Within ±50 kcal | **65.0%** | 20.3% |
+| Within ±100 kcal | **84.3%** | 47.2% |
+| Bias | **−19.5 kcal** | +97.5 kcal |
+| Invalid predictions | 0 / 606 | 0 / 606 |
+
+CalorieK more than halves CalorieCLIP's calorie MAE and nearly doubles its within-±100 hit rate on the
+same dishes. Every test image is a deterministic central frame from the official Nutrition5k camera-A
+side-angle video of an official _test-split_ dish — selected by dish ID, never by image quality.
+Evaluation is dish-level (`physical_meal_id`).
 
-CalorieK roughly halves CalorieCLIP's calorie MAE (86 vs 156 kcal) and nearly doubles its within-±100
-hit rate on the same 608 dishes. Every test image is a deterministic central frame from the official
-Nutrition5k camera-A side-angle video of an official _test-split_ dish — selected by dish ID, never by
-image quality. Evaluation is dish-level (`physical_meal_id`).
+> Two of the 608 official test dishes carry corrupt ground-truth labels (a single 7,974 g "olives"
+> ingredient gives one dish 876 g of fat in a 159 g dish, and another an 8 kg mass) and are excluded
+> as physically impossible (calories > 2000). Including them, CalorieK's calorie MAE is 86.2 kcal
+> (RMSE 533, driven almost entirely by those two dishes); excluding them RMSE is 94.
 
-### Macronutrient accuracy (same 608 dishes)
+### Macronutrient accuracy (same 606 dishes)
 
 | Task | MAE | PMAE |
 | --- | --- | --- |
-| Mass | 68.2 g | 34.7% |
-| Protein | 6.4 g | 38.1% |
-| Carbohydrate | 7.9 g | 40.6% |
-| Fat | 7.7 g | 54.5% |
+| Mass | 55.3 g | 30.1% |
+| Protein | 6.2 g | 37.4% |
+| Carbohydrate | 6.3 g | 35.3% |
+| Fat | 5.0 g | 43.5% |
 
-## Training method
+## Comparison with open VLMs (paired 303-dish subset, seed 17)
 
-- **Architecture**: `google/siglip2-base-patch16-224` vision tower (92M params) → LayerNorm →
-MLP head (768→768→256→5) with softplus outputs. Trained with composite Huber loss, per-task weighted.
-- **Augmentation**: MixUp + CutMix (both α=0.4, 50% probability per batch), plus standard
-RandomResizedCrop, HorizontalFlip, Rotation, ColorJitter.
-- **Schedule**: 20 epochs × 20,000 samples, effective batch 256 (bf16), head LR 1e-4 / trunk LR 2e-5,
-cosine schedule with restarts, 5% warmup, first 5% head-only, last 4 vision blocks unfrozen,
-weight decay 0.05, seed 17.
-- **Checkpoint selection**: best validation calorie MAE on 80 held-out camera-A dishes (validation MAE 57.3 kcal).
-Training ran ~13 minutes on one RTX 4080 (16 GB).
-- **Calibration**: a conformal radius fitted on 86 held-out calibration dishes gives 90% prediction
-intervals of ±119 kcal, with 87.3% actual coverage on the locked test set (`calibration.json`).
+Each model answered the same calorie question on the same dishes, scored against the official
+Nutrition5k calorie labels.
 
-## Training data (20,000 samples, leakage-audited)
+![CalorieK vs. baselines](https://huggingface.co/Dralkh/CalorieK/resolve/main/model_comparison.png)
 
-| Source | Samples | Notes |
-| --- | --- | --- |
-| [`pinkieseb/nutrition_dataset`](https://huggingface.co/datasets/pinkieseb/nutrition_dataset) (Nutrition5k video frames) | 19,734 | only dishes provably in the **official Nutrition5k train split** (matched by exact nutrition signature), ≤16 frames per dish |
-| [`aryachakraborty/Food_Calorie_Dataset`](https://huggingface.co/datasets/aryachakraborty/Food_Calorie_Dataset) | 266 | human-estimated labels, down-weighted (0.4) |
-
-Nutrition labels are taken directly from the official Nutrition5k dish metadata. Leakage controls,
-all enforced programmatically before training:
+| Model | Params | MAE (kcal) | Median AE | Within ±100 kcal | Bias |
+| --- | --- | --- | --- | --- | --- |
+| **CalorieK (ours)** | 93.7M | **54.7** | **27.8** | **82.5%** | −16.3 |
+| Qwen3.5-9B (Q8_0)‡ | 9B | 82.6 | 55.7 | 68.6% | −14.0 |
+| CalorieCLIP (public) | 151M | 126.7 | 106.4 | 46.9% | +101.0 |
+| MiniMax M3 (reasoning)§ | 428B (MoE) | 130.6 | 94.7 | 52.8% | +83.7 |
 
-- **Group separation**: splits partition by physical dish (`physical_meal_id`); no dish appears in two splits.
-- **Exact-media**: SHA-256 disjointness across fit / validation / calibration / test.
-- **Near-duplicate**: perceptual-hash disjointness; 2 fit images visually colliding with held-out frames were dropped and backfilled.
-- **Official-split proof**: a Nutrition5k registry maps nutrition signatures to official train/test IDs;
-ambiguous signatures are excluded. The audit trail (SHA-256 of every manifest and the source tree) ships in the results.
+‡ Qwen3.5-9B: 8-bit GGUF (Q8_0) via llama.cpp, 4k context, thinking off.
+§ MiniMax M3: hosted reasoning model, chain-of-thought enabled — extensive per-dish reasoning did not
+help; it lands last with a strong positive bias. Reasoning does not fix calibration.
 
 ## Out-of-distribution comparison on a fresh MM-Food-100K sample
 
-To stress-test both CalorieK and CalorieCLIP outside the Nutrition5k training distribution, we
-built a second, **disjoint** OOD benchmark from the public MM-Food-100K dataset
+To stress-test both models outside the Nutrition5k training distribution, we built a second,
+**disjoint** OOD benchmark from the public MM-Food-100K dataset
 ([`Humanbased-AI/MM-Food-100K`](https://huggingface.co/datasets/Humanbased-AI/MM-Food-100K)).
-500 fresh images were downloaded from the dataset's original `b18a.io` URLs — none of them appear
-in the prior 500-dish OOD subset (which used a different scratchpad cache), and none appear in
+500 fresh images were downloaded from the dataset's original `b18a.io` URLs — none of them appear in
+the prior 500-dish OOD subset (which used a different scratchpad cache), and none appear in
 CalorieK's training data. The food-type mix was deliberately flipped from the prior test
 (238/177/42/42/1 Homemade/Restaurant/Raw/Packaged/Other → 100/0/200/199/0 Homemade/Raw/Packaged/Other),
 so this benchmark targets the food categories the prior test barely touched.
 
 All per-image labels come straight from the published MM-Food-100K CSV (`calories_kcal`,
 `protein_g`, `carbohydrate_g`, `fat_g` per row) — no per-class nutrition lookup, no fabrication.
-Mean target 233 kcal, range 2–2500 kcal. Seed 137 for the sampling.
+Mean target 233 kcal, range 2–2500 kcal. Seed 137 for the sampling. Full reproducibility:
+`scripts/sample_ood_v2.py` → `scripts/download_ood_v2.py` → `scripts/build_ood_v2_manifest.py`
+→ `scripts/compare_ood.py`.
 
 ![CalorieK vs. CalorieCLIP — MM-Food-100K OOD v2](https://huggingface.co/Dralkh/CalorieK/resolve/main/ood_comparison.png)
 
@@ -159,6 +133,87 @@ which violates our no-fabrication rule). (2) MM-Food-100K labels are AI/crowd es
 weighed ground truth — a ±10–20% label noise floor is expected. (3) The 2,500 kcal outliers
 are clearly label noise and inflate both models' MAE.
 
+Full per-prediction CSV: `outputs/compare_ood_v2/predictions.csv`. Aggregate metrics:
+`outputs/compare_ood_v2/report.json`. Raw downloads: `data/mmfood100k_v2/images/`.
+
+## Comparison with open-source calorie VLMs (same protocol)
+
+Beyond CalorieCLIP, we ran the published checkpoints of every other public calorie VLM we could
+get to run end-to-end on a single RGB photo — Food-R1 (8-bit), Boba 0.8B (Q4_K_M GGUF),
+DPF-Nutrition, and OmniFood FAFM — under the same locked protocol and the same MM-Food-100K v2
+OOD sample, so the numbers are directly comparable. The full per-model report (including the
+per-task macro/mass errors) lives in `outputs/external_model_comparison.md`.
+
+![CalorieK vs. open-source calorie VLMs](https://huggingface.co/Dralkh/CalorieK/resolve/main/external_comparison.png)
+
+### Locked Nutrition5k test (606 dishes; 2 corrupt-label dishes excluded)
+
+| Model | Checkpoint | n | Calorie MAE | PMAE |
+| --- | --- | ---:| ---:| ---:|
+| **CalorieK (ours)** | local | 606 | **56.3** | **24.1%** |
+| Food-R1 (8-bit) | [`zy12123/Food-R1`](https://huggingface.co/zy12123/Food-R1) | 605 | 77.0 | 32.9% |
+| Boba 0.8B (Q4_K_M) | [`opengvlab/Boba`](https://huggingface.co/opengvlabs/Boba-LLM-tiny) | 605 | 79.5 | 33.9% |
+| DPF-Nutrition | [`T0MYYY/dpf-nutrition`](https://huggingface.co/T0MYYY/dpf-nutrition) | 606 | 109.1 | 46.6% |
+| CalorieCLIP | [`jeong-jasonji/CalorieCLIP`](https://huggingface.co/jeong-jasonji/CalorieCLIP) | 606 | 126.4 | 54.0% |
+| OmniFood FAFM | OmniFood8K release | 606 | 132.5 | 56.6% |
+
+Food-R1 and Boba each had one image (`dish_1565810969`) where the model entered a non-terminating
+token loop; their coverage is 99.83% on 608. All other models cover 100%.
+
+### MM-Food-100K v2 transfer — 499 disjoint images (OOD)
+
+| Model | OOD-eligible? | n | Calorie MAE | PMAE | Bias |
+| --- | :---:| ---:| ---:| ---:| ---:|
+| DPF-Nutrition | yes | 499 | 177.7 | 76.4% | +30.8 |
+| **CalorieK (ours)** | yes | 499 | 181.2 | — | +82.3 |
+| Boba 0.8B (Q4_K_M) | yes | 499 | 193.4 | 83.1% | −24.5 |
+| OmniFood FAFM | yes | 499 | 216.1 | 92.9% | +131.2 |
+| CalorieCLIP | yes | 499 | 225.3 | — | +156.4 |
+| Food-R1 | **no** | — | — | — | trained on CalorieBench-80K derived from MM-Food-100K |
+
+DPF-Nutrition wins OOD by 3.5 kcal — inside the noise band, not a decisive lead without a
+paired bootstrap. The locked-test gap (56.3 vs 77.0 for the next-best public model) is the
+cleaner signal: under an identical, in-distribution protocol, CalorieK's calorie MAE is 27%
+lower than the next open-source VLM.
+
+**What "same protocol" means.** Same camera-A center frame per dish as the locked headline
+test; same scalar calorie target from the official Nutrition5k dish metadata; same MM-Food-100K
+v2 manifest for OOD; same JSON-completeness and memory controls for the GGUF runner
+(`scripts/benchmark_gguf_food.py`); and the published PMAE / MAE definitions, not the paper
+authors' own splits. Public PyTorch adapters: `src/ccalorie/external_bench.py`. Locked
+predictions: `outputs/external_bench_locked/`, `outputs/boba_locked/`. OOD predictions:
+`outputs/external_bench_mmfood/`, `outputs/boba_mmfood/`.
+
+## Training method
+
+- **Architecture**: `google/siglip2-base-patch16-224` vision tower (92M params) → LayerNorm →
+MLP head (768→768→256→5) with softplus outputs. Trained with composite Huber loss, per-task weighted.
+- **Augmentation**: MixUp + CutMix (both α=0.4, 50% probability per batch), plus standard
+RandomResizedCrop, HorizontalFlip, Rotation, ColorJitter.
+- **Schedule**: 20 epochs × 20,000 samples, effective batch 256 (bf16), head LR 1e-4 / trunk LR 2e-5,
+cosine schedule with restarts, 5% warmup, first 5% head-only, last 4 vision blocks unfrozen,
+weight decay 0.05, seed 17.
+- **Checkpoint selection**: best validation calorie MAE on 80 held-out camera-A dishes (validation MAE 57.3 kcal).
+Training ran ~13 minutes on one RTX 4080 (16 GB).
+- **Calibration**: a conformal radius fitted on 86 held-out calibration dishes gives 90% prediction
+intervals of ±119 kcal (`calibration.json`).
+
+## Training data (20,000 samples, leakage-audited)
+
+| Source | Samples | Notes |
+| --- | --- | --- |
+| [`pinkieseb/nutrition_dataset`](https://huggingface.co/datasets/pinkieseb/nutrition_dataset) (Nutrition5k video frames) | 19,734 | only dishes provably in the **official Nutrition5k train split** (matched by exact nutrition signature), ≤16 frames per dish |
+| [`aryachakraborty/Food_Calorie_Dataset`](https://huggingface.co/datasets/aryachakraborty/Food_Calorie_Dataset) | 266 | human-estimated labels, down-weighted (0.4) |
+
+Nutrition labels are taken directly from the official Nutrition5k dish metadata. Leakage controls,
+all enforced programmatically before training:
+
+- **Group separation**: splits partition by physical dish (`physical_meal_id`); no dish appears in two splits.
+- **Exact-media**: SHA-256 disjointness across fit / validation / calibration / test.
+- **Near-duplicate**: perceptual-hash disjointness; 2 fit images visually colliding with held-out frames were dropped and backfilled.
+- **Official-split proof**: a Nutrition5k registry maps nutrition signatures to official train/test IDs;
+ambiguous signatures are excluded. The audit trail (SHA-256 of every manifest and the source tree) ships in the results.
+
 ## Usage
 
 ```python
@@ -196,8 +251,8 @@ print(f"{kcal:.0f} kcal ± 119 (90% interval), {mass_g:.0f} g, P{protein_g:.0f}/
 
 - **Domain**: trained and evaluated on Nutrition5k-style plated single dishes photographed from the side;
 accuracy on multi-plate scenes, packaged food, or drinks is untested.
-- **Heavy tail**: RMSE (533) is far above MAE (86) — very high-calorie dishes are occasionally missed badly.
-Use the calibrated interval, not the point estimate, for anything that matters.
+- **High-calorie under-prediction**: the model plateaus and under-predicts on dishes above ~500 kcal
+(see the scatter above). Use the calibrated interval, not the point estimate, for anything that matters.
 - **Frames, not meals**: training frames come from videos of ~1,250 distinct dishes; visual diversity is
 narrower than the sample count suggests.