Fix calorie<->mass / protein<->carb label swap: retrain on corrected labels; honest metrics (calorie MAE 86.2 kcal); remove invalid comparison charts

README.md

@@ -24,7 +24,7 @@ model-index:
       type: Nutrition5k
     metrics:
     - type: mae
-      value: 49.8
+      value: 86.2
       name: Calorie MAE (kcal)
 ---
 
@@ -34,79 +34,52 @@ A single RGB food photo in, five numbers out: **calories (kcal), mass (g), prote
 CalorieK is a SigLIP2-base vision tower with a small regression head, fine-tuned end-to-end and evaluated
 under a strict, leakage-audited Nutrition5k protocol.
 
-[![Comparison on the locked Nutrition5k test set](https://huggingface.co/Dralkh/CalorieK/resolve/main/comparison.png)](https://huggingface.co/Dralkh/CalorieK/blob/main/comparison.png)
+> **⚠️ Correction (2026-07-13).** Earlier versions of this card reported a
+> "49.8 kcal calorie MAE." A label-mapping bug (`n5k_registry.py` parsed the
+> Nutrition5k CSV columns in the wrong order) had swapped **calories↔mass** and
+> **protein↔carbohydrate** throughout training and evaluation — so the published
+> "calorie" figure was really a **mass** error in grams. The bug is fixed, the
+> labels canonicalized against the official CSV, and a model retrained on the
+> corrected labels. Honest numbers are below. The VLM/CalorieCLIP/OOD comparison
+> tables further down were scored against the swapped labels and are being
+> re-evaluated — treat them as invalid until updated.
 
 ## Headline result — locked Nutrition5k test set (608 dishes, official test split, never seen)
 
-| Metric | CalorieK | CalorieCLIP (same 608 dishes) |
-| --- | --- | --- |
-| Calorie MAE | **49.8 kcal** (95% CI 34.4–78.0) | 181.2 kcal |
-| Median absolute error | **24.5 kcal** | 137.0 kcal |
-| Within ±50 kcal | **78.5%** | 21.4% |
-| Within ±100 kcal | **94.2%** | 40.6% |
-| Bias | **−6.6 kcal** | +134.8 kcal |
-| Calibrated 90% interval | **±105.9 kcal** (actual coverage 93.6%) | — |
-| Invalid predictions | 0 / 608 | 0 / 608 |
+Corrected model (`siglip2-base-patch16-224`, retrained on canonicalized labels), dish-level (`physical_meal_id`):
+
+| Metric | CalorieK (corrected) |
+| --- | --- |
+| Calorie MAE | **86.2 kcal** (95% CI 53.5–132.3) |
+| Median absolute error | 30.1 kcal |
+| Within ±50 kcal | 64.8% |
+| Within ±100 kcal | 84.0% |
+| Bias | −49.5 kcal (systematic under-prediction) |
+| Invalid predictions | 0 / 608 |
 
 Every test image is a deterministic central frame from the official Nutrition5k camera-A side-angle
 video of an official _test-split_ dish — selected by dish ID, never by image quality. Evaluation is
 dish-level (`physical_meal_id`).
 
-### Macronutrient accuracy (same 608 dishes)
-
-| Task | MAE | Median AE | Bias |
-| --- | --- | --- | --- |
-| Mass | **90.4 g** | 34.5 g | −47.3 g |
-| Protein | **7.8 g** | 3.9 g | −1.9 g |
-| Carbohydrate | **5.8 g** | 2.8 g | −1.2 g |
-| Fat | **7.2 g** | 2.2 g | −3.1 g |
-
-## Comparison with open VLMs (paired 304-dish subset, seed 17)
-
-Each model answered the same calorie question on the same dishes
-(prompt: _"Estimate the total calories in this food image. Return exactly one integer number in kcal…"_).
-
-| Model | n | MAE (kcal) | Median AE | Within ±100 kcal | Bias |
-| --- | --- | --- | --- | --- | --- |
-| **CalorieK (ours)** | 304 | **62.7** | **24.0** | **94.1%** | −11.0 |
-| Qwen3.5-2B | 304 | 116.4 | 65.0 | 69.4% | −48.1 |
-| Qwen3.5-9B (Q8_0, thinking off) ‡ | 304 | 117.2 | 64.5 | 65.8% | +6.1 |
-| MiniCPM-V 4.6 (1.3B) | 304 | 163.3 | 97.0 | 52.6% | +75.5 |
-| Gemma-4-12B † | 100 | 176.8 | 90.5 | 58.0% | −43.3 |
-| MiniMax M3 (hosted, thinking) | 304 | 193.5 | 120.5 | 44.7% | +103.7 |
-| CalorieCLIP | 304 | 196.2 | 133.2 | 42.1% | +121.1 |
-
-† Gemma-4-12B was scored on a seeded 100-dish paired subset of the same 304 dishes (compute budget);
-all parse rates were 100%.
-‡ Qwen3.5-9B ran as an 8-bit GGUF (Q8_0) via llama.cpp with 4k context and thinking disabled.
-MiniMax M3 (a hosted reasoning model, chain-of-thought enabled) reinforces the point from the other
-direction: extensive per-dish reasoning did not help — it lands second-to-last with a strong +104 kcal
-systematic overestimate. Reasoning does not fix calibration.
-
-## Out-of-distribution evaluation (MM-Food-100K, 500 wild food photos)
-
-To measure generalization beyond Nutrition5k plating, both regression models were scored on a seeded
-500-image sample of [MM-Food-100K](https://huggingface.co/datasets/Codatta/MM-Food-100K)
-(restaurant/home photos; `food_prob ≥ 0.9`). **Caveat: these labels are crowd/AI estimates, not weighed**
-**measurements** — absolute errors include label noise no model can beat, so the relative comparison is
-the meaningful signal.
-
-|  | CalorieK | CalorieCLIP |
+### Macronutrient accuracy (same 608 dishes, corrected labels)
+
+| Task | MAE | PMAE |
 | --- | --- | --- |
-| MAE | **162.5 kcal** | 206.4 kcal |
-| Median error | **135.0 kcal** | 179.9 kcal |
-| Within ±100 kcal | **38.8%** | 25.6% |
-| Bias | **−8.1 kcal** | +98.3 kcal |
-| R² | **+0.24** | −0.08 |
+| Mass | **68.2 g** | 34.7% |
+| Protein | **6.4 g** | 38.1% |
+| Carbohydrate | **7.9 g** | 40.6% |
+| Fat | **7.7 g** | 54.5% |
 
-Findings, stated honestly:
+These are the true, correctly-labeled numbers. Calories (86 kcal, 33% PMAE) and
+fat (54% PMAE) are the weakest and are the focus of the ongoing architecture work
+(portion decomposition, ingredient-semantic fusion, depth fusion).
 
-- CalorieK beats CalorieCLIP out-of-distribution on every metric, keeps **near-zero bias** on food it
-has never seen, and retains real signal (R² +0.24) where CalorieCLIP is worse than predicting the
-mean (R² −0.08).
-- But CalorieK degrades ~3× versus in-distribution (49.8 → 162.5 kcal MAE): it is substantially a
-specialist in Nutrition5k-style plated dishes. For wild-photo deployment, mixing diverse weakly
-labeled data into training is the indicated next step.
+## Comparisons (under re-evaluation)
+
+The previous open-VLM, CalorieCLIP, and out-of-distribution (MM-Food-100K)
+comparison tables were computed against the swapped labels and are therefore
+invalid. They have been removed and will be republished once re-run against the
+corrected labels and the improved model.
 
 ## Training method (improved)
 
@@ -117,20 +90,17 @@ RandomResizedCrop, HorizontalFlip, Rotation, ColorJitter.
 - **Schedule**: 20 epochs × 20,000 samples, effective batch 256 (bf16), head LR 1e-4 / trunk LR 2e-5,
 cosine schedule with restarts, 5% warmup, first 5% head-only, last 4 vision blocks unfrozen,
 weight decay 0.05, seed 17.
-- **Checkpoint selection**: best validation MAE on 80 held-out camera-A dishes (validation MAE 39.3 kcal).
-Training ran ~12 minutes on one RTX 4080 (16 GB).
+- **Checkpoint selection**: best validation calorie MAE on 80 held-out camera-A dishes (validation MAE 57.3 kcal).
+Training ran ~13 minutes on one RTX 4080 (16 GB).
 - **Calibration**: a conformal radius fitted on 86 held-out calibration dishes gives 90% prediction
-intervals of ±105.9 kcal, with 93.6% actual coverage on the locked test set (`calibration.json`).
-
-[![Training loss](https://huggingface.co/Dralkh/CalorieK/resolve/main/loss_curve.png)](https://huggingface.co/Dralkh/CalorieK/blob/main/loss_curve.png)
+intervals of ±119 kcal, with 87.3% actual coverage on the locked test set (`calibration.json`).
 
 ## Training data (20,000 samples, leakage-audited)
 
 | Source | Samples | Notes |
 | --- | --- | --- |
-| [`mmathys/food-nutrients`](https://huggingface.co/datasets/mmathys/food-nutrients) (Nutrition5k overhead images) | 9,798 | Weighed plate totals |
-| [`pinkieseb/nutrition_dataset`](https://huggingface.co/datasets/pinkieseb/nutrition_dataset) (Nutrition5k video frames) | 9,936 | Weighed plate totals, ≤16 frames per dish |
-| [`aryachakraborty/Food_Calorie_Dataset`](https://huggingface.co/datasets/aryachakraborty/Food_Calorie_Dataset) | 266 | Human-estimated labels, down-weighted (0.4) |
+| [`pinkieseb/nutrition_dataset`](https://huggingface.co/datasets/pinkieseb/nutrition_dataset) (Nutrition5k video frames) | 19,734 | only dishes provably in the **official Nutrition5k train split** (matched by exact nutrition signature), ≤16 frames per dish |
+| [`aryachakraborty/Food_Calorie_Dataset`](https://huggingface.co/datasets/aryachakraborty/Food_Calorie_Dataset) | 266 | human-estimated labels, down-weighted (0.4) |
 
 Leakage controls, all enforced programmatically before training:
 
@@ -170,14 +140,14 @@ with torch.inference_mode():
     out = vision(pixel_values=pixels)
     feats = out.pooler_output if out.pooler_output is not None else out.last_hidden_state[:, 0]
     kcal, mass_g, protein_g, carb_g, fat_g = head(feats.float())[0].tolist()
-print(f"{kcal:.0f} kcal ± 106 (90% interval), {mass_g:.0f} g, P{protein_g:.0f}/C{carb_g:.0f}/F{fat_g:.0f}")
+print(f"{kcal:.0f} kcal ± 119 (90% interval), {mass_g:.0f} g, P{protein_g:.0f}/C{carb_g:.0f}/F{fat_g:.0f}")
 ```
 
 ## Limitations
 
 - **Domain**: trained and evaluated on Nutrition5k-style plated single dishes photographed from the side;
 accuracy on multi-plate scenes, packaged food, or drinks is untested.
-- **Heavy tail**: RMSE (326) is far above MAE (50) — very high-calorie dishes are occasionally missed badly.
+- **Heavy tail**: RMSE (533) is far above MAE (86) — very high-calorie dishes are occasionally missed badly.
 Use the calibrated interval, not the point estimate, for anything that matters.
 - **Frames, not meals**: training frames come from videos of ~1,250 distinct dishes; visual diversity is
 narrower than the sample count suggests.
@@ -186,10 +156,10 @@ narrower than the sample count suggests.
 ## Citation
 
 ```bibtex
-@misc{alkharji2026caloriek,
+@misc{caloriek2025,
   title={CalorieK: Accurate Food Calorie Estimation from a Single RGB Image},
-  author={Alkharji, Faisal},
-  year={2026},
+  author={Dralkh},
+  year={2025},
   url={https://huggingface.co/Dralkh/CalorieK}
 }
 ```

calibration.json

@@ -1,6 +1,6 @@
 {
   "coverage": 0.9,
-  "radius_kcal": 105.9188232421875,
-  "manifest_sha256": "6002cd3b3d33917f888573a49bee41fc23b26055a7c626408d6d71641e2b09d0",
-  "checkpoint": "outputs/siglip2_b0/checkpoint_best.pt"
+  "radius_kcal": 118.95077547851565,
+  "manifest_sha256": "1678ff2a4c82da6bb6e9acd8842a9b20c09624d11d692f8db10e5fa67423207a",
+  "checkpoint": "outputs/siglip2_corrected/checkpoint_best.pt"
 }

caloriek.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:17ac63ec7034919aae7436ecf43ea08809a4c7b2fd59abaed44c40a0bd0ac344
-size 607962335
+oid sha256:0d5a1f26f5b1a27d1914a18f28f5218ff6df23ee05b0932212bc0c1dd61b053a
+size 374774573

config.json

@@ -4,8 +4,15 @@
   "base_model": "google/siglip2-base-patch16-224",
   "image_size": 224,
   "hidden_size": 768,
-  "tasks": ["calories_kcal", "mass_g", "protein_g", "carbohydrate_g", "fat_g"],
+  "tasks": [
+    "calories_kcal",
+    "mass_g",
+    "protein_g",
+    "carbohydrate_g",
+    "fat_g"
+  ],
   "checkpoint_file": "caloriek.pt",
   "calibration_file": "calibration.json",
-  "interval_90_radius_kcal": 105.92
+  "interval_90_radius_kcal": 118.95,
+  "note": "labels canonicalized from official Nutrition5k CSV 2026-07-13; retrained (see README correction)"
 }