README.md

---
tags:
- image-to-text
- food
- calorie-estimation
- macronutrient-estimation
- siglip2
- nutrition5k
license: mit
base_model: google/siglip2-base-patch16-224
datasets:
- pinkieseb/nutrition_dataset
- aryachakraborty/Food_Calorie_Dataset
metrics:
- mae
---

# CalorieK — food-photo calorie & macro estimation

A single RGB food photo in, five numbers out: **calories (kcal), mass (g), protein (g), carbohydrate (g), fat (g)**.
CalorieK is a SigLIP2-base vision tower with a small regression head, fine-tuned end-to-end and evaluated
under a strict, leakage-audited Nutrition5k protocol.

![CalorieK predicted vs. actual calories](https://huggingface.co/Dralkh/CalorieK/resolve/main/calorie_accuracy.png)

## Headline result — locked Nutrition5k test set (606 dishes, official test split, never seen)

| Metric | CalorieK | CalorieCLIP (same dishes) |
| --- | --- | --- |
| Calorie MAE | **56.3 kcal** | 126.4 kcal |
| Median absolute error | **30.0 kcal** | 106.2 kcal |
| Within ±50 kcal | **65.0%** | 20.3% |
| Within ±100 kcal | **84.3%** | 47.2% |
| Bias | **−19.5 kcal** | +97.5 kcal |
| Invalid predictions | 0 / 606 | 0 / 606 |

CalorieK more than halves CalorieCLIP's calorie MAE and nearly doubles its within-±100 hit rate on the
same dishes. Every test image is a deterministic central frame from the official Nutrition5k camera-A
side-angle video of an official _test-split_ dish — selected by dish ID, never by image quality.
Evaluation is dish-level (`physical_meal_id`).

> Two of the 608 official test dishes carry corrupt ground-truth labels (a single 7,974 g "olives"
> ingredient gives one dish 876 g of fat in a 159 g dish, and another an 8 kg mass) and are excluded
> as physically impossible (calories > 2000). Including them, CalorieK's calorie MAE is 86.2 kcal
> (RMSE 533, driven almost entirely by those two dishes); excluding them RMSE is 94.

### Macronutrient accuracy (same 606 dishes)

| Task | MAE | PMAE |
| --- | --- | --- |
| Mass | 55.3 g | 30.1% |
| Protein | 6.2 g | 37.4% |
| Carbohydrate | 6.3 g | 35.3% |
| Fat | 5.0 g | 43.5% |

## Comparison with open VLMs (paired 303-dish subset, seed 17)

Each model answered the same calorie question on the same dishes, scored against the official
Nutrition5k calorie labels.

![CalorieK vs. baselines](https://huggingface.co/Dralkh/CalorieK/resolve/main/model_comparison.png)

| Model | Params | MAE (kcal) | Median AE | Within ±100 kcal | Bias |
| --- | --- | --- | --- | --- | --- |
| **CalorieK (ours)** | 93.7M | **54.7** | **27.8** | **82.5%** | −16.3 |
| Qwen3.5-9B (Q8_0)‡ | 9B | 82.6 | 55.7 | 68.6% | −14.0 |
| CalorieCLIP (public) | 151M | 126.7 | 106.4 | 46.9% | +101.0 |
| MiniMax M3 (reasoning)§ | 428B (MoE) | 130.6 | 94.7 | 52.8% | +83.7 |

‡ Qwen3.5-9B: 8-bit GGUF (Q8_0) via llama.cpp, 4k context, thinking off.
§ MiniMax M3: hosted reasoning model, chain-of-thought enabled — extensive per-dish reasoning did not
help; it lands last with a strong positive bias. Reasoning does not fix calibration.

## Out-of-distribution comparison on a fresh MM-Food-100K sample

To stress-test both models outside the Nutrition5k training distribution, we built a second,
**disjoint** OOD benchmark from the public MM-Food-100K dataset
([`Humanbased-AI/MM-Food-100K`](https://huggingface.co/datasets/Humanbased-AI/MM-Food-100K)).
500 fresh images were downloaded from the dataset's original `b18a.io` URLs — none of them appear in
the prior 500-dish OOD subset (which used a different scratchpad cache), and none appear in
CalorieK's training data. The food-type mix was deliberately flipped from the prior test
(238/177/42/42/1 Homemade/Restaurant/Raw/Packaged/Other → 100/0/200/199/0 Homemade/Raw/Packaged/Other),
so this benchmark targets the food categories the prior test barely touched.

All per-image labels come straight from the published MM-Food-100K CSV (`calories_kcal`,
`protein_g`, `carbohydrate_g`, `fat_g` per row) — no per-class nutrition lookup, no fabrication.
Mean target 233 kcal, range 2–2500 kcal. Seed 137 for the sampling. Full reproducibility:
`scripts/sample_ood_v2.py` → `scripts/download_ood_v2.py` → `scripts/build_ood_v2_manifest.py`
→ `scripts/compare_ood.py`.

![CalorieK vs. CalorieCLIP — MM-Food-100K OOD v2](https://huggingface.co/Dralkh/CalorieK/resolve/main/ood_comparison.png)

![Per food-type MAE](https://huggingface.co/Dralkh/CalorieK/resolve/main/ood_by_food_type.png)

![Predicted vs. true calories](https://huggingface.co/Dralkh/CalorieK/resolve/main/ood_scatter.png)

| Metric (n = 499) | CalorieK | CalorieCLIP | Δ (CK − CC) |
| --- | --- | --- | --- |
| **Calorie MAE** | **181.2 kcal** | 225.3 kcal | −44.1 |
| Median AE | **155.5** | 194.6 | −39.1 |
| RMSE | **272.6** | 304.8 | −32.2 |
| **Within ±50 kcal** | **19.2%** | 9.8% | +9.4 pp |
| **Within ±100 kcal** | **34.9%** | 22.8% | +12.0 pp |
| Bias | **+82.3** | +156.4 | −74.1 |
| Invalid predictions | 0 / 499 | 0 / 499 | — |

**Per food-type (where the new mix actually shifts the failure modes):**

| Food type | n | CalorieK MAE | CalorieK w/±100 | CalorieCLIP MAE | CalorieCLIP w/±100 |
| --- | ---:| ---:| ---:| ---:| ---:|
| Homemade food | 100 | **171** | **37%** | 251 | 23% |
| Packaged food | 199 | **123** | **59%** | 204 | 27% |
| Raw vegetables & fruits | 200 | 244 | 10% | **234** | **19%** |

CalorieK wins on Homemade and Packaged by 80 and 81 kcal MAE respectively, and almost doubles
CalorieCLIP's within-±100 hit rate on those categories. On Raw vegetables & fruit both models
fall apart (MAE 234–244 on a mean target of ~150 kcal, within-±100 under 20%) — this is the
long-tail small-portion regime that neither model handles, and the scatter shows both regress to
the 200–400 kcal range even for items like a single 62 kcal "Orange Slices" or 68 kcal "Fresh
Peaches." Four outliers in the 1,800–2,500 kcal range (e.g. "Pork Belly" labelled 2500 kcal and
categorised as Raw vegetables) drag both models' overall MAE up; these are the rare cases where
the MM-Food-100K ground truth is implausible (a single 2,000 g "Pork Belly" portion is
plausible, but its `food_type` labelling is clearly wrong). CalorieCLIP is also systematically
over-confident: it predicts 760 kcal on a 68 kcal peach and 771 kcal on a 150 kcal dairy drink,
a +156 kcal bias against CalorieK's +82.

**Caveats on this benchmark.** (1) MM-Food-100K is the *same source dataset* as the prior OOD
test, just a fresh, disjoint sample — it is not a wholly new dataset. We could not find a third,
truly-distinct, downloadable per-image-calorie food dataset (OmniFood8K is author-only via
Google Drive; CGMacros is CGM-centric; the only other plausible option, Food-101, has class
labels not per-image calories and would have required a fabricated per-class nutrition table,
which violates our no-fabrication rule). (2) MM-Food-100K labels are AI/crowd estimates, not
weighed ground truth — a ±10–20% label noise floor is expected. (3) The 2,500 kcal outliers
are clearly label noise and inflate both models' MAE.

Full per-prediction CSV: `outputs/compare_ood_v2/predictions.csv`. Aggregate metrics:
`outputs/compare_ood_v2/report.json`. Raw downloads: `data/mmfood100k_v2/images/`.

## Comparison with open-source calorie VLMs (same protocol)

Beyond CalorieCLIP, we ran the published checkpoints of every other public calorie VLM we could
get to run end-to-end on a single RGB photo — Food-R1 (8-bit), Boba 0.8B (Q4_K_M GGUF),
DPF-Nutrition, and OmniFood FAFM — under the same locked protocol and the same MM-Food-100K v2
OOD sample, so the numbers are directly comparable. The full per-model report (including the
per-task macro/mass errors) lives in `outputs/external_model_comparison.md`.

![CalorieK vs. open-source calorie VLMs](https://huggingface.co/Dralkh/CalorieK/resolve/main/external_comparison.png)

### Locked Nutrition5k test (606 dishes; 2 corrupt-label dishes excluded)

| Model | Checkpoint | n | Calorie MAE | PMAE |
| --- | --- | ---:| ---:| ---:|
| **CalorieK (ours)** | local | 606 | **56.3** | **24.1%** |
| Food-R1 (8-bit) | [`zy12123/Food-R1`](https://huggingface.co/zy12123/Food-R1) | 605 | 77.0 | 32.9% |
| Boba 0.8B (Q4_K_M) | [`opengvlab/Boba`](https://huggingface.co/opengvlabs/Boba-LLM-tiny) | 605 | 79.5 | 33.9% |
| DPF-Nutrition | [`T0MYYY/dpf-nutrition`](https://huggingface.co/T0MYYY/dpf-nutrition) | 606 | 109.1 | 46.6% |
| CalorieCLIP | [`jeong-jasonji/CalorieCLIP`](https://huggingface.co/jeong-jasonji/CalorieCLIP) | 606 | 126.4 | 54.0% |
| OmniFood FAFM | OmniFood8K release | 606 | 132.5 | 56.6% |

Food-R1 and Boba each had one image (`dish_1565810969`) where the model entered a non-terminating
token loop; their coverage is 99.83% on 608. All other models cover 100%.

### MM-Food-100K v2 transfer — 499 disjoint images (OOD)

| Model | OOD-eligible? | n | Calorie MAE | PMAE | Bias |
| --- | :---:| ---:| ---:| ---:| ---:|
| DPF-Nutrition | yes | 499 | 177.7 | 76.4% | +30.8 |
| **CalorieK (ours)** | yes | 499 | 181.2 | — | +82.3 |
| Boba 0.8B (Q4_K_M) | yes | 499 | 193.4 | 83.1% | −24.5 |
| OmniFood FAFM | yes | 499 | 216.1 | 92.9% | +131.2 |
| CalorieCLIP | yes | 499 | 225.3 | — | +156.4 |
| Food-R1 | **no** | — | — | — | trained on CalorieBench-80K derived from MM-Food-100K |

DPF-Nutrition wins OOD by 3.5 kcal — inside the noise band, not a decisive lead without a
paired bootstrap. The locked-test gap (56.3 vs 77.0 for the next-best public model) is the
cleaner signal: under an identical, in-distribution protocol, CalorieK's calorie MAE is 27%
lower than the next open-source VLM.

**What "same protocol" means.** Same camera-A center frame per dish as the locked headline
test; same scalar calorie target from the official Nutrition5k dish metadata; same MM-Food-100K
v2 manifest for OOD; same JSON-completeness and memory controls for the GGUF runner
(`scripts/benchmark_gguf_food.py`); and the published PMAE / MAE definitions, not the paper
authors' own splits. Public PyTorch adapters: `src/ccalorie/external_bench.py`. Locked
predictions: `outputs/external_bench_locked/`, `outputs/boba_locked/`. OOD predictions:
`outputs/external_bench_mmfood/`, `outputs/boba_mmfood/`.

## Training method

- **Architecture**: `google/siglip2-base-patch16-224` vision tower (92M params) → LayerNorm →
MLP head (768→768→256→5) with softplus outputs. Trained with composite Huber loss, per-task weighted.
- **Augmentation**: MixUp + CutMix (both α=0.4, 50% probability per batch), plus standard
RandomResizedCrop, HorizontalFlip, Rotation, ColorJitter.
- **Schedule**: 20 epochs × 20,000 samples, effective batch 256 (bf16), head LR 1e-4 / trunk LR 2e-5,
cosine schedule with restarts, 5% warmup, first 5% head-only, last 4 vision blocks unfrozen,
weight decay 0.05, seed 17.
- **Checkpoint selection**: best validation calorie MAE on 80 held-out camera-A dishes (validation MAE 57.3 kcal).
Training ran ~13 minutes on one RTX 4080 (16 GB).
- **Calibration**: a conformal radius fitted on 86 held-out calibration dishes gives 90% prediction
intervals of ±119 kcal (`calibration.json`).

## Training data (20,000 samples, leakage-audited)

| Source | Samples | Notes |
| --- | --- | --- |
| [`pinkieseb/nutrition_dataset`](https://huggingface.co/datasets/pinkieseb/nutrition_dataset) (Nutrition5k video frames) | 19,734 | only dishes provably in the **official Nutrition5k train split** (matched by exact nutrition signature), ≤16 frames per dish |
| [`aryachakraborty/Food_Calorie_Dataset`](https://huggingface.co/datasets/aryachakraborty/Food_Calorie_Dataset) | 266 | human-estimated labels, down-weighted (0.4) |

Nutrition labels are taken directly from the official Nutrition5k dish metadata. Leakage controls,
all enforced programmatically before training:

- **Group separation**: splits partition by physical dish (`physical_meal_id`); no dish appears in two splits.
- **Exact-media**: SHA-256 disjointness across fit / validation / calibration / test.
- **Near-duplicate**: perceptual-hash disjointness; 2 fit images visually colliding with held-out frames were dropped and backfilled.
- **Official-split proof**: a Nutrition5k registry maps nutrition signatures to official train/test IDs;
ambiguous signatures are excluded. The audit trail (SHA-256 of every manifest and the source tree) ships in the results.

## Usage

```python
import torch, torch.nn as nn, torch.nn.functional as F
from transformers import AutoImageProcessor, SiglipVisionModel
from huggingface_hub import hf_hub_download
from PIL import Image

class NutritionHead(nn.Module):
    def __init__(self, input_dim: int = 768):
        super().__init__()
        self.net = nn.Sequential(
            nn.LayerNorm(input_dim), nn.Linear(input_dim, 768), nn.GELU(), nn.Dropout(.15),
            nn.Linear(768, 256), nn.GELU(), nn.Dropout(.10), nn.Linear(256, 5))
    def forward(self, x): return F.softplus(self.net(x))

MODEL_ID = "google/siglip2-base-patch16-224"
processor = AutoImageProcessor.from_pretrained(MODEL_ID)
vision = SiglipVisionModel.from_pretrained(MODEL_ID)
head = NutritionHead()
state = torch.load(hf_hub_download("Dralkh/CalorieK", "caloriek.pt"), map_location="cpu", weights_only=False)
vision.load_state_dict(state["vision_state"]); head.load_state_dict(state["head_state"])
vision.eval(); head.eval()

image = Image.open("meal.jpg").convert("RGB")
pixels = processor(images=image, return_tensors="pt").pixel_values
with torch.inference_mode():
    out = vision(pixel_values=pixels)
    feats = out.pooler_output if out.pooler_output is not None else out.last_hidden_state[:, 0]
    kcal, mass_g, protein_g, carb_g, fat_g = head(feats.float())[0].tolist()
print(f"{kcal:.0f} kcal ± 119 (90% interval), {mass_g:.0f} g, P{protein_g:.0f}/C{carb_g:.0f}/F{fat_g:.0f}")
```

## Limitations

- **Domain**: trained and evaluated on Nutrition5k-style plated single dishes photographed from the side;
accuracy on multi-plate scenes, packaged food, or drinks is untested.
- **High-calorie under-prediction**: the model plateaus and under-predicts on dishes above ~500 kcal
(see the scatter above). Use the calibrated interval, not the point estimate, for anything that matters.
- **Frames, not meals**: training frames come from videos of ~1,250 distinct dishes; visual diversity is
narrower than the sample count suggests.

## Citation

```bibtex
@misc{caloriek2026,
  title={CalorieK: Accurate Food Calorie Estimation from a Single RGB Image},
  author={Faisal Alkharji},
  year={2026},
  url={https://huggingface.co/Dralkh/CalorieK}
}
```