FoodSense-VL

Predicting Taste, Smell, Texture, and Sound from Food Images

Model Summary

FoodSense-VL is an open-source vision-language model designed to predict four sensory dimensions of food—Taste, Smell, Texture, and Sound—directly from a single image. It outputs both quantitative scores (on a 1–5 scale) and qualitative natural-language justifications grounded in the visual evidence of the dish.

The model is built by fine-tuning Gemma 3 27B-IT using QLoRA. It was trained on the human-annotated FoodSense Dataset, demonstrating state-of-the-art correlation with human sensory judgments compared to leading open-weights VLMs like Qwen2.5-VL, LLaVA, and InternVL.

Accepted to the CVPR 2026 Workshop on Meta Food.

Usage: Quick Start

You can run inference using the standard transformers library with peft to load the adapter weights.

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
from PIL import Image
import requests

# 1. Load base model and processor
base_model_id = "google/gemma-3-27b-it"
adapter_id = "sababishraq/foodsense-vl"

processor = AutoProcessor.from_pretrained(base_model_id)
base_model = AutoModelForImageTextToText.from_pretrained(
    base_model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16
)

# 2. Load FoodSense-VL adapter
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

# 3. Prepare an image and prompt
url = "https://huggingface.co/datasets/sababishraq/foodsense-dataset/resolve/main/0001_01lamiW2bWW0rXlllNHYMA.jpg"
image = Image.open(requests.get(url, stream=True).raw)

prompt = "Predict the taste, smell, texture, and sound of this food on a scale from 1 to 5. Justify your ratings."
messages = [
    {"role": "user", "content": [{"type": "image"}, {"type": "text", "text": prompt}]}
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, return_dict=True, tokenize=True, return_tensors="pt"
).to(model.device)
inputs["pixel_values"] = processor(images=image, return_tensors="pt")["pixel_values"].to(model.device, dtype=torch.bfloat16)

# 4. Generate sensory prediction
with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.2)

print(processor.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Sample Output

Taste (4.2/5.0): The golden-brown sear on the steak suggests a savory, umami-rich flavor with a caramelized exterior. 
Smell (4.0/5.0): The visible grill marks and rendering fat indicate a strong, smoky, and meaty aroma.
Texture (3.8/5.0): The thick cut and visible grain imply a firm, chewy bite, though the juices suggest it remains tender inside.
Sound (2.5/5.0): Cutting through the slight crust might produce a muffled slicing sound, and chewing would be relatively quiet.

Benchmark Results

FoodSense-VL was evaluated on a held-out test set of 438 images against several leading open-source models. It achieves the highest correlation with human judgments (Pearson, Spearman, Lin's CCC) across all four sensory dimensions.

Model	MAE ↓	Pearson r ↑	Spearman ρ ↑	Lin's CCC ↑
LLaVA-v1.6-34B	0.435	0.229	0.197	0.113
InternVL2.5-26B	0.507	0.226	0.177	0.078
Qwen2.5-VL-32B	0.589	0.246	0.236	0.124
Gemma 3 27B (zero-shot)	0.602	0.211	0.181	0.136
Food-LLaMA-11B	0.740	0.080	0.089	0.055
FoodSense-VL (Ours)	0.538	0.372	0.360	0.343

Training Details

FoodSense-VL employs a two-stage QLoRA fine-tuning methodology:

1. Stage 1 (Human Sensory Alignment): The model is trained directly on the raw human ratings and short descriptors from the FoodSense dataset to align its internal representations with human sensory perception.
2. Stage 2 (MAmmoTH Expansion): Using a MAmmoTH-v2-inspired paradigm, we synthetically expand the short human descriptions into full, image-grounded reasoning traces. The model learns to link visual cues to the final 1–5 score.

Hyperparameters:

• GPU: 1x H100 (80GB)
• LoRA Rank: 16 (Alpha: 32)
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Learning Rate: 5e-6 (Stage 1), 2e-6 (Stage 2)
• Effective Batch Size: 64

Intended Use and Limitations

Intended Use: This model is intended for research in multi-modal vision-language understanding, specifically focusing on cross-sensory inferences (predicting non-visual properties from visual data).

Limitations:

• Not for safety/health advice: The model cannot genuinely determine if food is spoiled, safe to eat, or accurately estimate nutritional/caloric content.
• Subjective nature of sensory perception: Taste, smell, texture, and sound are highly subjective. The model predicts the average human perception expectations, not an absolute ground truth truth.

Citation

If you use this model, please cite the original paper:

@inproceedings{ishraq2026foodsense,
  title     = {FoodSense: A Multisensory Food Dataset and Benchmark for
               Predicting Taste, Smell, Texture, and Sound from Images},
  author    = {Ishraq, Sabab and Aarushi, Aarushi and Jiang, Juncai and Chen, Chen},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and
               Pattern Recognition (CVPR) Workshops},
  year      = {2026}
}