---
library_name: transformers
pipeline_tag: image-text-to-text
base_model: HuggingFaceTB/SmolVLM2-500M-Video-Instruct
license: apache-2.0
tags:
- vision-language-model
- lora
- food
- ingredient-recognition
---

# Kimchi-V2 — SmolVLM2-500M (ingredient recognition, ~353-class)

A LoRA fine-tune of [SmolVLM2-500M-Video-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM2-500M-Video-Instruct),
merged into a standalone model. Given a photo of ingredients arranged on a surface,
it returns a JSON list of the ingredients it recognizes. Built as an end-to-end ML
portfolio project (data → fine-tune → inference endpoint → web app).

**V2 change:** the vocabulary was expanded from 51 to **~353 ingredient classes** by
unioning the original 51-class set with a 316-class dataset (14 ingredients shared and
merged to a single canonical label). This directly targets V1's main failure mode —
padding the output with frequent in-vocab guesses whenever an out-of-vocabulary
ingredient appeared — by giving many of those previously-unknown items a real label.

- **Task:** image → JSON ingredient list
- **Classes:** ~353 single ingredients (51 legacy + 316 new − 14 overlapping, merged)
- **Adapter:** LoRA on q/k/v/o + gate/up/down projections, merged
- **Trained from base**, not continued from V1 (avoids forgetting the non-overlapping legacy classes)

## Usage

```python
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image

repo = "LongGrainRice/kimchi-test"
processor = AutoProcessor.from_pretrained(repo)
# bf16 needs an Ampere+ GPU (e.g. L4). On a T4 or older card, use torch.float16.
model = AutoModelForImageTextToText.from_pretrained(repo, torch_dtype=torch.bfloat16).to("cuda")

# Use the exact instruction the model was trained with — it keys on this wording.
INSTRUCTION = ("You are a food recognition assistant. List every food ingredient in this image. "
               'Respond ONLY with a JSON array of lowercase strings, e.g. ["milk", "eggs", "tomato"].')

image = Image.open("slab.jpg").convert("RGB")
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": INSTRUCTION}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=[prompt], images=[[image]], return_tensors="pt", padding=True).to(model.device)

ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.batch_decode(ids[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
```

De-duplicate the output before using it: `sorted(set(...))`.

## Training data

Synthetic composites built by unioning two single-item sources, both center-cropped with an
oval mask and pasted onto slab backgrounds with varied scale, rotation, and shadow; the label
is the set of pasted items.

- [`liamboyd1/singular-food-items`](https://www.kaggle.com/datasets/liamboyd1/singular-food-items)
  — 51 classes, image-rich (~1.3k/class), capped per class for variety.
- [`Scuccorese/food-ingredients-dataset`](https://huggingface.co/datasets/Scuccorese/food-ingredients-dataset)
  — 316 classes, ~21 images/class, with a 12-category / 28-subcategory hierarchy.

The 14 ingredients shared between the two sets are normalized to a single canonical name so
they don't fragment into duplicate classes. Because the two sources differ ~60× in per-class
image count, the compositor **samples classes uniformly** when building scenes rather than
sampling images uniformly — otherwise the image-rich legacy classes would dominate and the new
vocabulary would rarely appear. A small set of real photos is mixed in and upweighted.

## Training

- Base: SmolVLM2-500M-Video-Instruct, LoRA (q/k/v/o + gate/up/down), merged after training.
- ~3 epochs, effective batch 8, cosine schedule, gradient checkpointing.
- bf16 on L4 (24 GB); the pipeline auto-falls back to fp16 + smaller batch on a T4.

## Limitations

- **Vocabulary is fixed at ~353 classes.** The expansion covers many items V1 couldn't name,
  but anything outside the set still has no label.
- **Pantry staples aren't detected.** Salt, oil, sugar, flour and similar are generally not
  visible as distinct items and aren't in scope; the downstream recipe step introduces them
  separately, tagged as pantry/extra, so users aren't misled about what was actually detected.
- **New classes are data-thin (~21 images each).** Visually adjacent ingredients (e.g. kale vs
  collard greens, scallion vs leek) can be confused or both emitted for one item. This is a
  different, milder error mode than V1's out-of-vocab padding, but it means precision on
  fine-grained near-neighbors is the weak spot. Near-synonyms may be worth collapsing to a
  coarser label for a given use case.
- **Always de-duplicate the output** (`sorted(set(...))`).

Best used on scenes composed from the known classes. A base-vs-fine-tuned comparison script is
included in the project repo for evaluating on your own slab photos.