LongGrainRice/kimchi-test

Kimchi-V2 — SmolVLM2-500M (ingredient recognition, ~353-class)

A LoRA fine-tune of SmolVLM2-500M-Video-Instruct, merged into a standalone model. Given a photo of ingredients arranged on a surface, it returns a JSON list of the ingredients it recognizes. Built as an end-to-end ML portfolio project (data → fine-tune → inference endpoint → web app).

V2 change: the vocabulary was expanded from 51 to ~353 ingredient classes by unioning the original 51-class set with a 316-class dataset (14 ingredients shared and merged to a single canonical label). This directly targets V1's main failure mode — padding the output with frequent in-vocab guesses whenever an out-of-vocabulary ingredient appeared — by giving many of those previously-unknown items a real label.

• Task: image → JSON ingredient list
• Classes: ~353 single ingredients (51 legacy + 316 new − 14 overlapping, merged)
• Adapter: LoRA on q/k/v/o + gate/up/down projections, merged
• Trained from base, not continued from V1 (avoids forgetting the non-overlapping legacy classes)

Usage

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image

repo = "LongGrainRice/kimchi-test"
processor = AutoProcessor.from_pretrained(repo)
# bf16 needs an Ampere+ GPU (e.g. L4). On a T4 or older card, use torch.float16.
model = AutoModelForImageTextToText.from_pretrained(repo, torch_dtype=torch.bfloat16).to("cuda")

# Use the exact instruction the model was trained with — it keys on this wording.
INSTRUCTION = ("You are a food recognition assistant. List every food ingredient in this image. "
               'Respond ONLY with a JSON array of lowercase strings, e.g. ["milk", "eggs", "tomato"].')

image = Image.open("slab.jpg").convert("RGB")
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": INSTRUCTION}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=[prompt], images=[[image]], return_tensors="pt", padding=True).to(model.device)

ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.batch_decode(ids[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

De-duplicate the output before using it: sorted(set(...)).

Training data

Synthetic composites built by unioning two single-item sources, both center-cropped with an oval mask and pasted onto slab backgrounds with varied scale, rotation, and shadow; the label is the set of pasted items.

• liamboyd1/singular-food-items — 51 classes, image-rich (~1.3k/class), capped per class for variety.
• Scuccorese/food-ingredients-dataset — 316 classes, ~21 images/class, with a 12-category / 28-subcategory hierarchy.

The 14 ingredients shared between the two sets are normalized to a single canonical name so they don't fragment into duplicate classes. Because the two sources differ ~60× in per-class image count, the compositor samples classes uniformly when building scenes rather than sampling images uniformly — otherwise the image-rich legacy classes would dominate and the new vocabulary would rarely appear. A small set of real photos is mixed in and upweighted.

Training

• Base: SmolVLM2-500M-Video-Instruct, LoRA (q/k/v/o + gate/up/down), merged after training.
• ~3 epochs, effective batch 8, cosine schedule, gradient checkpointing.
• bf16 on L4 (24 GB); the pipeline auto-falls back to fp16 + smaller batch on a T4.

Limitations

• Vocabulary is fixed at ~353 classes. The expansion covers many items V1 couldn't name, but anything outside the set still has no label.
• Pantry staples aren't detected. Salt, oil, sugar, flour and similar are generally not visible as distinct items and aren't in scope; the downstream recipe step introduces them separately, tagged as pantry/extra, so users aren't misled about what was actually detected.
• New classes are data-thin (~21 images each). Visually adjacent ingredients (e.g. kale vs collard greens, scallion vs leek) can be confused or both emitted for one item. This is a different, milder error mode than V1's out-of-vocab padding, but it means precision on fine-grained near-neighbors is the weak spot. Near-synonyms may be worth collapsing to a coarser label for a given use case.
• Always de-duplicate the output (sorted(set(...))).

Best used on scenes composed from the known classes. A base-vs-fine-tuned comparison script is included in the project repo for evaluating on your own slab photos.