Instructions to use LongGrainRice/kimchi-test with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LongGrainRice/kimchi-test with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="LongGrainRice/kimchi-test")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("LongGrainRice/kimchi-test")
model = AutoModelForMultimodalLM.from_pretrained("LongGrainRice/kimchi-test")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use LongGrainRice/kimchi-test with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LongGrainRice/kimchi-test"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LongGrainRice/kimchi-test",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/LongGrainRice/kimchi-test

SGLang

How to use LongGrainRice/kimchi-test with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LongGrainRice/kimchi-test" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LongGrainRice/kimchi-test",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LongGrainRice/kimchi-test" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LongGrainRice/kimchi-test",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use LongGrainRice/kimchi-test with Docker Model Runner:
```
docker model run hf.co/LongGrainRice/kimchi-test
```

kimchi-test / README.md

LongGrainRice

Update README.md

74219ed verified 6 days ago

preview code

Raw

History Blame Contribute Delete

5.01 kB

metadata

library_name: transformers
pipeline_tag: image-text-to-text
base_model: HuggingFaceTB/SmolVLM2-500M-Video-Instruct
license: apache-2.0
tags:
  - vision-language-model
  - lora
  - food
  - ingredient-recognition

Kimchi-V2 — SmolVLM2-500M (ingredient recognition, ~353-class)

A LoRA fine-tune of SmolVLM2-500M-Video-Instruct, merged into a standalone model. Given a photo of ingredients arranged on a surface, it returns a JSON list of the ingredients it recognizes. Built as an end-to-end ML portfolio project (data → fine-tune → inference endpoint → web app).

V2 change: the vocabulary was expanded from 51 to ~353 ingredient classes by unioning the original 51-class set with a 316-class dataset (14 ingredients shared and merged to a single canonical label). This directly targets V1's main failure mode — padding the output with frequent in-vocab guesses whenever an out-of-vocabulary ingredient appeared — by giving many of those previously-unknown items a real label.

Task: image → JSON ingredient list
Classes: ~353 single ingredients (51 legacy + 316 new − 14 overlapping, merged)
Adapter: LoRA on q/k/v/o + gate/up/down projections, merged
Trained from base, not continued from V1 (avoids forgetting the non-overlapping legacy classes)

Usage

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image

repo = "LongGrainRice/kimchi-test"
processor = AutoProcessor.from_pretrained(repo)
# bf16 needs an Ampere+ GPU (e.g. L4). On a T4 or older card, use torch.float16.
model = AutoModelForImageTextToText.from_pretrained(repo, torch_dtype=torch.bfloat16).to("cuda")

# Use the exact instruction the model was trained with — it keys on this wording.
INSTRUCTION = ("You are a food recognition assistant. List every food ingredient in this image. "
               'Respond ONLY with a JSON array of lowercase strings, e.g. ["milk", "eggs", "tomato"].')

image = Image.open("slab.jpg").convert("RGB")
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": INSTRUCTION}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=[prompt], images=[[image]], return_tensors="pt", padding=True).to(model.device)

ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.batch_decode(ids[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

De-duplicate the output before using it: sorted(set(...)).

Training data

Synthetic composites built by unioning two single-item sources, both center-cropped with an oval mask and pasted onto slab backgrounds with varied scale, rotation, and shadow; the label is the set of pasted items.

liamboyd1/singular-food-items — 51 classes, image-rich (~1.3k/class), capped per class for variety.
Scuccorese/food-ingredients-dataset — 316 classes, ~21 images/class, with a 12-category / 28-subcategory hierarchy.

The 14 ingredients shared between the two sets are normalized to a single canonical name so they don't fragment into duplicate classes. Because the two sources differ ~60× in per-class image count, the compositor samples classes uniformly when building scenes rather than sampling images uniformly — otherwise the image-rich legacy classes would dominate and the new vocabulary would rarely appear. A small set of real photos is mixed in and upweighted.

Training

Base: SmolVLM2-500M-Video-Instruct, LoRA (q/k/v/o + gate/up/down), merged after training.
~3 epochs, effective batch 8, cosine schedule, gradient checkpointing.
bf16 on L4 (24 GB); the pipeline auto-falls back to fp16 + smaller batch on a T4.

Limitations

Vocabulary is fixed at ~353 classes. The expansion covers many items V1 couldn't name, but anything outside the set still has no label.
Pantry staples aren't detected. Salt, oil, sugar, flour and similar are generally not visible as distinct items and aren't in scope; the downstream recipe step introduces them separately, tagged as pantry/extra, so users aren't misled about what was actually detected.
New classes are data-thin (~21 images each). Visually adjacent ingredients (e.g. kale vs collard greens, scallion vs leek) can be confused or both emitted for one item. This is a different, milder error mode than V1's out-of-vocab padding, but it means precision on fine-grained near-neighbors is the weak spot. Near-synonyms may be worth collapsing to a coarser label for a given use case.
Always de-duplicate the output (sorted(set(...))).

Best used on scenes composed from the known classes. A base-vs-fine-tuned comparison script is included in the project repo for evaluating on your own slab photos.