paint-match-minicpm

MiniCPM-V-4.6 (1B) finetuned to extract paint codes from scale model instruction sheets and shop screenshots. Runs via llama.cpp on CPU — no GPU required.

Part of the Paint Match project for the Hugging Face "Build Small" hackathon.

What it does

Given a photo of a scale model instruction sheet or a hobby shop screenshot, the model returns a structured JSON list of paint codes (Humbrol, Tamiya, Meng, etc.). Used as the inference core of a fully local paint-matching tool running on a Radxa Dragon Q6A ARM board.

Benchmark

Metric	Score
F1 — benchmark (10 images)	0.935
F1 — shop holdout (53 images)	0.927
F1 — paper holdout (7 images)	0.928
Avg latency (Radxa CPU)	~64s

Base model comparison (InternVL3.5-2B): F1=0.873, ~190s latency.

Files

File	Description
model.safetensors	Merged model weights (LoRA baked in)
config.json + tokenizer files	Model config and tokenizer
minicpm-ft-Q6_K.gguf	Quantized GGUF for llama.cpp inference

Fine-tuning details

• Base model: openbmb/MiniCPM-V-4.6
• Method: LoRA (r=16, alpha=32, dropout=0.05)
• Target modules: q_proj, v_proj (+ language model projection layers)
• Training: 5 epochs, lr=2e-4, cosine scheduler, batch size 8
• Dataset: ~5,866 examples scraped from Airfix instruction sheets + shop photos, image resolution 960px
• Hardware: Modal.com H100, ~16 min training time

Usage (llama.cpp / GGUF)

llama-cli \
  -m minicpm-ft-Q6_K.gguf \
  --mmproj mmproj-model-f16.gguf \
  --image instruction_sheet.jpg \
  -p "Extract all paint codes from this image as JSON."

Usage (transformers)

from transformers import AutoModel, AutoTokenizer
from PIL import Image

model = AutoModel.from_pretrained(
    "build-small-hackathon/paint-match-minicpm",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "build-small-hackathon/paint-match-minicpm",
    trust_remote_code=True,
)

image = Image.open("instruction_sheet.jpg").convert("RGB")
msgs = [{"role": "user", "content": [image, "Extract all paint codes from this image as JSON."]}]
result = model.chat(image=None, msgs=msgs, tokenizer=tokenizer)
print(result)