docs/model_selection.md

Model Selection Journey: IMDB AutoFill VLM Pipeline

Context

The IMDB AutoFill pipeline extracts structured product data (brand, manufacturer, weight, country of origin, etc.) from retail product images. Each product has 3-5 images taken from different angles; images are grouped by a dataset label (tag_text) stuck to the edge of the packaging before being sent to the VLM.

The target dataset is 45 products from a Ghanaian retail context: brands imported from China, Indonesia, Vietnam, Cote d'Ivoire, Nigeria, and Sri Lanka, typically sold and distributed by local importers (e.g. LGD LIMITED, FAGIP VENTURES).

Accuracy is measured field-by-field against a ground-truth Excel sheet (eval/eval_from_org.xlsx).

---

Phase 1: llama3.2-vision:11b (Local, Ollama)

Why we started here: Zero API cost, runs on the local RTX 5090 32GB GPU via Ollama. Fast iteration without rate limits or spend.

Configuration:

• Model: llama3.2-vision:11b
• Backend: Ollama (localhost:11434)
• Concurrency: 4 parallel workers, 1 image per call
• Image grouping: by tag_text similarity

Results: 41 matched pairs / 45 GT products: 48.1% overall accuracy

Why we moved on:

• The 11B model struggled to read small edge labels (tag_text) reliably, causing images to split into separate products instead of grouping correctly.
• Field extraction quality was low: manufacturer, country, fragrance_flavor all scored poorly.
• The model's vision capability at 11B was simply insufficient for dense retail label text.

---

Phase 2: qwen2.5vl:32b (Local, Ollama)

Why we tried this: Qwen2.5-VL is a purpose-built vision-language model with significantly better OCR and label-reading capability than llama3.2-vision. At 32B it fits within the 32GB VRAM of the RTX 5090.

Configuration:

• Model: qwen2.5vl:32b
• Backend: Ollama (localhost:11434)
• Concurrency: reduced to 2 (32B model is slower per call)
• Image grouping: by tag_text similarity (unchanged)

Results: 55 matched pairs / 45 GT products: 51.9% overall accuracy

Note: 55 matched pairs > 45 GT because the model was over-splitting some products (empty tag_text means each image became its own product). Matched pairs counts how many predictions align to a GT row, not unique products.

What improved:

• Brand and product name accuracy up significantly
• tag_text extraction more reliable, leading to better image grouping

Why we moved on:

• Despite better OCR, the model was still leaving tag_text empty on ~30% of images, causing over-splitting (82+ predictions for 45 actual products).
• Inference speed was slow (32B model, 2 concurrent workers).
• The accuracy ceiling for local 32B models seemed to be around 52-55% given the visual complexity of the labels.

---

Phase 3: Gemini 2.5 Flash and 3.5 Flash (Cloud, Google AI)

Why we tried this: Gemini offered a cost-effective path to a stronger frontier model. Integrated via Google's OpenAI-compatible endpoint (https://generativelanguage.googleapis.com/v1beta/openai/), no new SDK required.

Configuration:

• Backend: gemini (via VLM_BACKEND=gemini)
• Models tested: gemini-2.5-flash, gemini-3.5-flash
• Batch size: 8 images per API call

Results:

Model	Matched pairs	Overall
Gemini 2.5 Flash	31	47.8%
Gemini 3.5 Flash	32	41.6%

Why we moved on:

• Gemini models consistently failed to read edge labels (tag_text) even with explicit prompting, causing severe over-splitting (116-122 predictions for 45 products).
• With only 31-32 matched pairs vs. 45 GT products, many products simply could not be evaluated.
• The larger Gemini 3.5 model actually scored lower than 2.5, likely a different training emphasis.
• Overall accuracy was worse than the local qwen model.

---

Phase 4: GPT-5.5 (OpenAI, Cloud)

Why we chose this: After Gemini's failure on tag_text reading, GPT-5.5 was evaluated as the strongest available vision model. It demonstrated noticeably better spatial awareness and OCR of small printed text, including the edge dataset labels.

Configuration:

• Model: gpt-5.5-2026-04-23 via OpenAI API
• Backend: openai (via VLM_BACKEND=openai)
• Batch size: 8 images per API call (all session images in one request)
• API params: max_completion_tokens=4096, no temperature (GPT-5.5 only supports default)
• Response format: {"type": "json_object"}

Key integration notes:

• GPT-5.5 does not support max_tokens; use max_completion_tokens
• GPT-5.5 does not support custom temperature; only the default (1) is supported
• Images are sent as base64 data URLs in the message content array

v1: Initial GPT-5.5 run

Results: 45 matched pairs / 45 GT products: 59.5% overall accuracy

All 45 GT products matched; tag_text reading was reliable enough to group images correctly. This was the first run where we achieved a 1:1 mapping between predictions and GT.

v2: Prompt improvements

Three targeted changes based on failure analysis of v1 predictions:

1. Manufacturer prompt: Updated to emphasise local African importer/distributor over foreign factory. Added examples of the expected company name format (LGD LIMITED, FAGIP VENTURES, SISTER SARDINE & MACKEREL VENTURES). Directed model to look for "Manufactured by", "Manufactured for", "Imported by", "Distributed by", "Packed by" phrasing.

2. Country normalisation: Added P.R.C., PRC, P.R.C, PEOPLES REPUBLIC OF CHINA, PEOPLE'S REPUBLIC OF CHINA -> CHINA mappings to backend/normalizer.py. Many products from Chinese factories use the PRC abbreviation on pack. Country accuracy: 38.5% -> 50%.

3. Fragrance/flavor broadening: Expanded the field definition to cover food flavors (BEEF, STEW RAGOUT, SOUP, NOODLES, CHICKEN), beverage flavors (COLA, LEMON, MALT, MULTI FRUIT), cosmetic fragrances (ROSE, CITRUS, JAPANESE CAMELLIA), and detergent scents (POWDER, FRESH). The GT definition is broader than traditional "fragrance"; any taste/scent descriptor on pack counts.

Results: 41 matched pairs / 45 GT products: 63.4% overall accuracy

Note: matched pairs dropped from 45 to 41 in v2 (minor grouping regression to investigate). Overall accuracy still improved because field-level extraction quality went up.

v2 Field breakdown

Field	Accuracy	Notes
category_type	90.0%	Strong
brand	85.4%	Strong
packaging_type	82.9%	Strong
product_name	78.0%	Good
weight	63.4%	Good
barcode	60.0%	Good
country_of_origin	50.0%	PRC->CHINA fix helped
fragrance_flavor	42.3%	Broadened definition helped
manufacturer	39.0%	Still reading foreign factory
tagline	0.0%	Regressed: definition too strict
promotional_messages	0.0%	Not yet extracted
addons	0.0%	Not yet extracted

Why GPT-5.5 was promising:

1. tag_text reliability: reads small edge labels that other models miss, enabling correct image grouping
2. Structured JSON compliance: consistently returns valid JSON arrays without retry failures
3. OCR quality: correctly reads dense small-print text (manufacturer, country, barcode) on diverse packaging
4. Spatial awareness: correctly identifies which text is on the front vs. back vs. edge of a product

Why we moved on:

• 6 sessions (8-image batches) consistently returned empty content across all 3 retries, likely OpenAI's content filter silently refusing certain product images
• Each failed session took 1300s before giving up (3x retry x ~430s/attempt)
• 34/45 products matched due to the 6 total failures

---

Architecture

The backend is designed to be model-agnostic via VLM_BACKEND env var:

VLM_BACKEND=anthropic  -> Claude Sonnet 4.6 (current default)
VLM_BACKEND=openai     -> GPT-5.5
VLM_BACKEND=gemini     -> Gemini 2.5 Flash via OpenAI-compat endpoint

Batch size is set to 8 images per call for all cloud backends.

---

API Pricing (as of June 2026)

Token rates per 1M tokens

Model	Input	Output	Batch Input	Batch Output
Claude Sonnet 4.6	$3.00	$15.00	$1.50	$7.50
GPT-5.5	$5.00	$30.00	$2.50	$15.00
Gemini 2.5 Flash	$0.30	$2.50	$0.15	$1.25

Batch pricing (50% off standard) applies when using the provider's async Batch API, available for all three models in the Bulk Batch tab.

Estimated cost per image

Each extraction call sends approximately 1,560 input tokens (image tiles + shared prompt overhead across the batch of 8) and receives approximately 300 output tokens (JSON for one product item).

Model	Quick Upload	Bulk Batch
Claude Sonnet 4.6	~$0.0090	~$0.0045
GPT-5.5	~$0.0170	~$0.0085
Gemini 2.5 Flash	~$0.0012	~$0.0006

Derivation (Claude example):

input:  1,560 tokens * ($3.00 / 1,000,000) = $0.00468
output:   300 tokens * ($15.00 / 1,000,000) = $0.00450
total per image                              = $0.00918  (~$0.009)
batch (50% off)                              = $0.00459  (~$0.0045)

Actual costs will vary with image resolution and extraction complexity. The per-row cost shown in the results table reflects real token usage from each API call.

---

Phase 5: Claude Sonnet 4.6 (Anthropic, Cloud) [Current]

Why we chose this: After diagnosing the GPT-5.5 empty content failures, Claude Sonnet 4.6 was tested on the 6 failed sessions. All 6 completed successfully in 15-75s each, with no empty responses. The decision was made to switch the full pipeline to Claude.

Configuration:

• Model: claude-sonnet-4-6 via Anthropic SDK
• Backend: anthropic (via VLM_BACKEND=anthropic)
• Batch size: 8 images per API call (unchanged)
• API params: max_tokens=4096, system prompt as top-level system parameter
• Images: base64-encoded JPEG blocks in user content array

Key integration notes:

• Anthropic's native SDK uses a different image format than OpenAI (source.type: "base64" blocks)
• System prompt is passed as the system parameter, not as a role: "system" message
• No response_format equivalent; prompt instructs JSON output directly
• Message Batches API (50% off): eval/run_eval_batch.py submits all 42 VLM requests as one batch

Eval progression (Claude Sonnet 4.6):

Version	Key changes	Matched	Overall
v1 (baseline)	Switch from GPT-5.5	41	76.3%
v2	Country: company address > "Made in" + Example 6	42	76.7%
v3	Tagline: non-strict + shortest descriptor; brand accent norm	42	77.3%
v4	fragrance_flavor: non-strict + detergent POWDER; manufacturer normalizer	42	79.0%
v5	Weight normalizer (GMS->G, G->KG, ML->L); addons: non-strict	42	79.6%
v6	Brand: trade name vs manufacturer distinction	44	79.9%

v6 Field Breakdown (final, 44/45 matched)

Field	Accuracy
variant	100.0%
brand	97.7%
category_type	90.9%
country_of_origin	88.9%
packaging_type	88.6%
product_name	86.4%
weight	81.8%
addons	75.0%
tagline	72.7%
fragrance_flavor	73.1%
manufacturer	63.6%
barcode	58.1%
promotional_messages	33.3%
OVERALL	79.9%

Why Claude Sonnet 4.6 Wins

1. Zero empty-content failures: processed all 41 sessions (including 8-10 image batches) with no silent refusals
2. Speed: 15-75s per session vs 300-1300s for GPT-5.5
3. Instruction following: responds well to field priority rules (tagline shortest-first, country company address, brand trade name)
4. Cost: Anthropic Batch API at 50% discount makes full evals roughly 2x cheaper than GPT-5.5 standard

Remaining hard limits (not model-dependent):

• Manufacturer (63.6%): GT is inconsistent; sometimes wants the manufacturer, sometimes the local distributor
• Barcode (58.1%): 16 sessions have no barcode visible in any image
• Promotional messages (33.3%): only 3 GT products have promos, high variance