MODEL_CATALOG.md

Model Catalog — Usage-Based View

Purpose: Map every model in shopstack/model_registry.py to the product workflows it powers. This is the decision maker's view: given a workflow, which model handles it, and what are the alternatives?

---

1. Provider Capability Map

Models enter the app through provider backends, each wired by ShopStack.config.SHOPSTACK_*_BACKEND. The table below shows which backend handles which capability group.

Backend Config	Provider Class	Capabilities	Runtime	Status
SHOPSTACK_PLANNER_BACKEND	LocalProvider	Text generation, planning, embeddings	mlx/llama.cpp	Active
SHOPSTACK_PLANNER_BACKEND	OpenAIProvider	Text generation, vision, embeddings	Cloud (API)	Available
SHOPSTACK_PLANNER_BACKEND	MockPlannerProvider	Mock text, planning	Mock	Default (dev)
SHOPSTACK_STT_BACKEND	LocalWhisperProvider	Speech-to-text	mlx/faster-whisper	Active
SHOPSTACK_STT_BACKEND	WhisperProvider	Speech-to-text (cloud)	Cloud (API)	Available
SHOPSTACK_STT_BACKEND	MockSTTProvider	Mock STT	Mock	Default (dev)
SHOPSTACK_VISION_BACKEND	OpenAIProvider	Vision understanding (object detection, grounding)	Cloud (API)	Available
SHOPSTACK_VISION_BACKEND	MockVisionProvider	Mock vision, detection	Mock	Default (dev)
SHOPSTACK_OCR_BACKEND	MockOCRProvider	OCR/extraction	Mock	Default (dev)
SHOPSTACK_SEGMENTATION_BACKEND	MockSegmentationProvider	Image segmentation	Mock	Default (dev)
SHOPSTACK_TTS_BACKEND	MockTTSProvider	Text-to-speech	Mock	Default (dev)
SHOPSTACK_EMBEDDINGS_BACKEND	LocalProvider / OpenAIProvider	Embeddings (planned fallback)	Shared	Inherited
SHOPSTACK_IMAGE_EDIT_BACKEND	MockImageEditProvider	Image generation, annotation	Mock	Default (dev)

Key: "Active" = wired to real local inference. "Available" = dependencies installable. "Default (dev)" = mock provider used during development.

---

2. Full Model Registry

All entries from shopstack/model_registry.py, organized by provider group.

2a. STT — Speech-to-Text

Model ID	HF ID	Params	License	Runtime	Status	Notes
local-whisper-tiny	mlx-community/whisper-tiny-mlx	0.04B	MIT	mlx	Active	Default local Whisper backend
qwen3-asr-1.7b	Qwen/Qwen3-ASR-1.7B	1.7B	Apache-2.0	transformers	Candidate	Top candidate for household commands
parakeet-0.6b	nvidia/parakeet-ctc-0.6b	0.6B	CC-BY-4.0	custom	Candidate	Lightweight streaming ASR
sense-voice-small	funasr/SenseVoiceSmall	0.2B	MIT	transformers	Candidate	Very fast, multilingual
whisper-large-v3-turbo	openai/whisper-large-v3-turbo	0.8B	MIT	transformers	Candidate	Baseline only

2b. TTS — Text-to-Speech

Model ID	HF ID	Params	License	Runtime	Status	Notes
qwen3-tts-0.6b	Qwen/Qwen3-TTS-0.6B	0.6B	Apache-2.0	transformers	Candidate	Lightweight TTS candidate
kokoro-82m	—	0.082B	Apache-2.0	custom	Candidate	Extremely lightweight (off_grid)

2c. Vision / Object Detection / Grounding

Model ID	HF ID	Params	License	Runtime	Status	Notes
minicpm-v-8b	openbmb/MiniCPM-V-2_6	8.0B	Apache-2.0	transformers	Candidate	Strong VLM for household items

2d. Planner — LLM / Text Generation

Model ID	HF ID	Params	License	Runtime	Status	Notes
llama-3.2-3b-instruct	mlx-community/Llama-3.2-3B-Instruct-4bit	3.0B	Llama 3.2 Community	mlx	Active	Default MLX backend (auto-downloaded)
llama-3.2-3b-gguf	unsloth/Llama-3.2-3B-Instruct-GGUF	3.0B	Llama 3.2 Community	gguf	Active	Downloaded GGUF, llama.cpp fallback
minicpm5-1b	openbmb/MiniCPM5-1B	1.0B	Apache-2.0	transformers	Candidate	Lightweight planner / parser
lfm2.5-8b-a1b-gguf	unsloth/LFM2.5-8B-A1B-GGUF	8.3B	Apache-2.0	gguf	Candidate	GGUF planner for llama.cpp path
shopstack-parser-lora	—	—	Apache-2.0 (planned)	transformers	Candidate	Future fine-tuned command parser (well_tuned)

2e. OCR — Optical Character Recognition

Model ID	HF ID	Params	License	Runtime	Status	Notes
nuextract3-4b	nuance/NuExtract3-4B	4.0B	CC-BY-NC-4.0	transformers	Candidate	Strong receipt extraction (non-commercial)

2f. Segmentation

Model ID	HF ID	Params	License	Runtime	Status	Notes
rmbg-1.4	briaai/RMBG-1.4	0.3B	Apache-2.0	transformers	Candidate	Background removal for item cards

2g. Embeddings

Model ID	HF ID	Params	License	Runtime	Status	Notes
bge-m3	BAAI/bge-m3	0.6B	MIT	transformers	Candidate	Multilingual embeddings

2h. Image Generation

Model ID	HF ID	Params	License	Runtime	Status	Notes
flux.2-klein-4b	black-forest-labs/FLUX.2-klein-4B	4.0B	FLUX.2-dev NC	diffusers	Candidate	Visual card generation

---

3. Workflow → Model Mapping

Each product workflow requires one or more model capabilities. The table below shows the mapping, the default model that powers it, and alternatives for tradeoffs (speed vs. quality, local vs. cloud).

Product Workflow	Capabilities Required	Default Model (Local)	Alternatives
Today Dashboard	Planning (text gen)	llama-3.2-3b (MLX)	llama-3.2-3b-gguf (llama.cpp), minicpm5-1b (faster, lighter)
Shopping List	Planning, tool-call parsing	llama-3.2-3b (MLX)	lfm2.5-8b-a1b-gguf (higher quality), shopstack-parser-lora (future)
Market Lens	Vision, object detection, OCR, barcode	minicpm-v-8b (vision — candidate)	GPT-4o (cloud, best quality), Mock (dev fallback)
Voice Commands (Ask)	STT → Planning → Tool-call parse	local-whisper-tiny (STT) + llama-3.2-3b (planning)	sense-voice-small (faster STT), qwen3-asr-1.7b (higher quality STT)
Add Purchase	Planning (classification)	llama-3.2-3b	minicpm5-1b (lighter), Mock (no model needed for form mode)
Price Intelligence	Planning (analysis)	llama-3.2-3b	— (primarily SQL + heuristic, model optional)
Inventory View/Search	Embeddings (semantic search)	bge-m3 (candidate)	LocalProvider.embed() (zero-vector fallback)
Use Soon / Alerts	Heuristic (no model)	—	—
Household Map	Heuristic (no model)	—	—
Field Notes	Planning (summarization)	llama-3.2-3b	—
Trace Export	Heuristic (no model)	—	—
Barcode Decoding	pyzbar / zbarimg	System zbar	—

3a. Detailed Workflow Flow

Voice Command
  └─ SHOPSTACK_STT_BACKEND → transcribe audio
       └─ SHOPSTACK_PLANNER_BACKEND → parse intent, execute action
            └─ SHOPSTACK_PLANNER_BACKEND (tool-call parsing if enabled)

Market Lens Scan
  ├─ SHOPSTACK_VISION_BACKEND → detect objects in image
  ├─ System zbar → decode barcode
  ├─ SHOPSTACK_OCR_BACKEND → extract text (receipts, labels)
  └─ SHOPSTACK_PLANNER_BACKEND → classify buy/skip decisions

Shopping List Creation
  └─ SHOPSTACK_PLANNER_BACKEND → classify items (buy/skip/use_soon)
       └─ Swiggy price enrichment (external data, no model)

Price Intelligence
  └─ SHOPSTACK_PLANNER_BACKEND → optional analysis
       └─ DB query → heuristic comparison (no model required)

---

4. Active Stack — Budget & Status

The total active model parameter budget is capped at 32B params (MAX_ACTIVE_MODEL_PARAMS_B).

Currently Active

Model	Group	Params	Runtime	Backend Config	Notes
llama-3.2-3b-instruct (MLX)	Planner	3.0B	mlx	SHOPSTACK_PLANNER_BACKEND=local	Default on Apple Silicon
llama-3.2-3b-gguf	Planner	3.0B	gguf	(llama.cpp fallback)	493 ms / 49 tokens via llama.cpp
local-whisper-tiny (MLX)	STT	0.04B	mlx	SHOPSTACK_STT_BACKEND=local_whisper	On-demand model loading
Total active		≤ 6.04B			Well within 32B cap

Candidate Pipeline

Priority	Model	Group	Params	Runtime	Why
P0	minicpm-v-8b	Vision	8.0B	transformers	Enables local Market Lens
P0	bge-m3	Embeddings	0.6B	transformers	Semantic search for inventory
P1	qwen3-asr-1.7b	STT	1.7B	transformers	Higher quality local STT
P1	sense-voice-small	STT	0.2B	transformers	Faster multilingual STT
P1	nuextract3-4b	OCR	4.0B	transformers	Receipt scanning (non-commercial)
P2	qwen3-tts-0.6b	TTS	0.6B	transformers	Text-to-speech responses
P2	kokoro-82m	TTS	0.082B	custom	Ultra-lightweight TTS
P2	minicpm5-1b	Planner	1.0B	transformers	Lightweight planner
P3	lfm2.5-8b-a1b-gguf	Planner	8.3B	gguf	Higher-quality planning
P3	parakeet-0.6b	STT	0.6B	custom	Streaming ASR
P3	whisper-large-v3-turbo	STT	0.8B	transformers	Baseline STT benchmark
P3	rmbg-1.4	Segmentation	0.3B	transformers	Item card polish
P3	flux.2-klein-4b	Image Edit	4.0B	diffusers	Visual card generation
P4	shopstack-parser-lora	Planner	~0.1B	transformers	Fine-tuned command parser

Budget Projection

Active (P0 deployed):    6.04B params
P0 candidates:           + 8.6B  =  14.6B  ← next milestone target
P1 candidates:           + 5.9B  =  20.5B
P2 candidates:           + 1.68B =  22.2B
P3 candidates:           + 5.7B  =  27.9B
P4 fine-tune:            + 0.1B  ≈  28.0B  ← still under 32B cap

---

5. Env Configuration Reference

# ── Planner (text gen / planning) ─────────────
SHOPSTACK_PLANNER_BACKEND=local    # LocalProvider (MLX or llama.cpp)
# SHOPSTACK_PLANNER_BACKEND=openai # Cloud fallback (requires API key)

# ── STT (speech-to-text) ─────────────────────
# SHOPSTACK_STT_BACKEND=local_whisper  # On-device whisper
# SHOPSTACK_LOCAL_WHISPER_SIZE=tiny    # tiny / base / small / medium / large

# ── Cloud fallback (optional) ────────────────
# SHOPSTACK_OPENAI_API_KEY=sk-...

# ── Model paths ──────────────────────────────
# SHOPSTACK_LOCAL_MODEL_DIR=              # default: shopstack/data/models/
# SHOPSTACK_LOCAL_MODEL_REPO=unsloth/Llama-3.2-3B-Instruct-GGUF
# SHOPSTACK_LOCAL_MODEL_FILE=Llama-3.2-3B-Instruct-Q4_K_M.gguf
# SHOPSTACK_LOCAL_AUTO_DOWNLOAD=false     # auto-download GGUF if missing

# ── Off-the-grid (mock mode) ────────────────
SHOPSTACK_OFF_THE_GRID=false  # false = allow real backends

---

6. Adding a New Model

1. Add a ModelEntry to shopstack/model_registry.py
2. If the model powers a new capability, add a provider class + backend wiring in shopstack/providers/registry.py
3. If the model replaces an existing backend, update the SHOPSTACK_*_BACKEND env default in shopstack/config.py
4. Update this catalog with the new model's row and workflow mapping
5. Verify the parameter budget: total_active_params() <= 32B