Spaces:
Running
Running
| # Model Catalog — Usage-Based View | |
| > **Purpose:** Map every model in `shopstack/model_registry.py` to the product | |
| > workflows it powers. This is the decision maker's view: given a workflow, | |
| > which model handles it, and what are the alternatives? | |
| --- | |
| ## 1. Provider Capability Map | |
| Models enter the app through **provider backends**, each wired by | |
| `ShopStack.config.SHOPSTACK_*_BACKEND`. The table below shows which backend | |
| handles which capability group. | |
| | Backend Config | Provider Class | Capabilities | Runtime | Status | | |
| |---------------------------------|-------------------------|----------------------------------------------------------|----------------|-----------------| | |
| | `SHOPSTACK_PLANNER_BACKEND` | `LocalProvider` | Text generation, planning, embeddings | `mlx`/`llama.cpp` | Active | | |
| | `SHOPSTACK_PLANNER_BACKEND` | `OpenAIProvider` | Text generation, vision, embeddings | Cloud (API) | Available | | |
| | `SHOPSTACK_PLANNER_BACKEND` | `MockPlannerProvider` | Mock text, planning | Mock | Default (dev) | | |
| | `SHOPSTACK_STT_BACKEND` | `LocalWhisperProvider` | Speech-to-text | `mlx`/`faster-whisper` | Active | | |
| | `SHOPSTACK_STT_BACKEND` | `WhisperProvider` | Speech-to-text (cloud) | Cloud (API) | Available | | |
| | `SHOPSTACK_STT_BACKEND` | `MockSTTProvider` | Mock STT | Mock | Default (dev) | | |
| | `SHOPSTACK_VISION_BACKEND` | `OpenAIProvider` | Vision understanding (object detection, grounding) | Cloud (API) | Available | | |
| | `SHOPSTACK_VISION_BACKEND` | `MockVisionProvider` | Mock vision, detection | Mock | Default (dev) | | |
| | `SHOPSTACK_OCR_BACKEND` | `MockOCRProvider` | OCR/extraction | Mock | Default (dev) | | |
| | `SHOPSTACK_SEGMENTATION_BACKEND`| `MockSegmentationProvider` | Image segmentation | Mock | Default (dev) | | |
| | `SHOPSTACK_TTS_BACKEND` | `MockTTSProvider` | Text-to-speech | Mock | Default (dev) | | |
| | `SHOPSTACK_EMBEDDINGS_BACKEND` | `LocalProvider` / `OpenAIProvider` | Embeddings (planned fallback) | Shared | Inherited | | |
| | `SHOPSTACK_IMAGE_EDIT_BACKEND` | `MockImageEditProvider` | Image generation, annotation | Mock | Default (dev) | | |
| > **Key:** "Active" = wired to real local inference. "Available" = dependencies | |
| > installable. "Default (dev)" = mock provider used during development. | |
| --- | |
| ## 2. Full Model Registry | |
| All entries from `shopstack/model_registry.py`, organized by provider group. | |
| ### 2a. STT — Speech-to-Text | |
| | Model ID | HF ID | Params | License | Runtime | Status | Notes | | |
| |-------------------------|-----------------------------------------|--------|--------------|---------------|------------|--------------------------------------| | |
| | `local-whisper-tiny` | `mlx-community/whisper-tiny-mlx` | 0.04B | MIT | `mlx` | **Active** | Default local Whisper backend | | |
| | `qwen3-asr-1.7b` | `Qwen/Qwen3-ASR-1.7B` | 1.7B | Apache-2.0 | `transformers`| Candidate | Top candidate for household commands | | |
| | `parakeet-0.6b` | `nvidia/parakeet-ctc-0.6b` | 0.6B | CC-BY-4.0 | `custom` | Candidate | Lightweight streaming ASR | | |
| | `sense-voice-small` | `funasr/SenseVoiceSmall` | 0.2B | MIT | `transformers`| Candidate | Very fast, multilingual | | |
| | `whisper-large-v3-turbo`| `openai/whisper-large-v3-turbo` | 0.8B | MIT | `transformers`| Candidate | Baseline only | | |
| ### 2b. TTS — Text-to-Speech | |
| | Model ID | HF ID | Params | License | Runtime | Status | Notes | | |
| |------------------|----------------------------|---------|------------|---------------|-----------|------------------------------------| | |
| | `qwen3-tts-0.6b` | `Qwen/Qwen3-TTS-0.6B` | 0.6B | Apache-2.0 | `transformers`| Candidate | Lightweight TTS candidate | | |
| | `kokoro-82m` | — | 0.082B | Apache-2.0 | `custom` | Candidate | Extremely lightweight (`off_grid`) | | |
| ### 2c. Vision / Object Detection / Grounding | |
| | Model ID | HF ID | Params | License | Runtime | Status | Notes | | |
| |---------------------|----------------------------------|--------|------------|---------------|-----------|------------------------------------| | |
| | `minicpm-v-8b` | `openbmb/MiniCPM-V-2_6` | 8.0B | Apache-2.0 | `transformers`| Candidate | Strong VLM for household items | | |
| ### 2d. Planner — LLM / Text Generation | |
| | Model ID | HF ID | Params | License | Runtime | Status | Notes | | |
| |-------------------------|---------------------------------------------|--------|--------------------|---------------|------------|------------------------------------------| | |
| | `llama-3.2-3b-instruct` | `mlx-community/Llama-3.2-3B-Instruct-4bit` | 3.0B | Llama 3.2 Community| `mlx` | **Active** | Default MLX backend (auto-downloaded) | | |
| | `llama-3.2-3b-gguf` | `unsloth/Llama-3.2-3B-Instruct-GGUF` | 3.0B | Llama 3.2 Community| `gguf` | **Active** | Downloaded GGUF, llama.cpp fallback | | |
| | `minicpm5-1b` | `openbmb/MiniCPM5-1B` | 1.0B | Apache-2.0 | `transformers`| Candidate | Lightweight planner / parser | | |
| | `lfm2.5-8b-a1b-gguf` | `unsloth/LFM2.5-8B-A1B-GGUF` | 8.3B | Apache-2.0 | `gguf` | Candidate | GGUF planner for llama.cpp path | | |
| | `shopstack-parser-lora` | — | — | Apache-2.0 (planned)| `transformers`| Candidate | Future fine-tuned command parser (`well_tuned`) | | |
| ### 2e. OCR — Optical Character Recognition | |
| | Model ID | HF ID | Params | License | Runtime | Status | Notes | | |
| |---------------------|--------------------------------|--------|--------------|---------------|-----------|------------------------------------| | |
| | `nuextract3-4b` | `nuance/NuExtract3-4B` | 4.0B | CC-BY-NC-4.0 | `transformers`| Candidate | Strong receipt extraction (non-commercial) | | |
| ### 2f. Segmentation | |
| | Model ID | HF ID | Params | License | Runtime | Status | Notes | | |
| |-------------|--------------------------|--------|------------|---------------|-----------|----------------------------------------| | |
| | `rmbg-1.4` | `briaai/RMBG-1.4` | 0.3B | Apache-2.0 | `transformers`| Candidate | Background removal for item cards | | |
| ### 2g. Embeddings | |
| | Model ID | HF ID | Params | License | Runtime | Status | Notes | | |
| |-----------|--------------------|--------|---------|---------------|-----------|------------------------------------| | |
| | `bge-m3` | `BAAI/bge-m3` | 0.6B | MIT | `transformers`| Candidate | Multilingual embeddings | | |
| ### 2h. Image Generation | |
| | Model ID | HF ID | Params | License | Runtime | Status | Notes | | |
| |-----------------------|------------------------------------------|--------|------------------------|-------------|-----------|------------------------------------| | |
| | `flux.2-klein-4b` | `black-forest-labs/FLUX.2-klein-4B` | 4.0B | FLUX.2-dev NC | `diffusers` | Candidate | Visual card generation | | |
| --- | |
| ## 3. Workflow → Model Mapping | |
| Each product workflow requires one or more model capabilities. The table below | |
| shows the mapping, the **default** model that powers it, and **alternatives** | |
| for tradeoffs (speed vs. quality, local vs. cloud). | |
| | Product Workflow | Capabilities Required | Default Model (Local) | Alternatives | | |
| |-----------------------------|----------------------------------|-------------------------------------|-------------------------------------------------------------------------| | |
| | **Today Dashboard** | Planning (text gen) | `llama-3.2-3b` (MLX) | `llama-3.2-3b-gguf` (llama.cpp), `minicpm5-1b` (faster, lighter) | | |
| | **Shopping List** | Planning, tool-call parsing | `llama-3.2-3b` (MLX) | `lfm2.5-8b-a1b-gguf` (higher quality), `shopstack-parser-lora` (future)| | |
| | **Market Lens** | Vision, object detection, OCR, barcode | `minicpm-v-8b` (vision — candidate) | GPT-4o (cloud, best quality), Mock (dev fallback) | | |
| | **Voice Commands (Ask)** | STT → Planning → Tool-call parse | `local-whisper-tiny` (STT) + `llama-3.2-3b` (planning) | `sense-voice-small` (faster STT), `qwen3-asr-1.7b` (higher quality STT)| | |
| | **Add Purchase** | Planning (classification) | `llama-3.2-3b` | `minicpm5-1b` (lighter), Mock (no model needed for form mode) | | |
| | **Price Intelligence** | Planning (analysis) | `llama-3.2-3b` | — (primarily SQL + heuristic, model optional) | | |
| | **Inventory View/Search** | Embeddings (semantic search) | `bge-m3` (candidate) | `LocalProvider.embed()` (zero-vector fallback) | | |
| | **Use Soon / Alerts** | Heuristic (no model) | — | — | | |
| | **Household Map** | Heuristic (no model) | — | — | | |
| | **Field Notes** | Planning (summarization) | `llama-3.2-3b` | — | | |
| | **Trace Export** | Heuristic (no model) | — | — | | |
| | **Barcode Decoding** | `pyzbar` / `zbarimg` | System `zbar` | — | | |
| ### 3a. Detailed Workflow Flow | |
| ``` | |
| Voice Command | |
| └─ SHOPSTACK_STT_BACKEND → transcribe audio | |
| └─ SHOPSTACK_PLANNER_BACKEND → parse intent, execute action | |
| └─ SHOPSTACK_PLANNER_BACKEND (tool-call parsing if enabled) | |
| Market Lens Scan | |
| ├─ SHOPSTACK_VISION_BACKEND → detect objects in image | |
| ├─ System zbar → decode barcode | |
| ├─ SHOPSTACK_OCR_BACKEND → extract text (receipts, labels) | |
| └─ SHOPSTACK_PLANNER_BACKEND → classify buy/skip decisions | |
| Shopping List Creation | |
| └─ SHOPSTACK_PLANNER_BACKEND → classify items (buy/skip/use_soon) | |
| └─ Swiggy price enrichment (external data, no model) | |
| Price Intelligence | |
| └─ SHOPSTACK_PLANNER_BACKEND → optional analysis | |
| └─ DB query → heuristic comparison (no model required) | |
| ``` | |
| --- | |
| ## 4. Active Stack — Budget & Status | |
| The **total active model parameter budget** is capped at **32B params** | |
| (`MAX_ACTIVE_MODEL_PARAMS_B`). | |
| ### Currently Active | |
| | Model | Group | Params | Runtime | Backend Config | Notes | | |
| |--------------------------------|------------|--------|-----------|--------------------------|------------------------------------| | |
| | `llama-3.2-3b-instruct` (MLX) | Planner | 3.0B | `mlx` | `SHOPSTACK_PLANNER_BACKEND=local` | Default on Apple Silicon | | |
| | `llama-3.2-3b-gguf` | Planner | 3.0B | `gguf` | (llama.cpp fallback) | 493 ms / 49 tokens via llama.cpp | | |
| | `local-whisper-tiny` (MLX) | STT | 0.04B | `mlx` | `SHOPSTACK_STT_BACKEND=local_whisper` | On-demand model loading | | |
| | **Total active** | | **≤ 6.04B** | | | Well within 32B cap | | |
| ### Candidate Pipeline | |
| | Priority | Model | Group | Params | Runtime | Why | | |
| |----------|--------------------------------|------------|--------|---------------|-----------------------------------| | |
| | P0 | `minicpm-v-8b` | Vision | 8.0B | `transformers`| Enables local Market Lens | | |
| | P0 | `bge-m3` | Embeddings | 0.6B | `transformers`| Semantic search for inventory | | |
| | P1 | `qwen3-asr-1.7b` | STT | 1.7B | `transformers`| Higher quality local STT | | |
| | P1 | `sense-voice-small` | STT | 0.2B | `transformers`| Faster multilingual STT | | |
| | P1 | `nuextract3-4b` | OCR | 4.0B | `transformers`| Receipt scanning (non-commercial) | | |
| | P2 | `qwen3-tts-0.6b` | TTS | 0.6B | `transformers`| Text-to-speech responses | | |
| | P2 | `kokoro-82m` | TTS | 0.082B | `custom` | Ultra-lightweight TTS | | |
| | P2 | `minicpm5-1b` | Planner | 1.0B | `transformers`| Lightweight planner | | |
| | P3 | `lfm2.5-8b-a1b-gguf` | Planner | 8.3B | `gguf` | Higher-quality planning | | |
| | P3 | `parakeet-0.6b` | STT | 0.6B | `custom` | Streaming ASR | | |
| | P3 | `whisper-large-v3-turbo` | STT | 0.8B | `transformers`| Baseline STT benchmark | | |
| | P3 | `rmbg-1.4` | Segmentation| 0.3B | `transformers`| Item card polish | | |
| | P3 | `flux.2-klein-4b` | Image Edit | 4.0B | `diffusers` | Visual card generation | | |
| | P4 | `shopstack-parser-lora` | Planner | ~0.1B | `transformers`| Fine-tuned command parser | | |
| ### Budget Projection | |
| ``` | |
| Active (P0 deployed): 6.04B params | |
| P0 candidates: + 8.6B = 14.6B ← next milestone target | |
| P1 candidates: + 5.9B = 20.5B | |
| P2 candidates: + 1.68B = 22.2B | |
| P3 candidates: + 5.7B = 27.9B | |
| P4 fine-tune: + 0.1B ≈ 28.0B ← still under 32B cap | |
| ``` | |
| --- | |
| ## 5. Env Configuration Reference | |
| ```bash | |
| # ── Planner (text gen / planning) ───────────── | |
| SHOPSTACK_PLANNER_BACKEND=local # LocalProvider (MLX or llama.cpp) | |
| # SHOPSTACK_PLANNER_BACKEND=openai # Cloud fallback (requires API key) | |
| # ── STT (speech-to-text) ───────────────────── | |
| # SHOPSTACK_STT_BACKEND=local_whisper # On-device whisper | |
| # SHOPSTACK_LOCAL_WHISPER_SIZE=tiny # tiny / base / small / medium / large | |
| # ── Cloud fallback (optional) ──────────────── | |
| # SHOPSTACK_OPENAI_API_KEY=sk-... | |
| # ── Model paths ────────────────────────────── | |
| # SHOPSTACK_LOCAL_MODEL_DIR= # default: shopstack/data/models/ | |
| # SHOPSTACK_LOCAL_MODEL_REPO=unsloth/Llama-3.2-3B-Instruct-GGUF | |
| # SHOPSTACK_LOCAL_MODEL_FILE=Llama-3.2-3B-Instruct-Q4_K_M.gguf | |
| # SHOPSTACK_LOCAL_AUTO_DOWNLOAD=false # auto-download GGUF if missing | |
| # ── Off-the-grid (mock mode) ──────────────── | |
| SHOPSTACK_OFF_THE_GRID=false # false = allow real backends | |
| ``` | |
| --- | |
| ## 6. Adding a New Model | |
| 1. Add a `ModelEntry` to `shopstack/model_registry.py` | |
| 2. If the model powers a new capability, add a provider class + backend wiring | |
| in `shopstack/providers/registry.py` | |
| 3. If the model replaces an existing backend, update the `SHOPSTACK_*_BACKEND` | |
| env default in `shopstack/config.py` | |
| 4. Update this catalog with the new model's row and workflow mapping | |
| 5. Verify the parameter budget: `total_active_params() <= 32B` | |