# ShopStack Exploration Map

**Project:** ShopStack / GharStock  
**Purpose:** A living exploration map for agents and collaborators.  
**Rule:** Keep adding to this file whenever a model, product angle, dataset, architecture pattern, evaluation method, market idea, or risk deserves future investigation.

---

## 0. Product Summary

ShopStack is a small-model shopping copilot and household commerce memory layer.

It helps a household:

- know what is already at home,
- build shopping lists,
- scan shelves, markets, receipts, packets, fridges, and pantry spaces,
- ask questions by voice,
- decide buy / skip / compare / replace,
- add purchases into inventory,
- track freshness, expiry, price, quantity, and location,
- remember where items are stored,
- learn which stores are cheaper/better,
- understand travel, weather, timing, and convenience,
- create anonymized traces and field notes for the Build Small Hackathon.

The product is intentionally broad long-term, but every exploration should connect back to household shopping, inventory, market intelligence, or daily-use memory.

---

## 1. Capability Map

### 1.1 Language + Reasoning

Current / possible capabilities:

- NER for item, brand, unit, quantity, store, price, location, date.
- Intent classification.
- Command parsing.
- Tool-call planning.
- Natural-language inventory query.
- Shopping decision explanation.
- Multilingual / Hinglish / Indian household phrasing.
- Synonym and alias normalization.
- Structured JSON extraction.
- RAG over household memory.
- Safety disclaimers for nutrition/medicine/price uncertainty.

Exploration questions:

- Which small model is best at Indian household command parsing?
- Can a tiny fine-tuned parser outperform a larger general model on our exact commands?
- How much can rules + schema validation reduce model errors?
- Which model produces the cleanest JSON tool calls locally?
- Should we separate command parser and answer generator?

Candidate model families:

- MiniCPM / OpenBMB
- Qwen
- LFM / LiquidAI
- Gemma
- GPT-OSS
- Granite
- GLM
- Mistral / Voxtral for speech-heavy flows
- Llama / Phi GGUF models for llama.cpp badge

---

### 1.2 NER / Entity Extraction

Entities to extract:

- item name
- canonical item name
- item category
- brand
- local alias
- quantity
- unit
- price
- normalized unit price
- expiry date
- manufacturing date
- store
- location at home
- storage instruction
- nutrition facts
- household member preference
- purchase timestamp
- travel context
- weather context

Exploration questions:

- Should NER be rule-based, model-based, or hybrid?
- Can a fine-tuned small model map Hinglish utterances to canonical items?
- How do we handle ambiguous items like “Surf,” “Vim,” “Maggi,” “bread,” “pav,” “dahi,” “curd,” “yogurt”?
- Can we build a household-specific lexicon that improves over time?
- Should item names be canonicalized using embeddings?

Dataset ideas:

- Indian household item alias dataset.
- Grocery command dataset.
- Receipt-line-to-canonical-item dataset.
- Hinglish quantity normalization dataset.
- Expiry and storage instruction dataset.

---

### 1.3 Time Series

Time-series capabilities:

- consumption rate estimation,
- days-until-stockout,
- restock interval prediction,
- price trend by item,
- price trend by store,
- day-of-week price patterns,
- seasonality,
- freshness decay,
- expiry forecasting,
- travel effort over time,
- store quality over time,
- household category spend.

Exploration questions:

- Which items have predictable restock cycles?
- How much history is needed for useful next-buy predictions?
- Can we estimate stockout without exact daily consumption?
- What is the right confidence language: “likely low,” “probably enough,” “uncertain”?
- How do we handle multiple lots of the same item?
- Should predictions be rule-based initially and learned later?

Possible models / tools:

- statsmodels / Prophet-like approaches,
- DuckDB time-series queries,
- simple rolling averages,
- exponential smoothing,
- Bayesian inventory estimates,
- local notebooks / HF Jobs / Modal experiments.

---

### 1.4 Spatial Intelligence

Spatial layers:

- fridge shelf memory,
- pantry shelf memory,
- household location map,
- item last-seen memory,
- item movement events,
- room/shelf heatmap,
- “find this item” assistant,
- storage recommendation,
- household location graph,
- future AR/SLAM-like map.

Exploration questions:

- Can shelf/fridge photos become stable “location snapshots”?
- How do we identify the same item across days?
- Can we track “milk moved from shopping bag → fridge door”?
- How much user confirmation is needed?
- Should locations be user-defined first: fridge top shelf, pantry shelf 2, bathroom cabinet?
- Is a heatmap useful before AR/SLAM exists?
- Can we use image embeddings for “this looks like the same shelf”?

Possible tools:

- OpenCV,
- object detection,
- image embeddings,
- segmentation,
- H3/geospatial only for external market map,
- graph DB or SQLite graph tables,
- future WebAR / mobile capture.

---

### 1.5 Voice: STT, TTS, Audio

Voice capabilities:

- voice shopping list creation,
- voice correction,
- voice ask while shopping,
- voice answer in noisy market mode,
- voice-based item movement,
- voice-based price logging,
- Indian-language / Hinglish UX,
- audio confidence and retry UX.

STT candidates to evaluate:

- Qwen3-ASR-1.7B
- NVIDIA Parakeet / Nemotron 0.6B streaming ASR
- Mistral Voxtral Mini Realtime
- SenseVoiceSmall
- VibeVoice ASR
- Cohere Transcribe model
- Granite Speech
- Whisper large-v3-turbo baseline

TTS candidates to evaluate:

- MOSS-TTS-v1.5
- VoxCPM2
- Qwen3-TTS 0.6B / 1.7B
- Higgs Audio v3 TTS
- Kokoro-82M
- CosyVoice
- OmniVoice
- Chatterbox / XTTS-style models if useful

Exploration questions:

- Which model best handles Hinglish grocery commands?
- Can the response voice be short, warm, and market-friendly?
- Should we use browser/device TTS as fallback?
- How do we handle noisy market audio?
- Can audio be processed locally in the Space without cloud APIs?
- How do we benchmark STT/TTS quickly?

Benchmark phrases:

- “Doodh ghar pe hai kya?”
- “Tamatar aadha kilo add karo.”
- “Nahi, yeh pyaaz hai aloo nahi.”
- “Bread expiry kal ka hai, skip karo.”
- “Surf Excel already ghar pe hai kya?”
- “Isko pantry mein move karo.”
- “Kal breakfast ke liye kya hai?”

---

### 1.6 Vision Understanding

Vision capabilities:

- item detection,
- item classification,
- visual grounding,
- shelf/market scan,
- packet understanding,
- receipt understanding,
- fridge/pantry scan,
- freshness/ripeness hints,
- damaged/spoiled item detection,
- visual confirmation cards,
- annotated photos.

Candidate models / tools:

- Gemma multimodal models
- MiniCPM-V
- LocateAnything
- Qwen VL / image models
- RF-DETR
- YOLO variants
- Marlin-2B for video
- TimeLens / video grounding models
- OpenCV + OCR hybrids
- CLIP/embedding-based matching

Exploration questions:

- Which model works best on Indian market photos?
- Can open-vocabulary grounding identify “dhaniya,” “pav,” “dahi,” “atta”?
- Does a general VLM beat object detection for household goods?
- Can we use image crops + text model instead of one large VLM?
- How do we show uncertainty without frustrating users?

---

### 1.7 OCR / Extraction

OCR targets:

- receipts,
- packet labels,
- expiry dates,
- MRP,
- quantity,
- brand,
- nutrition facts,
- store name,
- receipt totals,
- bill line items,
- handwritten notes.

Candidate tools:

- PaddleOCR / PaddleOCR-VL
- NuExtract3
- Tesseract fallback
- Donut-like document models
- DocTR / LayoutLM-style pipelines
- OCR + LLM extraction hybrid

Exploration questions:

- Can receipt OCR handle local Indian store bills?
- How do we normalize quantities from OCR?
- Can we reliably detect expiry dates from packets?
- How do we distinguish MRP from sale price?
- Should packet OCR be a separate close-up mode?

---

### 1.8 Classification

Classification tasks:

- item category,
- storage location,
- food vs household vs medicine vs cleaning,
- perishable vs shelf-stable,
- urgent vs optional,
- buy / skip / compare,
- confidence class,
- freshness class,
- trace safety/redaction class,
- price anomaly class,
- user intent class.

Exploration questions:

- Which classifications can be rules?
- Which need fine-tuning?
- Should we use a lightweight classifier before LLM calls?
- Can the fine-tuned model handle both intent and item category?
- How do we evaluate classification in Field Notes?

---

### 1.9 Segmentation / Grounding

Segmentation capabilities:

- product crop cards,
- item cutouts,
- shelf zones,
- fridge zones,
- visual confirmation,
- background removal,
- annotated maps.

Candidate tools:

- RMBG
- BiRefNet
- ClipSeg
- SAM variants if feasible
- YOLO segmentation
- RF-DETR segmentation
- LocateAnything grounding
- OpenCV masks

Exploration questions:

- Is segmentation necessary for every flow, or only review cards?
- Which model runs reliably in Spaces?
- Can segmentation improve user trust?
- Can shelf-zone segmentation power household spatial memory?
- How should we handle overlapping groceries?

---

### 1.10 Image Generation / Editing

Use cases:

- annotated shopping photos,
- item cards,
- shelf maps,
- “use soon” visual cards,
- printable pantry labels,
- shopping summary posters,
- household heatmap illustrations,
- price comparison cards,
- Field Notes visuals.

Candidate models:

- Black Forest Labs FLUX.2-klein-4B
- FLUX.2-klein-9B
- Qwen Image Edit
- ControlLight
- Lightweight PIL/HTML card generation
- SVG-based generated layouts

Exploration questions:

- Is image generation useful enough for the product, or should deterministic visual cards come first?
- Can generated cards improve sharing and polish?
- How do we keep image edits from hallucinating wrong product details?
- Should all factual text be rendered by code, not generated into image pixels?

---

### 1.11 Embeddings / Retrieval

Embedding uses:

- item alias matching,
- receipt line matching,
- memory search,
- similar purchase recall,
- store note retrieval,
- trace retrieval,
- product substitute matching,
- voice phrase similarity,
- user preference retrieval,
- location snapshot matching.

Candidate embedding models:

- Qwen embeddings
- MiniLM / sentence-transformers
- multilingual E5 family
- BGE multilingual
- Jina embeddings
- small local embedding models
- CLIP / SigLIP for image similarity

Exploration questions:

- Which multilingual embedding model handles Hinglish item aliases best?
- Should text and image embeddings share one store?
- Is SQLite vector extension enough?
- Should we use FAISS, LanceDB, Chroma, or DuckDB extensions?
- Can embeddings help trace retrieval for Sharing is Caring?

---

### 1.12 Graph / Linkage Memory

Nodes:

- Item
- ItemLot
- Store
- MarketArea
- HouseholdLocation
- Shelf
- FridgeZone
- PurchaseEvent
- PriceObservation
- ShoppingTrip
- WeatherContext
- UserPreference
- Recipe
- Trace
- ModelRun

Edges:

- bought_at
- stored_in
- moved_to
- consumed_by
- expires_before
- substitutes
- usually_bought_with
- preferred_by
- cheaper_at
- best_quality_at
- seen_in_photo
- mentioned_in_voice
- derived_from_receipt
- weather_affected
- route_to
- worth_travelling_for

Exploration questions:

- Is a graph DB needed, or can SQLite edge tables do enough?
- Which queries need graph traversal?
- Can graph memory make the product feel smarter quickly?
- How do we visualize this without making it technical?

---

## 2. User Use Case Map

### 2.1 Before Shopping

Use cases:

- create list from voice,
- check what is already at home,
- predict next buys,
- suggest what to buy for meals,
- avoid buying duplicates,
- choose store based on list,
- compare travel effort,
- account for weather,
- generate shopping route,
- plan by budget.

Questions:

- “What should I buy today?”
- “What is low at home?”
- “Can we make dinner without shopping?”
- “Which store should I go to?”
- “Is the Sunday market worth it today?”

---

### 2.2 During Shopping

Use cases:

- scan shelf,
- ask if item is needed,
- compare with home inventory,
- compare price with memory,
- check expiry,
- identify unknown item,
- quantity advice,
- substitution advice,
- allergy/preference warning,
- budget warning.

Questions:

- “Do I need this?”
- “Is this price okay?”
- “Which one should I pick?”
- “Is this enough?”
- “Do we already have this?”
- “Is this near expiry?”
- “Can I skip this?”

---

### 2.3 After Shopping

Use cases:

- purchase photo ingestion,
- receipt ingestion,
- inventory update,
- expiry tracking,
- price observation logging,
- store rating,
- trip context logging,
- trace export,
- family summary.

Questions:

- “Add all this.”
- “What expires first?”
- “What did we spend?”
- “Where should this go?”
- “Did we overbuy anything?”

---

### 2.4 At Home

Use cases:

- find items,
- check stock,
- plan meals,
- use-soon reminders,
- move item location,
- consume item,
- estimate remaining quantity,
- household member asks questions,
- shelf/fridge scan,
- spatial memory.

Questions:

- “Where is the dahi?”
- “Do we have detergent?”
- “What should we use today?”
- “What is in the fridge?”
- “Move toothpaste to bathroom cabinet.”
- “How much rice is left?”

---

### 2.5 Market Intelligence

Use cases:

- price trends,
- store ranking,
- cheapest location,
- freshness/quality memory,
- travel-time decision,
- weather-aware recommendation,
- route-aware shopping,
- neighborhood price map,
- Market Intelligence Graph / Market Map: home inventory + live market cards + price memory + substitutions + freshness risk,
- snapshot-based benchmarking from real market dataset imports.

Questions:

- “Where was tomato cheapest?”
- “Is this price high?”
- “Which store is better for fruits?”
- “Should I travel to the market today?”
- “What did we learn from last month’s shopping?”
- “How much can a Swiggy price snapshot improve price memory and trip decisions?”

---

## 3. Build Small Hackathon Constraint Map

Non-negotiables:

- total loaded model parameters must be <= 32B,
- app must be built on Gradio,
- app must be hosted as a Hugging Face Space,
- short walkthrough video and social post required,
- main track should show a real person / real problem,
- Codex is a parallel track, not the product.

Bonus quests:

- Off the Grid: no cloud APIs in runtime path.
- Well-Tuned: use a fine-tuned model published on HF.
- Off-Brand: custom frontend beyond default Gradio.
- Llama Champion: run a model through llama.cpp.
- Sharing is Caring: publish anonymized agent traces.
- Field Notes: write report/blog about what was built and learned.

Exploration questions:

- Which bonus quests are product-aligned?
- How do we claim Off the Grid while using credits for build-time jobs?
- How do we make llama.cpp visible in the app/report?
- What is the smallest useful fine-tune?
- How should traces be anonymized?

---

## 4. Sponsor Alignment Map

### Hugging Face + Gradio

Explore:

- Spaces deployment,
- Space README metadata,
- dataset/model linking,
- model publishing,
- dataset publishing,
- Jobs,
- GPU Spaces,
- custom Gradio Blocks UI,
- Gradio API endpoints,
- traces as datasets.

### OpenBMB

Explore:

- MiniCPM5-1B as parser/planner,
- MiniCPM5-1B-GGUF for llama.cpp badge,
- MiniCPM-V for vision,
- VoxCPM2 for TTS,
- OpenBMB special category angle.

### OpenAI / Codex

Explore:

- Codex-attributed commits,
- AGENTS.md,
- codex build log,
- tests and docs generated/reviewed by Codex,
- public GitHub repo,
- “Built with Codex” README section,
- Codex as engineering lane only.

### NVIDIA

Explore:

- LocateAnything for grounding,
- Parakeet/Nemotron ASR,
- GPU experiments,
- accelerated vision/audio workflows,
- possible RTX 5080 relevance in final story.

### Modal

Explore:

- fine-tuning jobs,
- benchmark jobs,
- model comparison runs,
- trace generation,
- quantization,
- batch inference,
- dataset generation.

### Black Forest Labs

Explore:

- FLUX image edit / visual cards,
- use-soon cards,
- shelf maps,
- shopping summaries,
- annotated item visuals.

### Cohere

Explore:

- ASR/transcription model comparison,
- embeddings/reranking comparisons if available,
- not required for local-first runtime.

---

## 5. Dataset Exploration Map

Potential datasets to create/publish:

1. Indian household shopping utterances.
2. Hinglish grocery command parser dataset.
3. Item alias/canonicalization dataset.
4. Purchase photo annotation dataset.
5. Receipt OCR/extraction dataset.
6. Packet label/expiry extraction dataset.
7. Inventory tool-call traces.
8. Market decision traces.
9. Price observation synthetic dataset.
10. Shelf/fridge location memory dataset.
11. Voice benchmark dataset.
12. TTS pronunciation phrase set.
13. Store memory schema examples.
14. Redacted agent trace dataset for Sharing is Caring.
15. Field Notes dataset with examples and failure modes.
16. Swiggy Instamart fresh vegetables snapshot dataset for price/unit-price benchmarking and market intelligence experiments.

Dataset quality questions:

- What can be public?
- What must be synthetic?
- What must be anonymized?
- What needs user consent?
- Which datasets help Well-Tuned most?
- Which datasets help judges understand the product?

## 5.1 Decision‑First Today / ShopStack Topic Map

A decision-first Today experience should be built around the questions users actually ask in the moment, not around tabs.

- What I have: household stock and location memory for pantry, fridge, and kitchen items.
- What I should buy: buy recommendations shaped by stock, waste risk, price snapshots, and market intelligence.
- What I should skip: overbuy warnings, duplicate items, and suggestions to avoid waste.
- What I should use soon: use-soon actions, meal prompts, and practical expiry-aware reminders.
- My own list: personal shopping list, voice list edits, and familiar/repeat item memory.
- Compare prices / offers: price/unit-price benchmarking, Swiggy Instamart fresh vegetables snapshot comparisons, and store choice tradeoffs.
- Market basket: basket-level intelligence, category budget signals, and market snapshot analytics.
- What needs confirmation: uncertain OCR/image matches, missing quantities, and low-confidence items.
- What I usually buy: recurring purchase patterns, cadence, and habitual home shopping behavior.
- What I’m overbuying / wasting: waste risk, stale stock alerts, and spend leakage from items that are not being used.

This should explicitly tie the Today dashboard to exploration-level research, including the Swiggy Instamart fresh vegetables snapshot dataset and market intelligence experiments.

**2026-06-08 Update:** The decision-first navigation has been implemented. The 15 flat tabs were restructured into 5 primary tabs (Today → Basket → ShopLens → Reconcile → Memory) with sub-tabs preserving all functionality. See DR-005 in DECISION_RECORDS.md for the full rationale and migration record. The product now opens to Today (decision-first dashboard with integrated Ask) rather than a flat feature catalog. Next exploration targets:
- Wire purchase cadence and waste prevention widgets into the Today dashboard so it answers "what should I do today?" without switching tabs.
- Surface Basket-level decision classification (buy/skip/compare/use-soon) more prominently in the Today view.
- Track which sub-tabs get the most real-world usage to validate or adjust the navigation hierarchy.

---

## 6. Evaluation Map

### Product Evaluations

- Can the user create a list by voice?
- Can the app detect relevant visible items?
- Can the app correctly say buy/skip?
- Can it add confirmed purchases?
- Can it answer inventory questions?
- Can it find item locations?
- Can it explain uncertainty?

### Model Evaluations

- STT exactness and intent retention.
- TTS clarity and warmth.
- OCR field extraction accuracy.
- object detection recall.
- segmentation usability.
- tool-call JSON validity.
- parser intent accuracy.
- latency.
- memory use.
- parameter count.
- install/deploy pain.
- local-first compatibility.

### Trace Evaluations

- trace completeness,
- privacy redaction,
- reproducibility,
- educational value,
- tool-call correctness.

### Field Notes Evaluations

- real user used it,
- what worked,
- what failed,
- what changed after feedback,
- small-model fit,
- honest limitations.

---

## 7. Marketing / Positioning Exploration

Possible positioning:

- “Remember what’s at home while you shop.”
- “A shopping copilot for Indian homes.”
- “Photo + voice inventory for everyday shopping.”
- “Your fridge, pantry, market, and shopping list in one memory.”
- “Small models for small household decisions.”
- “The home commerce memory layer.”
- “Not another grocery app — a memory for what you buy, where it goes, and when to buy again.”

Potential audiences:

- Indian families,
- parents,
- students/hostels,
- shared flats,
- home cooks,
- small kirana shoppers,
- apartment households,
- elderly users,
- caregivers,
- domestic helpers managing stock,
- local sellers later as adjacent market.

Potential channels:

- hackathon demo,
- YouTube build stream,
- Twitter/X build thread,
- LinkedIn product post,
- HF Space leaderboard,
- Gradio Discord,
- Indian tech/ProductHunt style launch,
- Reddit India/frugal/mealprep communities,
- WhatsApp family-group proof-of-use story.

Exploration questions:

- Is “ShopStack” too general or perfect for scaling?
- Should the India-local layer be in the tagline, not name?
- Can demo be recorded with real household shopping?
- What short video moment makes people immediately understand?
- Which poster/screenshot is most shareable?

---

## 8. Risk / Constraint Exploration

Risks:

- too broad,
- over-reliance on imperfect vision,
- noisy market audio,
- privacy concerns,
- live price unreliability,
- too many models in one Space,
- 32B parameter accounting,
- non-commercial model licenses,
- Gradio UI becoming complex,
- agent anchoring to a reduced scope,
- overclaiming accuracy,
- app feeling like a demo instead of product direction.

Mitigations:

- confirmation-first UX,
- model registry,
- local-first mode,
- connected modes clearly separated,
- visible uncertainty,
- trace redaction,
- field notes,
- provider interfaces,
- benchmark scripts,
- privacy-first README,
- no auto-purchase,
- no medical/diet/legal claims,
- no private data in repo.

---

## 9. Open Questions

Agents should keep adding questions here.

1. Which model stack gives the best local-first performance under 32B?
2. Which model is easiest to run through llama.cpp for the parser?
3. What exact fine-tuned model should be published for Well-Tuned?
4. How much real household data can be safely used?
5. What is the first public trace dataset schema?
6. How should we count parameters when multiple models are optional but not loaded together?
7. Can Gradio handle the desired custom UI without too much friction?
8. Which STT model handles Hinglish best?
9. Which TTS model sounds warm enough for household use?
10. Should we use OCR or VLM-first for receipts?
11. How do we estimate quantity from photos without overclaiming?
12. Should price intelligence be mostly manual-memory-first?
13. How much map/heatmap functionality belongs in the app surface?
14. What is the simplest useful household map?
15. How do we benchmark live market/shelf scans?
16. What should be in the walkthrough video?
17. What should the social post emphasize?
18. What should the Field Notes title be?
19. How can Codex involvement be made obvious and authentic?
20. What is the strongest sponsor-alignment story?

---

## 10. Parking Lot

Use this for anything that may be interesting later.

- AR mode for finding items at home.
- Barcode scanning.
- Local store loyalty memory.
- Recipe planning from inventory.
- Waste tracking.
- Family member preferences.
- Domestic helper voice workflow.
- Shared household mode.
- WhatsApp integration.
- Calendar/reminder integration.
- Price community map.
- Privacy-preserving neighborhood price sharing.
- Offline mobile app.
- Browser extension for online grocery carts.
- Email receipt ingestion.
- Smart label printing.
- Shelf-life prediction from images.
- Freshness/ripeness model.
- Food waste report.
- Shopping carbon/effort score.
- Festival shopping planning.
- Monthly household budget intelligence.
- Elder-friendly voice-only mode.
- Accessibility mode for low-vision shoppers.
- Agentic shopping comparison, with user confirmation only.
- Local-language onboarding.
- Synthetic data generation pipeline.
- Human review UI for fine-tune data.
- Public leaderboard for household command parsing models.
- Sponsor-specific benchmark tables.
- Model replacement changelog.

---

## 11. Agent Contribution Protocol

When an agent adds an exploration item:

1. Add it under the relevant section.
2. Include why it matters for ShopStack.
3. Add model/tool links if known.
4. Mark whether it affects:
   - product,
   - model stack,
   - dataset,
   - evaluation,
   - UI,
   - privacy,
   - sponsor alignment,
   - bonus quest,
   - marketing.
5. Do not delete old ideas unless they are unsafe or clearly obsolete.
6. Move rejected ideas to a “rejected/paused” note with reason.
7. Keep this map broad; implementation tasks belong in task docs, not here.
8. Avoid anchoring language that frames the product as temporary or small.
9. Keep hackathon constraints visible.
10. Prefer swappable interfaces over hardcoded models.

Template:

```md
### Idea / Model / Tool / Angle

**Category:** model / product / dataset / marketing / eval / privacy / sponsor / UI  
**Why it matters:**  
**How to test:**  
**Risks:**  
**Links:**  
**Status:** explore / test / adopt / pause / reject  
```

---

## 12. Current Strongest Directions

1. Voice-first shopping list and correction.
2. Vision-based market/shelf scan.
3. Purchase photo ingestion.
4. Inventory and freshness memory.
5. Household spatial memory.
6. Price and store memory.
7. Fine-tuned Indian household command parser.
8. llama.cpp parser/planner.
9. Anonymized trace dataset.
10. Field Notes with real household use.
11. Off-brand custom Gradio UI.
12. Model benchmarking and replacement policy.

---

_Last updated: 2026-06-05_

---

## 13. Explorer Addendum — Product and Architecture Discovery Backlog (2026-06-06)

**Source mix:** current docs, current package shape, test inventory, product hardening notes, and first-principles household-commerce analysis.  
**Status:** exploration backlog; not implementation commitment.  
**Update policy:** append-only unless an item is later superseded with a dated note.

### 13.1 Decision Service Extraction: make Gradio an adapter, not the brain

**Category:** architecture / product / evaluation / UI  
**Why it matters:** Current docs already identify shopping service extraction as the canonical direction. The same boundary should cover Market Lens, Today dashboard, inventory recommendations, trace composition, and price intelligence so decisions are testable without UI wiring. This turns ShopStack from a demo UI into a durable household decision engine.  
**How to test:** Create service-level tests that accept typed inputs and return typed decision objects; UI tests should only verify rendering/wiring.  
**Risks:** Over-abstracting before the domain stabilizes. Keep services narrow and behavior-first.  
**Status:** adopt incrementally.

### 13.2 Confidence-calibrated household recommendations

**Category:** product / model stack / evaluation / UI  
**Why it matters:** “Buy,” “skip,” and “use soon” decisions need calibrated confidence language because household data is incomplete: unknown exact quantities, stale market prices, missing expiry dates, and approximate consumption rates. A clear uncertainty model increases trust and avoids fake precision.  
**How to test:** Build fixtures for complete, partial, stale, and contradictory household states; assert both recommendation and confidence wording.  
**Risks:** Too many caveats can make the product feel timid. Use concise labels: confident / likely / uncertain / needs confirmation.  
**Status:** explore → adopt.

### 13.3 Household memory quality score

**Category:** product / data / UI / evaluation  
**Why it matters:** The app should tell users whether its memory is good enough to trust. A “memory freshness / completeness” score can surface stale inventory, unpriced items, missing locations, uncertain quantities, and old market snapshots.  
**How to test:** Score deterministic seeded households with known gaps; verify the score changes after purchases, consumption, imports, and stale data.  
**Risks:** A score without actionable fixes becomes vanity UI. Pair every low score with the next best cleanup action.  
**Status:** explore.

### 13.4 Receipt and invoice ingestion pipeline

**Category:** product / vision / OCR / dataset / privacy  
**Why it matters:** Manual purchase entry is high friction. Receipts, app invoices, WhatsApp order summaries, and email receipts can populate inventory, price memory, store memory, and purchase cadence in one action.  
**How to test:** Start with local sample receipt text/images; evaluate extraction into PurchaseEvent + PriceObservation + InventoryLot candidates with user confirmation.  
**Risks:** PII leakage, messy receipt formats, and false inventory additions. Keep local-first, redacted, and confirmation-backed.  
**Status:** explore.

### 13.5 Post-shopping reconciliation flow

**Category:** product / UI / data integrity  
**Why it matters:** A shopping list is not complete until the household confirms what was actually bought, skipped, substituted, or already found at home. This flow would close the loop between plan → purchase → inventory → price memory.  
**How to test:** Simulate a list with bought/partial/substituted/skipped items and assert inventory lots, price observations, shopping list state, and traces update together.  
**Risks:** Too much bookkeeping. Use batch confirmation with sensible defaults.  
**Status:** adopt candidate.

### 13.6 Substitution intelligence

**Category:** product / model stack / dataset / evaluation  
**Why it matters:** Real shopping involves substitutions: tomato types, milk sizes, dal brands, detergent pack sizes, produce quality, and budget alternatives. ShopStack should recommend acceptable swaps based on household preference, price, unit price, shelf life, and recipe intent.  
**How to test:** Build a substitution fixture set with item families, unacceptable swaps, dietary constraints, and unit-price comparisons.  
**Risks:** Bad substitutions can break recipes or preferences. Require confirmation and learn from rejections.  
**Status:** explore.

### 13.7 Waste-aware meal/use suggestions

**Category:** product / model stack / UI / marketing  
**Why it matters:** “Use soon” becomes more valuable when it suggests practical actions: “use spinach tonight,” “make raita with curd,” “freeze bread,” or “move onions out of fridge.” This ties inventory memory to waste reduction.  
**How to test:** Fixture expiring items and assert suggestions respect available pantry items, shelf life, household preferences, and safety disclaimers.  
**Risks:** Recipe hallucination and unsafe food advice. Keep suggestions simple, local, and confidence-tagged.  
**Status:** explore.

### 13.8 Store reliability and quality memory

**Category:** product / data / evaluation / UI  
**Why it matters:** Cheapest is not always best. Produce quality, delivery reliability, substitutions, stale items, missing items, and refund friction should influence future recommendations.  
**How to test:** Add store observations and verify recommendation ranking balances price, quality, freshness, travel/time, and confidence.  
**Risks:** Sparse data can bias rankings unfairly. Show “based on N observations.”  
**Status:** explore.

### 13.9 Freshness-first market data layer

**Category:** data / architecture / UI / privacy  
**Why it matters:** Swiggy snapshots are useful but point-in-time. Every price/availability answer should carry source, captured-at timestamp, stale-data warning, and fallback behavior when freshness is insufficient.  
**How to test:** Snapshot fixtures at fresh/stale/expired ages; assert UI and services never present old prices as live.  
**Risks:** Over-warning can reduce usefulness. Use compact timestamp badges.  
**Status:** adopt.

### 13.10 Local model routing policy by task risk

**Category:** model stack / architecture / evaluation / privacy  
**Why it matters:** Not every task needs the same model. Deterministic parsers can handle quantity/unit extraction, embeddings can handle alias matching, and larger local planners can handle explanation. Routing reduces latency and improves reliability under the 32B cap.  
**How to test:** Benchmark task families separately: command parse, receipt parse, recommendation explanation, alias match, and trace summarization.  
**Risks:** Too many moving parts. Keep routing explicit and inspectable in Model Stack.  
**Status:** explore.

### 13.11 Household-specific alias and preference learning

**Category:** product / dataset / model stack / privacy  
**Why it matters:** Every household has names like “red dal,” “Amul wala dahi,” “kids bread,” “big Surf,” or “Friday sabzi.” A local adaptive lexicon can make voice and search feel personal without cloud training.  
**How to test:** Record accepted corrections and rejected matches; assert future parsing improves while preserving auditability.  
**Risks:** Wrong learned aliases can poison memory. Require reversible alias review.  
**Status:** explore.

### 13.12 Inventory state reconciliation and contradiction detection

**Category:** data integrity / architecture / UI / evaluation  
**Why it matters:** Household state will drift: items consumed without logging, duplicates, stale lots, impossible negative quantities, expired-but-used items, and contradictory locations. The app needs a cleanup assistant rather than silently accumulating bad memory.  
**How to test:** Seed contradictory databases and assert the assistant proposes safe, confirmable fixes without destructive writes.  
**Risks:** Cleanup flows can feel accusatory or tedious. Make them lightweight: “Is this still true?”  
**Status:** explore.

### 13.13 Trace replay and evaluation harness

**Category:** evaluation / architecture / model stack / privacy  
**Why it matters:** Existing anonymized traces can become a regression harness: replay planner inputs, compare tool calls, verify redaction, and benchmark model/provider changes before adoption.  
**How to test:** Store golden trace fixtures and assert parser/tool-call outputs remain valid across model/config changes.  
**Risks:** Golden traces can freeze bad behavior. Version them and review failures for intended improvements.  
**Status:** adopt candidate.

### 13.14 Offline-first backup, restore, and household portability

**Category:** product / privacy / architecture / operations  
**Why it matters:** Local-first users need safe backup/restore, device transfer, and human-readable exports. Portability already exists; next exploration is reliable restore semantics, conflict handling, and encrypted backups.  
**How to test:** Round-trip realistic households through export/import, including duplicate items, traces, locations, prices, and market snapshots.  
**Risks:** Bad imports can corrupt the source of truth. Use dry-run diff and confirmation before writes.  
**Status:** explore.

### 13.15 Mobile capture path without losing local-first guarantees

**Category:** product / UI / architecture / privacy  
**Why it matters:** Fridge scans, market scans, receipt photos, barcode scans, and voice commands are naturally mobile. The Gradio UI is useful for prototype velocity, but the product direction likely needs a mobile capture layer that syncs to local household memory.  
**How to test:** Prototype mobile-friendly capture screens first; later evaluate a native/offline shell or local network companion.  
**Risks:** Mobile sync can break the local-first trust model. Keep explicit sync boundaries and exportable data.  
**Status:** explore.

### 13.16 Barcode and package knowledge base

**Category:** product / dataset / vision / UI  
**Why it matters:** Barcodes can reduce ambiguity for packaged goods and enrich inventory with brand, size, category, nutrition, and repeat-purchase memory. A local cache avoids repeated lookups.  
**How to test:** Build local barcode fixtures for common household goods and assert scan → candidate item → confirmation → inventory path.  
**Risks:** Public barcode databases may be incomplete or noisy. Treat lookup as candidate data, not truth.  
**Status:** explore.

### 13.17 Pantry economics and household budget intelligence

**Category:** product / commercial / evaluation / UI  
**Why it matters:** ShopStack can become a household operating system for grocery spend: price inflation by item, category budgets, bulk-buy payoff, waste cost, and store choice tradeoffs. This is a strong retention surface beyond reminders.  
**How to test:** Generate monthly fixture data and assert summaries match known spend, savings, waste, and stockout events.  
**Risks:** Budget advice can feel judgmental. Use neutral, practical language.  
**Status:** explore.

### 13.18 Safety boundaries for food, medicine, and sensitive household data

**Category:** safety / privacy / product / evaluation  
**Why it matters:** Household inventory can include medicines, baby food, allergens, alcohol, and sensitive routines. The product needs explicit boundaries for advice, logging, redaction, and exports.  
**How to test:** Add fixtures for allergens, expired medicine, baby formula, and personal data; assert conservative warnings and redaction.  
**Risks:** Over-blocking benign grocery flows. Scope warnings to genuinely sensitive categories.  
**Status:** explore → policy.

### 13.19 “Why this recommendation?” explanation cards

**Category:** product / UI / evaluation / trust  
**Why it matters:** Users should see the concrete evidence behind decisions: at-home stock, last purchase, expected use rate, price snapshot age, store comparison, expiry/waste signal, and confidence. This improves trust and makes model behavior auditable.  
**How to test:** Snapshot render decision cards for representative buy/skip/use-soon/substitute cases.  
**Risks:** Cards can become verbose. Use progressive disclosure.  
**Status:** adopt candidate.

### 13.20 Market Intelligence Graph / Market Map

**Category:** product / data / evaluation / UI  
**Why it matters:** The next unique ShopStack wedge is not a plain price comparator. It is a living household graph that connects what is at home, what the market shows, what is stale, what is overpriced, what is already covered by inventory, what should be substituted, and what is likely to be wasted. This turns ShopCompare + ShopMemory + ShopStock into one decision surface.  
**How to test:** Build fixtures with stale snapshots, sponsored cards, sold-out items, combo packs, partial inventory overlap, and substitution candidates; assert the graph produces stable buy / skip / use-soon / compare / wait / substitute lanes and truth-score labels.  
**Risks:** If the graph reuses retailer language too directly, it can drift back into a shopping ad surface. Keep the default ranking household-first and explicitly label stale or sponsored signals.  
**Status:** adopt candidate.

### 13.20 Launch story: from inventory app to household commerce memory

**Category:** marketing / product / positioning  
**Why it matters:** “Inventory app” sounds like chores. “Household commerce memory” is more distinctive: it remembers what you have, what you paid, where it is, what spoils, and what to buy next. This framing should guide demos, docs, and feature prioritization.  
**How to test:** Compare landing/demo narratives: inventory tracker vs shopping copilot vs household commerce memory; evaluate which makes the product feel inevitable and differentiated.  
**Risks:** Too abstract for first-time users. Lead with concrete daily workflows, then name the deeper system.  
**Status:** explore.

### 13.21 Data source expansion map

**Category:** data / product / privacy / evaluation  
**Why it matters:** Swiggy fresh vegetables is a strong start, but durable market intelligence needs multiple source types: user-entered prices, receipts, quick-commerce snapshots, supermarket invoices, local kirana observations, barcode/package metadata, and seasonal produce calendars.  
**How to test:** Define a normalized source contract and fixture each source type into PriceObservation/MarketSnapshot without UI assumptions.  
**Risks:** Scraping or terms-of-service risk for live platforms. Prefer user-provided exports/snapshots and explicit provenance.  
**Status:** explore.

### 13.22 Operator/developer diagnostics tab

**Category:** operations / UI / architecture / evaluation  
**Why it matters:** Local-first apps need inspectability: DB path, provider mode, model availability, snapshot freshness, trace counts, redaction status, and recent errors. A diagnostics view makes failures understandable without cloud logs.  
**How to test:** Mock provider/model/data states and assert the diagnostics view surfaces actionable status.  
**Risks:** Too much internal detail in user UI. Gate as “Diagnostics” or “Local system status.”  
**Status:** explore.

### 13.23 Evaluation leaderboard for small local household models

**Category:** model stack / evaluation / sponsor / marketing  
**Why it matters:** The project can produce a reusable benchmark for local household agents: Hinglish command parsing, tool-call JSON validity, receipt extraction, alias matching, and recommendation explanation. This supports model choice and gives the project external credibility.  
**How to test:** Build a small, versioned eval dataset from synthetic + anonymized traces; run candidate local models with latency, validity, and correction metrics.  
**Risks:** Public benchmark data must not leak household details. Keep anonymization and synthetic generation strict.  
**Status:** explore.

### 13.24 Human correction as the primary learning loop

**Category:** product / dataset / model stack / UI  
**Why it matters:** The fastest path to a trustworthy household agent is not autonomous action; it is fast correction capture. Every “no, this is onions,” “skip this brand,” or “we call this atta” should improve future behavior locally.  
**How to test:** Build correction fixtures and assert corrections update aliases/preferences/traces without mutating unrelated history.  
**Risks:** Correction UI can interrupt flow. Offer one-tap corrections at decision points.  
**Status:** explore.

### 13.25 Household graph model

**Category:** architecture / data / product  
**Why it matters:** Items, lots, locations, stores, brands, prices, household members, preferences, recipes, and traces form a graph. SQLite can still store this, but a graph-shaped mental model helps avoid one-off tables and clarifies future recommendations.  
**How to test:** Draw core relationships and map them to current tables; identify missing edges such as item↔alias, item↔substitute, store↔quality, member↔preference.  
**Risks:** Premature graph infrastructure. Keep SQLite; use the graph as domain design, not necessarily a graph database.  
**Status:** explore.

### 13.26 Market graph-to-basket compare actions

**Category:** product / UI / architecture / evaluation  
**Why it matters:** The Market Intelligence Graph should not stay isolated. Compare lanes need direct “compare with home” and “compare with memory” actions so the graph can feed the shopping list and basket workflow, not just display intelligence.  
**How to test:** Build compare-lane fixtures with partial inventory overlap and assert the compare view can expose the graph explanation, truth score, and next action.  
**Risks:** Too much duplication between compare panels and the graph view. Keep the compare panel as a compact pointer into the graph rather than a second full graph UI.  
**Status:** explore.

### 13.27 Market sponsored/stale de-emphasis policy

**Category:** product / safety / UI / evaluation  
**Why it matters:** Sponsored and stale cards should be visually quieter and rank lower, but still remain inspectable. This is a trust and anti-manipulation policy, not a hard block.  
**How to test:** Render fixtures with stale, sponsored, and upgrade-tagged items and verify they are visibly muted while their data remains accessible.  
**Risks:** If the de-emphasis is too aggressive, legitimate deals get buried. Tune the signal carefully.  
**Status:** explore.

### 13.28 Signal drill-down cards

**Category:** product / UI / trust / evaluation  
**Why it matters:** Each market decision should explain itself with a compact breakdown of freshness, availability, sizing confidence, price confidence, memory confidence, and penalties. This makes the Market Intelligence Graph feel inspectable rather than magical.  
**How to test:** Render representative buy/compare/substitute cards and assert the signal breakdown is present behind a progressive-disclosure control.  
**Risks:** Too much detail can overwhelm the user. Keep the breakdown collapsed by default.  
**Status:** explore.

### 13.29 Compare-to-basket bridge

**Category:** product / architecture / UI / evaluation  
**Why it matters:** Compare-lane items should be able to flow into the shopping list, basket summary, or substitute confirmation with one clear step. This turns market intelligence into action rather than a dead-end display.  
**How to test:** Build compare fixtures with home overlap and verify the next action can point to basket, pantry, or substitute confirmation without duplicating list state.  
**Risks:** Action shortcuts can over-automate. Keep the bridge explicit and confirmable.  
**Status:** explore.

### 13.30 Graph substrate projections

**Category:** architecture / product / evaluation  
**Why it matters:** The Market Intelligence Graph should be the canonical substrate, with Today, Unified Shopping, Market Lens, and Ask ShopStack acting as projections over the same decision state. That keeps recommendations consistent across surfaces and makes graph reasoning reusable.  
**How to test:** Build one graph fixture and project it into Today, unified shopping, market lens, and ask-context views; assert the same canonical item lands in the same lane and next action family.  
**Risks:** Projection helpers can become another layer of indirection if they duplicate logic. Keep them thin wrappers over the canonical graph.  
**Status:** explore.

### 13.31 Structured evidence and reason atoms

**Category:** data / trust / evaluation / product  
**Why it matters:** Natural-language reasons are useful, but the graph needs structured reason atoms and evidence claims so traces, UI drill-downs, and corrections can all point at the same underlying facts.  
**How to test:** Render a fixture graph and assert each decision cluster carries reason atoms, claim types, and a breakdown that can be serialized losslessly.  
**Risks:** Over-structuring can produce ceremony without value. Keep the atom vocabulary small and readable.  
**Status:** explore.

### 13.32 Combo solver and substitution ranker

**Category:** product / architecture / evaluation  
**Why it matters:** Combos are where household intelligence beats a store page. A combo should be compared against home overlap and split purchase candidates, while substitutions should rank by cooking role, availability, freshness, and preference.  
**How to test:** Use fixtures like onion/potato/tomato, herb bundles, and sold-out premium variants; assert the solver chooses compare/skip/buy-separate and the substitute ranking prefers the closest usable alternative.  
**Risks:** Over-fancy ranking can confuse users if it is not explainable. Always show the reason for the ranking.  
**Status:** explore.

### 13.33 Decision run memory loop

**Category:** product / data / evaluation  
**Why it matters:** The graph becomes much more valuable once the app remembers what the household did after the recommendation: bought, skipped, substituted, or corrected. This turns one-off advice into learning.  
**How to test:** Feed reconciliation events back into the graph and verify the next run uses the updated memory and preference signals.  
**Risks:** If the feedback loop is too aggressive, noisy corrections can distort the model. Keep manual correction explicit.  
**Status:** explore.

### 13.34 Frontier HF sweep and benchmark journal

**Category:** model stack / evaluation / documentation  
**Why it matters:** The repo now has a fresh live Hugging Face frontier sweep in addition to the older local benchmarks. We should keep the frontier shortlist current without confusing it with the stable local runtime truth.  
**How to test:** Run the reusable frontier harness in `tools/model_lab_frontier.py`, compare the saved JSONL/markdown artifacts, and keep the sweep notes synchronized with `shopstack/model_registry.py` and `Docs/MODEL_CATALOG.md`.  
**Recent findings:**  
- `Qwen/Qwen3.5-9B` scored 40.0% on the production-style planner prompt suite.  
- `Qwen/Qwen3.6-35B-A3B` scored 20.0% but remained fast on HF inference.  
- Live HF checks also surfaced `Qwen/Qwen3-ASR-0.6B`, `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice`, `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign`, `openbmb/MiniCPM-V-4.6`, `deepseek-ai/DeepSeek-OCR-2`, `PaddlePaddle/PaddleOCR-VL-1.6`, and `briaai/RMBG-2.0` as current frontier candidates.  
**Artifacts:** `benchmarks/model_lab/results/frontier_planner_20260613_113615.jsonl`, `benchmarks/model_lab/results/frontier_planner_20260613_113615.md`, `Docs/audits/MODEL_FRONTIER_SWEEP_2026-06-13.md`.  
**Status:** adopt candidate.

### 13.35 Modal benchmark governance and GPU budgeting

**Category:** evaluation / infra / documentation  
**Why it matters:** The Modal sweeps are intentionally parallel per candidate, but that only stays useful if the suite launches are budgeted together. The current plan-limit warning is a sign to sequence the sweeps, not to stop benchmarking.  
**How to test:** Track the GPU count per suite, confirm the harness uses isolated per-candidate workers, and rerun one suite at a time until the artifact is written successfully. For STT, OCR, planner, vision, and TTS, record the result file path and the worker count in the run note.  
**Risks:** Running planner + OCR + vision + STT sweeps concurrently can saturate the Modal workspace limit and produce stalled or wasted GPU time.  
**Status:** adopt candidate.

---

## 14. Explorer Priority Cut (2026-06-06)

If only five exploration threads get near-term attention, choose these:

1. **Decision service extraction** — unlocks testability and keeps UI thin.
2. **Freshness-first market data layer** — prevents stale prices from becoming fake truth.
3. **Post-shopping reconciliation** — closes the core product loop.
4. **Trace replay/eval harness** — makes local model/provider changes safe.
5. **Human correction learning loop** — turns daily use into compounding household intelligence.

_Last updated: 2026-06-06_

---

## 15. Priority Addendum — Architecture Foundation + Visible User Loop (2026-06-06)

**Context:** Follow-up prioritization pass after reviewing the explorer backlog. This refines the earlier priority cut by separating architecture foundation from the visible user loop. The key adjustment is pulling confidence-calibrated recommendation cards earlier because they make the product’s trust model visible to users.

### Revised priority order

1. **Service / decision extraction**
   - **Why:** This remains the foundation. ShopStack’s brain should live in typed services, not scattered Gradio callbacks.
   - **Scope:** inventory state, market candidates, recommendation decisions, trace building, correction handling, reconciliation.
   - **Reason to do first:** if freshness, reconciliation, cards, and learning are built inside UI handlers, they will need to be rewritten for mobile/API/agent surfaces.

2. **Freshness + provenance market data model**
   - **Why:** Swiggy and future market snapshots are useful but time-sensitive. Recommendations must never imply stale snapshot data is live inventory.
   - **Target market datum shape:**
     ```text
     source = swiggy_instamart
     captured_at = 2026-06-06
     location/context = snapshot/browser/page context if known
     freshness_state = fresh / aging / stale / expired
     availability_state = available / sold_out / unknown
     confidence = high / medium / low
     ```
   - **Reason to do second:** bad market truth destroys user trust faster than missing features.

3. **Confidence-calibrated recommendation cards**
   - **Why:** This is the first visible “this is smarter than a grocery list” surface.
   - **Desired output style:**
     ```text
     Buy Onion — ₹31/kg from Swiggy snapshot, available at capture time,
     common staple, likely useful if home stock is low. Medium confidence
     because current availability may have changed.
     ```
   - **Reason to do third:** it turns backend correctness into visible product trust.

4. **Post-shopping reconciliation**
   - **Why:** This closes the real household loop: planned → bought → skipped → substituted → already had → price changed.
   - **Examples:**
     ```text
     Bought: onion, tomato, potato
     Skipped: cauliflower
     Substituted: hybrid tomato instead of Indian tomato
     Already had: carrot
     Price changed: onion was ₹34 not ₹31
     ```
   - **Reason to do fourth:** without reconciliation, inventory decays and ShopStack becomes a one-time planner instead of household memory.

5. **Human correction learning loop**
   - **Why:** Corrections become local durable preferences and aliases.
   - **Examples:**
     ```text
     “atta” = wheat flour
     Prefer Indian tomato over hybrid tomato
     Avoid expensive premium/chemical-free variants unless requested
     Don’t recommend sold-out items
     Usually keep onion/potato/tomato stocked
     ```
   - **Reason to do fifth:** this makes ShopStack personal without cloud training.

6. **Trace replay / evaluation harness**
   - **Why:** Critical for model/provider/parser experimentation, but most meaningful after the first complete decision loop exists.
   - **Replay contract:**
     ```text
     inventory state + market snapshot + user intent
       → planner decision
       → recommendation cards
       → expected trace
       → expected user-facing output
     ```
   - **Reason to do sixth:** evaluation should protect real product behavior, not incomplete scaffolding.

7. **Receipt / invoice / order ingestion**
   - **Why:** High-leverage ingestion path, but it should feed an already-solid purchase/reconciliation model.
   - **Output targets:** purchase_event, price_observation, inventory_delta, merchant_history, brand preference, quantity normalization.
   - **Reason to do later:** ingestion without a solid reconciliation model creates messy data faster.

8. **Inventory contradiction cleanup / Household Memory Health**
   - **Why:** Long-term reliability layer for stale lots, duplicate items, impossible quantities, and old produce.
   - **Examples:**
     ```text
     You marked tomatoes bought 9 days ago. Still have them?
     You usually consume coriander within 2 days. Remove from inventory?
     You said you already had carrots, but also bought 500g today. Merge?
     ```
   - **Reason to do last among this cut:** it becomes valuable once enough real usage data exists.

### Build sequence

```text
services → provenance → recommendation cards → reconciliation → corrections → trace replay → ingestion → memory health
```

### Product thesis

ShopStack’s core magic should be:

> It knows what I have, knows what the market snapshot says, admits uncertainty, recommends clearly, then learns from what actually happened.

This sequence preserves clean internal architecture while quickly creating a visible user-facing reason to believe ShopStack is more than a grocery list.

_Last updated: 2026-06-06_

---

## 11. Feature Audit + Implementation Delegation Map (2026-06-07)

### Context

Pranay wants the full feature backlog implemented, but not by this agent. This section is a durable implementation map for future agents: what to build, why it matters, likely module ownership, and the local/open model + library stack that can support each feature while preserving the active constraint that simultaneously loaded models must stay at **≤32B total parameters**.

This is documentation-first. Future implementers should turn each feature cluster into focused tickets/PRs, use existing canonical modules, avoid duplicate routes/parallel systems, and keep core behavior local-first. Connected/cloud modes may exist only as optional, explicitly labeled modes.

### Research anchors checked

- Hugging Face OCR open-model guide, 2025-10-21: documents modern open OCR/VLM models and local tooling; notable ≤32B candidates include PaddleOCR-VL (~0.9B), Granite-Docling-258M, dots.ocr (~3B), DeepSeek-OCR (~3B), OlmOCR-2 (~8B), Chandra (~9B), Qwen3-VL (~9B), and Nanonets-OCR2 (~3-4B depending card/source wording). It also calls out MLX-VLM for Apple Silicon and vLLM/SGLang for batch/served OCR.
- SenseVoice repository: SenseVoice-Small is open-source, multilingual, low-latency ASR with speech understanding capabilities.
- Qwen3-32B model card: Qwen3-32B is Apache-2.0, 32.8B total / 31.2B non-embedding; this is at/just over the project’s 32B cap depending counting method, so treat it as a boundary-case candidate requiring explicit budget accounting before activation.
- Current grocery app market scan: competitive apps emphasize real-time shared lists, smart autocomplete, price estimates, receipt scanning, item-level spending analytics, plan-vs-actual comparison, pantry tracking, offline mode, and recipe-to-list flows. ShopStack’s differentiator should be local household memory + price/inventory/trace intelligence, not another checklist.

### Implementation rule for future agents

Build these through existing canonical boundaries:

```text
shopstack/services/*          product logic and typed workflows
shopstack/persistence/*       durable local state
shopstack/schemas/models.py   domain contracts
shopstack/providers/*         model/provider adapters
shopstack/tools/registry.py   validated mutation/query tools
shopstack/ui/screens/*        Gradio adapters only
shopstack/traces/*            redaction/export primitives
shopstack/market/*            market normalization/analytics/sources
```

Do **not** add a second app, second database, second trace pipeline, or route-like duplicate surface. Extend the existing service/provider/tool architecture.

---

### 11.1 P0 — make the current product feel real

#### F-001 Receipt ingestion → inventory + price memory

**Build:** Upload/photo receipt ingestion that extracts merchant/date/line items/quantities/prices/totals and proposes confirmable purchase events, inventory lots, and price observations.

**Why:** This closes the highest-leverage data loop. Receipt apps prove demand for item-level spend analytics; ShopStack can go further by updating home stock and future shopping decisions locally.

**Likely owners:** `shopstack/services/receipt.py`, `shopstack/providers/interfaces.py`, `shopstack/providers/*ocr*`, `shopstack/persistence/database.py`, `shopstack/ui/screens/market_lens.py` or a new canonical tab only if module registry is extended deliberately.

**Local/open model candidates ≤32B:**

| Candidate | Params | Fit | Notes |
|---|---:|---|---|
| `PaddlePaddle/PaddleOCR-VL` | ~0.9B | primary OCR/document parser | Multilingual, outputs Markdown/JSON/HTML; strong first test for receipts and packet labels. |
| `ibm-granite/granite-docling-258M` | 0.258B | lightweight document extraction | Very small; good CPU/Mac candidate for structured receipt/label extraction experiments. |
| `rednote-hilab/dots.ocr` | ~3B | OCR VLM | Markdown/JSON, grounding, handwriting; promising for messy receipts. |
| `deepseek-ai/DeepSeek-OCR` | ~3B | OCR + general visual understanding | Multilingual; good fallback candidate. |
| `allenai/olmOCR-2-7B-1025` | ~8B | high-quality document OCR | English-first; use for benchmark/comparison, not necessarily Indian receipt default. |
| `datalab-to/chandra` | ~9B | document OCR | Strong score, OpenRAIL; verify license before commercial claims. |
| `Qwen/Qwen3-VL-*` small variants | ~2B/4B/8-9B depending variant | general VLM + OCR | Useful if one VLM handles receipt, shelf, packet, and explanation flows. |

**Libraries/packages:** `paddleocr`, `pytesseract`/Tesseract, `surya-ocr`, `marker-pdf`, `docling`, `doctr`, `opencv-python`, `pillow`, `pydantic`, `rapidfuzz`, `dateparser`, `price-parser`, `babel`, `pandas`.

**Acceptance:** No database writes happen until the user confirms extracted rows. Every proposed write is trace-backed and redacted export-safe.

#### F-002 Plan-vs-actual shopping review

**Build:** Compare planned shopping list against actual receipt/manual purchases: bought, forgot, substituted, impulse-bought, overpaid, already-had, price changed.

**Why:** This is a competitor-visible differentiator and creates memory quality. Current grocery-list competitors highlight planned-vs-actual receipt comparison; ShopStack should connect that to inventory and future decisions.

**Likely owners:** `shopstack/services/reconciliation.py`, `shopstack/services/shopping.py`, `shopstack/persistence/database.py`, `shopstack/ui/screens/shopping.py`.

**Model/library candidates:** Rule-first with `rapidfuzz` + embeddings (`BAAI/bge-m3`, `intfloat/multilingual-e5-small/base`, `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`). Optional LLM arbitration with current 3B planner or `Qwen2.5/3 3B-8B Instruct`.

**Acceptance:** Reconciliation must preserve both user intent and actual purchase evidence; do not silently merge ambiguous items.

#### F-003 Trace service boundary

**Build:** Move trace bundle/export composition behind `shopstack/services/trace.py`, while keeping low-level redaction/export in `shopstack/traces/export.py`.

**Why:** Traces are core product trust, hackathon evidence, and evaluation substrate. They should not remain screen-local composition logic.

**Likely owners:** `shopstack/services/trace.py`, `shopstack/ui/screens/traces.py`, `shopstack/traces/export.py`.

**Model/library candidates:** No model required. Use typed Pydantic view models and deterministic JSONL export.

**Acceptance:** Existing trace UI behavior preserved; service can be unit-tested without Gradio.

#### F-004 Real local provider dogfood path

**Build:** A verified flow where at least one real local model path performs user input → perception/parser → validated tool call → trace → UI output.

**Why:** Provider interfaces exist, but product credibility requires a real, repeatable local model demo beyond mock providers.

**Likely owners:** `shopstack/providers/registry.py`, `shopstack/providers/local_provider.py`, `shopstack/planner/engine.py`, `shopstack/ui/screens/model_stack.py`.

**Local/open model candidates ≤32B:**

| Role | Candidate | Params | Notes |
|---|---|---:|---|
| Planner/parser | current `Llama-3.2-3B-Instruct-Q4_K_M.gguf` | 3B | Already downloaded/tested. |
| Planner/parser | `Qwen/Qwen2.5-7B-Instruct` or `Qwen/Qwen3-8B` | 7-8B | Strong structured output/tool-call candidate. |
| Planner/parser | `mistralai/Mistral-Small-3.2-24B-Instruct` GGUF | 24B | Strong single-model reasoning under cap if no big VLM loaded concurrently. |
| Edge parser | `HuggingFaceTB/SmolLM3-3B` | 3B | Good small parser candidate if tool JSON is reliable. |
| Boundary max | `Qwen/Qwen3-32B` | 31.2B non-embedding / 32.8B total | Use only with explicit budget policy; leaves no room for other loaded models. |

**Libraries/packages:** `llama-cpp-python`, `mlx-lm`, `mlx-vlm`, `transformers`, `accelerate`, `bitsandbytes` where supported, `outlines`, `guidance`, `instructor`, `jsonschema`, `pydantic`.

**Acceptance:** Model Stack tab shows exact backend/model/params, and traces identify model/provider used.

#### F-005 Empty-state and honesty polish

**Build:** Make empty charts/tables, mock perception, stale snapshots, and unavailable providers explicit and visually consistent.

**Why:** Trust is damaged faster by fake-looking output than by missing features.

**Model/library candidates:** No model required.

**Acceptance:** No chart/table renders as a broken blank shell when data is empty; mock/stale/preview mode is labeled wherever it affects decisions.

---

### 11.2 P1 — product intelligence features

#### F-006 Household-specific item aliases and canonicalization

**Build:** Local alias memory for “dahi/curd/yogurt,” “Surf,” “Vim,” “pav,” “atta,” etc. Add correction capture and confidence.

**Models/libraries:** `rapidfuzz`, `textdistance`, `sqlite-vec` or `sqlite-vss`, `BAAI/bge-m3` (0.6B), multilingual E5, MiniLM, `sentence-transformers`, optional local LLM parser (`Qwen3-4B/8B`, current Llama 3B).

**Acceptance:** Corrections persist and influence future shopping, receipt, and market matching.

#### F-007 Purchase cadence + stockout forecast

**Build:** Days-until-stockout and restock interval predictions using local history.

**Models/libraries:** Start deterministic: SQLite queries, `pandas`, `numpy`, `statsmodels`, simple exponential smoothing. Optional later: `sktime`, `statsforecast`, `prophet`-like alternatives.

**Acceptance:** Recommendations include confidence language: “likely low,” “uncertain,” not false precision.

#### F-008 Waste / overbuy detection

**Build:** Detect old active lots, duplicate purchases, unused perishables, and recurring waste patterns.

**Models/libraries:** Rule-first with `shopstack/decisions.py`; optional embeddings for item equivalence.

**Acceptance:** User can confirm cleanup/merge/consume; no automatic deletion.

#### F-009 Store memory

**Build:** Store entities with type, quality/freshness/price/trust/convenience scores, notes, and item-store edges.

**Models/libraries:** No model required initially. Later embeddings for store-note retrieval.

**Acceptance:** Store memory feeds “where should I buy?” and price intelligence surfaces.

#### F-010 Price fairness bands

**Build:** For scanned/manual price: compare against household memory and source snapshots; classify fair/high/low/unknown.

**Models/libraries:** `pandas`, rolling averages, robust median/IQR, unit normalization already in `shopstack/market/normalization.py`.

**Acceptance:** Always show source and freshness: household observation vs Swiggy snapshot vs manual entry.

#### F-011 Budget-aware list planning

**Build:** Estimate list total, compare against target budget, downgrade optional items, show “must buy / optional / wait” budget plan.

**Models/libraries:** Deterministic first; optional planner LLM for explanation only.

**Acceptance:** No hidden price assumptions; every estimate has source/freshness.

#### F-012 Substitutes and staples

**Build:** Item relationships: substitutes, usually-bought-with, staples, recipe-use groups.

**Models/libraries:** SQLite edge table, `networkx` for analysis if useful, embeddings for semantic substitute suggestions, optional small LLM for candidate generation.

**Acceptance:** Generated substitutes must be confirmable before becoming durable household rules.

---

### 11.3 P2 — ShopLens expansion

#### F-013 Packet label close-up mode

**Build:** OCR MRP, expiry, quantity, brand, nutrition/storage text from packet labels.

**Models:** PaddleOCR-VL, Granite-Docling, dots.ocr, DeepSeek-OCR, Qwen3-VL small, MiniCPM-V.

**Libraries:** `opencv-python` preprocessing, `pillow`, `dateparser`, `pydantic` validators.

#### F-014 Barcode catalog memory

**Build:** First scan creates barcode → item mapping; later scans resolve instantly.

**Libraries:** Existing `shopstack/scanner.py`; add/compare `pyzbar`, `zxing-cpp`, `opencv`, JS `html5-qrcode`, `quagga2`, `zxing-js/browser`.

**Acceptance:** Barcode mappings are user-correctable and local.

#### F-015 Fridge/pantry photo snapshot

**Build:** Save visual snapshots and confirmed item detections per household location.

**Models:** YOLOv8/YOLO11, RF-DETR, GroundingDINO/MM-GroundingDINO, OWL-ViT, Florence-2, MiniCPM-V, Qwen3-VL small.

**Current grounding note (2026-06-15):** the real-photo household grounding benchmark now exists with three imagegen-created scenes (fridge, cupboard, tabletop). `GroundingDINO` reached `0.2042` mean box IoU / `0.3333` success after provider fixes; `Qwen3-VL` stayed helper-only at `0.0136` mean box IoU.

**Libraries:** `ultralytics`, `rfdetr`, `transformers`, `supervision`, `opencv-python`, `faiss`/`sqlite-vec` for image-snapshot matching.

#### F-016 Shelf/location recommendations

**Build:** Recommend storage location based on category/perishability/history.

**Models/libraries:** Rule-first from category metadata; optional classifier/LLM for ambiguous items.

#### F-017 Home Scan scene understanding

**Build:** A broader ShopLens Home Scan that understands kitchen, bathroom, medicine, cleaning, bedroom, and document scenes; count visible items, read labels/expiry, accept local-language voice corrections, and propose safe inventory updates.

**Models:** Object detection + segmentation + OCR + STT first, then VLM scene summarization and multilingual correction parsing.

**Libraries:** Existing provider interfaces (`object_detection`, `segmentation`, `ocr`, `stt`, `image_edit`), `opencv-python`, `pydantic`, `dateparser`, optional VLM backends for frame captioning and cross-frame consolidation.

**Current implementation note (2026-06-14):** Shelf Scan now accepts a short video sweep, extracts multiple frames locally, merges repeated detections into one household-memory result, and uses the real OCR pipeline instead of the old mock-style compatibility fallback. The still-photo path remains intact.

**Current grounding note (2026-06-15):** generated fridge/cupboard/table photos are now saved under `benchmarks/modal/assets/household_grounding` and used in the dedicated grounding benchmark harness (`benchmarks/modal/bench_grounding_v1.py`).

**Acceptance:** Shelf scans stay confirmation-first, can be zone-aware, can merge short video sweeps, and update household memory without assuming the home is only pantry/fridge.

#### F-017 Batch confirmation UI

**Build:** Show proposed structured actions from OCR/image/list parsing and let user accept/edit/reject before writes.

**Models/libraries:** Pydantic action schemas, Gradio DataFrame/editable components, trace service.

---

### 11.4 P3 — market and context intelligence

#### F-018 Multi-source retailer adapters

**Build:** Add a common adapter interface for Blinkit, Zepto, BigBasket, local CSV/manual imports, and future connected sources.

**Libraries:** `pydantic`, `pandas`, `duckdb`, `httpx` only in connected mode, Playwright/browser automation only if explicitly allowed.

**Acceptance:** Every source record carries source, captured_at, freshness, confidence, and terms/collection notes.

#### F-019 Basket-level retailer choice

**Build:** Compare basket total across sources and split basket if savings justify it.

**Libraries:** `pandas`, `scipy`/simple optimization, existing `shopstack/market/basket.py`.

#### F-020 Travel-effort decision

**Build:** Manual-first trip effort model; optional route engines later.

**Libraries:** `osmnx`, `networkx`, OSRM/OpenRouteService/Valhalla only as optional connected mode; `geopy`, `h3` for coarse privacy cells.

#### F-021 Weather-aware shopping note

**Build:** Manual/mock weather context; optional weather lookup later.

**Libraries:** `meteostat`, Open-Meteo client in connected mode, local config/manual input in Off the Grid.

#### F-022 Private price heatmap

**Build:** Coarse local price cells with minimum observation counts.

**Libraries:** `h3`, `folium` or deterministic SVG/HTML cards; avoid exact home/route exposure in traces.

---

### 11.5 P4 — launch, evaluation, and hackathon assets

#### F-023 Walkthrough mode

**Build:** Guided demo story: ask what to buy → scan item → skip duplicate → add purchase → export trace.

**Models:** Use the smallest reliable active stack: current 3B planner + mock/real OCR path as available.

#### F-024 Public redacted trace dataset

**Build:** Curated anonymized JSONL trace examples and dataset card.

**Libraries:** existing redaction + `datasets` library for HF publishing.

#### F-025 Field Notes report generator

**Build:** Generate polished build report from traces, model stack, decisions, failures, and screenshots/outputs.

**Models:** Local LLM optional for drafting, but factual claims must come from deterministic trace/model metadata.

#### F-026 Well-Tuned narrow model

**Build:** Fine-tune a small command/entity parser for Hinglish grocery commands and publish it.

**Base candidates ≤32B:** `Qwen2.5-1.5B/3B-Instruct`, `Qwen3-1.7B/4B`, `SmolLM3-3B`, `MiniCPM5-1B`, current Llama 3B if license permits desired artifact.

**Libraries:** `trl`, `peft`, `unsloth`, `axolotl`, `datasets`, `evaluate`, `jsonschema`, `pydantic`.

**Dataset:** Indian/Hinglish household command parser dataset with tool-call JSON labels.

#### F-027 README / HF Space launch checklist

**Build:** Launch checklist covering privacy, local-first, model budget, data freshness, demo flow, trace dataset, licenses, and known limitations.

**Models:** None required.

---

## 12. Local/Open Model and Library Selection Matrix (≤32B constraint)

### 12.1 Recommended active-stack scenarios

The cap is about **simultaneously loaded active models**, not every candidate in the repo. Future agents should declare one active scenario before implementation/testing.

| Scenario | Active models | Approx params | Use when |
|---|---|---:|---|
| Minimal reliable | Llama 3.2 3B planner + BGE-M3 embeddings + Kokoro TTS optional | ~3.7B | Current local-first baseline, fast demos. |
| Receipt-first | PaddleOCR-VL 0.9B + Llama 3B planner + BGE-M3 | ~4.5B | Receipt/packet extraction with structured reconciliation. |
| Vision-heavy | MiniCPM-V 8B or Qwen3-VL 8-9B + Llama 3B + BGE-M3 | ~12-13B | Market Lens, shelf/fridge photos, OCR fallback. |
| Strong planner | Mistral Small 24B GGUF + lightweight OCR 0.9B + BGE-M3 | ~25.5B | Harder reasoning, still under cap if no large VLM. |
| Boundary single-brain | Qwen3-32B alone | 31.2B non-embedding / 32.8B total | Only if project explicitly accepts counting method and gives up simultaneous specialist models. |
| Full practical multimodal | Qwen3-VL 9B + Qwen/Llama planner 7-8B + SenseVoice 0.2B + Kokoro 0.082B + BGE-M3 0.6B + segmentation 0.3B | ~18-19B | Rich local multimodal stack under cap. |

### 12.2 Capability-by-capability candidate shortlist

| Capability | Best first candidate | Strong alternatives | Libraries/runtime |
|---|---|---|---|
| Planner/tool JSON | current Llama 3B GGUF | Qwen3 4B/8B, Qwen2.5 7B, Mistral Small 24B | `llama-cpp-python`, `mlx-lm`, `transformers`, `outlines`, `instructor` |
| Receipt OCR | PaddleOCR-VL | Granite-Docling, dots.ocr, DeepSeek-OCR, OlmOCR-2 | `paddleocr`, `mlx-vlm`, `vllm`, `sglang`, `docling`, `marker` |
| Packet label OCR | PaddleOCR-VL | dots.ocr, Qwen3-VL, MiniCPM-V | `opencv-python`, `dateparser`, OCR/VLM runtime |
| General VLM | MiniCPM-V or Qwen3-VL small | Florence-2, SmolVLM variants | `transformers`, `mlx-vlm` |
| Object detection | YOLO11/YOLOv8 | RF-DETR, GroundingDINO, OWL-ViT | `ultralytics`, `rfdetr`, `supervision`, `opencv-python` |
| Segmentation/cards | RMBG-1.4 | BiRefNet, SAM variants, ClipSeg | `transformers`, `opencv-python`, `pillow` |
| ASR | SenseVoice-Small | Parakeet 0.6B/V3, Distil-Whisper, Whisper large-v3-turbo | `funasr`, `faster-whisper`, `mlx-audio`, `transformers` |
| TTS | Kokoro-82M | Qwen3-TTS 0.6B, CosyVoice, Piper, KittenTTS | `kokoro`, `piper-tts`, `onnxruntime`, `mlx-audio` |
| Embeddings | BGE-M3 | multilingual E5, MiniLM, Jina embeddings | `sentence-transformers`, `faiss-cpu`, `sqlite-vec`, `lancedb`, `chromadb` |
| Reranking | BGE reranker small/base | Jina reranker small | `sentence-transformers`, `FlagEmbedding` |
| Time series | deterministic stats | statsmodels/sktime/statsforecast | `pandas`, `numpy`, `statsmodels`, `sktime` |
| Route/geo optional | manual + H3 | OSRM/OpenRouteService/Valhalla | `h3`, `osmnx`, `networkx`, `geopy`, `folium` |
| Barcode | existing scanner + pyzbar | zxing-cpp, html5-qrcode, Quagga2, ZXing JS | `pyzbar`, `zxing-cpp`, `opencv`, JS browser libs |

### 12.3 Specific model notes for future benchmarking

- **PaddleOCR-VL (~0.9B):** highest-priority OCR test because it is small, multilingual, and supports JSON/Markdown/HTML-style structured output. Best first model for receipts and packet labels.
- **Granite-Docling-258M:** extremely small document-understanding candidate. Best for CPU/Mac fallback and quick tests.
- **dots.ocr (~3B):** good OCR VLM candidate for Markdown/JSON and grounding. Test on messy local receipts.
- **DeepSeek-OCR (~3B):** useful for OCR + general visual understanding; test on labels and screenshots.
- **OlmOCR-2 (~8B):** strong document OCR but English-oriented; use as benchmark/reference more than default Indian household model.
- **Chandra (~9B):** strong document OCR, but verify OpenRAIL restrictions before product claims.
- **Qwen3-VL small family (~2B-9B variants):** promising single VLM for OCR, packet labels, shelf images, and explanation. Confirm exact chosen variant params/license before activation.
- **MiniCPM-V (~8B):** strong general VLM candidate and sponsor-aligned with OpenBMB angle.
- **SenseVoice-Small (~Whisper-small class):** strong ASR candidate for fast multilingual speech understanding. Benchmark Hinglish grocery phrases.
- **Parakeet 0.6B/V3:** very fast ASR candidate; confirm language coverage for Hindi/Hinglish before relying on it.
- **Kokoro-82M:** excellent tiny local TTS baseline. Use for short, warm household answers if pronunciation is acceptable.
- **Qwen3-TTS 0.6B / CosyVoice:** better voice quality candidates; benchmark latency and Hindi/Hinglish pronunciation.
- **BGE-M3 (0.6B):** already aligned with workspace embedding use; use for aliases, trace retrieval, and household memory search.
- **Qwen3-32B:** boundary case. It is useful as a “single strong brain” experiment but total params are listed as 32.8B and non-embedding 31.2B. Do not activate without updating the budget rule/counter and documenting whether total or non-embedding params are counted.

### 12.4 Non-model libraries that can implement much of the backlog

Many features should be deterministic first:

- **Parsing and validation:** `pydantic`, `jsonschema`, `dateparser`, `price-parser`, `babel`, `python-slugify`.
- **Matching/canonicalization:** `rapidfuzz`, `textdistance`, `unidecode`, `regex`, `sentence-transformers`.
- **Analytics/time series:** `pandas`, `numpy`, `duckdb`, `statsmodels`, `sktime`, `statsforecast`.
- **Vector/local memory:** `sqlite-vec`, `sqlite-vss`, `faiss-cpu`, `lancedb`, `chromadb`.
- **Vision preprocessing:** `opencv-python`, `pillow`, `scikit-image`, `imagehash`.
- **OCR baselines:** `pytesseract`, `paddleocr`, `surya-ocr`, `docling`, `marker-pdf`, `doctr`.
- **Barcode:** `pyzbar`, `zxing-cpp`, browser `html5-qrcode`, `quagga2`, `@zxing/browser`.
- **Maps/geo:** `h3`, `geopy`, `osmnx`, `networkx`, `folium` for optional map experiments.
- **Voice pipeline:** `funasr`, `faster-whisper`, `piper-tts`, `kokoro`, `onnxruntime`, `mlx-audio`, `pipecat` for a more advanced local voice loop.
- **Model serving:** `llama-cpp-python`, `mlx-lm`, `mlx-vlm`, `transformers`, `vllm`, `sglang`, `ollama` as optional local dev runtime.
- **Constrained outputs:** `outlines`, `guidance`, `instructor`, `lm-format-enforcer`.
- **Training/fine-tuning:** `datasets`, `peft`, `trl`, `unsloth`, `axolotl`, `evaluate`.

### 12.5 Implementation sequencing recommendation

Future agents should implement in this order:

```text
1. Trace service boundary
2. Receipt extraction proposal schema, no writes
3. Receipt confirmation → purchase/price/inventory writes
4. Plan-vs-actual reconciliation
5. Alias/canonicalization correction loop
6. Price fairness + budget-aware list planning
7. Packet label mode + barcode catalog memory
8. Fridge/pantry snapshot and storage recommendations
9. Store memory + basket-level retailer choice
10. Trace replay/eval harness + public redacted trace dataset
11. Fine-tuned narrow command parser
12. Optional travel/weather/heatmap intelligence
```

This order protects data integrity first, then adds intelligence, then expands perception/context. It also gives a future non-this-agent implementation team clean seams for independent work.

_Last updated: 2026-06-07_

---

## 13. Fresh HF / 2026 Model Launch Addendum (2026-06-07)

### Why this addendum exists

Pranay challenged whether the previous model scan was fresh enough. That challenge was valid. The first pass covered strong late-2025 OCR/VLM candidates, but it underweighted newer/open launches and HF ecosystem updates relevant to ShopStack, especially omni-modal, realtime speech, multimodal embeddings/rerankers, and HF Pro/Jobs workflows.

This section supersedes the older shortlist where it is more specific. Keep the older section as historical context, but future agents should check this addendum first when choosing models.

### Fresh research anchors checked

- **Qwen3-Omni-30B-A3B-Instruct / Thinking / Captioner** model cards and GitHub: Apache-2.0, omni-modal, text/image/audio/video input, text/speech output, 119 text languages, 19 speech-input languages, 10 speech-output languages, real-time streaming; 30B MoE with ~3B active parameters. Strong candidate for a single multimodal “one brain does a lot” experiment, but likely heavy operationally.
- **Mistral Voxtral Mini 4B Realtime 2602** model card: Apache-2.0, multilingual realtime ASR, 4B params, <500ms delay target, 13 languages, designed for on-device/minimal hardware compared with heavier speech stacks.
- **Hugging Face OCR open-model guide + community comments:** added/flagged models not previously emphasized enough: `opendatalab/MinerU2.5-2509-1.2B`, `lightonai/LightOnOCR-1B-1025`, `tencent/POINTS-Reader`, `Logics-MLLM/Logics-Parsing`, `SmolDocling-256M-preview`, `RolmOCR-7B`, `Typhoon-OCR-7B`, `MonkeyOCR-Recognition`.
- **Hugging Face multimodal Sentence Transformers v5.4 blog (2026-04):** new multimodal embedding/reranker support for text/images/audio/video. `Qwen/Qwen3-VL-Embedding-2B` is explicitly shown and needs ~8GB VRAM; 8B variants need ~20GB. This matters for shelf image retrieval, receipt visual retrieval, and trace/memory search.
- **HF Jobs pricing/docs:** Jobs can run batch compute on HF infrastructure when the account has positive credit balance; billed by minute while starting/running. This fits Pranay’s HF Pro/credits for benchmarking and batch OCR experiments without adding cloud dependency to the Off-the-Grid runtime.
- **vLLM-Omni speech API docs:** supports serving Qwen3-TTS, Fish Speech, Voxtral TTS, CosyVoice3, OmniVoice, VoxCPM2, MOSS-TTS-Nano, Qwen3-Omni and related omni/audio models via OpenAI-style APIs. Useful for benchmark rigs and optional connected/dev mode.
- **Aya Vision 8B/32B:** multilingual VLM family from CohereForAI. 8B is relevant for multilingual image/text tasks; 32B is boundary/too tight for simultaneous loaded stack and should be treated carefully.
- **SmolVLM / SmolVLM2 family:** tiny VLMs (256M/500M/2.2B class) useful for on-device-ish image caption/doc QA/video-lite experiments, not likely enough for robust receipt extraction alone but excellent baseline/performance fallback.

---

### 13.1 Updated “newer launch” candidates for ShopStack

#### Omni / single-model-does-a-lot candidates

| Model | Params / active | License | Why it matters for ShopStack | Caveat |
|---|---:|---|---|---|
| `Qwen/Qwen3-Omni-30B-A3B-Instruct` | 30B total, ~3B active MoE | Apache-2.0 | One model can cover text, image, audio, video, ASR-like input, and speech output. Strong candidate for a “single local/offline brain” experiment and sponsor-aligned demo. | Transformers inference can be slow; vLLM/Docker recommended. Counts as 30B loaded for project budget unless policy explicitly counts active params. Leaves little room under 32B if total params counted. |
| `Qwen/Qwen3-Omni-30B-A3B-Thinking` | 30B total, ~3B active MoE | Apache-2.0 | Better for complex multimodal reasoning over receipt/photo/audio traces. | Same budget/runtime caveat; likely slower. |
| `Qwen/Qwen3-Omni-30B-A3B-Captioner` | 30B total, ~3B active MoE | Apache-2.0 | Audio captioning and market/environment audio notes; useful for “what happened in this noisy shopping audio?” | More specialized; not a default core runtime. |
| `openbmb/MiniCPM-o-2_6` | ~8B | check model card | Any-to-any style multimodal candidate with vision/speech/language. Good OpenBMB sponsor alignment. | Must verify current license/runtime before product use. |

**Opinionated take:** Qwen3-Omni is the most exciting newer launch for the product vision, but it is not the first implementation model. Use it with HF Pro/credits for benchmarking and one “wow” demo. For reliable local-first core, keep specialist small models + deterministic services.

#### Fresh OCR / document-understanding candidates to add to benchmark set

| Model | Params | Why it matters | Suggested ShopStack test |
|---|---:|---|---|
| `opendatalab/MinerU2.5-2509-1.2B` | ~1.2B | Mentioned in HF OCR guide comments as missing from comparison; compact document parser. | Indian grocery receipt with line items + packet label close-up. |
| `lightonai/LightOnOCR-1B-1025` | ~1B | Community-flagged “punches above weight” OCR candidate. | Low-quality local receipt photo and multilingual label. |
| `tencent/POINTS-Reader` | ~3B text backbone + vision encoder | Efficient end-to-end document AI replacing larger 7B backbone with Qwen2.5-3B; Chinese/English strengths. | Receipt OCR and structured text extraction speed/accuracy. |
| `Logics-MLLM/Logics-Parsing` | verify card | Single-model document parsing for complex docs. | Use as secondary benchmark, not default. |
| `SmolDocling-256M-preview` | ~256M | Tiny document understanding candidate. | CPU/Mac fallback for basic labels/receipts. |
| `RolmOCR-7B` | ~7B | Lightweight VLM-level OCR option. | Compare against OlmOCR and PaddleOCR-VL. |
| `Typhoon-OCR-7B` | ~7B | OCR model seen in HF OCR demo collections. | Test if receipts are multilingual/noisy. |
| `MonkeyOCR-Recognition` | verify card | OCR recognition candidate in HF demo collections. | Baseline OCR recognition, not full structured extraction. |

**Updated OCR priority:**

```text
1. PaddleOCR-VL (~0.9B)
2. MinerU2.5-1.2B + LightOnOCR-1B
3. Granite/SmolDocling tiny fallbacks
4. dots.ocr / DeepSeek-OCR / POINTS-Reader
5. OlmOCR-2 / RolmOCR / Typhoon-OCR as heavier benchmarks
6. Qwen3-VL / Qwen3-Omni for general multimodal fallback
```

#### Fresh ASR / realtime voice candidates

| Model | Params | License | Why it matters | Caveat |
|---|---:|---|---|---|
| `mistralai/Voxtral-Mini-4B-Realtime-2602` | 4B | Apache-2.0 | Newer realtime ASR, 13 languages, <500ms delay target, open weights. Stronger fresh candidate than older Whisper-only baseline. | Hindi/Hinglish support must be tested; 4B is heavier than SenseVoice/Parakeet. |
| `Qwen/Qwen3-Omni-30B-A3B-Instruct` | 30B total / 3B active | Apache-2.0 | Speech input and speech output in one model; supports Urdu but not explicitly Hindi speech input in listed languages. | Heavy; use as benchmark/wow path. |
| `FunAudioLLM/SenseVoiceSmall` | ~0.2B class | likely permissive; verify exact card | Fast multilingual ASR + emotion/audio events. | Language list may not cover Hindi directly; benchmark Hinglish. |
| NVIDIA Parakeet V3 / 0.6B family | ~0.6B | check card | Very fast ASR; good low-latency candidate. | Often English-focused; verify multilingual/Hinglish. |
| Whisper large-v3-turbo | ~0.8B | MIT | Mature baseline with broad language support. | Not native realtime; use faster-whisper/whisper.cpp. |

#### Fresh TTS / speech output candidates

| Model | Params | License | Why it matters | Caveat |
|---|---:|---|---|---|
| `Qwen/Qwen3-TTS-12Hz-0.6B-*` | 0.6B | check card; Qwen family often Apache-2.0 | Small Qwen TTS; vLLM-Omni supports CustomVoice/Base. | Benchmark Hindi/Hinglish pronunciation and runtime complexity. |
| `Qwen/Qwen3-TTS-12Hz-1.7B-*` | 1.7B | check card | Better quality than 0.6B; supports CustomVoice/VoiceDesign/Base variants via vLLM-Omni. | Heavier, still fine under 32B. |
| `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | 0.5B | verify card | vLLM-Omni supported; voice cloning style path. | More complex request requirements. |
| `mistralai/Voxtral-4B-TTS-2603` | ~4.1B | reported CC-BY-NC 4.0 in third-party scan; verify HF card | High-quality multilingual/Hindi-capable TTS candidate. | Non-commercial likely; not default for product unless license accepted. |
| `OpenMOSS-Team/MOSS-TTS-Nano` | small; verify | verify | vLLM-Omni supported; potential lightweight voice cloning. | Needs ref audio; not preset voice default. |
| Kokoro-82M | 82M | Apache-2.0 | Still best tiny default TTS baseline. | Pronunciation/voice quality may be weaker than newer TTS. |
| Piper | small | permissive | Reliable local TTS fallback. | Less “wow,” but robust. |

#### Fresh multimodal embedding / reranker candidates

| Model / stack | Params | Why it matters |
|---|---:|---|
| `Qwen/Qwen3-VL-Embedding-2B` | 2B | Directly relevant to visual receipt/shelf/image retrieval, multimodal memory search, and trace retrieval. HF Sentence Transformers v5.4 shows loading with image support. |
| Qwen3-VL Embedding 8B variants | ~8B | Higher-quality multimodal retrieval if GPU memory allows. |
| Sentence Transformers v5.4 multimodal rerankers | varies | Lets ShopStack rerank text-image/audio/video memories with one API. |
| CLIP/SigLIP/SigLIP2 family | small-medium | Good fast image/text similarity for shelf snapshots and visual duplicate matching. |
| BGE-M3 | 0.6B | Still best text-heavy multilingual local memory default. |

**New product implication:** ShopStack should not only do text embeddings. It should plan for **multimodal household memory**: receipt images, shelf photos, voice snippets/transcripts, and traces should be retrievable across modalities.

---

### 13.2 HF Pro / credits usage plan

Pranay has HF Pro and credits. Use them for research, benchmarking, and artifact creation — not as hidden runtime dependency for the Off-the-Grid path.

#### Good uses of HF Pro/credits

1. **Batch OCR bakeoff with HF Jobs**
   - Create a small representative dataset: 20 receipt photos, 20 packet labels, 10 shelf/fridge photos, 10 Swiggy screenshots/product cards.
   - Run PaddleOCR-VL, MinerU2.5, LightOnOCR, dots.ocr, DeepSeek-OCR, Granite/SmolDocling, POINTS-Reader via Jobs or temporary GPU Space.
   - Save outputs as a versioned benchmark dataset/artifact.

2. **Temporary GPU Space for demo/model exploration**
   - Spin up model-specific Spaces for Qwen3-Omni, Voxtral Realtime, Qwen3-TTS, Qwen3-VL Embedding.
   - Record latency, memory, model output, failure modes.
   - Tear down or downgrade hardware after experiment.

3. **Dataset publishing**
   - Publish redacted/synthetic command parser dataset.
   - Publish redacted trace examples.
   - Publish OCR benchmark fixtures if privacy-safe.

4. **Fine-tuning / LoRA training**
   - Narrow parser LoRA on Qwen/SmolLM/MiniCPM-sized base.
   - Do not fine-tune a broad “do everything” model first.

5. **Model comparison reports**
   - Use HF Jobs for repeatable batch inference and store metrics.
   - Feed results back into `Docs/MODEL_CATALOG.md` and `shopstack/model_registry.py` only after local/runtime verification.

#### Bad uses of HF Pro/credits

- Making HF Inference Endpoint mandatory for core app behavior while claiming Off the Grid.
- Using credits to bypass implementing local confirmation, validation, and traceability.
- Benchmarking on generic OCR samples only; ShopStack needs grocery/household-specific data.
- Promoting a model to active runtime without testing memory, latency, license, and JSON/tool-call reliability.

### 13.3 Revised model strategy

The correct strategy is not “pick one best model.” Use a three-lane model strategy:

```text
Lane A — Reliable local core
  deterministic services + Llama/Qwen 3B-8B planner + PaddleOCR-VL/Granite/SmolDocling + BGE-M3 + Kokoro/Piper

Lane B — Specialist excellence
  Voxtral Realtime for ASR, Qwen3-TTS/CosyVoice3 for TTS, Qwen3-VL Embedding for multimodal memory, YOLO/RF-DETR for detection

Lane C — Wow / future omni demo
  Qwen3-Omni-30B-A3B for audio/image/video/text/speech unified experience, benchmarked with HF credits, optional not required
```

### 13.4 Updated benchmark tasks for future agents

Create a repo-local benchmark spec before model promotion:

| Task | Inputs | Required output | Metrics |
|---|---|---|---|
| Receipt line extraction | receipt photos | merchant/date/items/qty/unit/line price/total | field F1, human correction count, latency |
| Packet label extraction | packet label closeups | brand/item/MRP/expiry/quantity/storage | exact date/price accuracy, confidence calibration |
| Hinglish voice command | short audio phrases | transcript + intent + tool JSON | WER, intent accuracy, JSON validity |
| Market photo scan | shelf/product photo | candidate item + price/freshness uncertainty | top-1/top-3 accuracy, false certainty rate |
| Alias matching | text/receipt/list variants | canonical item ID | accuracy, ambiguous deferral rate |
| Multimodal retrieval | text query over images/traces | relevant receipt/shelf/trace | recall@k, rerank quality |
| Short TTS answer | generated household answers | natural speech | latency, intelligibility, Hindi/Hinglish pronunciation |
| Omni demo | photo/audio/video prompt | useful answer + optional speech | latency, trustworthiness, failure modes |

### 13.5 Immediate doc follow-ups

Future agents should update these artifacts after each real benchmark:

- `Docs/MODEL_CATALOG.md`: add the fresh model, params, license, runtime, status, benchmark notes.
- `shopstack/model_registry.py`: add only candidates that passed basic source/license checks; mark active only after runtime verification.
- `Docs/ShopStack_Exploration_Map.md`: add findings, rejected models, and changed priorities.
- `tests/`: add fixtures for model-independent parsing/validation paths so model swaps do not break product contracts.

### 13.6 Bottom line

The previous shortlist was useful but incomplete. The biggest newer additions are:

1. **Qwen3-Omni-30B-A3B** as the ambitious single-model omni experiment.
2. **Voxtral Mini 4B Realtime** as a fresh ASR candidate.
3. **Qwen3-TTS / CosyVoice3 / vLLM-Omni speech stack** as a newer TTS evaluation lane.
4. **MinerU2.5, LightOnOCR, POINTS-Reader, SmolDocling, RolmOCR, Typhoon-OCR** as added OCR benchmark candidates.
5. **Qwen3-VL Embedding + Sentence Transformers multimodal v5.4** as the right direction for multimodal household memory.
6. **HF Jobs/Spaces with Pro credits** as the right way to benchmark without making cloud a product dependency.

_Last updated: 2026-06-07_

---

## 14. Living Model Research Intake + Leaderboard-Driven Discovery Protocol (2026-06-07)

### Why this section exists

Pranay pushed back that the model research was not thorough enough. That pushback is correct: generic “best model” lists are too shallow for ShopStack. The project needs a repeatable intake process that can absorb user-provided model names, watch new Hugging Face launches, compare leaderboards by modality and size, and still stay grounded in the product’s actual tasks.

This is a **living research queue**, not an implementation instruction. Future agents should document new findings here first, then only promote models into `Docs/MODEL_CATALOG.md` / `shopstack/model_registry.py` after license and runtime verification.

### Hard constraints for model candidates

- Prefer **open-source / open-weight / locally runnable** models.
- Prefer models with **≤32B total loaded parameters**. MoE models with ≤32B active params but much larger total params may be listed, but must be marked as **active-param-only fit** and treated as eval/API/large-rig candidates unless a local hardware plan exists.
- Keep the **Off-the-Grid** promise: HF Pro/credits may be used for research, benchmarking, fine-tuning, and batch evals, but must not become a hidden runtime dependency for core behavior.
- Rank by **ShopStack usefulness**, not generic benchmark prestige.
- Every candidate must eventually be tested on grocery/household data, not just public benchmark samples.

### Intake template for names Pranay provides

When Pranay gives a model/library name, append a row here or in a dated subtable using this schema:

| Field | Required note |
|---|---|
| Name / link | Exact HF repo, GitHub repo, paper, or package name. |
| Modality | Text, OCR/doc, VLM, ASR, TTS, embedding/rerank, detection, segmentation, time-series, agent/tooling. |
| Size | Total params, active params if MoE, expected quantized RAM/VRAM. |
| License | Exact license and commercial risk. Do not infer. |
| Runtime | `transformers`, `llama.cpp`, `mlx`, `vLLM`, `SGLang`, ONNX, WebGPU, library API, etc. |
| ShopStack fit | Receipt OCR, packet label OCR, pantry photo, Hinglish voice, TTS answers, planner/tool calls, multimodal memory, price intelligence, etc. |
| Evidence source | Official model card/paper, leaderboard, GitHub, independent eval, or user-provided hunch. |
| Verdict | `evaluate now`, `watch`, `reject`, `eval-only`, or `blocked on license/runtime`. |
| Benchmark task | The first concrete ShopStack eval this model should run. |

### Leaderboard/source watchlist by capability

Use these sources first before relying on blog roundups:

| Capability | Primary sources to check | What to extract | Caveat |
|---|---|---|---|
| General small LLM / reasoning | Hugging Face Open LLM Leaderboard org/results, Artificial Analysis with open-weights filter, Vellum Open Source LLM Leaderboard, Onyx self-hosted/open-source leaderboards | Params, open-weight status, reasoning/coding scores, context, latency/cost where available | General reasoning scores do not prove tool-call reliability or JSON correctness. |
| Small local / edge LLMs | HF model cards, small-model leaderboards, llama.cpp/MLX/Ollama community benchmarks | Quantized memory, speed, instruction following, multimodal support | Many “under 10B” lists mix official, vendor-reported, and community scores. Verify model cards. |
| Tool calling / structured output | Berkeley Function Calling Leaderboard, Gorilla/OpenFunctions/xLAM/Hammer reports, independent local LM Studio/Open WebUI tests | Tool selection, argument validity, multi-tool support, format compatibility | Chat-template/runtime mismatch can make a leaderboard winner fail locally. Test with ShopStack’s actual server. |
| VLM / multimodal | OpenCompass Open VLM Leaderboard, VLMEvalKit, HF OpenVLM records, model cards for Qwen/Gemma/MiniCPM/SmolVLM/Aya/Phi | OCR, chart/doc QA, visual reasoning, GUI grounding, video/image support | VLM leaderboard rank is not the same as receipt OCR or packet label extraction. |
| OCR / document AI | OCRBench-style reports, OmniDocBench, DocVQA/InfoVQA/ChartQA benchmarks, HF OCR guides, Docling/MinerU/PaddleOCR docs | Field extraction, layout retention, multilingual OCR, structured Markdown/JSON support | Product needs receipts and labels; academic docs may not transfer. |
| ASR / speech input | HF Open ASR Leaderboard, ASR Leaderboard paper/code, model cards for Canary/Qwen, Parakeet, Whisper, SenseVoice, Voxtral | WER, real-time factor, multilingual support, streaming support | Hindi/Hinglish and noisy market audio must be tested separately. |
| TTS / speech output | TTS Arena, HF trending TTS, model cards for Kokoro/Piper/Qwen3-TTS/CosyVoice/Chatterbox | Naturalness, latency, voice cloning, language/pronunciation, license | Subjective voice quality needs human listening tests. |
| Embeddings / rerankers | MTEB leaderboard, Sentence Transformers docs/blogs, BGE/Jina/Qwen embedding model cards | Retrieval score, multilingual score, multimodal support, rerank ability | Overall MTEB score hides retrieval-specific performance. Test household memory queries. |
| Detection/segmentation | COCO/LVIS/open-vocabulary detection reports, Ultralytics/RF-DETR/GroundingDINO/SAM docs | Object detection, grounding, segmentation, runtime | Model may detect generic objects but not branded groceries. |
| Time series / forecasting | Nixtla/statsforecast docs, sktime, NeuralForecast, Chronos/TinyTimeMixers model cards | Forecast accuracy, local CPU viability, probabilistic outputs | Use deterministic statistical baselines before neural forecasting. |

### Current leaderboard-driven candidate intake

This table captures newer candidates and source directions surfaced by the broader leaderboard scan. It is deliberately conservative: **candidate ≠ approved runtime**.

| Candidate | Modality | Size / fit | Evidence/source direction | ShopStack fit | Verdict / first eval |
|---|---|---|---|---|---|
| `Qwen3.5-4B` / `Qwen3.5-9B` small series | Planner, tool calling, possible multimodal depending exact repo | ≤10B | Small-model rankings and independent local tool-calling tests report very strong structured/tool behavior for 4B-class Qwen | Local planner, command parser, JSON/tool calls, cheap local assistant | **Evaluate now**: tool JSON harness + shopping command parser. Verify exact HF repos/licensing. |
| `NVIDIA Nemotron Nano 4B` / `Nemotron Nano 9B/12B/VL` | Tool calling, general LLM, VLM variants | ≤12B | Artificial Analysis and local tool-calling evals surface strong speed/structured behavior for Nano class | Local planner fallback, possible VLM route if VL variant is open/licensed | **Evaluate** after license/runtime check: tool call schema + receipt/photo QA. |
| `GPT-OSS-20B` | General/reasoning/tooling | 20B | Vellum/Artificial Analysis list strong open-weight cost/speed; local tool eval shows decent but not best tool calls | Larger local brain for planning/evals, not first core model | **Watch/evaluate**: compare against Qwen 4B/9B on JSON and latency. Confirm license/runtime. |
| `Gemma 3 4B / 12B / 27B` and `Gemma 3n E4B` | Text + vision, edge/local | ≤27B total; 3n effective smaller | Small-model and open-source leaderboards consistently cite strong local multimodal and edge viability | Pantry/shelf photo QA, on-device assistant, fallback VLM | **Evaluate now** for vision + local memory; test receipts but do not assume OCR excellence. |
| `Phi-4-mini` / `Phi-4 multimodal` / `Phi-4 reasoning-vision` | Small reasoning, multimodal | ~3.8B to ~15B | Small-model rankings cite math/reasoning; multimodal variants appear useful for GUI/vision | Lightweight reasoning, possible product-photo understanding | **Watch/evaluate**: tool-call reliability looked weaker in one local test; verify current variants. |
| `Mistral Nemo 12B` | Planner/tooling | 12B | Independent local tool-call eval ranked it highly under LM Studio | Structured local planner alternative | **Evaluate** if Qwen fails or license/runtime is simpler. |
| `Mistral Small 3.x / 24B` | Planner/general | 24B | Self-hosted leaderboards list it, but local tool-call tests may underperform via incompatible templates | Possible high-quality planner if native runtime works | **Blocked on runtime-template validation**. Do not assume BFCL/official score transfers. |
| `xLAM-2 8B` / Salesforce function-calling models | Tool calling | ~8B | BFCL-oriented function calling family; independent local tests warn about template incompatibility | Specialist local tool caller | **Evaluate with native template**, not generic OpenAI-compatible server only. |
| `Hammer 2.x 7B` | Tool calling | ~7B | Function-calling specialist with benchmark claims; local runtime mismatch risk | Specialist command-to-action parser | **Evaluate only after chat-template support confirmed**. |
| `Qwen3-VL-Embedding-2B` | Multimodal embedding/rerank | 2B | HF Sentence Transformers v5.4 multimodal docs/blog explicitly support multimodal embeddings | Multimodal household memory: retrieve receipt images, shelf photos, voice traces | **Evaluate now**: recall@k on text query → receipt/photo/trace. |
| Sentence Transformers v5.4 multimodal stack | Library/runtime | model-dependent | HF blog adds text/image/audio/video embedding and reranker support | One API for multimodal memory experiments | **Adopt in benchmark tooling** before app runtime. |
| `NVIDIA Canary-Qwen 2.5B` | ASR | 2.5B | HF Open ASR Leaderboard / ASR paper sources cite strong open ASR accuracy | Voice receipt notes, spoken shopping commands | **Evaluate now**: Hinglish/noisy command WER + latency. |
| `NVIDIA Parakeet TDT/CTC 0.6B–1.1B` | ASR | ≤1.1B | ASR leaderboard cites very high real-time factor for fast transcription | Realtime local voice if language coverage fits | **Evaluate speed lane**: English/Hinglish WER and streaming. |
| `IBM Granite Speech 3.3 8B` | ASR/audio LLM | ~8B/9B class | ASR leaderboard/blog sources cite strong WER and Apache license claims; verify exact card | Accurate voice command transcription | **Watch/evaluate** after license/runtime check. |
| `Voxtral Mini 4B Realtime` | ASR/audio | 4B | Mistral/HF realtime speech model candidate | Low-latency voice loop | **Evaluate**: latency + Hindi/Hinglish support; compare to Canary/Parakeet/SenseVoice. |
| `Chatterbox` | TTS | ~0.5B class | TTS roundups/trending cite strong small voice cloning/naturalness | Warm local household voice, optional personalized voice | **Evaluate**: license, voice quality, latency, pronunciation. |
| `Kokoro-82M` | TTS | 82M | Still strong tiny baseline | Default local TTS baseline | **Keep baseline**; benchmark against Qwen3-TTS/CosyVoice/Chatterbox. |
| `Qwen3-TTS 0.6B/1.7B` | TTS | ≤1.7B | vLLM-Omni supported Qwen TTS family | Higher-quality local speech output | **Evaluate**: runtime complexity + Hinglish pronunciation. |
| `PaddleOCR-VL` | OCR/doc | ~0.9B | Prior section already flagged; still top first OCR candidate | Receipt/label structured extraction | **Evaluate first** on grocery dataset. |
| `MinerU2.5-1.2B` / `LightOnOCR-1B` | OCR/doc | ~1B | HF OCR guide/community-cited newer compact OCR candidates | Receipt/label extraction, document parsing | **Evaluate now** in OCR bakeoff. |
| `SmolDocling` / `Granite-Docling` | OCR/doc | 256M-ish | Tiny document understanding candidates | CPU fallback, quick document parse | **Evaluate as fallback**, not likely top accuracy. |
| `Qwen3-Omni-30B-A3B` | Omni multimodal | 30B total / ~3B active | Fresh HF model cards; omni audio/image/video/text/speech | “One model does a lot” wow demo and benchmark oracle | **Eval-only/wow lane**: total loaded params nearly consumes cap; not reliable first core. |
| `Qwen3-VL 235B-A22B` | VLM/OCR/GUI | 235B total / 22B active | Open-source VLM leaderboards and provider writeups cite strong OCR/GUI | Reference oracle for vision/OCR quality | **Active-param-only / cloud eval**, not local Off-the-Grid default due total size. |
| `MiniMax-M2.5` | Agentic LLM | 229B total / 10B active | Open-source/self-hosted leaderboards cite strong coding/function-calling | Reference for complex planning | **API/eval-only** unless hardware plan exists. Total params exceed local cap. |
| `Kimi K2 Thinking/K2.5` | Agentic LLM | ~1T total / 32B active | Open-source leaderboards place it high for reasoning/agentic coding | High-end external judge/planner reference | **Out of local cap by total size**; can be benchmark reference with credits/API only. |

### ShopStack-specific model eval suite to create before promotion

A future implementation agent should create this as a benchmark harness before adding active runtime models:

| Eval | Dataset size | Pass criteria | Why it matters |
|---|---:|---|---|
| Receipt OCR bakeoff | 20–50 real/synthetic receipts | item/date/merchant/total field F1, correction count, latency | Core automation must not corrupt inventory/price history. |
| Packet label extraction | 20–50 packet labels | MRP/expiry/brand/size exactness, ambiguity detection | Prevent stale/expired item and wrong unit bugs. |
| Hinglish voice command | 50–100 short clips | WER + intent accuracy + valid JSON tool calls | Voice UX depends on this. |
| Tool-call schema harness | 40–100 prompts | exact tool selection, required args, no unauthorized action | Planner safety and trace correctness. |
| Multimodal memory retrieval | 30–100 image/text/audio trace entries | recall@5 and rerank quality | Enables “where did we buy this?” and “show last receipt for atta.” |
| Shelf/fridge image understanding | 30–50 photos | top-k item detection/caption confidence, false-certainty rate | Supports pantry/photo features without hallucinated writes. |
| TTS listening test | 20 generated responses | intelligibility, warmth, pronunciation, latency | Voice output should feel useful, not gimmicky. |
| Local runtime budget | all promoted models | RAM/VRAM, cold start, tokens/sec, fallback behavior | Keeps Off-the-Grid credible. |

### Research quality rules for future passes

- Do not cite a model as “best” without saying **best for what benchmark/task**.
- Always separate **official/vendor-reported** metrics from **third-party** and **our own** metrics.
- For MoE models, record both **total params** and **active params**. The project cap should default to total loaded params unless explicitly overridden for a demo/eval lane.
- A model with strong BFCL/function-calling score still needs local runtime validation because chat templates often break OpenAI-compatible serving.
- A VLM with strong MMMU/MMBench score still needs receipt/label OCR testing.
- An ASR model with low English WER still needs Hinglish and noisy-market audio tests.
- A TTS model with high community preference still needs licensing and pronunciation checks.
- Prefer deterministic libraries for core arithmetic, matching, time-series, and validation; models should propose, classify, transcribe, extract, or explain — not silently mutate state.

### Immediate next research actions

1. As Pranay provides model names, append them to the intake table with exact links and verdicts.
2. Pull exact HF model cards/licenses for the top candidates above, starting with Qwen3.5 small, Nemotron Nano, Canary-Qwen, Granite Speech, Chatterbox, Qwen3-VL-Embedding, and the OCR candidates.
3. Build the benchmark fixture list before any implementation agent wires models into runtime.
4. Update `Docs/MODEL_CATALOG.md` only after exact model card/license/runtime evidence is collected.
5. Keep a rejected/blocked list; avoiding bad models is as useful as finding good ones.

_Last updated: 2026-06-07_

---

## Design System & UI Architecture (2026-07-12)

### Context
Full design audit completed (see `Docs/audits/audit_05_full_design_audit.md`). UI surface has solid foundation but needs component library expansion, design token completion, and architectural cleanup.

### Completed
- **Split `other.py`** → `price_memory.py`, `household_map.py`, `field_notes.py`, `swiggy_market.py`
- **Moved `DEMO_SEED_INVENTORY`** → `shopstack/data/seed_demo.py`
- **CSS tokens v2**: spacing, font-size, shadow, z-index, animation scales
- **Unified `DECISION_COLORS`** with CSS variables, created `theme_tokens.py`
- **Fixed Audit 3**: removed 3 wrapper fns, private imports, extracted `_runtime_label`
- **P1 components**: `item_row`, `stat_card`, `data_table`, `confirm_dialog`, `toast`, `loading_skeleton`, `empty_state_enhanced`

### Open
- Phase 2/3 components, dark mode, keyboard shortcuts, a11y audit
- Remaining hardcoded color migration in swiggy_market, image_cards, market_lens, shopping
- Component adoption: migrate existing screens to use new components

_Last updated: 2026-07-12_

---

## Architecture Consolidation & Stash Recovery (2026-06-08)

### Context
An intentional architectural branch (stashed as WIP) performed three major
transformations: screen module extraction, design system v2, and 5-tab UX
restructure. Full architecture documented in `Docs/STASH_ARCHITECTURE_RECOVERY.md`.

### Completed Architecture Changes
- **5-tab nested UX**: 13 flat tabs → 5 nested tabs (Today, Basket, Market Lens,
  Reconcile, Memory). Groups related product surfaces.
- **other.py** → 51L re-export shim. Functions extracted to canonical modules:
  `price_memory.py`, `household_map.py`, `field_notes.py`, `swiggy_market.py`.
- **Design system v2**: 728-line token-based theme with Emil Kowalski animations,
  WCAG AA focus-visible, reduced-motion, responsive breakpoints.
- **Lazy provider loading**: ProviderRegistry defers heavy imports (funasr, torch,
  kokoro) to first `get()` call. `import app` dropped from 90s to 2.3s.
- **Local model defaults**: STT=mlx-whisper, TTS=kokoro-82M, planner=MLX/llama.cpp.
  All real models by default, mock only for off-trunk providers (vision/OCR).
- **Cost tracking + tracing**: OpenTelemetry trace_call() spans in local and
  OpenAI providers with model tier estimation.
- **semantic_find_item**: Tiered search (exact → prefix → semantic embedding)
  registered as canonical tool with ToolSpec.
- **Decision cards**: SVG-based visual cards in dashboard for buy/skip/use-soon.

### New Capabilities Added (Aug 2026-06-08)
- **Receipt scanning**: OCR → purchase events pipeline with `services/receipt.py`
- **Multi-retailer adapters**: Blinkit, Zepto, DMart source adapters with
  cross-source price comparison via `market/sources/_comparison.py`
- **Semantic inventory search**: BGE-M3 embedding provider with tiered fallback
  to prefix search
- **Dashboard intelligence**: Cadence detection, waste warnings, weather context
  all wired into DashboardState via service layer
- **Nutrition tracking**: 30-item reference database + service + UI tab
- **Weather/trip context**: Open-meteo API integration with caching and offline
  mock fallback
- **Demo walkthrough**: `scripts/demo_walkthrough.py` — idempotent, seeds inventory,
  prices, shopping list
- **Performance benchmarks**: 30 pytest-benchmark tests with results in
  `benchmarks/RESULTS.md`

### Test Health
619 passed, 0 failed, zero regressions from 597 baseline

### Decision Records (DR-005 through DR-012)
- DR-005: Screen module extraction — split other.py into canonical modules
- DR-006: Design system v2 — token-based CSS replacing flat hardcoded values
- DR-007: 5-tab nested UX — 13 flat tabs collapsed to 5 primary tabs
- DR-008: Lazy provider loading — deferred heavy imports for fast app init
- DR-009: Local model defaults — real models by default, optical upgrades
- DR-010: SVG decision cards — visual card system with PNG fallback via Flux
- DR-011: semantic_find_item — tiered search registered as canonical tool
- DR-012: Cost tracking + tracing — OpenTelemetry spans in LLM providers

### Exploration Areas Opened
- Dark mode theme variant using design tokens
- Keyboard shortcut system for power users
- Accessibility audit against WCAG 2.1 AA
- Component migration: adopt primitives (ItemRow, Toast, etc.) across screens
- Model download UX: progress indicators for initial local model downloads
- Offline-first backup/restore: encrypted household snapshots
- Mobile capture path: PWA wrapper for Gradio with local-first guarantees

---

## 14. Expansion Sweep — Cross-Project Inspired Areas (2026-06-08)

This section expands the open exploration surface using patterns from sibling
Projects docs such as `family-dinner-os` exploration audits, `invoice-intelligence`
autoresearch/evaluation docs, `idea_pad` domain census, and local ShopStack docs.
All items remain documentation-only until promoted into `ROADMAP.md` or decision
records.

### 14.1 Household Operating-System Verticals

ShopStack can stay anchored in shopping while growing into adjacent household
systems:

- **Family meal operating system** — meal debates, preferences, leftovers,
  nutrition, budget, and available stock converge into one dinner decision.
- **Caregiver / elder household mode** — low-vision voice workflows, medicine-like
  caution labels, easy “what is at home?” answers, and remote family check-ins.
- **Shared flat / hostel mode** — roommate ownership, split costs, communal vs
  personal shelves, expiry responsibility, and “who used the milk?” event trails.
- **Domestic helper / staff workflow** — voice-first Hindi/Hinglish commands,
  confirmation cards, shopping handoff lists, and audit-friendly purchase logs.
- **Baby / pet / dietary inventory** — formula, diapers, pet food, allergy-safe
  items, repeat cadence, and “never run out” thresholds.
- **Emergency prep / monsoon stock** — critical household reserves, power-cut
  foods, water, medicines, flashlights, and weather-aware restock prompts.
- **Festival / event planning** — Diwali, Eid, weddings, birthdays, guests, bulk
  shopping, perishability, and post-event leftover/waste analysis.
- **Home ops ledger** — groceries plus household consumables, repairs, cleaning,
  subscriptions, appliance filters, warranty docs, and recurring errands.

Exploration questions:

- Which vertical keeps the strongest wedge while preserving “shopping memory” as
  the core mental model?
- Which mode can be demoed with real household evidence in under 3 minutes?
- What user roles need separate permissions, not just separate labels?
- Which verticals create sensitive claims requiring stronger disclaimers?

### 14.2 Retail, Kirana, and Community Commerce Angles

ShopStack can horizontally extend from one household into local commerce memory:

- **Kirana companion** — compare home inventory against nearby kirana price/quality
  memory, not only quick-commerce snapshots.
- **Neighborhood price commons** — privacy-preserving aggregate prices for staples,
  normalized by unit, quality, and freshness.
- **Store quality memory** — subjective freshness, staff helpfulness, return issues,
  packaging quality, substitution reliability, and delivery timing.
- **Local seller mini-CRM** — seller notes, WhatsApp order templates, credit ledger,
  preferred brands, and recurring household baskets.
- **Market route planner** — “buy vegetables here, detergent there” based on
  travel time, weather, price variance, and perishability.
- **Community-supported buying** — bulk rice/oil/dal buys across apartments,
  savings split, expiry/storage planning, and consented group orders.
- **Retailer API / scraping ethics map** — what data is official, user-provided,
  manually entered, public, or not safe to automate.

Exploration questions:

- What is the trust boundary between personal memory and community data?
- How do we prevent stale community prices from misleading users?
- Can quality/freshness be represented without pretending to be objective?
- Which integrations are product-critical vs demo theater?

### 14.3 Autoresearch and Evaluation Harness

Borrow the invoice-extraction pattern: fixed benchmark, fixed evaluator, one
candidate change, keep/discard logging.

Candidate benchmark tracks:

- **Command parser benchmark** — Hinglish and English shopping commands → JSON tool
  calls, with exact intent, item, quantity, unit, location, and confidence scoring.
- **Receipt extraction benchmark** — receipt photos/text → purchase events with
  item, quantity, price, store, date, total, tax/fee ambiguity, and review flags.
- **Packet label OCR benchmark** — expiry/MRP/nutrition extraction with false-positive
  penalty for confusing MRP, sale price, manufacturing date, and best-before date.
- **Market Lens benchmark** — shelf/photo/barcode inputs → identify item, compare
  with inventory, classify buy/skip/compare, and explain uncertainty.
- **Semantic alias benchmark** — pav/bread, dahi/curd/yogurt, Surf/detergent,
  dhaniya/coriander, etc. → canonical item IDs.
- **Voice benchmark** — noisy-market audio → preserved intent and slot accuracy.
- **Decision benchmark** — inventory + market price + cadence + weather → expected
  buy/skip/use-soon/compare/watch classification.

Immutable surfaces for each run:

- benchmark manifest,
- ground-truth labels,
- scoring formula,
- dataset split,
- validation schema,
- evaluator code.

Mutable candidate surfaces:

- prompt text,
- model/provider,
- parser/OCR/vision routing,
- validation thresholds,
- retry/fallback policy,
- normalization maps,
- confidence language templates.

Artifacts to store in-project:

- `experiments/results.tsv`,
- per-run `summary.json`,
- `error_analysis.json`,
- candidate prompt/config snapshots,
- keep/discard rationale,
- hard examples promoted to fixtures.

### 14.4 Data and Schema Expansion

Potential canonical data layers:

- household item taxonomy with Indian/local variants,
- alias and brand synonym map,
- unit normalization and pack-size grammar,
- storage-location ontology,
- shelf-life and freshness rules by category,
- price-quality observation schema,
- receipt line item schema,
- packet label schema,
- nutrition/allergen schema,
- household preference schema,
- user role/permission schema,
- consent and anonymization schema for traces,
- retailer source provenance schema,
- model-run provenance schema,
- uncertainty/explanation schema.

High-leverage datasets to publish or keep internal:

- 500–2,000 human-reviewed grocery commands,
- 100 receipt examples with synthetic/redacted variants,
- 300 packet label close-ups / synthetic crops,
- 1,000 alias-to-canonical mappings,
- 100 household shelf/fridge snapshots with zones,
- 100 market photo examples with review cards,
- 50 noisy voice snippets for benchmark phrases,
- real/synthetic price observations across Swiggy, Blinkit, Zepto, DMart, kirana,
- anonymized trace set showing tool-call planning and corrections.

Exploration questions:

- Which maps belong in SQLite tables vs versioned YAML/JSON?
- Which maps need user-editable UI because every household differs?
- Which data can be safely public for hackathon/storytelling?
- What validation catches stale or contradictory data before it reaches decisions?

### 14.5 UX, Interface, and Interaction Surfaces

Open product surfaces:

- **Today as command center** — one screen for buy, skip, use soon, compare, confirm,
  and “why this recommendation?”
- **Review queue** — uncertain OCR/vision/voice events become cards to approve,
  merge, split, edit, or discard.
- **Cinematic household timeline** — family-dinner-style event playback showing how
  ShopStack reasoned from inventory, price memory, weather, and preferences.
- **Shelf map editor** — user-defined locations, shelf zones, item cards, last-seen
  photos, and confidence badges.
- **Market mode** — large buttons, short voice responses, offline-safe fallback,
  high contrast, and no dense tables.
- **Kitchen mode** — hands-free voice, recipe/use-soon prompts, timer-adjacent flow,
  and “what can I cook now?”
- **Family summary** — shareable daily/weekly household digest: what to use, buy,
  avoid, and what was wasted/saved.
- **Operator/debug view** — model used, fallback path, confidence, source data age,
  and trace export without leaking private content.
- **Onboarding wizard** — household type, preferred stores, staple items, language,
  dietary constraints, budget, and reminder preferences.

Design exploration topics:

- warm household visual language vs utilitarian inventory tool,
- mobile-first capture vs desktop-first Gradio demo,
- confidence language and iconography,
- accessibility for low vision / elder users,
- no-shame waste messaging,
- deterministic factual cards vs generated visuals,
- offline/connected mode indicators,
- empty states that teach the mental model.

### 14.6 Agent, Model, and Pipeline Architecture

Potential specialized agents/pipelines:

- **Inventory Reconciler** — merges receipt, voice, photo, and manual updates into
  lots without double-counting.
- **Market Analyst** — compares unit prices, source freshness, travel cost, and
  quality memory.
- **Waste Coach** — detects overbuy patterns and suggests practical use-soon actions.
- **Meal Planner** — proposes meals from current stock, preferences, and expiry.
- **Data Steward** — flags duplicates, stale prices, bad aliases, and low-confidence
  observations for review.
- **Privacy Guard** — redacts traces, receipts, addresses, phone numbers, faces, and
  household member names before export.
- **Benchmark Runner** — executes candidate prompts/models against fixed datasets
  and writes keep/discard results.
- **Source Adapter Auditor** — validates retailer adapters for freshness, schema
  drift, and unit-price correctness.

Pipeline questions:

- Where should rules beat models outright?
- Which model outputs must never mutate inventory without confirmation?
- How should fallback paths be visible to the user and trace logs?
- Can one canonical event ledger drive inventory, prices, traces, and explanations?
- Should ShopStack adopt event sourcing for all household changes?

### 14.7 Privacy, Safety, Trust, and Governance

Trust surfaces to design explicitly:

- local-first mode and clear cloud-mode disclosure,
- household data export/delete,
- encrypted backup snapshots,
- consent per household member,
- child/elder-safe voice interactions,
- face/address/phone redaction in images and receipts,
- private vs shareable trace boundaries,
- nutrition/allergen caution language,
- medicine-like item handling without medical claims,
- price freshness and source disclaimers,
- no auto-purchase without explicit confirmation,
- audit trail for who added/consumed/moved items,
- conflict handling when multiple users edit the same household.

Open questions:

- What is the minimum privacy story credible for real household photos?
- How do we make anonymized hackathon traces useful but safe?
- Which features should be disabled in public demo mode?
- What should be impossible for the agent to do automatically?

### 14.8 Business, Distribution, and Packaging Exploration

Potential packages:

- free local household inventory app,
- paid family sync / encrypted backup,
- premium market intelligence and retailer comparison,
- caregiver/elder household plan,
- apartment/community price commons,
- kirana/seller companion,
- meal-planning add-on,
- privacy-first offline appliance / NAS-style version,
- API/data layer for other home apps.

Distribution channels:

- Hugging Face Space demo,
- Product Hunt / X build thread,
- WhatsApp-family proof video,
- Indian home-cooking creators,
- Reddit/Discord frugal and meal-prep communities,
- apartment society pilots,
- local kirana pilot,
- caregiver and elder-tech communities,
- student hostel beta.

Positioning experiments:

- “Your home’s shopping memory.”
- “Know what you have before you buy again.”
- “The anti-overbuy grocery assistant.”
- “A local-first inventory copilot for Indian homes.”
- “From receipt to fridge to next shopping decision.”
- “A memory layer for household commerce.”

### 14.9 Platform and Integration Backlog

Potential integrations:

- WhatsApp list import/export,
- email receipt ingestion,
- browser extension for online grocery carts,
- mobile PWA capture,
- barcode/QR product databases,
- calendar/reminders,
- weather APIs,
- maps/travel-time APIs,
- smartwatch/voice shortcut,
- printer/label maker,
- CSV/JSON/SQLite backup,
- Home Assistant / smart fridge / camera feeds,
- supermarket loyalty emails where user-consented,
- nutrition databases,
- recipe APIs or local recipe notebooks.

Integration rules to define before building:

- canonical source of truth,
- data freshness,
- user consent,
- failure/fallback behavior,
- duplicate detection,
- manual correction path,
- export portability,
- test fixture strategy.

### 14.10 Repo, Documentation, and Quality Exploration

Process surfaces worth improving:

- keep `FEATURE_MAP.md`, `ROADMAP.md`, decision records, and this exploration map
  synchronized without rewriting history,
- add exploration area IDs so roadmap items can cite their origin,
- create an `experiments/README.md` describing benchmark promotion rules,
- create a data catalog for all JSON/CSV/YAML datasets,
- track “built / partially built / only explored” status per idea,
- maintain screenshot/visual evidence under `Docs/review/assets/`, not root,
- add docs for provider fallback behavior and model parameter accounting,
- add a privacy review checklist for trace/dataset publication,
- add a demo-script checklist for hackathon video reproducibility,
- create hard-example fixtures whenever a model or OCR failure is found.

### 14.11 New Open Questions

21. Should ShopStack become an event-sourced household ledger, or stay CRUD-first
    with derived event logs?
22. Which single vertical expansion is the strongest: caregiver, shared flat,
    kirana/community, meal planning, or festival planning?
23. What is the smallest fixed benchmark that prevents model/prompt regressions?
24. Which decisions require hard rule gates instead of model judgment?
25. How should household member permissions work without making onboarding heavy?
26. Can price memory combine subjective quality and objective unit price cleanly?
27. What public demo data feels real without exposing private household details?
28. Should receipt ingestion be the primary acquisition wedge because it creates
    instant inventory and price memory?
29. What is the right event model for correction: amend, reverse, merge, split, or
    overwrite?
30. Which integrations are worth building first because they compound the memory
    layer rather than distract from it?

---

## 15. HF Pipeline Model Sweep (2026-06-09)

Created a dedicated pipeline/model exploration note at
`Docs/exploration/HF_PIPELINE_MODEL_EXPLORATION_2026-06-09.md` after querying
Hugging Face Hub metadata with the local `SHOPSTACK_HF_API_KEY` from `.env`.
The key was used only for read-only discovery and was not printed or stored.

Coverage added:

- speech-to-text / market voice capture,
- text-to-speech / spoken household responses,
- planner, command parser, slot extraction, and reranking,
- OCR, receipt extraction, packet labels, document layout,
- vision, VQA, object detection, grounding, and visual similarity,
- segmentation, cropping, and background removal,
- text/image embeddings and semantic retrieval,
- image generation/editing for non-factual cards,
- privacy/redaction model surfaces,
- candidate active stacks under the 32B parameter cap,
- benchmark promotion plan for fixed evals.

Important exploration decisions:

- Do not choose models by downloads alone; require product-fit benchmarks.
- Keep factual prices, expiry dates, and inventory counts rendered by code, not
  generated into image pixels.
- Keep direct inventory mutation behind validated tool calls and confirmation.
- Treat each pipeline stage as separately benchmarked: ASR, parser, OCR, vision,
  embeddings, segmentation, and decision reasoning should have different evals.
- Distinguish submitted active models, optional fallbacks, and research-only
  candidates in future model docs.

New model families/lanes opened:

- WhisperKit/CoreML and pyannote VAD/diarization for future mobile/voice capture.
- Indic Parler TTS, VibeVoice, OmniVoice, Chatterbox for voice quality and Indian
  language support.
- PaddleOCR v5, TrOCR, Donut, LayoutLM, Table Transformer, and PP-DocLayout for
  receipts and packet labels.
- BLIP/ViLT/Pix2Struct/DETR/YOLOS/RT-DETR/CLIP/SigLIP for visual understanding and
  shelf/product matching.
- RMBG-2.0, BiRefNet, ClipSeg, SegFormer, Mask2Former for item cards and shelf zones.
- Qwen3-Embedding, multilingual-e5, Jina v3, MiniLM, Nomic, and BGE rerankers for
  semantic memory and alias search.
- Qwen Image Edit, FLUX.2 variants, Z-Image Turbo, SDXL/SD-turbo for optional visual
  assets, with deterministic overlays for factual content.

---

## 16. Travel Agency Agent Pattern Transfer (2026-06-09)

Sampled `travel_agency_agent` and `exploration_maps/travel_agency_agent` for
transferable exploration patterns. The strongest reusable ideas are not travel
features themselves; they are operating-system patterns: lifecycle pipelines,
dual-output safety boundaries, destination/context intelligence, supplier scoring,
governance, continuity, and research-status discipline.

### 16.1 Pipeline Lifecycle: Inquiry → Decision → Strategy

Travel pattern:

```text
Inquiry → Intake Normalization → Decision/Gaps → Strategy/Bundle → Safe Output
```

ShopStack equivalent:

```text
Household signal → Normalize → Gap/Decision → Action Bundle → User-safe Output
```

Candidate ShopStack lifecycle stages:

- **Signal intake** — voice command, receipt photo, packet label, fridge photo,
  shopping list edit, manual inventory update, market price observation.
- **Normalization** — canonical item, unit, quantity, store/source, location,
  confidence, provenance, timestamp.
- **Gap analysis** — missing quantity, ambiguous item, stale price, duplicate lot,
  expiry uncertainty, source freshness, low-confidence OCR/vision.
- **Decision state** — buy, skip, use soon, compare, watch, confirm, reconcile,
  split/merge, discard.
- **Action bundle** — inventory mutation proposal, price observation, reminder,
  shopping list update, review card, trace event.
- **Safe output** — household-facing explanation that excludes internal-only model
  prompts, raw OCR noise, private metadata, and unsupported claims.

Exploration question: should ShopStack formalize a canonical `HouseholdSignal →
DecisionBundle` contract similar to Travel Agency Agent's trip packet pipeline?

### 16.2 Dual-Output Safety Boundary

Travel pattern: internal operator bundle can contain richer decision context, while
traveler-facing output must be sanitized and leakage-checked.

ShopStack transfer:

- **Internal/debug bundle** — raw OCR text, model output, confidence vectors,
  fallback path, source age, prompt/run metadata, private notes.
- **Household-safe bundle** — concise explanation, action recommendation, source
  caveat, and confirmation affordance.
- **Public/demo-safe bundle** — anonymized screenshots/traces with household names,
  addresses, phone numbers, receipts, faces, and payment identifiers removed.

Potential feature:

- Add a `Strict Public Demo Mode` that blocks export/share if unsafe terms or
  unredacted receipt fields are detected.

### 16.3 Gap-State and Review-Queue Taxonomy

Travel Agent emphasizes blockers, confidence, next-action state, and operator
workbench review. ShopStack should mirror this for household automation.

Candidate gap states:

- `missing_quantity`
- `ambiguous_item`
- `ambiguous_unit`
- `possible_duplicate_lot`
- `stale_price_source`
- `expiry_date_conflict`
- `ocr_layout_uncertain`
- `vision_low_confidence`
- `needs_location_confirmation`
- `needs_household_member_confirmation`
- `unsafe_to_auto_apply`
- `source_adapter_schema_drift`

Review queue actions:

- approve,
- edit,
- merge with existing lot,
- split into multiple lots,
- mark as price-only observation,
- save as field note,
- discard,
- promote to benchmark fixture.

This would turn model uncertainty into a productive UI surface instead of hidden
backend ambiguity.

### 16.4 Destination Intelligence → Household Context Intelligence

Travel Agent has destination intelligence: weather, AQI, crime, disease,
political stability, crowding, cost of living, local laws, digital infra,
accessibility, emergencies.

ShopStack equivalent context layers:

- **Weather/seasonality** — rain, heat, humidity, monsoon spoilage, festival demand.
- **Neighborhood context** — local market day, store closures, delivery delays,
  apartment/community bulk buys.
- **Household calendar** — guests, travel, school/work schedules, fasting days,
  festivals, parties.
- **Health/diet context** — allergies, low-salt/diabetic preferences, baby/elder
  household needs, without making medical claims.
- **Appliance/storage context** — fridge space, freezer availability, power cuts,
  pantry capacity.
- **Mobility/accessibility context** — who is shopping, carrying capacity, elder
  user constraints, delivery vs walk/drive tradeoff.
- **Budget/pay-cycle context** — monthly stocking, weekly top-up, sale timing,
  price volatility.

Potential score systems:

- `buy_now_score`,
- `skip_risk_score`,
- `waste_risk_score`,
- `travel_worth_it_score`,
- `freshness_risk_score`,
- `stockout_risk_score`,
- `confidence_to_auto_apply`.

### 16.5 Supplier Reliability → Store and Source Reliability

Travel Agent explores supplier ecosystems, dark inventory, vendor reliability,
commission economics, and disruption risk. ShopStack has an analogous store/source
layer.

Store/source reliability dimensions:

- price accuracy,
- delivery punctuality,
- produce freshness,
- substitution quality,
- return/refund friction,
- stockout frequency,
- packaging quality,
- unit-size honesty,
- stale-data risk,
- adapter schema stability,
- household preference fit.

Candidate source types:

- manual household observation,
- receipt-derived observation,
- Swiggy/Blinkit/Zepto/DMart snapshot,
- kirana WhatsApp quote,
- community aggregate,
- loyalty/email receipt,
- public product database,
- synthetic/demo fixture.

Exploration question: should every price/availability recommendation carry a
`source_reliability` and `freshness_timestamp`, not just a numeric price?

### 16.6 Autonomy Gradient and Governance

Travel Agent has explicit autonomy/governance research: when AI acts, when it asks,
and how settings control autonomy.

ShopStack autonomy levels:

1. **Observe only** — save scan/receipt/voice as note, no action.
2. **Suggest** — show recommendation, no mutation.
3. **Draft action** — prefill add/consume/move/list update for approval.
4. **Auto-apply low-risk** — apply only rule-safe actions, e.g. mark a manually
   checked shopping item as purchased.
5. **Never automate** — purchases, deletes, public trace export, medical/nutrition
   claims, ambiguous inventory reductions.

Governance settings to explore:

- per-household autonomy level,
- per-source trust level,
- per-member permissions,
- confirmation thresholds by action type,
- “strict mode” for public demos,
- audit log visibility.

### 16.7 Operational Continuity and Disruption Handling

Travel Agent includes continuity/resilience thinking for supplier outages,
geopolitical disruption, and operational recovery. ShopStack should do the same for
household data and sources.

Continuity scenarios:

- quick-commerce source unavailable,
- OCR/vision model fails or times out,
- local model missing/download failed,
- stale price snapshot,
- corrupted SQLite/backup restore,
- duplicate household edits from two users,
- offline market mode,
- fridge/pantry photo too poor,
- receipt not readable,
- user disputes an auto-applied action.

Recovery UX:

- visible fallback path,
- “save for later review”,
- manual entry fallback,
- source freshness warning,
- undo/reverse event,
- backup/export prompt,
- hard-example capture for benchmark improvement.

### 16.8 Real-Validation Status Discipline

The Travel Agent exploration map is useful because it marks the gap between deep
theory and real validation. ShopStack should adopt that discipline.

Recommended statuses for exploration/roadmap items:

- `idea_only`
- `researched`
- `prototype_exists`
- `wired_in_app`
- `tested_with_fixtures`
- `tested_with_real_household_data`
- `dogfooded_in_market`
- `public_demo_safe`
- `production_ready`

Add validation metadata:

- evidence path,
- test command,
- data source,
- last verified date,
- known failure modes,
- privacy status,
- benchmark score if applicable.

### 16.9 Content Prism and Free-Tool Strategy

Travel Agent maps every research topic into content, SEO tools, and multiple
audiences. ShopStack can do a household-commerce version.

Three-audience prism:

- **Household user** — practical: save money, avoid waste, know what is at home.
- **Builder/judge** — small-model, local-first, multimodal, benchmarked pipeline.
- **Retail/community partner** — source reliability, price memory, local commerce.

Free-tool ideas:

- grocery unit-price calculator,
- “is this price high?” checker,
- receipt-to-spend summary,
- food waste estimator,
- pantry stockout calculator,
- festival shopping planner,
- rainy-day grocery checklist,
- bulk-buy break-even calculator,
- expiry label parser demo,
- household alias dataset explorer.

### 16.10 New Open Questions from Travel-Agent Transfer

31. Should ShopStack create a canonical `DecisionBundle` object that separates
    internal/debug fields from household-safe output?
32. What is the exact autonomy gradient for add, consume, move, delete, export,
    and purchase-like actions?
33. Should stale source data automatically downgrade recommendations to “watch” or
    “confirm” instead of buy/skip?
34. What score formulas are explainable enough for `travel_worth_it`, `waste_risk`,
    and `stockout_risk`?
35. Which review-queue actions are essential for real household correction flows?
36. Can every failed OCR/vision/parser case become a benchmark fixture with one
    click?
37. What is the strict public-demo leakage guard for receipts, traces, images, and
    voice?
38. Which household context layers matter first: weather, calendar, budget,
    storage capacity, or mobility?
39. Should store/source reliability become a first-class table alongside price
    observations?
40. How do we track exploration maturity without making docs heavy to maintain?

---

## 17. ShopStack Deep Exploration Status Map — Travel-Agent-Style Inventory (2026-06-09)

This is the deeper exploration structure that should have been added first: a
status-and-gap map like `travel_agency_agent/EXPLORATION_MAP.md`, but for
ShopStack. It is intentionally broad and detailed. It distinguishes what exists,
what is only theorized, what is missing, and what deserves priority.

### 17.0 Status Legend

| Symbol | Meaning |
|---|---|
| ✅ | Built/researched with concrete docs, code, tests, or verified evidence |
| 🟡 | Partially explored or partially built; needs validation or completion |
| ❌ | Identified gap; little/no durable research or implementation |
| 🔵 | New angle opened by this exploration pass |
| 🧪 | Needs benchmark/dogfood evidence before promotion |

Priority scale:

- **P0** — existential for credible demo/product wedge.
- **P1** — critical for launch-quality product.
- **P2** — strategic expansion / moat.
- **P3** — moonshot or later-stage platform option.

---

### A. Household Customer & User Research

| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| A1. Real household interviews | ❌ | No structured interviews with Indian households, parents, students, shared flats, helpers, elder/caregiver users | **P0** |
| A2. Day-in-the-life shopping shadowing | ❌ | No observed shopping trips from list-making through purchase/reconcile/use-soon | **P0** |
| A3. Domestic helper workflow research | 🔵 | Helper-led shopping/inventory is a distinct user role with language/trust constraints | P1 |
| A4. Shared-flat / hostel behavior | 🔵 | Ownership, split-cost, “who used it,” and low-friction correction patterns unknown | P1 |
| A5. Elder/caregiver household research | 🔵 | Accessibility, low-vision, medicine-adjacent safety, and remote check-in needs unvalidated | P1 |
| A6. Willingness-to-pay | ❌ | No pricing validation: free local tool vs family sync vs market intelligence vs caregiver plan | P1 |
| A7. Non-consumption research | 🔵 | Why do households not use inventory apps? Friction, shame, setup burden, habit failure | **P0** |
| A8. Cultural/language phrase research | 🟡 | Hinglish examples exist, but no corpus of real household commands | **P0** |

### B. Core Product Wedge & Feature Validation

| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| B1. Wedge product identification | 🟡 | Many features exist; not validated which one makes users come back daily | **P0** |
| B2. Receipt-to-inventory wedge | 🟡 | Pipeline discussed/built partially; needs real receipt dogfood and review UX | **P0** |
| B3. “Do I need this?” Market Lens wedge | 🟡 | Strong concept; needs real market photo/audio tests | **P0** |
| B4. Use-soon / waste prevention wedge | 🟡 | Logic exists; needs household behavior validation and no-shame UX | P1 |
| B5. Price memory wedge | 🟡 | Swiggy/source adapters exist; real user perceived value untested | P1 |
| B6. Household map / find item wedge | 🟡 | Built conceptually; needs spatial UX validation | P1 |
| B7. Meal planning from stock | 🔵 | High adjacent value but may distract unless tied to use-soon/waste | P2 |
| B8. Festival/event planning | 🔵 | Strong India-specific seasonal wedge; not researched | P2 |
| B9. Caregiver “never run out” mode | 🔵 | High trust/value vertical; requires safety and permissions | P2 |

### C. Inventory, Memory, and Data Model

| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| C1. Inventory lots and CRUD | ✅ | Built/tested; needs real-world correction flow stress tests | P0 |
| C2. Event ledger / audit trail | 🟡 | Traces exist, but canonical event-sourced household ledger is not formalized | **P0** |
| C3. Lot merge/split/correction model | ❌ | Real receipts/photos will create duplicate/partial lots; correction semantics missing | **P0** |
| C4. Multi-user household model | ❌ | User columns exist but no real roles/permissions/conflict model | P1 |
| C5. Household location ontology | 🟡 | Locations seeded; user-editable shelf/zone model needs research | P1 |
| C6. Item taxonomy and aliases | 🟡 | Canonicalization exists, but no robust Indian household taxonomy/dataset | **P0** |
| C7. Unit and pack grammar | 🟡 | Unit parsing exists in market data; packet/receipt/general grammar needs hard examples | P1 |
| C8. Shelf-life/freshness rules | 🟡 | Produce metadata exists; broader item shelf-life database incomplete | P1 |
| C9. Preference memory | 🟡 | Field notes/preference ideas exist; no canonical preference schema | P2 |
| C10. Backup/restore/encryption | 🟡 | Portability exists; encrypted household snapshots need exploration | P1 |

### D. Shopping List, Basket, and Decision Logic

| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| D1. Shopping list CRUD/complete | ✅ | Built/tested; needs multi-user and real purchase reconciliation | P0 |
| D2. Buy/skip/use-soon classification | ✅/🟡 | Built; needs real-world benchmark and explanation calibration | **P0** |
| D3. Basket-level recommendations | 🟡 | Item-level decisions exist; basket budget/travel/source optimization incomplete | P1 |
| D4. Confirmation thresholds | ❌ | No formal autonomy/confirmation policy by action risk | **P0** |
| D5. Review queue for uncertain decisions | ❌ | Critical missing UI/contract for OCR/vision/model ambiguity | **P0** |
| D6. Budget-aware decisions | 🔵 | No household budget/pay-cycle integration | P2 |
| D7. Substitute recommendations | 🔵 | Needs item taxonomy, preferences, price, and availability | P2 |
| D8. “Watch” state for stale/uncertain data | 🔵 | Decision taxonomy should downgrade stale-source recommendations | P1 |

### E. Receipts, OCR, Packet Labels, and Purchase Reconciliation

| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| E1. Real receipt OCR | 🟡 | Tesseract works better than GLM-OCR on real photos; robust pipeline not finalized | **P0** |
| E2. Receipt-to-purchase event schema | 🟡 | Service exists; needs review, merge/split, tax/fees, store/source provenance | **P0** |
| E3. Packet expiry/MRP extraction | ❌ | Core in-store/home scan use case; no benchmark or UX | **P0** |
| E4. Indian receipt layout corpus | ❌ | Need kirana, supermarket, pharmacy, delivery, thermal, mixed-language examples | **P0** |
| E5. OCR preprocessing | 🔵 | Deskew, crop, contrast, binarize, dewarp, table regions not evaluated | P1 |
| E6. MRP vs sale price disambiguation | ❌ | High-risk; wrong price memory if confused | P1 |
| E7. Manufacturing vs expiry/best-before | ❌ | High-risk; needs hard rule/model benchmark | P1 |
| E8. Receipt privacy redaction | 🟡 | Trace redaction exists; receipt image/text redaction needs stricter rules | P1 |
| E9. Email/WhatsApp receipt ingestion | 🔵 | Strong acquisition path; not researched | P2 |

### F. Voice, Conversational UX, and Multilingual Support

| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| F1. Voice add command | ✅/🟡 | Tests exist; needs real audio and Hinglish corpus | **P0** |
| F2. Market-mode voice | ❌ | No noisy outdoor benchmark or short-response UX validation | **P0** |
| F3. Hindi/Hinglish support | 🟡 | Examples exist; no real corpus, ASR eval, or fallback strategy | **P0** |
| F4. Voice correction flow | ❌ | “No, this is onion not potato” needs explicit state machine | P1 |
| F5. TTS model selection | 🟡 | HF candidates mapped; no pronunciation/listening benchmark | P1 |
| F6. Wake-word / push-to-talk | 🔵 | Hands-free kitchen mode vs privacy risk not explored | P2 |
| F7. Multi-speaker household audio | 🔵 | Speaker diarization/user attribution not explored | P3 |

### G. Vision, Market Lens, Shelf/Fridge Understanding

| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| G1. Barcode + image scan surface | ✅/🟡 | Built/tested partially; real scan quality unknown | P0 |
| G2. Product photo identification | 🟡 | VLM candidates exist; no fixed household/market image benchmark | **P0** |
| G3. Fridge/pantry scan | 🔵 | Huge value; needs zone model, confidence, and correction UX | P1 |
| G4. Shelf/location snapshot memory | 🔵 | Same-shelf/same-item embeddings and last-seen memory not built | P2 |
| G5. Freshness/ripeness detection | 🔵 | Valuable for produce; high false-claim risk | P2 |
| G6. Segmentation/cropping | 🟡 | RMBG exists; ClipSeg/BiRefNet/RMBG-2.0 not benchmarked | P1 |
| G7. Visual uncertainty UX | ❌ | Need review cards and “not sure” states; avoid overconfident wrong labels | **P0** |
| G8. AR/find-item mode | 🔵 | Interesting later; depends on spatial memory maturity | P3 |

### H. Market Intelligence, Retail Sources, and Store Reliability

| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| H1. Swiggy data source | ✅/🟡 | Built snapshot loader; freshness and real coverage limits need product copy | P0 |
| H2. Blinkit/Zepto/DMart adapters | 🟡 | Mentioned/built in docs/tests; need current verification and source reliability map | P1 |
| H3. Kirana/manual quote memory | 🔵 | Critical India-local angle; no schema/workflow | P1 |
| H4. Store/source reliability scoring | 🔵 | Price alone is insufficient; need freshness, substitutions, returns, punctuality | P1 |
| H5. Community price commons | 🔵 | Strong moat; privacy/staleness/trust issues unsolved | P2 |
| H6. Basket-level store choice | 🔵 | Compare total basket + travel + freshness, not item price only | P1 |
| H7. Price anomaly detection | 🟡 | Basic price memory exists; no robust anomaly/freshness scoring | P1 |
| H8. Scraping/API ethics | ❌ | Need official/user-provided/public/unsafe source classification | P1 |
| H9. Delivery vs walk/drive tradeoff | 🔵 | Travel-worth-it score needs weather/time/carrying capacity | P2 |

### I. Household Context Intelligence

| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| I1. Weather/trip context | 🟡 | Schema/docs exist; needs decision integration and validation | P1 |
| I2. Calendar/event context | 🔵 | Guests, festivals, fasting, school/work schedules not modeled | P2 |
| I3. Storage capacity context | 🔵 | Fridge/freezer/pantry capacity affects bulk-buy and freshness decisions | P2 |
| I4. Health/diet/allergy context | 🔵 | Useful but safety-sensitive; no medical claims | P2 |
| I5. Budget/pay-cycle context | 🔵 | Monthly stocking and payday cycles likely matter; unresearched | P2 |
| I6. Mobility/accessibility context | 🔵 | Elder/caregiver and carrying capacity decisions unmodeled | P2 |
| I7. Seasonality/festival demand | 🔵 | India-specific product moat; no dataset | P2 |
| I8. Power-cut/monsoon resilience | 🔵 | Household ops vertical; freshness and emergency stock implications | P3 |

### J. AI Pipeline, Model Ops, and Benchmarks

| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| J1. Model registry | ✅/🟡 | Exists; docs drift vs code must be continuously synced | P0 |
| J2. HF pipeline model sweep | ✅ | Added 2026-06-09 exploration doc; still needs benchmarks | P1 |
| J3. Command parser benchmark | ❌ | Must exist before changing planner defaults | **P0** |
| J4. Receipt OCR benchmark | 🟡 | Some evidence exists; needs fixed manifest and real labels | **P0** |
| J5. Vision benchmark | ❌ | Need product/photo benchmark for Market Lens/fridge scans | **P0** |
| J6. Voice benchmark | ❌ | Need audio fixtures for ASR/TTS choices | **P0** |
| J7. Alias/embedding benchmark | ❌ | Needed before semantic search promotion | P1 |
| J8. Decision benchmark | ❌ | Need expected buy/skip/use-soon cases with inventory/source context | **P0** |
| J9. Cost/latency model routing | 🟡 | Local/cloud providers exist; routing by task complexity not fully documented | P1 |
| J10. Fine-tuned parser LoRA | 🟡 | Candidate exists; dataset/training/eval not done | P1 |
| J11. Model drift/error capture | 🔵 | Corrections should feed benchmark/fine-tune data | P1 |

### K. Privacy, Safety, Trust, and Governance

| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| K1. Trace redaction | ✅/🟡 | Exists; needs image/receipt/audio extension | P0 |
| K2. Public demo strict mode | 🔵 | Needed for hackathon/social sharing safety | **P0** |
| K3. Household data export/delete | 🟡 | Portability exists; privacy UX and deletion semantics need work | P1 |
| K4. Encrypted backup | 🔵 | Important for trust; not built | P1 |
| K5. Nutrition/health claim guardrails | ❌ | Nutrition fields exist; no safety language/policy | P1 |
| K6. Auto-action governance | ❌ | Need autonomy levels, confirmation thresholds, never-automate list | **P0** |
| K7. Multi-user permissions | ❌ | Needed before shared household/caregiver modes | P1 |
| K8. Receipt/image PII redaction | ❌ | Critical for public trace/dataset publication | **P0** |
| K9. Source freshness disclaimers | 🟡 | Some copy exists; should be uniform across all recommendations | P1 |

### L. UX, Design, Accessibility, and Operator Workbench

| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| L1. Decision-first Today | ✅/🟡 | Implemented per docs; needs real usage validation | P0 |
| L2. Review queue/workbench | ❌ | Missing central surface for uncertain signals | **P0** |
| L3. Market-mode mobile UX | ❌ | Gradio desktop-ish UI may fail in market; needs mobile/touch flow | **P0** |
| L4. Kitchen hands-free UX | 🔵 | Useful; needs voice/privacy tradeoff research | P2 |
| L5. Accessibility / low vision | 🟡 | CSS/design notes exist; no real audit for elder/low-vision users | P1 |
| L6. Empty-state onboarding | 🔵 | Setup burden is existential; needs sample data/import wizard | P1 |
| L7. No-shame waste UX | 🔵 | Waste prevention can feel judgmental; needs tone exploration | P1 |
| L8. Debug/operator view | 🟡 | Runtime diagnostics exist; per-decision provenance view missing | P1 |
| L9. Shareable cards/reports | 🟡 | Image gen/card ideas exist; factual deterministic rendering policy needed | P2 |

### M. Integrations and Platform Surface

| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| M1. WhatsApp import/export | 🔵 | High India fit; no implementation/research | P1 |
| M2. Email receipt ingestion | 🔵 | Strong passive data capture; privacy/auth complexity | P2 |
| M3. Browser extension for grocery carts | 🔵 | Strong online-shopping wedge; not researched | P2 |
| M4. Mobile/PWA capture path | 🟡 | Mentioned; needs concrete architecture and test | P1 |
| M5. Calendar/reminder integration | 🔵 | Use-soon and shopping reminders; not researched | P2 |
| M6. Label printer / pantry labels | 🔵 | Nice household ops extension | P3 |
| M7. Home Assistant/smart fridge/camera | 🔵 | Future ambient inventory path | P3 |
| M8. Public product/barcode DBs | 🟡 | Barcode decode exists; product metadata source strategy missing | P1 |

### N. Business Model, Distribution, and Growth

| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| N1. Hackathon positioning | 🟡 | Product story exists; needs final wedge and proof video | **P0** |
| N2. Product naming/positioning | 🟡 | ShopStack/ShopStock architecture clarified; tagline tests missing | P1 |
| N3. Pricing/packaging | ❌ | No validation for free/local vs paid sync/market/caregiver plans | P2 |
| N4. Community launch channels | 🟡 | Channels listed; no execution plan or asset map | P1 |
| N5. Free tools / SEO | 🔵 | Many ideas; no priority or build plan | P2 |
| N6. Apartment/community pilot | 🔵 | Strong distribution wedge; needs privacy/community research | P2 |
| N7. Kirana/seller partnership | 🔵 | Potential B2B2C angle; needs validation | P3 |
| N8. Data/dataset publishing story | 🟡 | Dataset ideas exist; no public/private split finalized | P1 |

### O. Operations, QA, Docs, and Project Hygiene

| Area | Status | Critical Gap | Priority |
|---|---|---|---|
| O1. Feature map / roadmap / decision records | ✅/🟡 | Exist; some drift visible, needs regular sync | P0 |
| O2. Exploration maturity tracking | 🔵 | Need statuses/evidence paths like Travel Agent | P1 |
| O3. Demo reproducibility | 🟡 | Demo script exists; final hackathon walkthrough checklist needed | P1 |
| O4. Data catalog | ❌ | JSON/CSV/YAML sources need canonical catalog and validation | P1 |
| O5. Benchmark results registry | 🟡 | Benchmark docs exist; model/task eval registry missing | P1 |
| O6. CI/deployment verification | 🟡 | Docs mention gaps/drift; current status needs verification | P1 |
| O7. Screenshot/artifact handling | 🟡 | Need disciplined `Docs/review/assets/` policy | P2 |
| O8. Public limitation/claims docs | 🟡 | Need clear limitations: stale prices, OCR errors, no medical guarantees | **P0** |

### P. New Research Areas Not Previously Covered Deeply

1. **Household correction semantics** 🔵 — amend/reverse/merge/split/overwrite event
   model for all inventory and receipt mistakes.
2. **Strict public-demo safety system** 🔵 — block unsafe trace/image/receipt export.
3. **Source reliability and freshness scoring** 🔵 — every external price/availability
   needs trust metadata.
4. **Review queue as product core** 🔵 — turn uncertain model outputs into a daily
   correction workbench.
5. **Household autonomy gradient** 🔵 — action risk classification and governance.
6. **Real household command corpus** 🔵 — Hinglish, English, Hindi, corrections,
   noisy-market utterances.
7. **Indian receipt/packet label corpus** 🔵 — kirana, supermarket, pharmacy,
   quick-commerce, bilingual, thermal, crumpled, low-light.
8. **No-shame waste coaching** 🔵 — behavior design for reducing waste without guilt.
9. **Kirana/manual quote memory** 🔵 — bridge offline retail and quick-commerce data.
10. **Household context scoring** 🔵 — weather, calendar, storage, budget, mobility,
    seasonality.
11. **Community price commons** 🔵 — privacy-preserving neighborhood price data.
12. **Caregiver/elder household mode** 🔵 — accessibility, permissions, remote family
    trust, never-run-out items.
13. **Apartment/shared-flat mode** 🔵 — split costs, ownership, accountability,
    shared vs personal shelves.
14. **Passive ingestion moat** 🔵 — email receipts, WhatsApp orders, browser carts,
    loyalty messages.
15. **Model-error-to-benchmark loop** 🔵 — one-click promotion of failures into evals.

---

### 17.1 Research Priority Recommendations

#### Tier 1 — Existential / Must Do for Credible Demo

| # | Research Area | Why | Effort |
|---|---|---|---|
| 1 | Wedge validation: receipt-to-inventory vs Market Lens vs use-soon | Current scope is broad; need one irresistible demo path | 2-4 days |
| 2 | Real household command corpus + parser benchmark | Voice/text is core; without eval, model choices are vibes | 2-3 days |
| 3 | Real receipt + packet OCR benchmark | Receipt ingestion is likely the strongest passive wedge | 3-5 days |
| 4 | Review queue contract and UX | Every imperfect model needs correction; this is core product safety | 2-4 days |
| 5 | Autonomy/confirmation policy | Prevent unsafe mutation and overclaiming | 1-2 days |
| 6 | Public demo strict privacy mode | Required before sharing traces/screenshots/receipts | 1-2 days |
| 7 | Market Lens real-photo dogfood | Validate “Do I need this?” in actual shopping context | 2-3 days |
| 8 | Limitations/claims doc | Must be honest about stale prices, OCR, nutrition, privacy | 1 day |

#### Tier 2 — Critical for Launch Quality

| # | Research Area | Why | Effort |
|---|---|---|---|
| 9 | Item taxonomy + aliases dataset | Powers parser, search, receipts, embeddings, substitutions | 3-5 days |
| 10 | Source reliability scoring | Price intelligence needs trust/freshness, not just numbers | 2-3 days |
| 11 | Lot correction semantics | Real data will be messy; correction must be first-class | 2-3 days |
| 12 | Mobile/market-mode UX | Desktop Gradio is not enough for shopping context | 2-4 days |
| 13 | TTS/STT benchmark | Voice-first claim requires real audio evidence | 2-3 days |
| 14 | Data catalog and validation | Data/config is product architecture | 1-2 days |
| 15 | Encrypted backup/export/delete story | Trust and portability for private household data | 2-4 days |

#### Tier 3 — Strategic / Moat

| # | Research Area | Why | Effort |
|---|---|---|---|
| 16 | Kirana/manual quote memory | India-local differentiation | 1 week |
| 17 | Community price commons | Network/data moat, but trust-heavy | 1-2 weeks |
| 18 | Caregiver/elder mode | High-value vertical with trust needs | 1 week |
| 19 | Festival/event planning | Strong seasonal Indian household behavior | 3-5 days |
| 20 | Browser/WhatsApp/email passive ingestion | Creates compounding memory with less manual entry | 1-2 weeks |
| 21 | Household context scoring | Differentiates from plain inventory apps | 1 week |
| 22 | Programmatic free tools/content | Acquisition engine after product wedge is clear | ongoing |

---

### 17.2 Exploration Statistics Snapshot

Approximate status across the table above:

| Metric | Approx count |
|---|---:|
| Deep categories | 15 |
| Specific rows/status items | 120+ |
| Clearly built or strongly documented | 15-20 |
| Partial / needs validation | 35-45 |
| True gaps | 35-45 |
| New angles opened | 40+ |
| P0 existential items | 20+ |

Takeaway: ShopStack has a lot built, but the main gap is **real-world validation
and correction infrastructure**. The permanent product path is not “add more
models”; it is **household signal → normalized memory → safe decision bundle →
review/confirmation → benchmark feedback loop**.

---

## 18. Exhaustive Scenario, Persona, Edge-Case, and Benchmark Expansion (2026-06-09)

This section fills the remaining obvious exploration gaps: concrete scenarios,
personas, lifecycle maps, failure catalogs, model-by-task benchmarks, data moat,
commercial expansion, free tools, operations, and safety/regulatory surfaces.

### 18.1 Persona Matrix

| Persona | Core job | Pain | ShopStack wedge | Special constraints |
|---|---|---|---|---|
| Primary household shopper | Buy what is needed, avoid duplicates | Mental inventory burden | “Do we need this?” + list + receipt reconcile | Speed, trust, low setup |
| Secondary family member | Check/add items occasionally | Doesn’t know where things are | Ask/search/find item | Simple UI, limited permissions |
| Domestic helper | Execute shopping/restocking tasks | Language, confirmation, accountability | Voice list, photo receipt, review queue | Hindi/Hinglish, no complex forms |
| Elder user | Know stock and avoid running out | Low vision, memory, mobility | Voice-first never-run-out mode | Accessibility, caregiver permissions |
| Caregiver/remote family | Monitor essentials for someone else | Can’t see what’s at home | Shared stock dashboard + alerts | Privacy/consent, low false alarms |
| Student/hostel user | Manage cheap staples | Budget, shared fridge chaos | Budget basket + shared shelf | Low cost, mobile-first |
| Shared-flat roommate | Split shared vs personal items | Disputes and freeloading | Ownership, consume log, split cost | Social friction, audit trail |
| Parent/home cook | Plan meals from stock | Waste and repeated shopping | Use-soon meals + festival/event planning | No-shame tone, dietary preferences |
| Budget optimizer | Spend less over time | Unit-price confusion | Price memory + high-price alerts | Accurate source freshness |
| Health-conscious shopper | Track nutrition/allergens | Label overload | Packet label parser + caution flags | No medical overclaims |
| Quick-commerce power user | Compare apps quickly | App switching, dynamic prices | Basket comparison + source memory | Stale data warnings |
| Kirana-first shopper | Buy locally/offline | No digital data | Manual quote/receipt memory | Offline-first, WhatsApp-friendly |
| Apartment/community admin | Coordinate bulk buys | Demand aggregation, fairness | Community basket + split logistics | Consent, settlement, governance |
| Kirana owner | Serve repeat households | Remember preferences/orders | Seller-side household memory | Separate product, B2B trust |
| Hackathon judge/builder | Understand small-model value | Demo clarity | Transparent model/pipeline evidence | Field notes, benchmark artifacts |

### 18.2 Household Scenario Library

#### Before Shopping

- User asks “what should I buy today?” and expects low-stock, use-soon, and price-aware suggestions.
- User creates list from voice while cooking.
- User checks whether staples are enough for the week.
- User plans shopping around rain/heat/market day.
- User needs a budget basket under ₹500.
- User checks if Sunday market is worth the trip.
- User wants to avoid buying duplicate detergent/rice/oil.
- User plans groceries before guests arrive.
- User plans festival shopping with bulk sweets/oil/flour/dry fruits.
- User needs a “quick top-up” list after returning from travel.

#### During Shopping

- User scans tomatoes and asks if price is high.
- User scans a packet and asks whether it is already at home.
- User asks if the visible expiry date is too soon.
- User asks which pack size is cheaper by unit.
- User sees substitute brand and asks if household usually accepts it.
- User records a kirana quote by voice.
- User asks whether to walk to another store for cheaper produce.
- User corrects model: “no, this is coriander not spinach.”
- User cannot read small MRP/expiry text and needs OCR.
- User is in noisy market and wants one-sentence answer.

#### After Shopping

- User photographs receipt and wants inventory updated.
- User photographs all groceries on the counter.
- User marks purchased list items and adds prices.
- User splits receipt into inventory vs household expense only.
- User records store freshness rating.
- User reconciles duplicate lots after OCR error.
- User adds expiry dates from packet closeups.
- User moves items to fridge/pantry/bathroom cabinet.
- User shares weekly spend/waste summary.
- User exports redacted trace for demo/field notes.

#### At Home

- User asks where the dahi is.
- User asks what can be cooked without shopping.
- User asks what expires first.
- User consumes half a packet and updates quantity.
- User finds old item hidden behind shelf.
- User scans fridge before leaving for store.
- User wants lunchbox ideas from use-soon items.
- User asks if baby formula/pet food is enough.
- User gets reminder before milk/bread stockout.
- User wants to reduce waste without guilt.

#### Shared Household / Caregiver

- Roommate consumes shared milk; others need visibility.
- One user buys item another already bought.
- Helper buys groceries and sends receipt photo.
- Caregiver checks elder household essentials remotely.
- Parent asks kid to check fridge with photo.
- Apartment group coordinates bulk rice/oil order.
- Household member disputes an auto-applied consume event.
- User wants private personal shelf hidden from others.
- Remote family wants alerts only for critical items.
- Helper needs voice-only shopping confirmation.

### 18.3 Lifecycle Maps

#### Item Lifecycle

```text
Need detected → list candidate → bought → receipt/price observed → stored → moved
→ opened → partially consumed → use-soon → consumed / wasted / donated → learned
```

State questions:

- What event created the item?
- Which source is trusted for quantity/price/expiry?
- Is this a new lot or duplicate?
- Who confirmed it?
- What correction history exists?
- What did the system learn for next time?

#### Shopping Decision Lifecycle

```text
Signal → normalize → compare against inventory → enrich with price/source/context
→ classify → explain → confirm → apply → trace → benchmark failure if corrected
```

#### Receipt Lifecycle

```text
Image/email/WhatsApp → preprocess → OCR → line grouping → structured extraction
→ source/price provenance → duplicate detection → review queue → inventory/price events
```

#### Model Failure Lifecycle

```text
Bad output → user correction → capture raw input + expected output → redact → fixture
→ benchmark run → candidate prompt/model/config → keep/discard log
```

### 18.4 Edge-Case and Failure Catalog

| Category | Edge cases to explore |
|---|---|
| OCR/receipts | crumpled receipt, thermal fading, mixed Hindi/English, right-aligned prices, discounts, taxes, delivery fees, refund lines, unreadable totals, duplicate receipt photo |
| Packet labels | MRP vs sale price, mfg vs expiry vs best-before, date format ambiguity, batch number mistaken as date, tiny font, curved packet, reflective packaging |
| Voice | negation missed, noisy market, mixed language, brand homophones, quantity omitted, correction utterance, multi-speaker command, child voice |
| Inventory | duplicate lot, partial consume, item moved without logging, stale quantity, opened vs unopened, household member disagreement, unit conversion error |
| Market price | stale app snapshot, unavailable item, combo pack, delivery fee hidden, quality lower despite cheaper price, store substitution, region mismatch |
| Vision | similar produce, occluded shelf, transparent containers, unlabeled jars, multiple items in one frame, overconfident wrong VLM label |
| Shared household | two users edit same item, private vs shared item, helper permissions, remote caregiver override, split-cost dispute |
| Privacy | receipt phone/address, UPI/card token, face in fridge photo, child voice/audio, household names in trace, location metadata in image |
| Safety | nutrition overclaim, medicine-like item advice, allergy false negative, spoiled food missed, auto-delete or auto-purchase risk |
| Offline/runtime | model missing, HF download fails, OCR timeout, DB locked, backup restore conflict, source adapter schema drift |

### 18.5 Feature Universe Backlog

| Cluster | Feature ideas |
|---|---|
| Capture | voice command, receipt photo, packet scan, fridge photo, shelf photo, barcode, email receipt, WhatsApp receipt/list, browser cart import |
| Normalize | item aliases, brand mapping, unit grammar, pack-size parser, expiry parser, store/source canonicalization, household location ontology |
| Decide | buy/skip/use-soon/watch/compare/confirm, budget basket, stockout prediction, waste risk, travel-worth-it, source reliability, substitution |
| Reconcile | review queue, merge/split lots, undo/reverse event, duplicate receipt detection, price-only observation, correction-to-fixture |
| Remember | price history, store quality, household preferences, recurring cadence, shelf locations, last-seen photo, field notes, model traces |
| Act | shopping list, reminders, label print, calendar reminder, family summary, helper handoff, caregiver alert, community bulk-buy request |
| Explain | why buy/skip, source freshness, confidence, fallback path, data provenance, no-medical-claim caveat, stale price warning |
| Share | redacted trace export, weekly digest, spend/waste report, public demo dataset, field notes, shareable card |
| Admin | model status, source adapter health, data catalog, backup/export/delete, autonomy settings, permissions, privacy guard |

### 18.6 Model-by-Task Benchmark Matrix

| Task | Candidate models/tools | Metric | Must-have fixture set |
|---|---|---|---|
| ASR command transcription | SenseVoice, Qwen3-ASR, Parakeet, Whisper, WhisperKit | WER + slot retention + latency | 100 Hinglish/English/noisy commands |
| Intent classification | Qwen small, Gemma small, DistilBERT, rules | intent F1, confusion matrix | add/move/consume/find/price/check/correct |
| Slot extraction | parser LoRA, Qwen, token classifier, rules | exact item/qty/unit/location accuracy | household command corpus |
| Tool-call generation | Qwen3.5/4B, MiniCPM5, GGUF Llama/LFM | valid JSON, executable calls, no unsafe action | 200 command/action cases |
| Receipt OCR | Tesseract, PaddleOCR, GLM-OCR, TrOCR, Donut | field accuracy, row grouping, latency | 100 Indian receipts |
| Packet OCR | OCR + VLM | expiry/MRP/date disambiguation | 100 packet labels |
| VLM item ID | MiniCPM-V, Qwen2.5-VL, BLIP/ViLT baselines | top-1/top-3, abstain quality | 100 market/fridge photos |
| Object detection | DETR, YOLOS, RT-DETR, table/document detectors | box recall/precision | shelf/receipt/table crops |
| Segmentation | RMBG, BiRefNet, ClipSeg, SegFormer | mask IoU, visual usefulness | item card/shelf zone images |
| Embeddings | BGE-M3, Qwen3 Embedding, E5, Jina, MiniLM | top-k alias retrieval | 1k alias/canonical pairs |
| Reranking | BGE reranker | top-1 after retrieve | hard alias negatives |
| Decision classifier | rules + model explanations | expected class accuracy + explanation faithfulness | inventory+price+context scenarios |

- **Current STT note:** `Qwen3-ASR-1.7B` is now benchmarked on the 20-command Hinglish set with transliteration-aware scoring: **34.46% WER**, **55.6% slot retention**, **1.11s** latency. It outputs Devanagari script, so the quality score only makes sense after Latin transliteration.
| TTS | Kokoro, Qwen TTS, Indic Parler, CosyVoice, VibeVoice | pronunciation, MOS, latency | grocery terms + short replies |
| Privacy redaction | regex, NER, OCR boxes, face detector | leak recall, false positives | receipts/images/traces/audio transcripts |

### 18.7 Data Moat and Dataset Strategy

| Dataset | Public? | Why valuable | Risks |
|---|---|---|---|
| Household command corpus | Public synthetic + private real | Parser/fine-tune moat | Household language privacy |
| Indian item alias graph | Public | Core search/parser value | Regional bias/staleness |
| Receipt OCR corpus | Mostly private/redacted synthetic | Receipt-to-inventory benchmark | Phone/address/payment leakage |
| Packet label corpus | Public/private mix | Expiry/MRP extraction | Brand/copyright/photos |
| Price observation history | Private/aggregate public | Store/source intelligence | Stale/misleading prices |
| Store reliability observations | Private/aggregate public | Quality moat beyond price | Defamation/fairness concerns |
| Fridge/shelf images | Private only by default | Spatial memory/VLM eval | Faces/home privacy/location metadata |
| Correction traces | Redacted public subset | Model improvement and Field Notes | Raw private content leakage |
| Use-soon/waste outcomes | Private/aggregate | Waste coaching efficacy | Shame/sensitive household behavior |
| Community basket data | Consent-only aggregate | Group buying/network effects | Consent/governance/trust |

### 18.8 Commercial and Expansion Map

| Segment | Product | Why it could work | Research needed |
|---|---|---|---|
| Individual households | Free/local inventory + shopping memory | Broad wedge, privacy-friendly | Setup friction, retention |
| Families | Sync, shared permissions, reminders | Multi-user value | Permissions, pricing |
| Caregivers | Never-run-out essentials + remote alerts | High pain/trust | Safety, elder UX |
| Shared flats/hostels | Shared pantry + split costs | Clear social pain | Dispute avoidance |
| Apartment societies | Bulk-buy coordination | Network effects | Governance/payment logistics |
| Kirana stores | Repeat household memory / WhatsApp ordering | Offline commerce bridge | Seller workflows |
| Quick-commerce users | Basket comparison and price memory | Existing digital behavior | Source data access |
| Health/wellness users | Label/nutrition caution | High engagement | Claim boundaries |
| Hackathon/builders | Small-model multimodal case study | Demo/credibility | Field notes/benchmarks |
| Data/API product | Aggregated price/alias/source intelligence | Moat if consented | Privacy/legal/commercial demand |

### 18.9 Programmatic Content and Free Tools

| Tool/content | Audience | Data needed | Product tie-in |
|---|---|---|---|
| Unit-price calculator | Budget shoppers | pack sizes, units | Price memory |
| “Is this price high?” checker | Market shoppers | price history/snapshots | Market Lens |
| Receipt-to-spend summary | Families | OCR receipt | Receipt ingestion |
| Expiry label parser demo | Anyone scanning packets | OCR labels | Packet scan |
| Food waste estimator | Home cooks | inventory/use-soon | Waste coach |
| Festival shopping planner | Indian households | festival lists | Event planning |
| Rainy-day grocery checklist | Weather-aware shoppers | weather + staples | Context intelligence |
| Bulk-buy break-even calculator | Apartments/families | unit prices/storage | Community buying |
| Pantry audit checklist | New users | item taxonomy | Onboarding |
| Grocery alias explorer | Builders/judges | alias graph | Dataset story |
| “What can I cook?” mini-tool | Home cooks | inventory + recipes | Meal planning |
| Caregiver essentials checklist | Caregivers | essential categories | Elder mode |

### 18.10 Regulatory, Safety, and Claims Expansion

| Surface | Rule to research/document |
|---|---|
| Nutrition | Provide informational label extraction, not diet/medical advice. |
| Allergens | Warn only from explicit label/user data; never guarantee safety. |
| Medicines/pharmacy | Track stock/expiry only; no dosage/medical guidance. |
| Prices | Always show source and freshness; avoid “best price” guarantee if stale. |
| Community data | Consent, aggregation thresholds, removal rights, anti-abuse. |
| Receipts | Redact phone/address/payment IDs before sharing/export. |
| Household images | Strip EXIF/location; detect faces/private content before public use. |
| Child/elder data | Extra consent and conservative sharing defaults. |
| Auto-actions | No auto-purchase; no destructive delete without confirmation. |
| Public demo | Synthetic/redacted data by default; strict leakage guard. |

### 18.11 Operations and Quality System

| Operational system | Purpose |
|---|---|
| Data catalog | Every JSON/CSV/YAML/source snapshot has owner, schema, freshness, tests. |
| Benchmark registry | Every model/pipeline candidate has fixed eval, score, latency, keep/discard. |
| Source adapter monitor | Detect schema drift, stale snapshots, unit parsing failures. |
| Redaction review | Gate traces/images/receipts before public export. |
| Demo mode | Deterministic seeded data and walkthrough reset. |
| Backup/restore drill | Prove household memory can be exported/restored. |
| Failure fixture promotion | Corrections become regression tests. |
| Claims checklist | Validate UI/docs copy against actual capabilities. |
| Exploration maturity tracker | Each idea has status, evidence path, priority, and last verified date. |
| Dogfood log | Real household/market sessions with outcomes and failures. |

### 18.12 More Open Questions

41. What exact 3-minute demo proves value fastest: receipt, market scan, or use-soon?
42. Which persona should the product optimize for first: primary shopper, helper,
    caregiver, or shared-flat user?
43. What is the minimum review queue that makes imperfect models useful rather than annoying?
44. Which household corrections should alter future model behavior automatically?
45. How much setup can users tolerate before seeing value?
46. What data should never leave the device, even with consent?
47. Can stale price/source uncertainty be explained in one short line?
48. Is the moat item aliases, receipt corpus, price memory, source reliability, or
    correction traces?
49. Should ShopStack sell sync/backup, intelligence, or community buying first?
50. What feature would a household still use after the hackathon novelty fades?

_Last updated: 2026-06-09_