Docs/archive/ShopStack_Exploration_Map.md

ShopStack Exploration Map

Project: ShopStack / GharStock
Purpose: A living exploration map for agents and collaborators.
Rule: Keep adding to this file whenever a model, product angle, dataset, architecture pattern, evaluation method, market idea, or risk deserves future investigation.

---

0. Product Summary

ShopStack is a small-model shopping copilot and household commerce memory layer.

It helps a household:

• know what is already at home,
• build shopping lists,
• scan shelves, markets, receipts, packets, fridges, and pantry spaces,
• ask questions by voice,
• decide buy / skip / compare / replace,
• add purchases into inventory,
• track freshness, expiry, price, quantity, and location,
• remember where items are stored,
• learn which stores are cheaper/better,
• understand travel, weather, timing, and convenience,
• create anonymized traces and field notes for the Build Small Hackathon.

The product is intentionally broad long-term, but every exploration should connect back to household shopping, inventory, market intelligence, or daily-use memory.

---

1. Capability Map

1.1 Language + Reasoning

Current / possible capabilities:

• NER for item, brand, unit, quantity, store, price, location, date.
• Intent classification.
• Command parsing.
• Tool-call planning.
• Natural-language inventory query.
• Shopping decision explanation.
• Multilingual / Hinglish / Indian household phrasing.
• Synonym and alias normalization.
• Structured JSON extraction.
• RAG over household memory.
• Safety disclaimers for nutrition/medicine/price uncertainty.

Exploration questions:

• Which small model is best at Indian household command parsing?
• Can a tiny fine-tuned parser outperform a larger general model on our exact commands?
• How much can rules + schema validation reduce model errors?
• Which model produces the cleanest JSON tool calls locally?
• Should we separate command parser and answer generator?

Candidate model families:

• MiniCPM / OpenBMB
• Qwen
• LFM / LiquidAI
• Gemma
• GPT-OSS
• Granite
• GLM
• Mistral / Voxtral for speech-heavy flows
• Llama / Phi GGUF models for llama.cpp badge

---

1.2 NER / Entity Extraction

Entities to extract:

• item name
• canonical item name
• item category
• brand
• local alias
• quantity
• unit
• price
• normalized unit price
• expiry date
• manufacturing date
• store
• location at home
• storage instruction
• nutrition facts
• household member preference
• purchase timestamp
• travel context
• weather context

Exploration questions:

• Should NER be rule-based, model-based, or hybrid?
• Can a fine-tuned small model map Hinglish utterances to canonical items?
• How do we handle ambiguous items like “Surf,” “Vim,” “Maggi,” “bread,” “pav,” “dahi,” “curd,” “yogurt”?
• Can we build a household-specific lexicon that improves over time?
• Should item names be canonicalized using embeddings?

Dataset ideas:

• Indian household item alias dataset.
• Grocery command dataset.
• Receipt-line-to-canonical-item dataset.
• Hinglish quantity normalization dataset.
• Expiry and storage instruction dataset.

---

1.3 Time Series

Time-series capabilities:

• consumption rate estimation,
• days-until-stockout,
• restock interval prediction,
• price trend by item,
• price trend by store,
• day-of-week price patterns,
• seasonality,
• freshness decay,
• expiry forecasting,
• travel effort over time,
• store quality over time,
• household category spend.

Exploration questions:

• Which items have predictable restock cycles?
• How much history is needed for useful next-buy predictions?
• Can we estimate stockout without exact daily consumption?
• What is the right confidence language: “likely low,” “probably enough,” “uncertain”?
• How do we handle multiple lots of the same item?
• Should predictions be rule-based initially and learned later?

Possible models / tools:

• statsmodels / Prophet-like approaches,
• DuckDB time-series queries,
• simple rolling averages,
• exponential smoothing,
• Bayesian inventory estimates,
• local notebooks / HF Jobs / Modal experiments.

---

1.4 Spatial Intelligence

Spatial layers:

• fridge shelf memory,
• pantry shelf memory,
• household location map,
• item last-seen memory,
• item movement events,
• room/shelf heatmap,
• “find this item” assistant,
• storage recommendation,
• household location graph,
• future AR/SLAM-like map.

Exploration questions:

• Can shelf/fridge photos become stable “location snapshots”?
• How do we identify the same item across days?
• Can we track “milk moved from shopping bag → fridge door”?
• How much user confirmation is needed?
• Should locations be user-defined first: fridge top shelf, pantry shelf 2, bathroom cabinet?
• Is a heatmap useful before AR/SLAM exists?
• Can we use image embeddings for “this looks like the same shelf”?

Possible tools:

• OpenCV,
• object detection,
• image embeddings,
• segmentation,
• H3/geospatial only for external market map,
• graph DB or SQLite graph tables,
• future WebAR / mobile capture.

---

1.5 Voice: STT, TTS, Audio

Voice capabilities:

• voice shopping list creation,
• voice correction,
• voice ask while shopping,
• voice answer in noisy market mode,
• voice-based item movement,
• voice-based price logging,
• Indian-language / Hinglish UX,
• audio confidence and retry UX.

STT candidates to evaluate:

• Qwen3-ASR-1.7B
• NVIDIA Parakeet / Nemotron 0.6B streaming ASR
• Mistral Voxtral Mini Realtime
• SenseVoiceSmall
• VibeVoice ASR
• Cohere Transcribe model
• Granite Speech
• Whisper large-v3-turbo baseline

TTS candidates to evaluate:

• MOSS-TTS-v1.5
• VoxCPM2
• Qwen3-TTS 0.6B / 1.7B
• Higgs Audio v3 TTS
• Kokoro-82M
• CosyVoice
• OmniVoice
• Chatterbox / XTTS-style models if useful

Exploration questions:

• Which model best handles Hinglish grocery commands?
• Can the response voice be short, warm, and market-friendly?
• Should we use browser/device TTS as fallback?
• How do we handle noisy market audio?
• Can audio be processed locally in the Space without cloud APIs?
• How do we benchmark STT/TTS quickly?

Benchmark phrases:

• “Doodh ghar pe hai kya?”
• “Tamatar aadha kilo add karo.”
• “Nahi, yeh pyaaz hai aloo nahi.”
• “Bread expiry kal ka hai, skip karo.”
• “Surf Excel already ghar pe hai kya?”
• “Isko pantry mein move karo.”
• “Kal breakfast ke liye kya hai?”

---

1.6 Vision Understanding

Vision capabilities:

• item detection,
• item classification,
• visual grounding,
• shelf/market scan,
• packet understanding,
• receipt understanding,
• fridge/pantry scan,
• freshness/ripeness hints,
• damaged/spoiled item detection,
• visual confirmation cards,
• annotated photos.

Candidate models / tools:

• Gemma multimodal models
• MiniCPM-V
• LocateAnything
• Qwen VL / image models
• RF-DETR
• YOLO variants
• Marlin-2B for video
• TimeLens / video grounding models
• OpenCV + OCR hybrids
• CLIP/embedding-based matching

Exploration questions:

• Which model works best on Indian market photos?
• Can open-vocabulary grounding identify “dhaniya,” “pav,” “dahi,” “atta”?
• Does a general VLM beat object detection for household goods?
• Can we use image crops + text model instead of one large VLM?
• How do we show uncertainty without frustrating users?

---

1.7 OCR / Extraction

OCR targets:

• receipts,
• packet labels,
• expiry dates,
• MRP,
• quantity,
• brand,
• nutrition facts,
• store name,
• receipt totals,
• bill line items,
• handwritten notes.

Candidate tools:

• PaddleOCR / PaddleOCR-VL
• NuExtract3
• Tesseract fallback
• Donut-like document models
• DocTR / LayoutLM-style pipelines
• OCR + LLM extraction hybrid

Exploration questions:

• Can receipt OCR handle local Indian store bills?
• How do we normalize quantities from OCR?
• Can we reliably detect expiry dates from packets?
• How do we distinguish MRP from sale price?
• Should packet OCR be a separate close-up mode?

---

1.8 Classification

Classification tasks:

• item category,
• storage location,
• food vs household vs medicine vs cleaning,
• perishable vs shelf-stable,
• urgent vs optional,
• buy / skip / compare,
• confidence class,
• freshness class,
• trace safety/redaction class,
• price anomaly class,
• user intent class.

Exploration questions:

• Which classifications can be rules?
• Which need fine-tuning?
• Should we use a lightweight classifier before LLM calls?
• Can the fine-tuned model handle both intent and item category?
• How do we evaluate classification in Field Notes?

---

1.9 Segmentation / Grounding

Segmentation capabilities:

• product crop cards,
• item cutouts,
• shelf zones,
• fridge zones,
• visual confirmation,
• background removal,
• annotated maps.

Candidate tools:

• RMBG
• BiRefNet
• ClipSeg
• SAM variants if feasible
• YOLO segmentation
• RF-DETR segmentation
• LocateAnything grounding
• OpenCV masks

Exploration questions:

• Is segmentation necessary for every flow, or only review cards?
• Which model runs reliably in Spaces?
• Can segmentation improve user trust?
• Can shelf-zone segmentation power household spatial memory?
• How should we handle overlapping groceries?

---

1.10 Image Generation / Editing

Use cases:

• annotated shopping photos,
• item cards,
• shelf maps,
• “use soon” visual cards,
• printable pantry labels,
• shopping summary posters,
• household heatmap illustrations,
• price comparison cards,
• Field Notes visuals.

Candidate models:

• Black Forest Labs FLUX.2-klein-4B
• FLUX.2-klein-9B
• Qwen Image Edit
• ControlLight
• Lightweight PIL/HTML card generation
• SVG-based generated layouts

Exploration questions:

• Is image generation useful enough for the product, or should deterministic visual cards come first?
• Can generated cards improve sharing and polish?
• How do we keep image edits from hallucinating wrong product details?
• Should all factual text be rendered by code, not generated into image pixels?

---

1.11 Embeddings / Retrieval

Embedding uses:

• item alias matching,
• receipt line matching,
• memory search,
• similar purchase recall,
• store note retrieval,
• trace retrieval,
• product substitute matching,
• voice phrase similarity,
• user preference retrieval,
• location snapshot matching.

Candidate embedding models:

• Qwen embeddings
• MiniLM / sentence-transformers
• multilingual E5 family
• BGE multilingual
• Jina embeddings
• small local embedding models
• CLIP / SigLIP for image similarity

Exploration questions:

• Which multilingual embedding model handles Hinglish item aliases best?
• Should text and image embeddings share one store?
• Is SQLite vector extension enough?
• Should we use FAISS, LanceDB, Chroma, or DuckDB extensions?
• Can embeddings help trace retrieval for Sharing is Caring?

---

1.12 Graph / Linkage Memory

Nodes:

• Item
• ItemLot
• Store
• MarketArea
• HouseholdLocation
• Shelf
• FridgeZone
• PurchaseEvent
• PriceObservation
• ShoppingTrip
• WeatherContext
• UserPreference
• Recipe
• Trace
• ModelRun

Edges:

• bought_at
• stored_in
• moved_to
• consumed_by
• expires_before
• substitutes
• usually_bought_with
• preferred_by
• cheaper_at
• best_quality_at
• seen_in_photo
• mentioned_in_voice
• derived_from_receipt
• weather_affected
• route_to
• worth_travelling_for

Exploration questions:

• Is a graph DB needed, or can SQLite edge tables do enough?
• Which queries need graph traversal?
• Can graph memory make the product feel smarter quickly?
• How do we visualize this without making it technical?

---

2. User Use Case Map

2.1 Before Shopping

Use cases:

• create list from voice,
• check what is already at home,
• predict next buys,
• suggest what to buy for meals,
• avoid buying duplicates,
• choose store based on list,
• compare travel effort,
• account for weather,
• generate shopping route,
• plan by budget.

Questions:

• “What should I buy today?”
• “What is low at home?”
• “Can we make dinner without shopping?”
• “Which store should I go to?”
• “Is the Sunday market worth it today?”

---

2.2 During Shopping

Use cases:

• scan shelf,
• ask if item is needed,
• compare with home inventory,
• compare price with memory,
• check expiry,
• identify unknown item,
• quantity advice,
• substitution advice,
• allergy/preference warning,
• budget warning.

Questions:

• “Do I need this?”
• “Is this price okay?”
• “Which one should I pick?”
• “Is this enough?”
• “Do we already have this?”
• “Is this near expiry?”
• “Can I skip this?”

---

2.3 After Shopping

Use cases:

• purchase photo ingestion,
• receipt ingestion,
• inventory update,
• expiry tracking,
• price observation logging,
• store rating,
• trip context logging,
• trace export,
• family summary.

Questions:

• “Add all this.”
• “What expires first?”
• “What did we spend?”
• “Where should this go?”
• “Did we overbuy anything?”

---

2.4 At Home

Use cases:

• find items,
• check stock,
• plan meals,
• use-soon reminders,
• move item location,
• consume item,
• estimate remaining quantity,
• household member asks questions,
• shelf/fridge scan,
• spatial memory.

Questions:

• “Where is the dahi?”
• “Do we have detergent?”
• “What should we use today?”
• “What is in the fridge?”
• “Move toothpaste to bathroom cabinet.”
• “How much rice is left?”

---

2.5 Market Intelligence

Use cases:

• price trends,
• store ranking,
• cheapest location,
• freshness/quality memory,
• travel-time decision,
• weather-aware recommendation,
• route-aware shopping,
• neighborhood price map,
• Market Intelligence Graph / Market Map: home inventory + live market cards + price memory + substitutions + freshness risk,
• snapshot-based benchmarking from real market dataset imports.

Questions:

• “Where was tomato cheapest?”
• “Is this price high?”
• “Which store is better for fruits?”
• “Should I travel to the market today?”
• “What did we learn from last month’s shopping?”
• “How much can a Swiggy price snapshot improve price memory and trip decisions?”

---

3. Build Small Hackathon Constraint Map

Non-negotiables:

• total loaded model parameters must be <= 32B,
• app must be built on Gradio,
• app must be hosted as a Hugging Face Space,
• short walkthrough video and social post required,
• main track should show a real person / real problem,
• Codex is a parallel track, not the product.

Bonus quests:

• Off the Grid: no cloud APIs in runtime path.
• Well-Tuned: use a fine-tuned model published on HF.
• Off-Brand: custom frontend beyond default Gradio.
• Llama Champion: run a model through llama.cpp.
• Sharing is Caring: publish anonymized agent traces.
• Field Notes: write report/blog about what was built and learned.

Exploration questions:

• Which bonus quests are product-aligned?
• How do we claim Off the Grid while using credits for build-time jobs?
• How do we make llama.cpp visible in the app/report?
• What is the smallest useful fine-tune?
• How should traces be anonymized?

---

4. Sponsor Alignment Map

Hugging Face + Gradio

Explore:

• Spaces deployment,
• Space README metadata,
• dataset/model linking,
• model publishing,
• dataset publishing,
• Jobs,
• GPU Spaces,
• custom Gradio Blocks UI,
• Gradio API endpoints,
• traces as datasets.

OpenBMB

Explore:

• MiniCPM5-1B as parser/planner,
• MiniCPM5-1B-GGUF for llama.cpp badge,
• MiniCPM-V for vision,
• VoxCPM2 for TTS,
• OpenBMB special category angle.

OpenAI / Codex

Explore:

• Codex-attributed commits,
• AGENTS.md,
• codex build log,
• tests and docs generated/reviewed by Codex,
• public GitHub repo,
• “Built with Codex” README section,
• Codex as engineering lane only.

NVIDIA

Explore:

• LocateAnything for grounding,
• Parakeet/Nemotron ASR,
• GPU experiments,
• accelerated vision/audio workflows,
• possible RTX 5080 relevance in final story.

Modal

Explore:

• fine-tuning jobs,
• benchmark jobs,
• model comparison runs,
• trace generation,
• quantization,
• batch inference,
• dataset generation.

Black Forest Labs

Explore:

• FLUX image edit / visual cards,
• use-soon cards,
• shelf maps,
• shopping summaries,
• annotated item visuals.

Cohere

Explore:

• ASR/transcription model comparison,
• embeddings/reranking comparisons if available,
• not required for local-first runtime.

---

5. Dataset Exploration Map

Potential datasets to create/publish:

1. Indian household shopping utterances.
2. Hinglish grocery command parser dataset.
3. Item alias/canonicalization dataset.
4. Purchase photo annotation dataset.
5. Receipt OCR/extraction dataset.
6. Packet label/expiry extraction dataset.
7. Inventory tool-call traces.
8. Market decision traces.
9. Price observation synthetic dataset.
10. Shelf/fridge location memory dataset.
11. Voice benchmark dataset.
12. TTS pronunciation phrase set.
13. Store memory schema examples.
14. Redacted agent trace dataset for Sharing is Caring.
15. Field Notes dataset with examples and failure modes.
16. Swiggy Instamart fresh vegetables snapshot dataset for price/unit-price benchmarking and market intelligence experiments.

Dataset quality questions:

• What can be public?
• What must be synthetic?
• What must be anonymized?
• What needs user consent?
• Which datasets help Well-Tuned most?
• Which datasets help judges understand the product?

5.1 Decision‑First Today / ShopStack Topic Map

A decision-first Today experience should be built around the questions users actually ask in the moment, not around tabs.

• What I have: household stock and location memory for pantry, fridge, and kitchen items.
• What I should buy: buy recommendations shaped by stock, waste risk, price snapshots, and market intelligence.
• What I should skip: overbuy warnings, duplicate items, and suggestions to avoid waste.
• What I should use soon: use-soon actions, meal prompts, and practical expiry-aware reminders.
• My own list: personal shopping list, voice list edits, and familiar/repeat item memory.
• Compare prices / offers: price/unit-price benchmarking, Swiggy Instamart fresh vegetables snapshot comparisons, and store choice tradeoffs.
• Market basket: basket-level intelligence, category budget signals, and market snapshot analytics.
• What needs confirmation: uncertain OCR/image matches, missing quantities, and low-confidence items.
• What I usually buy: recurring purchase patterns, cadence, and habitual home shopping behavior.
• What I’m overbuying / wasting: waste risk, stale stock alerts, and spend leakage from items that are not being used.

This should explicitly tie the Today dashboard to exploration-level research, including the Swiggy Instamart fresh vegetables snapshot dataset and market intelligence experiments.

2026-06-08 Update: The decision-first navigation has been implemented. The 15 flat tabs were restructured into 5 primary tabs (Today → Basket → ShopLens → Reconcile → Memory) with sub-tabs preserving all functionality. See DR-005 in DECISION_RECORDS.md for the full rationale and migration record. The product now opens to Today (decision-first dashboard with integrated Ask) rather than a flat feature catalog. Next exploration targets:

• Wire purchase cadence and waste prevention widgets into the Today dashboard so it answers "what should I do today?" without switching tabs.
• Surface Basket-level decision classification (buy/skip/compare/use-soon) more prominently in the Today view.
• Track which sub-tabs get the most real-world usage to validate or adjust the navigation hierarchy.

---

6. Evaluation Map

Product Evaluations

• Can the user create a list by voice?
• Can the app detect relevant visible items?
• Can the app correctly say buy/skip?
• Can it add confirmed purchases?
• Can it answer inventory questions?
• Can it find item locations?
• Can it explain uncertainty?

Model Evaluations

• STT exactness and intent retention.
• TTS clarity and warmth.
• OCR field extraction accuracy.
• object detection recall.
• segmentation usability.
• tool-call JSON validity.
• parser intent accuracy.
• latency.
• memory use.
• parameter count.
• install/deploy pain.
• local-first compatibility.

Trace Evaluations

• trace completeness,
• privacy redaction,
• reproducibility,
• educational value,
• tool-call correctness.

Field Notes Evaluations

• real user used it,
• what worked,
• what failed,
• what changed after feedback,
• small-model fit,
• honest limitations.

---

7. Marketing / Positioning Exploration

Possible positioning:

• “Remember what’s at home while you shop.”
• “A shopping copilot for Indian homes.”
• “Photo + voice inventory for everyday shopping.”
• “Your fridge, pantry, market, and shopping list in one memory.”
• “Small models for small household decisions.”
• “The home commerce memory layer.”
• “Not another grocery app — a memory for what you buy, where it goes, and when to buy again.”

Potential audiences:

• Indian families,
• parents,
• students/hostels,
• shared flats,
• home cooks,
• small kirana shoppers,
• apartment households,
• elderly users,
• caregivers,
• domestic helpers managing stock,
• local sellers later as adjacent market.

Potential channels:

• hackathon demo,
• YouTube build stream,
• Twitter/X build thread,
• LinkedIn product post,
• HF Space leaderboard,
• Gradio Discord,
• Indian tech/ProductHunt style launch,
• Reddit India/frugal/mealprep communities,
• WhatsApp family-group proof-of-use story.

Exploration questions:

• Is “ShopStack” too general or perfect for scaling?
• Should the India-local layer be in the tagline, not name?
• Can demo be recorded with real household shopping?
• What short video moment makes people immediately understand?
• Which poster/screenshot is most shareable?

---

8. Risk / Constraint Exploration

Risks:

• too broad,
• over-reliance on imperfect vision,
• noisy market audio,
• privacy concerns,
• live price unreliability,
• too many models in one Space,
• 32B parameter accounting,
• non-commercial model licenses,
• Gradio UI becoming complex,
• agent anchoring to a reduced scope,
• overclaiming accuracy,
• app feeling like a demo instead of product direction.

Mitigations:

• confirmation-first UX,
• model registry,
• local-first mode,
• connected modes clearly separated,
• visible uncertainty,
• trace redaction,
• field notes,
• provider interfaces,
• benchmark scripts,
• privacy-first README,
• no auto-purchase,
• no medical/diet/legal claims,
• no private data in repo.

---

9. Open Questions

Agents should keep adding questions here.

1. Which model stack gives the best local-first performance under 32B?
2. Which model is easiest to run through llama.cpp for the parser?
3. What exact fine-tuned model should be published for Well-Tuned?
4. How much real household data can be safely used?
5. What is the first public trace dataset schema?
6. How should we count parameters when multiple models are optional but not loaded together?
7. Can Gradio handle the desired custom UI without too much friction?
8. Which STT model handles Hinglish best?
9. Which TTS model sounds warm enough for household use?
10. Should we use OCR or VLM-first for receipts?
11. How do we estimate quantity from photos without overclaiming?
12. Should price intelligence be mostly manual-memory-first?
13. How much map/heatmap functionality belongs in the app surface?
14. What is the simplest useful household map?
15. How do we benchmark live market/shelf scans?
16. What should be in the walkthrough video?
17. What should the social post emphasize?
18. What should the Field Notes title be?
19. How can Codex involvement be made obvious and authentic?
20. What is the strongest sponsor-alignment story?

---

10. Parking Lot

Use this for anything that may be interesting later.

• AR mode for finding items at home.
• Barcode scanning.
• Local store loyalty memory.
• Recipe planning from inventory.
• Waste tracking.
• Family member preferences.
• Domestic helper voice workflow.
• Shared household mode.
• WhatsApp integration.
• Calendar/reminder integration.
• Price community map.
• Privacy-preserving neighborhood price sharing.
• Offline mobile app.
• Browser extension for online grocery carts.
• Email receipt ingestion.
• Smart label printing.
• Shelf-life prediction from images.
• Freshness/ripeness model.
• Food waste report.
• Shopping carbon/effort score.
• Festival shopping planning.
• Monthly household budget intelligence.
• Elder-friendly voice-only mode.
• Accessibility mode for low-vision shoppers.
• Agentic shopping comparison, with user confirmation only.
• Local-language onboarding.
• Synthetic data generation pipeline.
• Human review UI for fine-tune data.
• Public leaderboard for household command parsing models.
• Sponsor-specific benchmark tables.
• Model replacement changelog.

---

11. Agent Contribution Protocol

When an agent adds an exploration item:

1. Add it under the relevant section.
2. Include why it matters for ShopStack.
3. Add model/tool links if known.
4. Mark whether it affects:
   • product,
   • model stack,
   • dataset,
   • evaluation,
   • UI,
   • privacy,
   • sponsor alignment,
   • bonus quest,
   • marketing.
5. Do not delete old ideas unless they are unsafe or clearly obsolete.
6. Move rejected ideas to a “rejected/paused” note with reason.
7. Keep this map broad; implementation tasks belong in task docs, not here.
8. Avoid anchoring language that frames the product as temporary or small.
9. Keep hackathon constraints visible.
10. Prefer swappable interfaces over hardcoded models.

Template:

### Idea / Model / Tool / Angle

**Category:** model / product / dataset / marketing / eval / privacy / sponsor / UI  
**Why it matters:**  
**How to test:**  
**Risks:**  
**Links:**  
**Status:** explore / test / adopt / pause / reject  

---

12. Current Strongest Directions

1. Voice-first shopping list and correction.
2. Vision-based market/shelf scan.
3. Purchase photo ingestion.
4. Inventory and freshness memory.
5. Household spatial memory.
6. Price and store memory.
7. Fine-tuned Indian household command parser.
8. llama.cpp parser/planner.
9. Anonymized trace dataset.
10. Field Notes with real household use.
11. Off-brand custom Gradio UI.
12. Model benchmarking and replacement policy.

---

Last updated: 2026-06-05

---

13. Explorer Addendum — Product and Architecture Discovery Backlog (2026-06-06)

Source mix: current docs, current package shape, test inventory, product hardening notes, and first-principles household-commerce analysis.
Status: exploration backlog; not implementation commitment.
Update policy: append-only unless an item is later superseded with a dated note.

13.1 Decision Service Extraction: make Gradio an adapter, not the brain

Category: architecture / product / evaluation / UI
Why it matters: Current docs already identify shopping service extraction as the canonical direction. The same boundary should cover Market Lens, Today dashboard, inventory recommendations, trace composition, and price intelligence so decisions are testable without UI wiring. This turns ShopStack from a demo UI into a durable household decision engine.
How to test: Create service-level tests that accept typed inputs and return typed decision objects; UI tests should only verify rendering/wiring.
Risks: Over-abstracting before the domain stabilizes. Keep services narrow and behavior-first.
Status: adopt incrementally.

13.2 Confidence-calibrated household recommendations

Category: product / model stack / evaluation / UI
Why it matters: “Buy,” “skip,” and “use soon” decisions need calibrated confidence language because household data is incomplete: unknown exact quantities, stale market prices, missing expiry dates, and approximate consumption rates. A clear uncertainty model increases trust and avoids fake precision.
How to test: Build fixtures for complete, partial, stale, and contradictory household states; assert both recommendation and confidence wording.
Risks: Too many caveats can make the product feel timid. Use concise labels: confident / likely / uncertain / needs confirmation.
Status: explore → adopt.

13.3 Household memory quality score

Category: product / data / UI / evaluation
Why it matters: The app should tell users whether its memory is good enough to trust. A “memory freshness / completeness” score can surface stale inventory, unpriced items, missing locations, uncertain quantities, and old market snapshots.
How to test: Score deterministic seeded households with known gaps; verify the score changes after purchases, consumption, imports, and stale data.
Risks: A score without actionable fixes becomes vanity UI. Pair every low score with the next best cleanup action.
Status: explore.

13.4 Receipt and invoice ingestion pipeline

Category: product / vision / OCR / dataset / privacy
Why it matters: Manual purchase entry is high friction. Receipts, app invoices, WhatsApp order summaries, and email receipts can populate inventory, price memory, store memory, and purchase cadence in one action.
How to test: Start with local sample receipt text/images; evaluate extraction into PurchaseEvent + PriceObservation + InventoryLot candidates with user confirmation.
Risks: PII leakage, messy receipt formats, and false inventory additions. Keep local-first, redacted, and confirmation-backed.
Status: explore.

13.5 Post-shopping reconciliation flow

Category: product / UI / data integrity
Why it matters: A shopping list is not complete until the household confirms what was actually bought, skipped, substituted, or already found at home. This flow would close the loop between plan → purchase → inventory → price memory.
How to test: Simulate a list with bought/partial/substituted/skipped items and assert inventory lots, price observations, shopping list state, and traces update together.
Risks: Too much bookkeeping. Use batch confirmation with sensible defaults.
Status: adopt candidate.

13.6 Substitution intelligence

Category: product / model stack / dataset / evaluation
Why it matters: Real shopping involves substitutions: tomato types, milk sizes, dal brands, detergent pack sizes, produce quality, and budget alternatives. ShopStack should recommend acceptable swaps based on household preference, price, unit price, shelf life, and recipe intent.
How to test: Build a substitution fixture set with item families, unacceptable swaps, dietary constraints, and unit-price comparisons.
Risks: Bad substitutions can break recipes or preferences. Require confirmation and learn from rejections.
Status: explore.

13.7 Waste-aware meal/use suggestions

Category: product / model stack / UI / marketing
Why it matters: “Use soon” becomes more valuable when it suggests practical actions: “use spinach tonight,” “make raita with curd,” “freeze bread,” or “move onions out of fridge.” This ties inventory memory to waste reduction.
How to test: Fixture expiring items and assert suggestions respect available pantry items, shelf life, household preferences, and safety disclaimers.
Risks: Recipe hallucination and unsafe food advice. Keep suggestions simple, local, and confidence-tagged.
Status: explore.

13.8 Store reliability and quality memory

Category: product / data / evaluation / UI
Why it matters: Cheapest is not always best. Produce quality, delivery reliability, substitutions, stale items, missing items, and refund friction should influence future recommendations.
How to test: Add store observations and verify recommendation ranking balances price, quality, freshness, travel/time, and confidence.
Risks: Sparse data can bias rankings unfairly. Show “based on N observations.”
Status: explore.

13.9 Freshness-first market data layer

Category: data / architecture / UI / privacy
Why it matters: Swiggy snapshots are useful but point-in-time. Every price/availability answer should carry source, captured-at timestamp, stale-data warning, and fallback behavior when freshness is insufficient.
How to test: Snapshot fixtures at fresh/stale/expired ages; assert UI and services never present old prices as live.
Risks: Over-warning can reduce usefulness. Use compact timestamp badges.
Status: adopt.

13.10 Local model routing policy by task risk

Category: model stack / architecture / evaluation / privacy
Why it matters: Not every task needs the same model. Deterministic parsers can handle quantity/unit extraction, embeddings can handle alias matching, and larger local planners can handle explanation. Routing reduces latency and improves reliability under the 32B cap.
How to test: Benchmark task families separately: command parse, receipt parse, recommendation explanation, alias match, and trace summarization.
Risks: Too many moving parts. Keep routing explicit and inspectable in Model Stack.
Status: explore.

13.11 Household-specific alias and preference learning

Category: product / dataset / model stack / privacy
Why it matters: Every household has names like “red dal,” “Amul wala dahi,” “kids bread,” “big Surf,” or “Friday sabzi.” A local adaptive lexicon can make voice and search feel personal without cloud training.
How to test: Record accepted corrections and rejected matches; assert future parsing improves while preserving auditability.
Risks: Wrong learned aliases can poison memory. Require reversible alias review.
Status: explore.

13.12 Inventory state reconciliation and contradiction detection

Category: data integrity / architecture / UI / evaluation
Why it matters: Household state will drift: items consumed without logging, duplicates, stale lots, impossible negative quantities, expired-but-used items, and contradictory locations. The app needs a cleanup assistant rather than silently accumulating bad memory.
How to test: Seed contradictory databases and assert the assistant proposes safe, confirmable fixes without destructive writes.
Risks: Cleanup flows can feel accusatory or tedious. Make them lightweight: “Is this still true?”
Status: explore.

13.13 Trace replay and evaluation harness

Category: evaluation / architecture / model stack / privacy
Why it matters: Existing anonymized traces can become a regression harness: replay planner inputs, compare tool calls, verify redaction, and benchmark model/provider changes before adoption.
How to test: Store golden trace fixtures and assert parser/tool-call outputs remain valid across model/config changes.
Risks: Golden traces can freeze bad behavior. Version them and review failures for intended improvements.
Status: adopt candidate.

13.14 Offline-first backup, restore, and household portability

Category: product / privacy / architecture / operations
Why it matters: Local-first users need safe backup/restore, device transfer, and human-readable exports. Portability already exists; next exploration is reliable restore semantics, conflict handling, and encrypted backups.
How to test: Round-trip realistic households through export/import, including duplicate items, traces, locations, prices, and market snapshots.
Risks: Bad imports can corrupt the source of truth. Use dry-run diff and confirmation before writes.
Status: explore.

13.15 Mobile capture path without losing local-first guarantees

Category: product / UI / architecture / privacy
Why it matters: Fridge scans, market scans, receipt photos, barcode scans, and voice commands are naturally mobile. The Gradio UI is useful for prototype velocity, but the product direction likely needs a mobile capture layer that syncs to local household memory.
How to test: Prototype mobile-friendly capture screens first; later evaluate a native/offline shell or local network companion.
Risks: Mobile sync can break the local-first trust model. Keep explicit sync boundaries and exportable data.
Status: explore.

13.16 Barcode and package knowledge base

Category: product / dataset / vision / UI
Why it matters: Barcodes can reduce ambiguity for packaged goods and enrich inventory with brand, size, category, nutrition, and repeat-purchase memory. A local cache avoids repeated lookups.
How to test: Build local barcode fixtures for common household goods and assert scan → candidate item → confirmation → inventory path.
Risks: Public barcode databases may be incomplete or noisy. Treat lookup as candidate data, not truth.
Status: explore.

13.17 Pantry economics and household budget intelligence

Category: product / commercial / evaluation / UI
Why it matters: ShopStack can become a household operating system for grocery spend: price inflation by item, category budgets, bulk-buy payoff, waste cost, and store choice tradeoffs. This is a strong retention surface beyond reminders.
How to test: Generate monthly fixture data and assert summaries match known spend, savings, waste, and stockout events.
Risks: Budget advice can feel judgmental. Use neutral, practical language.
Status: explore.

13.18 Safety boundaries for food, medicine, and sensitive household data

Category: safety / privacy / product / evaluation
Why it matters: Household inventory can include medicines, baby food, allergens, alcohol, and sensitive routines. The product needs explicit boundaries for advice, logging, redaction, and exports.
How to test: Add fixtures for allergens, expired medicine, baby formula, and personal data; assert conservative warnings and redaction.
Risks: Over-blocking benign grocery flows. Scope warnings to genuinely sensitive categories.
Status: explore → policy.

13.19 “Why this recommendation?” explanation cards

Category: product / UI / evaluation / trust
Why it matters: Users should see the concrete evidence behind decisions: at-home stock, last purchase, expected use rate, price snapshot age, store comparison, expiry/waste signal, and confidence. This improves trust and makes model behavior auditable.
How to test: Snapshot render decision cards for representative buy/skip/use-soon/substitute cases.
Risks: Cards can become verbose. Use progressive disclosure.
Status: adopt candidate.

13.20 Market Intelligence Graph / Market Map

Category: product / data / evaluation / UI
Why it matters: The next unique ShopStack wedge is not a plain price comparator. It is a living household graph that connects what is at home, what the market shows, what is stale, what is overpriced, what is already covered by inventory, what should be substituted, and what is likely to be wasted. This turns ShopCompare + ShopMemory + ShopStock into one decision surface.
How to test: Build fixtures with stale snapshots, sponsored cards, sold-out items, combo packs, partial inventory overlap, and substitution candidates; assert the graph produces stable buy / skip / use-soon / compare / wait / substitute lanes and truth-score labels.
Risks: If the graph reuses retailer language too directly, it can drift back into a shopping ad surface. Keep the default ranking household-first and explicitly label stale or sponsored signals.
Status: adopt candidate.

13.20 Launch story: from inventory app to household commerce memory

Category: marketing / product / positioning
Why it matters: “Inventory app” sounds like chores. “Household commerce memory” is more distinctive: it remembers what you have, what you paid, where it is, what spoils, and what to buy next. This framing should guide demos, docs, and feature prioritization.
How to test: Compare landing/demo narratives: inventory tracker vs shopping copilot vs household commerce memory; evaluate which makes the product feel inevitable and differentiated.
Risks: Too abstract for first-time users. Lead with concrete daily workflows, then name the deeper system.
Status: explore.

13.21 Data source expansion map

Category: data / product / privacy / evaluation
Why it matters: Swiggy fresh vegetables is a strong start, but durable market intelligence needs multiple source types: user-entered prices, receipts, quick-commerce snapshots, supermarket invoices, local kirana observations, barcode/package metadata, and seasonal produce calendars.
How to test: Define a normalized source contract and fixture each source type into PriceObservation/MarketSnapshot without UI assumptions.
Risks: Scraping or terms-of-service risk for live platforms. Prefer user-provided exports/snapshots and explicit provenance.
Status: explore.

13.22 Operator/developer diagnostics tab

Category: operations / UI / architecture / evaluation
Why it matters: Local-first apps need inspectability: DB path, provider mode, model availability, snapshot freshness, trace counts, redaction status, and recent errors. A diagnostics view makes failures understandable without cloud logs.
How to test: Mock provider/model/data states and assert the diagnostics view surfaces actionable status.
Risks: Too much internal detail in user UI. Gate as “Diagnostics” or “Local system status.”
Status: explore.

13.23 Evaluation leaderboard for small local household models

Category: model stack / evaluation / sponsor / marketing
Why it matters: The project can produce a reusable benchmark for local household agents: Hinglish command parsing, tool-call JSON validity, receipt extraction, alias matching, and recommendation explanation. This supports model choice and gives the project external credibility.
How to test: Build a small, versioned eval dataset from synthetic + anonymized traces; run candidate local models with latency, validity, and correction metrics.
Risks: Public benchmark data must not leak household details. Keep anonymization and synthetic generation strict.
Status: explore.

13.24 Human correction as the primary learning loop

Category: product / dataset / model stack / UI
Why it matters: The fastest path to a trustworthy household agent is not autonomous action; it is fast correction capture. Every “no, this is onions,” “skip this brand,” or “we call this atta” should improve future behavior locally.
How to test: Build correction fixtures and assert corrections update aliases/preferences/traces without mutating unrelated history.
Risks: Correction UI can interrupt flow. Offer one-tap corrections at decision points.
Status: explore.

13.25 Household graph model

Category: architecture / data / product
Why it matters: Items, lots, locations, stores, brands, prices, household members, preferences, recipes, and traces form a graph. SQLite can still store this, but a graph-shaped mental model helps avoid one-off tables and clarifies future recommendations.
How to test: Draw core relationships and map them to current tables; identify missing edges such as item↔alias, item↔substitute, store↔quality, member↔preference.
Risks: Premature graph infrastructure. Keep SQLite; use the graph as domain design, not necessarily a graph database.
Status: explore.

13.26 Market graph-to-basket compare actions

Category: product / UI / architecture / evaluation
Why it matters: The Market Intelligence Graph should not stay isolated. Compare lanes need direct “compare with home” and “compare with memory” actions so the graph can feed the shopping list and basket workflow, not just display intelligence.
How to test: Build compare-lane fixtures with partial inventory overlap and assert the compare view can expose the graph explanation, truth score, and next action.
Risks: Too much duplication between compare panels and the graph view. Keep the compare panel as a compact pointer into the graph rather than a second full graph UI.
Status: explore.

13.27 Market sponsored/stale de-emphasis policy

Category: product / safety / UI / evaluation
Why it matters: Sponsored and stale cards should be visually quieter and rank lower, but still remain inspectable. This is a trust and anti-manipulation policy, not a hard block.
How to test: Render fixtures with stale, sponsored, and upgrade-tagged items and verify they are visibly muted while their data remains accessible.
Risks: If the de-emphasis is too aggressive, legitimate deals get buried. Tune the signal carefully.
Status: explore.

13.28 Signal drill-down cards

Category: product / UI / trust / evaluation
Why it matters: Each market decision should explain itself with a compact breakdown of freshness, availability, sizing confidence, price confidence, memory confidence, and penalties. This makes the Market Intelligence Graph feel inspectable rather than magical.
How to test: Render representative buy/compare/substitute cards and assert the signal breakdown is present behind a progressive-disclosure control.
Risks: Too much detail can overwhelm the user. Keep the breakdown collapsed by default.
Status: explore.

13.29 Compare-to-basket bridge

Category: product / architecture / UI / evaluation
Why it matters: Compare-lane items should be able to flow into the shopping list, basket summary, or substitute confirmation with one clear step. This turns market intelligence into action rather than a dead-end display.
How to test: Build compare fixtures with home overlap and verify the next action can point to basket, pantry, or substitute confirmation without duplicating list state.
Risks: Action shortcuts can over-automate. Keep the bridge explicit and confirmable.
Status: explore.

13.30 Graph substrate projections

Category: architecture / product / evaluation
Why it matters: The Market Intelligence Graph should be the canonical substrate, with Today, Unified Shopping, Market Lens, and Ask ShopStack acting as projections over the same decision state. That keeps recommendations consistent across surfaces and makes graph reasoning reusable.
How to test: Build one graph fixture and project it into Today, unified shopping, market lens, and ask-context views; assert the same canonical item lands in the same lane and next action family.
Risks: Projection helpers can become another layer of indirection if they duplicate logic. Keep them thin wrappers over the canonical graph.
Status: explore.

13.31 Structured evidence and reason atoms

Category: data / trust / evaluation / product
Why it matters: Natural-language reasons are useful, but the graph needs structured reason atoms and evidence claims so traces, UI drill-downs, and corrections can all point at the same underlying facts.
How to test: Render a fixture graph and assert each decision cluster carries reason atoms, claim types, and a breakdown that can be serialized losslessly.
Risks: Over-structuring can produce ceremony without value. Keep the atom vocabulary small and readable.
Status: explore.

13.32 Combo solver and substitution ranker

Category: product / architecture / evaluation
Why it matters: Combos are where household intelligence beats a store page. A combo should be compared against home overlap and split purchase candidates, while substitutions should rank by cooking role, availability, freshness, and preference.
How to test: Use fixtures like onion/potato/tomato, herb bundles, and sold-out premium variants; assert the solver chooses compare/skip/buy-separate and the substitute ranking prefers the closest usable alternative.
Risks: Over-fancy ranking can confuse users if it is not explainable. Always show the reason for the ranking.
Status: explore.

13.33 Decision run memory loop

Category: product / data / evaluation
Why it matters: The graph becomes much more valuable once the app remembers what the household did after the recommendation: bought, skipped, substituted, or corrected. This turns one-off advice into learning.
How to test: Feed reconciliation events back into the graph and verify the next run uses the updated memory and preference signals.
Risks: If the feedback loop is too aggressive, noisy corrections can distort the model. Keep manual correction explicit.
Status: explore.

13.34 Frontier HF sweep and benchmark journal

Category: model stack / evaluation / documentation
Why it matters: The repo now has a fresh live Hugging Face frontier sweep in addition to the older local benchmarks. We should keep the frontier shortlist current without confusing it with the stable local runtime truth.
How to test: Run the reusable frontier harness in tools/model_lab_frontier.py, compare the saved JSONL/markdown artifacts, and keep the sweep notes synchronized with shopstack/model_registry.py and Docs/MODEL_CATALOG.md.
Recent findings:

• Qwen/Qwen3.5-9B scored 40.0% on the production-style planner prompt suite.
• Qwen/Qwen3.6-35B-A3B scored 20.0% but remained fast on HF inference.
• Live HF checks also surfaced Qwen/Qwen3-ASR-0.6B, Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice, Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign, openbmb/MiniCPM-V-4.6, deepseek-ai/DeepSeek-OCR-2, PaddlePaddle/PaddleOCR-VL-1.6, and briaai/RMBG-2.0 as current frontier candidates.
  Artifacts: benchmarks/model_lab/results/frontier_planner_20260613_113615.jsonl, benchmarks/model_lab/results/frontier_planner_20260613_113615.md, Docs/audits/MODEL_FRONTIER_SWEEP_2026-06-13.md.
  Status: adopt candidate.

13.35 Modal benchmark governance and GPU budgeting

Category: evaluation / infra / documentation
Why it matters: The Modal sweeps are intentionally parallel per candidate, but that only stays useful if the suite launches are budgeted together. The current plan-limit warning is a sign to sequence the sweeps, not to stop benchmarking.
How to test: Track the GPU count per suite, confirm the harness uses isolated per-candidate workers, and rerun one suite at a time until the artifact is written successfully. For STT, OCR, planner, vision, and TTS, record the result file path and the worker count in the run note.
Risks: Running planner + OCR + vision + STT sweeps concurrently can saturate the Modal workspace limit and produce stalled or wasted GPU time.
Status: adopt candidate.

---

14. Explorer Priority Cut (2026-06-06)

If only five exploration threads get near-term attention, choose these:

1. Decision service extraction — unlocks testability and keeps UI thin.
2. Freshness-first market data layer — prevents stale prices from becoming fake truth.
3. Post-shopping reconciliation — closes the core product loop.
4. Trace replay/eval harness — makes local model/provider changes safe.
5. Human correction learning loop — turns daily use into compounding household intelligence.

Last updated: 2026-06-06

---

15. Priority Addendum — Architecture Foundation + Visible User Loop (2026-06-06)

Context: Follow-up prioritization pass after reviewing the explorer backlog. This refines the earlier priority cut by separating architecture foundation from the visible user loop. The key adjustment is pulling confidence-calibrated recommendation cards earlier because they make the product’s trust model visible to users.

Revised priority order

1. Service / decision extraction

   • Why: This remains the foundation. ShopStack’s brain should live in typed services, not scattered Gradio callbacks.
   • Scope: inventory state, market candidates, recommendation decisions, trace building, correction handling, reconciliation.
   • Reason to do first: if freshness, reconciliation, cards, and learning are built inside UI handlers, they will need to be rewritten for mobile/API/agent surfaces.

2. Freshness + provenance market data model

   • Why: Swiggy and future market snapshots are useful but time-sensitive. Recommendations must never imply stale snapshot data is live inventory.
   • Target market datum shape:

     source = swiggy_instamart
     captured_at = 2026-06-06
     location/context = snapshot/browser/page context if known
     freshness_state = fresh / aging / stale / expired
     availability_state = available / sold_out / unknown
     confidence = high / medium / low
     

   • Reason to do second: bad market truth destroys user trust faster than missing features.

3. Confidence-calibrated recommendation cards

   • Why: This is the first visible “this is smarter than a grocery list” surface.
   • Desired output style:

     Buy Onion — ₹31/kg from Swiggy snapshot, available at capture time,
     common staple, likely useful if home stock is low. Medium confidence
     because current availability may have changed.
     

   • Reason to do third: it turns backend correctness into visible product trust.

4. Post-shopping reconciliation

   • Why: This closes the real household loop: planned → bought → skipped → substituted → already had → price changed.
   • Examples:

     Bought: onion, tomato, potato
     Skipped: cauliflower
     Substituted: hybrid tomato instead of Indian tomato
     Already had: carrot
     Price changed: onion was ₹34 not ₹31
     

   • Reason to do fourth: without reconciliation, inventory decays and ShopStack becomes a one-time planner instead of household memory.

5. Human correction learning loop

   • Why: Corrections become local durable preferences and aliases.
   • Examples:

     “atta” = wheat flour
     Prefer Indian tomato over hybrid tomato
     Avoid expensive premium/chemical-free variants unless requested
     Don’t recommend sold-out items
     Usually keep onion/potato/tomato stocked
     

   • Reason to do fifth: this makes ShopStack personal without cloud training.

6. Trace replay / evaluation harness

   • Why: Critical for model/provider/parser experimentation, but most meaningful after the first complete decision loop exists.
   • Replay contract:

     inventory state + market snapshot + user intent
       → planner decision
       → recommendation cards
       → expected trace
       → expected user-facing output
     

   • Reason to do sixth: evaluation should protect real product behavior, not incomplete scaffolding.

7. Receipt / invoice / order ingestion

   • Why: High-leverage ingestion path, but it should feed an already-solid purchase/reconciliation model.
   • Output targets: purchase_event, price_observation, inventory_delta, merchant_history, brand preference, quantity normalization.
   • Reason to do later: ingestion without a solid reconciliation model creates messy data faster.

8. Inventory contradiction cleanup / Household Memory Health

   • Why: Long-term reliability layer for stale lots, duplicate items, impossible quantities, and old produce.
   • Examples:

     You marked tomatoes bought 9 days ago. Still have them?
     You usually consume coriander within 2 days. Remove from inventory?
     You said you already had carrots, but also bought 500g today. Merge?
     

   • Reason to do last among this cut: it becomes valuable once enough real usage data exists.

Build sequence

services → provenance → recommendation cards → reconciliation → corrections → trace replay → ingestion → memory health

Product thesis

ShopStack’s core magic should be:

It knows what I have, knows what the market snapshot says, admits uncertainty, recommends clearly, then learns from what actually happened.

This sequence preserves clean internal architecture while quickly creating a visible user-facing reason to believe ShopStack is more than a grocery list.

Last updated: 2026-06-06

---

11. Feature Audit + Implementation Delegation Map (2026-06-07)

Context

Pranay wants the full feature backlog implemented, but not by this agent. This section is a durable implementation map for future agents: what to build, why it matters, likely module ownership, and the local/open model + library stack that can support each feature while preserving the active constraint that simultaneously loaded models must stay at ≤32B total parameters.

This is documentation-first. Future implementers should turn each feature cluster into focused tickets/PRs, use existing canonical modules, avoid duplicate routes/parallel systems, and keep core behavior local-first. Connected/cloud modes may exist only as optional, explicitly labeled modes.

Research anchors checked

• Hugging Face OCR open-model guide, 2025-10-21: documents modern open OCR/VLM models and local tooling; notable ≤32B candidates include PaddleOCR-VL (0.9B), Granite-Docling-258M, dots.ocr (3B), DeepSeek-OCR (3B), OlmOCR-2 (8B), Chandra (9B), Qwen3-VL (9B), and Nanonets-OCR2 (~3-4B depending card/source wording). It also calls out MLX-VLM for Apple Silicon and vLLM/SGLang for batch/served OCR.
• SenseVoice repository: SenseVoice-Small is open-source, multilingual, low-latency ASR with speech understanding capabilities.
• Qwen3-32B model card: Qwen3-32B is Apache-2.0, 32.8B total / 31.2B non-embedding; this is at/just over the project’s 32B cap depending counting method, so treat it as a boundary-case candidate requiring explicit budget accounting before activation.
• Current grocery app market scan: competitive apps emphasize real-time shared lists, smart autocomplete, price estimates, receipt scanning, item-level spending analytics, plan-vs-actual comparison, pantry tracking, offline mode, and recipe-to-list flows. ShopStack’s differentiator should be local household memory + price/inventory/trace intelligence, not another checklist.

Implementation rule for future agents

Build these through existing canonical boundaries:

shopstack/services/*          product logic and typed workflows
shopstack/persistence/*       durable local state
shopstack/schemas/models.py   domain contracts
shopstack/providers/*         model/provider adapters
shopstack/tools/registry.py   validated mutation/query tools
shopstack/ui/screens/*        Gradio adapters only
shopstack/traces/*            redaction/export primitives
shopstack/market/*            market normalization/analytics/sources

Do not add a second app, second database, second trace pipeline, or route-like duplicate surface. Extend the existing service/provider/tool architecture.

---

11.1 P0 — make the current product feel real

F-001 Receipt ingestion → inventory + price memory

Build: Upload/photo receipt ingestion that extracts merchant/date/line items/quantities/prices/totals and proposes confirmable purchase events, inventory lots, and price observations.

Why: This closes the highest-leverage data loop. Receipt apps prove demand for item-level spend analytics; ShopStack can go further by updating home stock and future shopping decisions locally.

Likely owners: shopstack/services/receipt.py, shopstack/providers/interfaces.py, shopstack/providers/*ocr*, shopstack/persistence/database.py, shopstack/ui/screens/market_lens.py or a new canonical tab only if module registry is extended deliberately.

Local/open model candidates ≤32B:

Candidate	Params	Fit	Notes
PaddlePaddle/PaddleOCR-VL	~0.9B	primary OCR/document parser	Multilingual, outputs Markdown/JSON/HTML; strong first test for receipts and packet labels.
ibm-granite/granite-docling-258M	0.258B	lightweight document extraction	Very small; good CPU/Mac candidate for structured receipt/label extraction experiments.
rednote-hilab/dots.ocr	~3B	OCR VLM	Markdown/JSON, grounding, handwriting; promising for messy receipts.
deepseek-ai/DeepSeek-OCR	~3B	OCR + general visual understanding	Multilingual; good fallback candidate.
allenai/olmOCR-2-7B-1025	~8B	high-quality document OCR	English-first; use for benchmark/comparison, not necessarily Indian receipt default.
datalab-to/chandra	~9B	document OCR	Strong score, OpenRAIL; verify license before commercial claims.
Qwen/Qwen3-VL-* small variants	~2B/4B/8-9B depending variant	general VLM + OCR	Useful if one VLM handles receipt, shelf, packet, and explanation flows.

Libraries/packages: paddleocr, pytesseract/Tesseract, surya-ocr, marker-pdf, docling, doctr, opencv-python, pillow, pydantic, rapidfuzz, dateparser, price-parser, babel, pandas.

Acceptance: No database writes happen until the user confirms extracted rows. Every proposed write is trace-backed and redacted export-safe.

F-002 Plan-vs-actual shopping review

Build: Compare planned shopping list against actual receipt/manual purchases: bought, forgot, substituted, impulse-bought, overpaid, already-had, price changed.

Why: This is a competitor-visible differentiator and creates memory quality. Current grocery-list competitors highlight planned-vs-actual receipt comparison; ShopStack should connect that to inventory and future decisions.

Likely owners: shopstack/services/reconciliation.py, shopstack/services/shopping.py, shopstack/persistence/database.py, shopstack/ui/screens/shopping.py.

Model/library candidates: Rule-first with rapidfuzz + embeddings (BAAI/bge-m3, intfloat/multilingual-e5-small/base, sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2). Optional LLM arbitration with current 3B planner or Qwen2.5/3 3B-8B Instruct.

Acceptance: Reconciliation must preserve both user intent and actual purchase evidence; do not silently merge ambiguous items.

F-003 Trace service boundary

Build: Move trace bundle/export composition behind shopstack/services/trace.py, while keeping low-level redaction/export in shopstack/traces/export.py.

Why: Traces are core product trust, hackathon evidence, and evaluation substrate. They should not remain screen-local composition logic.

Likely owners: shopstack/services/trace.py, shopstack/ui/screens/traces.py, shopstack/traces/export.py.

Model/library candidates: No model required. Use typed Pydantic view models and deterministic JSONL export.

Acceptance: Existing trace UI behavior preserved; service can be unit-tested without Gradio.

F-004 Real local provider dogfood path

Build: A verified flow where at least one real local model path performs user input → perception/parser → validated tool call → trace → UI output.

Why: Provider interfaces exist, but product credibility requires a real, repeatable local model demo beyond mock providers.

Likely owners: shopstack/providers/registry.py, shopstack/providers/local_provider.py, shopstack/planner/engine.py, shopstack/ui/screens/model_stack.py.

Local/open model candidates ≤32B:

Role	Candidate	Params	Notes
Planner/parser	current Llama-3.2-3B-Instruct-Q4_K_M.gguf	3B	Already downloaded/tested.
Planner/parser	Qwen/Qwen2.5-7B-Instruct or Qwen/Qwen3-8B	7-8B	Strong structured output/tool-call candidate.
Planner/parser	mistralai/Mistral-Small-3.2-24B-Instruct GGUF	24B	Strong single-model reasoning under cap if no big VLM loaded concurrently.
Edge parser	HuggingFaceTB/SmolLM3-3B	3B	Good small parser candidate if tool JSON is reliable.
Boundary max	Qwen/Qwen3-32B	31.2B non-embedding / 32.8B total	Use only with explicit budget policy; leaves no room for other loaded models.

Libraries/packages: llama-cpp-python, mlx-lm, mlx-vlm, transformers, accelerate, bitsandbytes where supported, outlines, guidance, instructor, jsonschema, pydantic.

Acceptance: Model Stack tab shows exact backend/model/params, and traces identify model/provider used.

F-005 Empty-state and honesty polish

Build: Make empty charts/tables, mock perception, stale snapshots, and unavailable providers explicit and visually consistent.

Why: Trust is damaged faster by fake-looking output than by missing features.

Model/library candidates: No model required.

Acceptance: No chart/table renders as a broken blank shell when data is empty; mock/stale/preview mode is labeled wherever it affects decisions.

---

11.2 P1 — product intelligence features

F-006 Household-specific item aliases and canonicalization

Build: Local alias memory for “dahi/curd/yogurt,” “Surf,” “Vim,” “pav,” “atta,” etc. Add correction capture and confidence.

Models/libraries: rapidfuzz, textdistance, sqlite-vec or sqlite-vss, BAAI/bge-m3 (0.6B), multilingual E5, MiniLM, sentence-transformers, optional local LLM parser (Qwen3-4B/8B, current Llama 3B).

Acceptance: Corrections persist and influence future shopping, receipt, and market matching.

F-007 Purchase cadence + stockout forecast

Build: Days-until-stockout and restock interval predictions using local history.

Models/libraries: Start deterministic: SQLite queries, pandas, numpy, statsmodels, simple exponential smoothing. Optional later: sktime, statsforecast, prophet-like alternatives.

Acceptance: Recommendations include confidence language: “likely low,” “uncertain,” not false precision.

F-008 Waste / overbuy detection

Build: Detect old active lots, duplicate purchases, unused perishables, and recurring waste patterns.

Models/libraries: Rule-first with shopstack/decisions.py; optional embeddings for item equivalence.

Acceptance: User can confirm cleanup/merge/consume; no automatic deletion.

F-009 Store memory

Build: Store entities with type, quality/freshness/price/trust/convenience scores, notes, and item-store edges.

Models/libraries: No model required initially. Later embeddings for store-note retrieval.

Acceptance: Store memory feeds “where should I buy?” and price intelligence surfaces.

F-010 Price fairness bands

Build: For scanned/manual price: compare against household memory and source snapshots; classify fair/high/low/unknown.

Models/libraries: pandas, rolling averages, robust median/IQR, unit normalization already in shopstack/market/normalization.py.

Acceptance: Always show source and freshness: household observation vs Swiggy snapshot vs manual entry.

F-011 Budget-aware list planning

Build: Estimate list total, compare against target budget, downgrade optional items, show “must buy / optional / wait” budget plan.

Models/libraries: Deterministic first; optional planner LLM for explanation only.

Acceptance: No hidden price assumptions; every estimate has source/freshness.

F-012 Substitutes and staples

Build: Item relationships: substitutes, usually-bought-with, staples, recipe-use groups.

Models/libraries: SQLite edge table, networkx for analysis if useful, embeddings for semantic substitute suggestions, optional small LLM for candidate generation.

Acceptance: Generated substitutes must be confirmable before becoming durable household rules.

---

11.3 P2 — ShopLens expansion

F-013 Packet label close-up mode

Build: OCR MRP, expiry, quantity, brand, nutrition/storage text from packet labels.

Models: PaddleOCR-VL, Granite-Docling, dots.ocr, DeepSeek-OCR, Qwen3-VL small, MiniCPM-V.

Libraries: opencv-python preprocessing, pillow, dateparser, pydantic validators.

F-014 Barcode catalog memory

Build: First scan creates barcode → item mapping; later scans resolve instantly.

Libraries: Existing shopstack/scanner.py; add/compare pyzbar, zxing-cpp, opencv, JS html5-qrcode, quagga2, zxing-js/browser.

Acceptance: Barcode mappings are user-correctable and local.

F-015 Fridge/pantry photo snapshot

Build: Save visual snapshots and confirmed item detections per household location.

Models: YOLOv8/YOLO11, RF-DETR, GroundingDINO/MM-GroundingDINO, OWL-ViT, Florence-2, MiniCPM-V, Qwen3-VL small.

Current grounding note (2026-06-15): the real-photo household grounding benchmark now exists with three imagegen-created scenes (fridge, cupboard, tabletop). GroundingDINO reached 0.2042 mean box IoU / 0.3333 success after provider fixes; Qwen3-VL stayed helper-only at 0.0136 mean box IoU.

Libraries: ultralytics, rfdetr, transformers, supervision, opencv-python, faiss/sqlite-vec for image-snapshot matching.

F-016 Shelf/location recommendations

Build: Recommend storage location based on category/perishability/history.

Models/libraries: Rule-first from category metadata; optional classifier/LLM for ambiguous items.

F-017 Home Scan scene understanding

Build: A broader ShopLens Home Scan that understands kitchen, bathroom, medicine, cleaning, bedroom, and document scenes; count visible items, read labels/expiry, accept local-language voice corrections, and propose safe inventory updates.

Models: Object detection + segmentation + OCR + STT first, then VLM scene summarization and multilingual correction parsing.

Libraries: Existing provider interfaces (object_detection, segmentation, ocr, stt, image_edit), opencv-python, pydantic, dateparser, optional VLM backends for frame captioning and cross-frame consolidation.

Current implementation note (2026-06-14): Shelf Scan now accepts a short video sweep, extracts multiple frames locally, merges repeated detections into one household-memory result, and uses the real OCR pipeline instead of the old mock-style compatibility fallback. The still-photo path remains intact.

Current grounding note (2026-06-15): generated fridge/cupboard/table photos are now saved under benchmarks/modal/assets/household_grounding and used in the dedicated grounding benchmark harness (benchmarks/modal/bench_grounding_v1.py).

Acceptance: Shelf scans stay confirmation-first, can be zone-aware, can merge short video sweeps, and update household memory without assuming the home is only pantry/fridge.

F-017 Batch confirmation UI

Build: Show proposed structured actions from OCR/image/list parsing and let user accept/edit/reject before writes.

Models/libraries: Pydantic action schemas, Gradio DataFrame/editable components, trace service.

---

11.4 P3 — market and context intelligence

F-018 Multi-source retailer adapters

Build: Add a common adapter interface for Blinkit, Zepto, BigBasket, local CSV/manual imports, and future connected sources.

Libraries: pydantic, pandas, duckdb, httpx only in connected mode, Playwright/browser automation only if explicitly allowed.

Acceptance: Every source record carries source, captured_at, freshness, confidence, and terms/collection notes.

F-019 Basket-level retailer choice

Build: Compare basket total across sources and split basket if savings justify it.

Libraries: pandas, scipy/simple optimization, existing shopstack/market/basket.py.

F-020 Travel-effort decision

Build: Manual-first trip effort model; optional route engines later.

Libraries: osmnx, networkx, OSRM/OpenRouteService/Valhalla only as optional connected mode; geopy, h3 for coarse privacy cells.

F-021 Weather-aware shopping note

Build: Manual/mock weather context; optional weather lookup later.

Libraries: meteostat, Open-Meteo client in connected mode, local config/manual input in Off the Grid.

F-022 Private price heatmap

Build: Coarse local price cells with minimum observation counts.

Libraries: h3, folium or deterministic SVG/HTML cards; avoid exact home/route exposure in traces.

---

11.5 P4 — launch, evaluation, and hackathon assets

F-023 Walkthrough mode

Build: Guided demo story: ask what to buy → scan item → skip duplicate → add purchase → export trace.

Models: Use the smallest reliable active stack: current 3B planner + mock/real OCR path as available.

F-024 Public redacted trace dataset

Build: Curated anonymized JSONL trace examples and dataset card.

Libraries: existing redaction + datasets library for HF publishing.

F-025 Field Notes report generator

Build: Generate polished build report from traces, model stack, decisions, failures, and screenshots/outputs.

Models: Local LLM optional for drafting, but factual claims must come from deterministic trace/model metadata.

F-026 Well-Tuned narrow model

Build: Fine-tune a small command/entity parser for Hinglish grocery commands and publish it.

Base candidates ≤32B: Qwen2.5-1.5B/3B-Instruct, Qwen3-1.7B/4B, SmolLM3-3B, MiniCPM5-1B, current Llama 3B if license permits desired artifact.

Libraries: trl, peft, unsloth, axolotl, datasets, evaluate, jsonschema, pydantic.

Dataset: Indian/Hinglish household command parser dataset with tool-call JSON labels.

F-027 README / HF Space launch checklist

Build: Launch checklist covering privacy, local-first, model budget, data freshness, demo flow, trace dataset, licenses, and known limitations.

Models: None required.

---

12. Local/Open Model and Library Selection Matrix (≤32B constraint)

12.1 Recommended active-stack scenarios

The cap is about simultaneously loaded active models, not every candidate in the repo. Future agents should declare one active scenario before implementation/testing.

Scenario	Active models	Approx params	Use when
Minimal reliable	Llama 3.2 3B planner + BGE-M3 embeddings + Kokoro TTS optional	~3.7B	Current local-first baseline, fast demos.
Receipt-first	PaddleOCR-VL 0.9B + Llama 3B planner + BGE-M3	~4.5B	Receipt/packet extraction with structured reconciliation.
Vision-heavy	MiniCPM-V 8B or Qwen3-VL 8-9B + Llama 3B + BGE-M3	~12-13B	Market Lens, shelf/fridge photos, OCR fallback.
Strong planner	Mistral Small 24B GGUF + lightweight OCR 0.9B + BGE-M3	~25.5B	Harder reasoning, still under cap if no large VLM.
Boundary single-brain	Qwen3-32B alone	31.2B non-embedding / 32.8B total	Only if project explicitly accepts counting method and gives up simultaneous specialist models.
Full practical multimodal	Qwen3-VL 9B + Qwen/Llama planner 7-8B + SenseVoice 0.2B + Kokoro 0.082B + BGE-M3 0.6B + segmentation 0.3B	~18-19B	Rich local multimodal stack under cap.

12.2 Capability-by-capability candidate shortlist

Capability	Best first candidate	Strong alternatives	Libraries/runtime
Planner/tool JSON	current Llama 3B GGUF	Qwen3 4B/8B, Qwen2.5 7B, Mistral Small 24B	llama-cpp-python, mlx-lm, transformers, outlines, instructor
Receipt OCR	PaddleOCR-VL	Granite-Docling, dots.ocr, DeepSeek-OCR, OlmOCR-2	paddleocr, mlx-vlm, vllm, sglang, docling, marker
Packet label OCR	PaddleOCR-VL	dots.ocr, Qwen3-VL, MiniCPM-V	opencv-python, dateparser, OCR/VLM runtime
General VLM	MiniCPM-V or Qwen3-VL small	Florence-2, SmolVLM variants	transformers, mlx-vlm
Object detection	YOLO11/YOLOv8	RF-DETR, GroundingDINO, OWL-ViT	ultralytics, rfdetr, supervision, opencv-python
Segmentation/cards	RMBG-1.4	BiRefNet, SAM variants, ClipSeg	transformers, opencv-python, pillow
ASR	SenseVoice-Small	Parakeet 0.6B/V3, Distil-Whisper, Whisper large-v3-turbo	funasr, faster-whisper, mlx-audio, transformers
TTS	Kokoro-82M	Qwen3-TTS 0.6B, CosyVoice, Piper, KittenTTS	kokoro, piper-tts, onnxruntime, mlx-audio
Embeddings	BGE-M3	multilingual E5, MiniLM, Jina embeddings	sentence-transformers, faiss-cpu, sqlite-vec, lancedb, chromadb
Reranking	BGE reranker small/base	Jina reranker small	sentence-transformers, FlagEmbedding
Time series	deterministic stats	statsmodels/sktime/statsforecast	pandas, numpy, statsmodels, sktime
Route/geo optional	manual + H3	OSRM/OpenRouteService/Valhalla	h3, osmnx, networkx, geopy, folium
Barcode	existing scanner + pyzbar	zxing-cpp, html5-qrcode, Quagga2, ZXing JS	pyzbar, zxing-cpp, opencv, JS browser libs

12.3 Specific model notes for future benchmarking

• PaddleOCR-VL (~0.9B): highest-priority OCR test because it is small, multilingual, and supports JSON/Markdown/HTML-style structured output. Best first model for receipts and packet labels.
• Granite-Docling-258M: extremely small document-understanding candidate. Best for CPU/Mac fallback and quick tests.
• dots.ocr (~3B): good OCR VLM candidate for Markdown/JSON and grounding. Test on messy local receipts.
• DeepSeek-OCR (~3B): useful for OCR + general visual understanding; test on labels and screenshots.
• OlmOCR-2 (~8B): strong document OCR but English-oriented; use as benchmark/reference more than default Indian household model.
• Chandra (~9B): strong document OCR, but verify OpenRAIL restrictions before product claims.
• Qwen3-VL small family (~2B-9B variants): promising single VLM for OCR, packet labels, shelf images, and explanation. Confirm exact chosen variant params/license before activation.
• MiniCPM-V (~8B): strong general VLM candidate and sponsor-aligned with OpenBMB angle.
• SenseVoice-Small (~Whisper-small class): strong ASR candidate for fast multilingual speech understanding. Benchmark Hinglish grocery phrases.
• Parakeet 0.6B/V3: very fast ASR candidate; confirm language coverage for Hindi/Hinglish before relying on it.
• Kokoro-82M: excellent tiny local TTS baseline. Use for short, warm household answers if pronunciation is acceptable.
• Qwen3-TTS 0.6B / CosyVoice: better voice quality candidates; benchmark latency and Hindi/Hinglish pronunciation.
• BGE-M3 (0.6B): already aligned with workspace embedding use; use for aliases, trace retrieval, and household memory search.
• Qwen3-32B: boundary case. It is useful as a “single strong brain” experiment but total params are listed as 32.8B and non-embedding 31.2B. Do not activate without updating the budget rule/counter and documenting whether total or non-embedding params are counted.

12.4 Non-model libraries that can implement much of the backlog

Many features should be deterministic first:

• Parsing and validation: pydantic, jsonschema, dateparser, price-parser, babel, python-slugify.
• Matching/canonicalization: rapidfuzz, textdistance, unidecode, regex, sentence-transformers.
• Analytics/time series: pandas, numpy, duckdb, statsmodels, sktime, statsforecast.
• Vector/local memory: sqlite-vec, sqlite-vss, faiss-cpu, lancedb, chromadb.
• Vision preprocessing: opencv-python, pillow, scikit-image, imagehash.
• OCR baselines: pytesseract, paddleocr, surya-ocr, docling, marker-pdf, doctr.
• Barcode: pyzbar, zxing-cpp, browser html5-qrcode, quagga2, @zxing/browser.
• Maps/geo: h3, geopy, osmnx, networkx, folium for optional map experiments.
• Voice pipeline: funasr, faster-whisper, piper-tts, kokoro, onnxruntime, mlx-audio, pipecat for a more advanced local voice loop.
• Model serving: llama-cpp-python, mlx-lm, mlx-vlm, transformers, vllm, sglang, ollama as optional local dev runtime.
• Constrained outputs: outlines, guidance, instructor, lm-format-enforcer.
• Training/fine-tuning: datasets, peft, trl, unsloth, axolotl, evaluate.

12.5 Implementation sequencing recommendation

Future agents should implement in this order:

1. Trace service boundary
2. Receipt extraction proposal schema, no writes
3. Receipt confirmation → purchase/price/inventory writes
4. Plan-vs-actual reconciliation
5. Alias/canonicalization correction loop
6. Price fairness + budget-aware list planning
7. Packet label mode + barcode catalog memory
8. Fridge/pantry snapshot and storage recommendations
9. Store memory + basket-level retailer choice
10. Trace replay/eval harness + public redacted trace dataset
11. Fine-tuned narrow command parser
12. Optional travel/weather/heatmap intelligence

This order protects data integrity first, then adds intelligence, then expands perception/context. It also gives a future non-this-agent implementation team clean seams for independent work.

Last updated: 2026-06-07

---

13. Fresh HF / 2026 Model Launch Addendum (2026-06-07)

Why this addendum exists

Pranay challenged whether the previous model scan was fresh enough. That challenge was valid. The first pass covered strong late-2025 OCR/VLM candidates, but it underweighted newer/open launches and HF ecosystem updates relevant to ShopStack, especially omni-modal, realtime speech, multimodal embeddings/rerankers, and HF Pro/Jobs workflows.

This section supersedes the older shortlist where it is more specific. Keep the older section as historical context, but future agents should check this addendum first when choosing models.

Fresh research anchors checked

• Qwen3-Omni-30B-A3B-Instruct / Thinking / Captioner model cards and GitHub: Apache-2.0, omni-modal, text/image/audio/video input, text/speech output, 119 text languages, 19 speech-input languages, 10 speech-output languages, real-time streaming; 30B MoE with ~3B active parameters. Strong candidate for a single multimodal “one brain does a lot” experiment, but likely heavy operationally.
• Mistral Voxtral Mini 4B Realtime 2602 model card: Apache-2.0, multilingual realtime ASR, 4B params, <500ms delay target, 13 languages, designed for on-device/minimal hardware compared with heavier speech stacks.
• Hugging Face OCR open-model guide + community comments: added/flagged models not previously emphasized enough: opendatalab/MinerU2.5-2509-1.2B, lightonai/LightOnOCR-1B-1025, tencent/POINTS-Reader, Logics-MLLM/Logics-Parsing, SmolDocling-256M-preview, RolmOCR-7B, Typhoon-OCR-7B, MonkeyOCR-Recognition.
• Hugging Face multimodal Sentence Transformers v5.4 blog (2026-04): new multimodal embedding/reranker support for text/images/audio/video. Qwen/Qwen3-VL-Embedding-2B is explicitly shown and needs ~8GB VRAM; 8B variants need ~20GB. This matters for shelf image retrieval, receipt visual retrieval, and trace/memory search.
• HF Jobs pricing/docs: Jobs can run batch compute on HF infrastructure when the account has positive credit balance; billed by minute while starting/running. This fits Pranay’s HF Pro/credits for benchmarking and batch OCR experiments without adding cloud dependency to the Off-the-Grid runtime.
• vLLM-Omni speech API docs: supports serving Qwen3-TTS, Fish Speech, Voxtral TTS, CosyVoice3, OmniVoice, VoxCPM2, MOSS-TTS-Nano, Qwen3-Omni and related omni/audio models via OpenAI-style APIs. Useful for benchmark rigs and optional connected/dev mode.
• Aya Vision 8B/32B: multilingual VLM family from CohereForAI. 8B is relevant for multilingual image/text tasks; 32B is boundary/too tight for simultaneous loaded stack and should be treated carefully.
• SmolVLM / SmolVLM2 family: tiny VLMs (256M/500M/2.2B class) useful for on-device-ish image caption/doc QA/video-lite experiments, not likely enough for robust receipt extraction alone but excellent baseline/performance fallback.

---

13.1 Updated “newer launch” candidates for ShopStack

Omni / single-model-does-a-lot candidates

Model	Params / active	License	Why it matters for ShopStack	Caveat
Qwen/Qwen3-Omni-30B-A3B-Instruct	30B total, ~3B active MoE	Apache-2.0	One model can cover text, image, audio, video, ASR-like input, and speech output. Strong candidate for a “single local/offline brain” experiment and sponsor-aligned demo.	Transformers inference can be slow; vLLM/Docker recommended. Counts as 30B loaded for project budget unless policy explicitly counts active params. Leaves little room under 32B if total params counted.
Qwen/Qwen3-Omni-30B-A3B-Thinking	30B total, ~3B active MoE	Apache-2.0	Better for complex multimodal reasoning over receipt/photo/audio traces.	Same budget/runtime caveat; likely slower.
Qwen/Qwen3-Omni-30B-A3B-Captioner	30B total, ~3B active MoE	Apache-2.0	Audio captioning and market/environment audio notes; useful for “what happened in this noisy shopping audio?”	More specialized; not a default core runtime.
openbmb/MiniCPM-o-2_6	~8B	check model card	Any-to-any style multimodal candidate with vision/speech/language. Good OpenBMB sponsor alignment.	Must verify current license/runtime before product use.

Opinionated take: Qwen3-Omni is the most exciting newer launch for the product vision, but it is not the first implementation model. Use it with HF Pro/credits for benchmarking and one “wow” demo. For reliable local-first core, keep specialist small models + deterministic services.

Fresh OCR / document-understanding candidates to add to benchmark set

Model	Params	Why it matters	Suggested ShopStack test
opendatalab/MinerU2.5-2509-1.2B	~1.2B	Mentioned in HF OCR guide comments as missing from comparison; compact document parser.	Indian grocery receipt with line items + packet label close-up.
lightonai/LightOnOCR-1B-1025	~1B	Community-flagged “punches above weight” OCR candidate.	Low-quality local receipt photo and multilingual label.
tencent/POINTS-Reader	~3B text backbone + vision encoder	Efficient end-to-end document AI replacing larger 7B backbone with Qwen2.5-3B; Chinese/English strengths.	Receipt OCR and structured text extraction speed/accuracy.
Logics-MLLM/Logics-Parsing	verify card	Single-model document parsing for complex docs.	Use as secondary benchmark, not default.
SmolDocling-256M-preview	~256M	Tiny document understanding candidate.	CPU/Mac fallback for basic labels/receipts.
RolmOCR-7B	~7B	Lightweight VLM-level OCR option.	Compare against OlmOCR and PaddleOCR-VL.
Typhoon-OCR-7B	~7B	OCR model seen in HF OCR demo collections.	Test if receipts are multilingual/noisy.
MonkeyOCR-Recognition	verify card	OCR recognition candidate in HF demo collections.	Baseline OCR recognition, not full structured extraction.

Updated OCR priority:

1. PaddleOCR-VL (~0.9B)
2. MinerU2.5-1.2B + LightOnOCR-1B
3. Granite/SmolDocling tiny fallbacks
4. dots.ocr / DeepSeek-OCR / POINTS-Reader
5. OlmOCR-2 / RolmOCR / Typhoon-OCR as heavier benchmarks
6. Qwen3-VL / Qwen3-Omni for general multimodal fallback

Fresh ASR / realtime voice candidates

Model	Params	License	Why it matters	Caveat
mistralai/Voxtral-Mini-4B-Realtime-2602	4B	Apache-2.0	Newer realtime ASR, 13 languages, <500ms delay target, open weights. Stronger fresh candidate than older Whisper-only baseline.	Hindi/Hinglish support must be tested; 4B is heavier than SenseVoice/Parakeet.
Qwen/Qwen3-Omni-30B-A3B-Instruct	30B total / 3B active	Apache-2.0	Speech input and speech output in one model; supports Urdu but not explicitly Hindi speech input in listed languages.	Heavy; use as benchmark/wow path.
FunAudioLLM/SenseVoiceSmall	~0.2B class	likely permissive; verify exact card	Fast multilingual ASR + emotion/audio events.	Language list may not cover Hindi directly; benchmark Hinglish.
NVIDIA Parakeet V3 / 0.6B family	~0.6B	check card	Very fast ASR; good low-latency candidate.	Often English-focused; verify multilingual/Hinglish.
Whisper large-v3-turbo	~0.8B	MIT	Mature baseline with broad language support.	Not native realtime; use faster-whisper/whisper.cpp.

Fresh TTS / speech output candidates

Model	Params	License	Why it matters	Caveat
Qwen/Qwen3-TTS-12Hz-0.6B-*	0.6B	check card; Qwen family often Apache-2.0	Small Qwen TTS; vLLM-Omni supports CustomVoice/Base.	Benchmark Hindi/Hinglish pronunciation and runtime complexity.
Qwen/Qwen3-TTS-12Hz-1.7B-*	1.7B	check card	Better quality than 0.6B; supports CustomVoice/VoiceDesign/Base variants via vLLM-Omni.	Heavier, still fine under 32B.
FunAudioLLM/Fun-CosyVoice3-0.5B-2512	0.5B	verify card	vLLM-Omni supported; voice cloning style path.	More complex request requirements.
mistralai/Voxtral-4B-TTS-2603	~4.1B	reported CC-BY-NC 4.0 in third-party scan; verify HF card	High-quality multilingual/Hindi-capable TTS candidate.	Non-commercial likely; not default for product unless license accepted.
OpenMOSS-Team/MOSS-TTS-Nano	small; verify	verify	vLLM-Omni supported; potential lightweight voice cloning.	Needs ref audio; not preset voice default.
Kokoro-82M	82M	Apache-2.0	Still best tiny default TTS baseline.	Pronunciation/voice quality may be weaker than newer TTS.
Piper	small	permissive	Reliable local TTS fallback.	Less “wow,” but robust.

Fresh multimodal embedding / reranker candidates

Model / stack	Params	Why it matters
Qwen/Qwen3-VL-Embedding-2B	2B	Directly relevant to visual receipt/shelf/image retrieval, multimodal memory search, and trace retrieval. HF Sentence Transformers v5.4 shows loading with image support.
Qwen3-VL Embedding 8B variants	~8B	Higher-quality multimodal retrieval if GPU memory allows.
Sentence Transformers v5.4 multimodal rerankers	varies	Lets ShopStack rerank text-image/audio/video memories with one API.
CLIP/SigLIP/SigLIP2 family	small-medium	Good fast image/text similarity for shelf snapshots and visual duplicate matching.
BGE-M3	0.6B	Still best text-heavy multilingual local memory default.

New product implication: ShopStack should not only do text embeddings. It should plan for multimodal household memory: receipt images, shelf photos, voice snippets/transcripts, and traces should be retrievable across modalities.

---

13.2 HF Pro / credits usage plan

Pranay has HF Pro and credits. Use them for research, benchmarking, and artifact creation — not as hidden runtime dependency for the Off-the-Grid path.

Good uses of HF Pro/credits

1. Batch OCR bakeoff with HF Jobs

   • Create a small representative dataset: 20 receipt photos, 20 packet labels, 10 shelf/fridge photos, 10 Swiggy screenshots/product cards.
   • Run PaddleOCR-VL, MinerU2.5, LightOnOCR, dots.ocr, DeepSeek-OCR, Granite/SmolDocling, POINTS-Reader via Jobs or temporary GPU Space.
   • Save outputs as a versioned benchmark dataset/artifact.

2. Temporary GPU Space for demo/model exploration

   • Spin up model-specific Spaces for Qwen3-Omni, Voxtral Realtime, Qwen3-TTS, Qwen3-VL Embedding.
   • Record latency, memory, model output, failure modes.
   • Tear down or downgrade hardware after experiment.

3. Dataset publishing

   • Publish redacted/synthetic command parser dataset.
   • Publish redacted trace examples.
   • Publish OCR benchmark fixtures if privacy-safe.

4. Fine-tuning / LoRA training

   • Narrow parser LoRA on Qwen/SmolLM/MiniCPM-sized base.
   • Do not fine-tune a broad “do everything” model first.

5. Model comparison reports

   • Use HF Jobs for repeatable batch inference and store metrics.
   • Feed results back into Docs/MODEL_CATALOG.md and shopstack/model_registry.py only after local/runtime verification.

Bad uses of HF Pro/credits

• Making HF Inference Endpoint mandatory for core app behavior while claiming Off the Grid.
• Using credits to bypass implementing local confirmation, validation, and traceability.
• Benchmarking on generic OCR samples only; ShopStack needs grocery/household-specific data.
• Promoting a model to active runtime without testing memory, latency, license, and JSON/tool-call reliability.

13.3 Revised model strategy

The correct strategy is not “pick one best model.” Use a three-lane model strategy:

Lane A — Reliable local core
  deterministic services + Llama/Qwen 3B-8B planner + PaddleOCR-VL/Granite/SmolDocling + BGE-M3 + Kokoro/Piper

Lane B — Specialist excellence
  Voxtral Realtime for ASR, Qwen3-TTS/CosyVoice3 for TTS, Qwen3-VL Embedding for multimodal memory, YOLO/RF-DETR for detection

Lane C — Wow / future omni demo
  Qwen3-Omni-30B-A3B for audio/image/video/text/speech unified experience, benchmarked with HF credits, optional not required

13.4 Updated benchmark tasks for future agents

Create a repo-local benchmark spec before model promotion:

Task	Inputs	Required output	Metrics
Receipt line extraction	receipt photos	merchant/date/items/qty/unit/line price/total	field F1, human correction count, latency
Packet label extraction	packet label closeups	brand/item/MRP/expiry/quantity/storage	exact date/price accuracy, confidence calibration
Hinglish voice command	short audio phrases	transcript + intent + tool JSON	WER, intent accuracy, JSON validity
Market photo scan	shelf/product photo	candidate item + price/freshness uncertainty	top-1/top-3 accuracy, false certainty rate
Alias matching	text/receipt/list variants	canonical item ID	accuracy, ambiguous deferral rate
Multimodal retrieval	text query over images/traces	relevant receipt/shelf/trace	recall@k, rerank quality
Short TTS answer	generated household answers	natural speech	latency, intelligibility, Hindi/Hinglish pronunciation
Omni demo	photo/audio/video prompt	useful answer + optional speech	latency, trustworthiness, failure modes

13.5 Immediate doc follow-ups

Future agents should update these artifacts after each real benchmark:

• Docs/MODEL_CATALOG.md: add the fresh model, params, license, runtime, status, benchmark notes.
• shopstack/model_registry.py: add only candidates that passed basic source/license checks; mark active only after runtime verification.
• Docs/ShopStack_Exploration_Map.md: add findings, rejected models, and changed priorities.
• tests/: add fixtures for model-independent parsing/validation paths so model swaps do not break product contracts.

13.6 Bottom line

The previous shortlist was useful but incomplete. The biggest newer additions are:

1. Qwen3-Omni-30B-A3B as the ambitious single-model omni experiment.
2. Voxtral Mini 4B Realtime as a fresh ASR candidate.
3. Qwen3-TTS / CosyVoice3 / vLLM-Omni speech stack as a newer TTS evaluation lane.
4. MinerU2.5, LightOnOCR, POINTS-Reader, SmolDocling, RolmOCR, Typhoon-OCR as added OCR benchmark candidates.
5. Qwen3-VL Embedding + Sentence Transformers multimodal v5.4 as the right direction for multimodal household memory.
6. HF Jobs/Spaces with Pro credits as the right way to benchmark without making cloud a product dependency.

Last updated: 2026-06-07

---

14. Living Model Research Intake + Leaderboard-Driven Discovery Protocol (2026-06-07)

Why this section exists

Pranay pushed back that the model research was not thorough enough. That pushback is correct: generic “best model” lists are too shallow for ShopStack. The project needs a repeatable intake process that can absorb user-provided model names, watch new Hugging Face launches, compare leaderboards by modality and size, and still stay grounded in the product’s actual tasks.

This is a living research queue, not an implementation instruction. Future agents should document new findings here first, then only promote models into Docs/MODEL_CATALOG.md / shopstack/model_registry.py after license and runtime verification.

Hard constraints for model candidates

• Prefer open-source / open-weight / locally runnable models.
• Prefer models with ≤32B total loaded parameters. MoE models with ≤32B active params but much larger total params may be listed, but must be marked as active-param-only fit and treated as eval/API/large-rig candidates unless a local hardware plan exists.
• Keep the Off-the-Grid promise: HF Pro/credits may be used for research, benchmarking, fine-tuning, and batch evals, but must not become a hidden runtime dependency for core behavior.
• Rank by ShopStack usefulness, not generic benchmark prestige.
• Every candidate must eventually be tested on grocery/household data, not just public benchmark samples.

Intake template for names Pranay provides

When Pranay gives a model/library name, append a row here or in a dated subtable using this schema:

Field	Required note
Name / link	Exact HF repo, GitHub repo, paper, or package name.
Modality	Text, OCR/doc, VLM, ASR, TTS, embedding/rerank, detection, segmentation, time-series, agent/tooling.
Size	Total params, active params if MoE, expected quantized RAM/VRAM.
License	Exact license and commercial risk. Do not infer.
Runtime	transformers, llama.cpp, mlx, vLLM, SGLang, ONNX, WebGPU, library API, etc.
ShopStack fit	Receipt OCR, packet label OCR, pantry photo, Hinglish voice, TTS answers, planner/tool calls, multimodal memory, price intelligence, etc.
Evidence source	Official model card/paper, leaderboard, GitHub, independent eval, or user-provided hunch.
Verdict	evaluate now, watch, reject, eval-only, or blocked on license/runtime.
Benchmark task	The first concrete ShopStack eval this model should run.

Leaderboard/source watchlist by capability

Use these sources first before relying on blog roundups:

Capability	Primary sources to check	What to extract	Caveat
General small LLM / reasoning	Hugging Face Open LLM Leaderboard org/results, Artificial Analysis with open-weights filter, Vellum Open Source LLM Leaderboard, Onyx self-hosted/open-source leaderboards	Params, open-weight status, reasoning/coding scores, context, latency/cost where available	General reasoning scores do not prove tool-call reliability or JSON correctness.
Small local / edge LLMs	HF model cards, small-model leaderboards, llama.cpp/MLX/Ollama community benchmarks	Quantized memory, speed, instruction following, multimodal support	Many “under 10B” lists mix official, vendor-reported, and community scores. Verify model cards.
Tool calling / structured output	Berkeley Function Calling Leaderboard, Gorilla/OpenFunctions/xLAM/Hammer reports, independent local LM Studio/Open WebUI tests	Tool selection, argument validity, multi-tool support, format compatibility	Chat-template/runtime mismatch can make a leaderboard winner fail locally. Test with ShopStack’s actual server.
VLM / multimodal	OpenCompass Open VLM Leaderboard, VLMEvalKit, HF OpenVLM records, model cards for Qwen/Gemma/MiniCPM/SmolVLM/Aya/Phi	OCR, chart/doc QA, visual reasoning, GUI grounding, video/image support	VLM leaderboard rank is not the same as receipt OCR or packet label extraction.
OCR / document AI	OCRBench-style reports, OmniDocBench, DocVQA/InfoVQA/ChartQA benchmarks, HF OCR guides, Docling/MinerU/PaddleOCR docs	Field extraction, layout retention, multilingual OCR, structured Markdown/JSON support	Product needs receipts and labels; academic docs may not transfer.
ASR / speech input	HF Open ASR Leaderboard, ASR Leaderboard paper/code, model cards for Canary/Qwen, Parakeet, Whisper, SenseVoice, Voxtral	WER, real-time factor, multilingual support, streaming support	Hindi/Hinglish and noisy market audio must be tested separately.
TTS / speech output	TTS Arena, HF trending TTS, model cards for Kokoro/Piper/Qwen3-TTS/CosyVoice/Chatterbox	Naturalness, latency, voice cloning, language/pronunciation, license	Subjective voice quality needs human listening tests.
Embeddings / rerankers	MTEB leaderboard, Sentence Transformers docs/blogs, BGE/Jina/Qwen embedding model cards	Retrieval score, multilingual score, multimodal support, rerank ability	Overall MTEB score hides retrieval-specific performance. Test household memory queries.
Detection/segmentation	COCO/LVIS/open-vocabulary detection reports, Ultralytics/RF-DETR/GroundingDINO/SAM docs	Object detection, grounding, segmentation, runtime	Model may detect generic objects but not branded groceries.
Time series / forecasting	Nixtla/statsforecast docs, sktime, NeuralForecast, Chronos/TinyTimeMixers model cards	Forecast accuracy, local CPU viability, probabilistic outputs	Use deterministic statistical baselines before neural forecasting.

Current leaderboard-driven candidate intake

This table captures newer candidates and source directions surfaced by the broader leaderboard scan. It is deliberately conservative: candidate ≠ approved runtime.

Candidate	Modality	Size / fit	Evidence/source direction	ShopStack fit	Verdict / first eval
Qwen3.5-4B / Qwen3.5-9B small series	Planner, tool calling, possible multimodal depending exact repo	≤10B	Small-model rankings and independent local tool-calling tests report very strong structured/tool behavior for 4B-class Qwen	Local planner, command parser, JSON/tool calls, cheap local assistant	Evaluate now: tool JSON harness + shopping command parser. Verify exact HF repos/licensing.
NVIDIA Nemotron Nano 4B / Nemotron Nano 9B/12B/VL	Tool calling, general LLM, VLM variants	≤12B	Artificial Analysis and local tool-calling evals surface strong speed/structured behavior for Nano class	Local planner fallback, possible VLM route if VL variant is open/licensed	Evaluate after license/runtime check: tool call schema + receipt/photo QA.
GPT-OSS-20B	General/reasoning/tooling	20B	Vellum/Artificial Analysis list strong open-weight cost/speed; local tool eval shows decent but not best tool calls	Larger local brain for planning/evals, not first core model	Watch/evaluate: compare against Qwen 4B/9B on JSON and latency. Confirm license/runtime.
Gemma 3 4B / 12B / 27B and Gemma 3n E4B	Text + vision, edge/local	≤27B total; 3n effective smaller	Small-model and open-source leaderboards consistently cite strong local multimodal and edge viability	Pantry/shelf photo QA, on-device assistant, fallback VLM	Evaluate now for vision + local memory; test receipts but do not assume OCR excellence.
Phi-4-mini / Phi-4 multimodal / Phi-4 reasoning-vision	Small reasoning, multimodal	~3.8B to ~15B	Small-model rankings cite math/reasoning; multimodal variants appear useful for GUI/vision	Lightweight reasoning, possible product-photo understanding	Watch/evaluate: tool-call reliability looked weaker in one local test; verify current variants.
Mistral Nemo 12B	Planner/tooling	12B	Independent local tool-call eval ranked it highly under LM Studio	Structured local planner alternative	Evaluate if Qwen fails or license/runtime is simpler.
Mistral Small 3.x / 24B	Planner/general	24B	Self-hosted leaderboards list it, but local tool-call tests may underperform via incompatible templates	Possible high-quality planner if native runtime works	Blocked on runtime-template validation. Do not assume BFCL/official score transfers.
xLAM-2 8B / Salesforce function-calling models	Tool calling	~8B	BFCL-oriented function calling family; independent local tests warn about template incompatibility	Specialist local tool caller	Evaluate with native template, not generic OpenAI-compatible server only.
Hammer 2.x 7B	Tool calling	~7B	Function-calling specialist with benchmark claims; local runtime mismatch risk	Specialist command-to-action parser	Evaluate only after chat-template support confirmed.
Qwen3-VL-Embedding-2B	Multimodal embedding/rerank	2B	HF Sentence Transformers v5.4 multimodal docs/blog explicitly support multimodal embeddings	Multimodal household memory: retrieve receipt images, shelf photos, voice traces	Evaluate now: recall@k on text query → receipt/photo/trace.
Sentence Transformers v5.4 multimodal stack	Library/runtime	model-dependent	HF blog adds text/image/audio/video embedding and reranker support	One API for multimodal memory experiments	Adopt in benchmark tooling before app runtime.
NVIDIA Canary-Qwen 2.5B	ASR	2.5B	HF Open ASR Leaderboard / ASR paper sources cite strong open ASR accuracy	Voice receipt notes, spoken shopping commands	Evaluate now: Hinglish/noisy command WER + latency.
NVIDIA Parakeet TDT/CTC 0.6B–1.1B	ASR	≤1.1B	ASR leaderboard cites very high real-time factor for fast transcription	Realtime local voice if language coverage fits	Evaluate speed lane: English/Hinglish WER and streaming.
IBM Granite Speech 3.3 8B	ASR/audio LLM	~8B/9B class	ASR leaderboard/blog sources cite strong WER and Apache license claims; verify exact card	Accurate voice command transcription	Watch/evaluate after license/runtime check.
Voxtral Mini 4B Realtime	ASR/audio	4B	Mistral/HF realtime speech model candidate	Low-latency voice loop	Evaluate: latency + Hindi/Hinglish support; compare to Canary/Parakeet/SenseVoice.
Chatterbox	TTS	~0.5B class	TTS roundups/trending cite strong small voice cloning/naturalness	Warm local household voice, optional personalized voice	Evaluate: license, voice quality, latency, pronunciation.
Kokoro-82M	TTS	82M	Still strong tiny baseline	Default local TTS baseline	Keep baseline; benchmark against Qwen3-TTS/CosyVoice/Chatterbox.
Qwen3-TTS 0.6B/1.7B	TTS	≤1.7B	vLLM-Omni supported Qwen TTS family	Higher-quality local speech output	Evaluate: runtime complexity + Hinglish pronunciation.
PaddleOCR-VL	OCR/doc	~0.9B	Prior section already flagged; still top first OCR candidate	Receipt/label structured extraction	Evaluate first on grocery dataset.
MinerU2.5-1.2B / LightOnOCR-1B	OCR/doc	~1B	HF OCR guide/community-cited newer compact OCR candidates	Receipt/label extraction, document parsing	Evaluate now in OCR bakeoff.
SmolDocling / Granite-Docling	OCR/doc	256M-ish	Tiny document understanding candidates	CPU fallback, quick document parse	Evaluate as fallback, not likely top accuracy.
Qwen3-Omni-30B-A3B	Omni multimodal	30B total / ~3B active	Fresh HF model cards; omni audio/image/video/text/speech	“One model does a lot” wow demo and benchmark oracle	Eval-only/wow lane: total loaded params nearly consumes cap; not reliable first core.
Qwen3-VL 235B-A22B	VLM/OCR/GUI	235B total / 22B active	Open-source VLM leaderboards and provider writeups cite strong OCR/GUI	Reference oracle for vision/OCR quality	Active-param-only / cloud eval, not local Off-the-Grid default due total size.
MiniMax-M2.5	Agentic LLM	229B total / 10B active	Open-source/self-hosted leaderboards cite strong coding/function-calling	Reference for complex planning	API/eval-only unless hardware plan exists. Total params exceed local cap.
Kimi K2 Thinking/K2.5	Agentic LLM	~1T total / 32B active	Open-source leaderboards place it high for reasoning/agentic coding	High-end external judge/planner reference	Out of local cap by total size; can be benchmark reference with credits/API only.

ShopStack-specific model eval suite to create before promotion

A future implementation agent should create this as a benchmark harness before adding active runtime models:

Eval	Dataset size	Pass criteria	Why it matters
Receipt OCR bakeoff	20–50 real/synthetic receipts	item/date/merchant/total field F1, correction count, latency	Core automation must not corrupt inventory/price history.
Packet label extraction	20–50 packet labels	MRP/expiry/brand/size exactness, ambiguity detection	Prevent stale/expired item and wrong unit bugs.
Hinglish voice command	50–100 short clips	WER + intent accuracy + valid JSON tool calls	Voice UX depends on this.
Tool-call schema harness	40–100 prompts	exact tool selection, required args, no unauthorized action	Planner safety and trace correctness.
Multimodal memory retrieval	30–100 image/text/audio trace entries	recall@5 and rerank quality	Enables “where did we buy this?” and “show last receipt for atta.”
Shelf/fridge image understanding	30–50 photos	top-k item detection/caption confidence, false-certainty rate	Supports pantry/photo features without hallucinated writes.
TTS listening test	20 generated responses	intelligibility, warmth, pronunciation, latency	Voice output should feel useful, not gimmicky.
Local runtime budget	all promoted models	RAM/VRAM, cold start, tokens/sec, fallback behavior	Keeps Off-the-Grid credible.

Research quality rules for future passes

• Do not cite a model as “best” without saying best for what benchmark/task.
• Always separate official/vendor-reported metrics from third-party and our own metrics.
• For MoE models, record both total params and active params. The project cap should default to total loaded params unless explicitly overridden for a demo/eval lane.
• A model with strong BFCL/function-calling score still needs local runtime validation because chat templates often break OpenAI-compatible serving.
• A VLM with strong MMMU/MMBench score still needs receipt/label OCR testing.
• An ASR model with low English WER still needs Hinglish and noisy-market audio tests.
• A TTS model with high community preference still needs licensing and pronunciation checks.
• Prefer deterministic libraries for core arithmetic, matching, time-series, and validation; models should propose, classify, transcribe, extract, or explain — not silently mutate state.

Immediate next research actions

1. As Pranay provides model names, append them to the intake table with exact links and verdicts.
2. Pull exact HF model cards/licenses for the top candidates above, starting with Qwen3.5 small, Nemotron Nano, Canary-Qwen, Granite Speech, Chatterbox, Qwen3-VL-Embedding, and the OCR candidates.
3. Build the benchmark fixture list before any implementation agent wires models into runtime.
4. Update Docs/MODEL_CATALOG.md only after exact model card/license/runtime evidence is collected.
5. Keep a rejected/blocked list; avoiding bad models is as useful as finding good ones.

Last updated: 2026-06-07

---

Design System & UI Architecture (2026-07-12)

Context

Full design audit completed (see Docs/audits/audit_05_full_design_audit.md). UI surface has solid foundation but needs component library expansion, design token completion, and architectural cleanup.

Completed

• Split other.py → price_memory.py, household_map.py, field_notes.py, swiggy_market.py
• Moved DEMO_SEED_INVENTORY → shopstack/data/seed_demo.py
• CSS tokens v2: spacing, font-size, shadow, z-index, animation scales
• Unified DECISION_COLORS with CSS variables, created theme_tokens.py
• Fixed Audit 3: removed 3 wrapper fns, private imports, extracted _runtime_label
• P1 components: item_row, stat_card, data_table, confirm_dialog, toast, loading_skeleton, empty_state_enhanced

Open

• Phase 2/3 components, dark mode, keyboard shortcuts, a11y audit
• Remaining hardcoded color migration in swiggy_market, image_cards, market_lens, shopping
• Component adoption: migrate existing screens to use new components

Last updated: 2026-07-12

---

Architecture Consolidation & Stash Recovery (2026-06-08)

Context

An intentional architectural branch (stashed as WIP) performed three major transformations: screen module extraction, design system v2, and 5-tab UX restructure. Full architecture documented in Docs/STASH_ARCHITECTURE_RECOVERY.md.

Completed Architecture Changes

• 5-tab nested UX: 13 flat tabs → 5 nested tabs (Today, Basket, Market Lens, Reconcile, Memory). Groups related product surfaces.
• other.py → 51L re-export shim. Functions extracted to canonical modules: price_memory.py, household_map.py, field_notes.py, swiggy_market.py.
• Design system v2: 728-line token-based theme with Emil Kowalski animations, WCAG AA focus-visible, reduced-motion, responsive breakpoints.
• Lazy provider loading: ProviderRegistry defers heavy imports (funasr, torch, kokoro) to first get() call. import app dropped from 90s to 2.3s.
• Local model defaults: STT=mlx-whisper, TTS=kokoro-82M, planner=MLX/llama.cpp. All real models by default, mock only for off-trunk providers (vision/OCR).
• Cost tracking + tracing: OpenTelemetry trace_call() spans in local and OpenAI providers with model tier estimation.
• semantic_find_item: Tiered search (exact → prefix → semantic embedding) registered as canonical tool with ToolSpec.
• Decision cards: SVG-based visual cards in dashboard for buy/skip/use-soon.

New Capabilities Added (Aug 2026-06-08)

• Receipt scanning: OCR → purchase events pipeline with services/receipt.py
• Multi-retailer adapters: Blinkit, Zepto, DMart source adapters with cross-source price comparison via market/sources/_comparison.py
• Semantic inventory search: BGE-M3 embedding provider with tiered fallback to prefix search
• Dashboard intelligence: Cadence detection, waste warnings, weather context all wired into DashboardState via service layer
• Nutrition tracking: 30-item reference database + service + UI tab
• Weather/trip context: Open-meteo API integration with caching and offline mock fallback
• Demo walkthrough: scripts/demo_walkthrough.py — idempotent, seeds inventory, prices, shopping list
• Performance benchmarks: 30 pytest-benchmark tests with results in benchmarks/RESULTS.md

Test Health

619 passed, 0 failed, zero regressions from 597 baseline

Decision Records (DR-005 through DR-012)

• DR-005: Screen module extraction — split other.py into canonical modules
• DR-006: Design system v2 — token-based CSS replacing flat hardcoded values
• DR-007: 5-tab nested UX — 13 flat tabs collapsed to 5 primary tabs
• DR-008: Lazy provider loading — deferred heavy imports for fast app init
• DR-009: Local model defaults — real models by default, optical upgrades
• DR-010: SVG decision cards — visual card system with PNG fallback via Flux
• DR-011: semantic_find_item — tiered search registered as canonical tool
• DR-012: Cost tracking + tracing — OpenTelemetry spans in LLM providers

Exploration Areas Opened

• Dark mode theme variant using design tokens
• Keyboard shortcut system for power users
• Accessibility audit against WCAG 2.1 AA
• Component migration: adopt primitives (ItemRow, Toast, etc.) across screens
• Model download UX: progress indicators for initial local model downloads
• Offline-first backup/restore: encrypted household snapshots
• Mobile capture path: PWA wrapper for Gradio with local-first guarantees

---

14. Expansion Sweep — Cross-Project Inspired Areas (2026-06-08)

This section expands the open exploration surface using patterns from sibling Projects docs such as family-dinner-os exploration audits, invoice-intelligence autoresearch/evaluation docs, idea_pad domain census, and local ShopStack docs. All items remain documentation-only until promoted into ROADMAP.md or decision records.

14.1 Household Operating-System Verticals

ShopStack can stay anchored in shopping while growing into adjacent household systems:

• Family meal operating system — meal debates, preferences, leftovers, nutrition, budget, and available stock converge into one dinner decision.
• Caregiver / elder household mode — low-vision voice workflows, medicine-like caution labels, easy “what is at home?” answers, and remote family check-ins.
• Shared flat / hostel mode — roommate ownership, split costs, communal vs personal shelves, expiry responsibility, and “who used the milk?” event trails.
• Domestic helper / staff workflow — voice-first Hindi/Hinglish commands, confirmation cards, shopping handoff lists, and audit-friendly purchase logs.
• Baby / pet / dietary inventory — formula, diapers, pet food, allergy-safe items, repeat cadence, and “never run out” thresholds.
• Emergency prep / monsoon stock — critical household reserves, power-cut foods, water, medicines, flashlights, and weather-aware restock prompts.
• Festival / event planning — Diwali, Eid, weddings, birthdays, guests, bulk shopping, perishability, and post-event leftover/waste analysis.
• Home ops ledger — groceries plus household consumables, repairs, cleaning, subscriptions, appliance filters, warranty docs, and recurring errands.

Exploration questions:

• Which vertical keeps the strongest wedge while preserving “shopping memory” as the core mental model?
• Which mode can be demoed with real household evidence in under 3 minutes?
• What user roles need separate permissions, not just separate labels?
• Which verticals create sensitive claims requiring stronger disclaimers?

14.2 Retail, Kirana, and Community Commerce Angles

ShopStack can horizontally extend from one household into local commerce memory:

• Kirana companion — compare home inventory against nearby kirana price/quality memory, not only quick-commerce snapshots.
• Neighborhood price commons — privacy-preserving aggregate prices for staples, normalized by unit, quality, and freshness.
• Store quality memory — subjective freshness, staff helpfulness, return issues, packaging quality, substitution reliability, and delivery timing.
• Local seller mini-CRM — seller notes, WhatsApp order templates, credit ledger, preferred brands, and recurring household baskets.
• Market route planner — “buy vegetables here, detergent there” based on travel time, weather, price variance, and perishability.
• Community-supported buying — bulk rice/oil/dal buys across apartments, savings split, expiry/storage planning, and consented group orders.
• Retailer API / scraping ethics map — what data is official, user-provided, manually entered, public, or not safe to automate.

Exploration questions:

• What is the trust boundary between personal memory and community data?
• How do we prevent stale community prices from misleading users?
• Can quality/freshness be represented without pretending to be objective?
• Which integrations are product-critical vs demo theater?

14.3 Autoresearch and Evaluation Harness

Borrow the invoice-extraction pattern: fixed benchmark, fixed evaluator, one candidate change, keep/discard logging.

Candidate benchmark tracks:

• Command parser benchmark — Hinglish and English shopping commands → JSON tool calls, with exact intent, item, quantity, unit, location, and confidence scoring.
• Receipt extraction benchmark — receipt photos/text → purchase events with item, quantity, price, store, date, total, tax/fee ambiguity, and review flags.
• Packet label OCR benchmark — expiry/MRP/nutrition extraction with false-positive penalty for confusing MRP, sale price, manufacturing date, and best-before date.
• Market Lens benchmark — shelf/photo/barcode inputs → identify item, compare with inventory, classify buy/skip/compare, and explain uncertainty.
• Semantic alias benchmark — pav/bread, dahi/curd/yogurt, Surf/detergent, dhaniya/coriander, etc. → canonical item IDs.
• Voice benchmark — noisy-market audio → preserved intent and slot accuracy.
• Decision benchmark — inventory + market price + cadence + weather → expected buy/skip/use-soon/compare/watch classification.

Immutable surfaces for each run:

• benchmark manifest,
• ground-truth labels,
• scoring formula,
• dataset split,
• validation schema,
• evaluator code.

Mutable candidate surfaces:

• prompt text,
• model/provider,
• parser/OCR/vision routing,
• validation thresholds,
• retry/fallback policy,
• normalization maps,
• confidence language templates.

Artifacts to store in-project:

• experiments/results.tsv,
• per-run summary.json,
• error_analysis.json,
• candidate prompt/config snapshots,
• keep/discard rationale,
• hard examples promoted to fixtures.

14.4 Data and Schema Expansion

Potential canonical data layers:

• household item taxonomy with Indian/local variants,
• alias and brand synonym map,
• unit normalization and pack-size grammar,
• storage-location ontology,
• shelf-life and freshness rules by category,
• price-quality observation schema,
• receipt line item schema,
• packet label schema,
• nutrition/allergen schema,
• household preference schema,
• user role/permission schema,
• consent and anonymization schema for traces,
• retailer source provenance schema,
• model-run provenance schema,
• uncertainty/explanation schema.

High-leverage datasets to publish or keep internal:

• 500–2,000 human-reviewed grocery commands,
• 100 receipt examples with synthetic/redacted variants,
• 300 packet label close-ups / synthetic crops,
• 1,000 alias-to-canonical mappings,
• 100 household shelf/fridge snapshots with zones,
• 100 market photo examples with review cards,
• 50 noisy voice snippets for benchmark phrases,
• real/synthetic price observations across Swiggy, Blinkit, Zepto, DMart, kirana,
• anonymized trace set showing tool-call planning and corrections.

Exploration questions:

• Which maps belong in SQLite tables vs versioned YAML/JSON?
• Which maps need user-editable UI because every household differs?
• Which data can be safely public for hackathon/storytelling?
• What validation catches stale or contradictory data before it reaches decisions?

14.5 UX, Interface, and Interaction Surfaces

Open product surfaces:

• Today as command center — one screen for buy, skip, use soon, compare, confirm, and “why this recommendation?”
• Review queue — uncertain OCR/vision/voice events become cards to approve, merge, split, edit, or discard.
• Cinematic household timeline — family-dinner-style event playback showing how ShopStack reasoned from inventory, price memory, weather, and preferences.
• Shelf map editor — user-defined locations, shelf zones, item cards, last-seen photos, and confidence badges.
• Market mode — large buttons, short voice responses, offline-safe fallback, high contrast, and no dense tables.
• Kitchen mode — hands-free voice, recipe/use-soon prompts, timer-adjacent flow, and “what can I cook now?”
• Family summary — shareable daily/weekly household digest: what to use, buy, avoid, and what was wasted/saved.
• Operator/debug view — model used, fallback path, confidence, source data age, and trace export without leaking private content.
• Onboarding wizard — household type, preferred stores, staple items, language, dietary constraints, budget, and reminder preferences.

Design exploration topics:

• warm household visual language vs utilitarian inventory tool,
• mobile-first capture vs desktop-first Gradio demo,
• confidence language and iconography,
• accessibility for low vision / elder users,
• no-shame waste messaging,
• deterministic factual cards vs generated visuals,
• offline/connected mode indicators,
• empty states that teach the mental model.

14.6 Agent, Model, and Pipeline Architecture

Potential specialized agents/pipelines:

• Inventory Reconciler — merges receipt, voice, photo, and manual updates into lots without double-counting.
• Market Analyst — compares unit prices, source freshness, travel cost, and quality memory.
• Waste Coach — detects overbuy patterns and suggests practical use-soon actions.
• Meal Planner — proposes meals from current stock, preferences, and expiry.
• Data Steward — flags duplicates, stale prices, bad aliases, and low-confidence observations for review.
• Privacy Guard — redacts traces, receipts, addresses, phone numbers, faces, and household member names before export.
• Benchmark Runner — executes candidate prompts/models against fixed datasets and writes keep/discard results.
• Source Adapter Auditor — validates retailer adapters for freshness, schema drift, and unit-price correctness.

Pipeline questions:

• Where should rules beat models outright?
• Which model outputs must never mutate inventory without confirmation?
• How should fallback paths be visible to the user and trace logs?
• Can one canonical event ledger drive inventory, prices, traces, and explanations?
• Should ShopStack adopt event sourcing for all household changes?

14.7 Privacy, Safety, Trust, and Governance

Trust surfaces to design explicitly:

• local-first mode and clear cloud-mode disclosure,
• household data export/delete,
• encrypted backup snapshots,
• consent per household member,
• child/elder-safe voice interactions,
• face/address/phone redaction in images and receipts,
• private vs shareable trace boundaries,
• nutrition/allergen caution language,
• medicine-like item handling without medical claims,
• price freshness and source disclaimers,
• no auto-purchase without explicit confirmation,
• audit trail for who added/consumed/moved items,
• conflict handling when multiple users edit the same household.

Open questions:

• What is the minimum privacy story credible for real household photos?
• How do we make anonymized hackathon traces useful but safe?
• Which features should be disabled in public demo mode?
• What should be impossible for the agent to do automatically?

14.8 Business, Distribution, and Packaging Exploration

Potential packages:

• free local household inventory app,
• paid family sync / encrypted backup,
• premium market intelligence and retailer comparison,
• caregiver/elder household plan,
• apartment/community price commons,
• kirana/seller companion,
• meal-planning add-on,
• privacy-first offline appliance / NAS-style version,
• API/data layer for other home apps.

Distribution channels:

• Hugging Face Space demo,
• Product Hunt / X build thread,
• WhatsApp-family proof video,
• Indian home-cooking creators,
• Reddit/Discord frugal and meal-prep communities,
• apartment society pilots,
• local kirana pilot,
• caregiver and elder-tech communities,
• student hostel beta.

Positioning experiments:

• “Your home’s shopping memory.”
• “Know what you have before you buy again.”
• “The anti-overbuy grocery assistant.”
• “A local-first inventory copilot for Indian homes.”
• “From receipt to fridge to next shopping decision.”
• “A memory layer for household commerce.”

14.9 Platform and Integration Backlog

Potential integrations:

• WhatsApp list import/export,
• email receipt ingestion,
• browser extension for online grocery carts,
• mobile PWA capture,
• barcode/QR product databases,
• calendar/reminders,
• weather APIs,
• maps/travel-time APIs,
• smartwatch/voice shortcut,
• printer/label maker,
• CSV/JSON/SQLite backup,
• Home Assistant / smart fridge / camera feeds,
• supermarket loyalty emails where user-consented,
• nutrition databases,
• recipe APIs or local recipe notebooks.

Integration rules to define before building:

• canonical source of truth,
• data freshness,
• user consent,
• failure/fallback behavior,
• duplicate detection,
• manual correction path,
• export portability,
• test fixture strategy.

14.10 Repo, Documentation, and Quality Exploration

Process surfaces worth improving:

• keep FEATURE_MAP.md, ROADMAP.md, decision records, and this exploration map synchronized without rewriting history,
• add exploration area IDs so roadmap items can cite their origin,
• create an experiments/README.md describing benchmark promotion rules,
• create a data catalog for all JSON/CSV/YAML datasets,
• track “built / partially built / only explored” status per idea,
• maintain screenshot/visual evidence under Docs/review/assets/, not root,
• add docs for provider fallback behavior and model parameter accounting,
• add a privacy review checklist for trace/dataset publication,
• add a demo-script checklist for hackathon video reproducibility,
• create hard-example fixtures whenever a model or OCR failure is found.

14.11 New Open Questions

21. Should ShopStack become an event-sourced household ledger, or stay CRUD-first with derived event logs?
22. Which single vertical expansion is the strongest: caregiver, shared flat, kirana/community, meal planning, or festival planning?
23. What is the smallest fixed benchmark that prevents model/prompt regressions?
24. Which decisions require hard rule gates instead of model judgment?
25. How should household member permissions work without making onboarding heavy?
26. Can price memory combine subjective quality and objective unit price cleanly?
27. What public demo data feels real without exposing private household details?
28. Should receipt ingestion be the primary acquisition wedge because it creates instant inventory and price memory?
29. What is the right event model for correction: amend, reverse, merge, split, or overwrite?
30. Which integrations are worth building first because they compound the memory layer rather than distract from it?

---

15. HF Pipeline Model Sweep (2026-06-09)

Created a dedicated pipeline/model exploration note at Docs/exploration/HF_PIPELINE_MODEL_EXPLORATION_2026-06-09.md after querying Hugging Face Hub metadata with the local SHOPSTACK_HF_API_KEY from .env. The key was used only for read-only discovery and was not printed or stored.

Coverage added:

• speech-to-text / market voice capture,
• text-to-speech / spoken household responses,
• planner, command parser, slot extraction, and reranking,
• OCR, receipt extraction, packet labels, document layout,
• vision, VQA, object detection, grounding, and visual similarity,
• segmentation, cropping, and background removal,
• text/image embeddings and semantic retrieval,
• image generation/editing for non-factual cards,
• privacy/redaction model surfaces,
• candidate active stacks under the 32B parameter cap,
• benchmark promotion plan for fixed evals.

Important exploration decisions:

• Do not choose models by downloads alone; require product-fit benchmarks.
• Keep factual prices, expiry dates, and inventory counts rendered by code, not generated into image pixels.
• Keep direct inventory mutation behind validated tool calls and confirmation.
• Treat each pipeline stage as separately benchmarked: ASR, parser, OCR, vision, embeddings, segmentation, and decision reasoning should have different evals.
• Distinguish submitted active models, optional fallbacks, and research-only candidates in future model docs.

New model families/lanes opened:

• WhisperKit/CoreML and pyannote VAD/diarization for future mobile/voice capture.
• Indic Parler TTS, VibeVoice, OmniVoice, Chatterbox for voice quality and Indian language support.
• PaddleOCR v5, TrOCR, Donut, LayoutLM, Table Transformer, and PP-DocLayout for receipts and packet labels.
• BLIP/ViLT/Pix2Struct/DETR/YOLOS/RT-DETR/CLIP/SigLIP for visual understanding and shelf/product matching.
• RMBG-2.0, BiRefNet, ClipSeg, SegFormer, Mask2Former for item cards and shelf zones.
• Qwen3-Embedding, multilingual-e5, Jina v3, MiniLM, Nomic, and BGE rerankers for semantic memory and alias search.
• Qwen Image Edit, FLUX.2 variants, Z-Image Turbo, SDXL/SD-turbo for optional visual assets, with deterministic overlays for factual content.

---

16. Travel Agency Agent Pattern Transfer (2026-06-09)

Sampled travel_agency_agent and exploration_maps/travel_agency_agent for transferable exploration patterns. The strongest reusable ideas are not travel features themselves; they are operating-system patterns: lifecycle pipelines, dual-output safety boundaries, destination/context intelligence, supplier scoring, governance, continuity, and research-status discipline.

16.1 Pipeline Lifecycle: Inquiry → Decision → Strategy

Travel pattern:

Inquiry → Intake Normalization → Decision/Gaps → Strategy/Bundle → Safe Output

ShopStack equivalent:

Household signal → Normalize → Gap/Decision → Action Bundle → User-safe Output

Candidate ShopStack lifecycle stages:

• Signal intake — voice command, receipt photo, packet label, fridge photo, shopping list edit, manual inventory update, market price observation.
• Normalization — canonical item, unit, quantity, store/source, location, confidence, provenance, timestamp.
• Gap analysis — missing quantity, ambiguous item, stale price, duplicate lot, expiry uncertainty, source freshness, low-confidence OCR/vision.
• Decision state — buy, skip, use soon, compare, watch, confirm, reconcile, split/merge, discard.
• Action bundle — inventory mutation proposal, price observation, reminder, shopping list update, review card, trace event.
• Safe output — household-facing explanation that excludes internal-only model prompts, raw OCR noise, private metadata, and unsupported claims.

Exploration question: should ShopStack formalize a canonical HouseholdSignal → DecisionBundle contract similar to Travel Agency Agent's trip packet pipeline?

16.2 Dual-Output Safety Boundary

Travel pattern: internal operator bundle can contain richer decision context, while traveler-facing output must be sanitized and leakage-checked.

ShopStack transfer:

• Internal/debug bundle — raw OCR text, model output, confidence vectors, fallback path, source age, prompt/run metadata, private notes.
• Household-safe bundle — concise explanation, action recommendation, source caveat, and confirmation affordance.
• Public/demo-safe bundle — anonymized screenshots/traces with household names, addresses, phone numbers, receipts, faces, and payment identifiers removed.

Potential feature:

• Add a Strict Public Demo Mode that blocks export/share if unsafe terms or unredacted receipt fields are detected.

16.3 Gap-State and Review-Queue Taxonomy

Travel Agent emphasizes blockers, confidence, next-action state, and operator workbench review. ShopStack should mirror this for household automation.

Candidate gap states:

• missing_quantity
• ambiguous_item
• ambiguous_unit
• possible_duplicate_lot
• stale_price_source
• expiry_date_conflict
• ocr_layout_uncertain
• vision_low_confidence
• needs_location_confirmation
• needs_household_member_confirmation
• unsafe_to_auto_apply
• source_adapter_schema_drift

Review queue actions:

• approve,
• edit,
• merge with existing lot,
• split into multiple lots,
• mark as price-only observation,
• save as field note,
• discard,
• promote to benchmark fixture.

This would turn model uncertainty into a productive UI surface instead of hidden backend ambiguity.

16.4 Destination Intelligence → Household Context Intelligence

Travel Agent has destination intelligence: weather, AQI, crime, disease, political stability, crowding, cost of living, local laws, digital infra, accessibility, emergencies.

ShopStack equivalent context layers:

• Weather/seasonality — rain, heat, humidity, monsoon spoilage, festival demand.
• Neighborhood context — local market day, store closures, delivery delays, apartment/community bulk buys.
• Household calendar — guests, travel, school/work schedules, fasting days, festivals, parties.
• Health/diet context — allergies, low-salt/diabetic preferences, baby/elder household needs, without making medical claims.
• Appliance/storage context — fridge space, freezer availability, power cuts, pantry capacity.
• Mobility/accessibility context — who is shopping, carrying capacity, elder user constraints, delivery vs walk/drive tradeoff.
• Budget/pay-cycle context — monthly stocking, weekly top-up, sale timing, price volatility.

Potential score systems:

• buy_now_score,
• skip_risk_score,
• waste_risk_score,
• travel_worth_it_score,
• freshness_risk_score,
• stockout_risk_score,
• confidence_to_auto_apply.

16.5 Supplier Reliability → Store and Source Reliability

Travel Agent explores supplier ecosystems, dark inventory, vendor reliability, commission economics, and disruption risk. ShopStack has an analogous store/source layer.

Store/source reliability dimensions:

• price accuracy,
• delivery punctuality,
• produce freshness,
• substitution quality,
• return/refund friction,
• stockout frequency,
• packaging quality,
• unit-size honesty,
• stale-data risk,
• adapter schema stability,
• household preference fit.

Candidate source types:

• manual household observation,
• receipt-derived observation,
• Swiggy/Blinkit/Zepto/DMart snapshot,
• kirana WhatsApp quote,
• community aggregate,
• loyalty/email receipt,
• public product database,
• synthetic/demo fixture.

Exploration question: should every price/availability recommendation carry a source_reliability and freshness_timestamp, not just a numeric price?

16.6 Autonomy Gradient and Governance

Travel Agent has explicit autonomy/governance research: when AI acts, when it asks, and how settings control autonomy.

ShopStack autonomy levels:

1. Observe only — save scan/receipt/voice as note, no action.
2. Suggest — show recommendation, no mutation.
3. Draft action — prefill add/consume/move/list update for approval.
4. Auto-apply low-risk — apply only rule-safe actions, e.g. mark a manually checked shopping item as purchased.
5. Never automate — purchases, deletes, public trace export, medical/nutrition claims, ambiguous inventory reductions.

Governance settings to explore:

• per-household autonomy level,
• per-source trust level,
• per-member permissions,
• confirmation thresholds by action type,
• “strict mode” for public demos,
• audit log visibility.

16.7 Operational Continuity and Disruption Handling

Travel Agent includes continuity/resilience thinking for supplier outages, geopolitical disruption, and operational recovery. ShopStack should do the same for household data and sources.

Continuity scenarios:

• quick-commerce source unavailable,
• OCR/vision model fails or times out,
• local model missing/download failed,
• stale price snapshot,
• corrupted SQLite/backup restore,
• duplicate household edits from two users,
• offline market mode,
• fridge/pantry photo too poor,
• receipt not readable,
• user disputes an auto-applied action.

Recovery UX:

• visible fallback path,
• “save for later review”,
• manual entry fallback,
• source freshness warning,
• undo/reverse event,
• backup/export prompt,
• hard-example capture for benchmark improvement.

16.8 Real-Validation Status Discipline

The Travel Agent exploration map is useful because it marks the gap between deep theory and real validation. ShopStack should adopt that discipline.

Recommended statuses for exploration/roadmap items:

• idea_only
• researched
• prototype_exists
• wired_in_app
• tested_with_fixtures
• tested_with_real_household_data
• dogfooded_in_market
• public_demo_safe
• production_ready

Add validation metadata:

• evidence path,
• test command,
• data source,
• last verified date,
• known failure modes,
• privacy status,
• benchmark score if applicable.

16.9 Content Prism and Free-Tool Strategy

Travel Agent maps every research topic into content, SEO tools, and multiple audiences. ShopStack can do a household-commerce version.

Three-audience prism:

• Household user — practical: save money, avoid waste, know what is at home.
• Builder/judge — small-model, local-first, multimodal, benchmarked pipeline.
• Retail/community partner — source reliability, price memory, local commerce.

Free-tool ideas:

• grocery unit-price calculator,
• “is this price high?” checker,
• receipt-to-spend summary,
• food waste estimator,
• pantry stockout calculator,
• festival shopping planner,
• rainy-day grocery checklist,
• bulk-buy break-even calculator,
• expiry label parser demo,
• household alias dataset explorer.

16.10 New Open Questions from Travel-Agent Transfer

31. Should ShopStack create a canonical DecisionBundle object that separates internal/debug fields from household-safe output?
32. What is the exact autonomy gradient for add, consume, move, delete, export, and purchase-like actions?
33. Should stale source data automatically downgrade recommendations to “watch” or “confirm” instead of buy/skip?
34. What score formulas are explainable enough for travel_worth_it, waste_risk, and stockout_risk?
35. Which review-queue actions are essential for real household correction flows?
36. Can every failed OCR/vision/parser case become a benchmark fixture with one click?
37. What is the strict public-demo leakage guard for receipts, traces, images, and voice?
38. Which household context layers matter first: weather, calendar, budget, storage capacity, or mobility?
39. Should store/source reliability become a first-class table alongside price observations?
40. How do we track exploration maturity without making docs heavy to maintain?

---

17. ShopStack Deep Exploration Status Map — Travel-Agent-Style Inventory (2026-06-09)

This is the deeper exploration structure that should have been added first: a status-and-gap map like travel_agency_agent/EXPLORATION_MAP.md, but for ShopStack. It is intentionally broad and detailed. It distinguishes what exists, what is only theorized, what is missing, and what deserves priority.

17.0 Status Legend

Symbol	Meaning
✅	Built/researched with concrete docs, code, tests, or verified evidence
🟡	Partially explored or partially built; needs validation or completion
❌	Identified gap; little/no durable research or implementation
🔵	New angle opened by this exploration pass
🧪	Needs benchmark/dogfood evidence before promotion

Priority scale:

• P0 — existential for credible demo/product wedge.
• P1 — critical for launch-quality product.
• P2 — strategic expansion / moat.
• P3 — moonshot or later-stage platform option.

---

A. Household Customer & User Research

Area	Status	Critical Gap	Priority
A1. Real household interviews	❌	No structured interviews with Indian households, parents, students, shared flats, helpers, elder/caregiver users	P0
A2. Day-in-the-life shopping shadowing	❌	No observed shopping trips from list-making through purchase/reconcile/use-soon	P0
A3. Domestic helper workflow research	🔵	Helper-led shopping/inventory is a distinct user role with language/trust constraints	P1
A4. Shared-flat / hostel behavior	🔵	Ownership, split-cost, “who used it,” and low-friction correction patterns unknown	P1
A5. Elder/caregiver household research	🔵	Accessibility, low-vision, medicine-adjacent safety, and remote check-in needs unvalidated	P1
A6. Willingness-to-pay	❌	No pricing validation: free local tool vs family sync vs market intelligence vs caregiver plan	P1
A7. Non-consumption research	🔵	Why do households not use inventory apps? Friction, shame, setup burden, habit failure	P0
A8. Cultural/language phrase research	🟡	Hinglish examples exist, but no corpus of real household commands	P0

B. Core Product Wedge & Feature Validation

Area	Status	Critical Gap	Priority
B1. Wedge product identification	🟡	Many features exist; not validated which one makes users come back daily	P0
B2. Receipt-to-inventory wedge	🟡	Pipeline discussed/built partially; needs real receipt dogfood and review UX	P0
B3. “Do I need this?” Market Lens wedge	🟡	Strong concept; needs real market photo/audio tests	P0
B4. Use-soon / waste prevention wedge	🟡	Logic exists; needs household behavior validation and no-shame UX	P1
B5. Price memory wedge	🟡	Swiggy/source adapters exist; real user perceived value untested	P1
B6. Household map / find item wedge	🟡	Built conceptually; needs spatial UX validation	P1
B7. Meal planning from stock	🔵	High adjacent value but may distract unless tied to use-soon/waste	P2
B8. Festival/event planning	🔵	Strong India-specific seasonal wedge; not researched	P2
B9. Caregiver “never run out” mode	🔵	High trust/value vertical; requires safety and permissions	P2

C. Inventory, Memory, and Data Model

Area	Status	Critical Gap	Priority
C1. Inventory lots and CRUD	✅	Built/tested; needs real-world correction flow stress tests	P0
C2. Event ledger / audit trail	🟡	Traces exist, but canonical event-sourced household ledger is not formalized	P0
C3. Lot merge/split/correction model	❌	Real receipts/photos will create duplicate/partial lots; correction semantics missing	P0
C4. Multi-user household model	❌	User columns exist but no real roles/permissions/conflict model	P1
C5. Household location ontology	🟡	Locations seeded; user-editable shelf/zone model needs research	P1
C6. Item taxonomy and aliases	🟡	Canonicalization exists, but no robust Indian household taxonomy/dataset	P0
C7. Unit and pack grammar	🟡	Unit parsing exists in market data; packet/receipt/general grammar needs hard examples	P1
C8. Shelf-life/freshness rules	🟡	Produce metadata exists; broader item shelf-life database incomplete	P1
C9. Preference memory	🟡	Field notes/preference ideas exist; no canonical preference schema	P2
C10. Backup/restore/encryption	🟡	Portability exists; encrypted household snapshots need exploration	P1

D. Shopping List, Basket, and Decision Logic

Area	Status	Critical Gap	Priority
D1. Shopping list CRUD/complete	✅	Built/tested; needs multi-user and real purchase reconciliation	P0
D2. Buy/skip/use-soon classification	✅/🟡	Built; needs real-world benchmark and explanation calibration	P0
D3. Basket-level recommendations	🟡	Item-level decisions exist; basket budget/travel/source optimization incomplete	P1
D4. Confirmation thresholds	❌	No formal autonomy/confirmation policy by action risk	P0
D5. Review queue for uncertain decisions	❌	Critical missing UI/contract for OCR/vision/model ambiguity	P0
D6. Budget-aware decisions	🔵	No household budget/pay-cycle integration	P2
D7. Substitute recommendations	🔵	Needs item taxonomy, preferences, price, and availability	P2
D8. “Watch” state for stale/uncertain data	🔵	Decision taxonomy should downgrade stale-source recommendations	P1

E. Receipts, OCR, Packet Labels, and Purchase Reconciliation

Area	Status	Critical Gap	Priority
E1. Real receipt OCR	🟡	Tesseract works better than GLM-OCR on real photos; robust pipeline not finalized	P0
E2. Receipt-to-purchase event schema	🟡	Service exists; needs review, merge/split, tax/fees, store/source provenance	P0
E3. Packet expiry/MRP extraction	❌	Core in-store/home scan use case; no benchmark or UX	P0
E4. Indian receipt layout corpus	❌	Need kirana, supermarket, pharmacy, delivery, thermal, mixed-language examples	P0
E5. OCR preprocessing	🔵	Deskew, crop, contrast, binarize, dewarp, table regions not evaluated	P1
E6. MRP vs sale price disambiguation	❌	High-risk; wrong price memory if confused	P1
E7. Manufacturing vs expiry/best-before	❌	High-risk; needs hard rule/model benchmark	P1
E8. Receipt privacy redaction	🟡	Trace redaction exists; receipt image/text redaction needs stricter rules	P1
E9. Email/WhatsApp receipt ingestion	🔵	Strong acquisition path; not researched	P2

F. Voice, Conversational UX, and Multilingual Support

Area	Status	Critical Gap	Priority
F1. Voice add command	✅/🟡	Tests exist; needs real audio and Hinglish corpus	P0
F2. Market-mode voice	❌	No noisy outdoor benchmark or short-response UX validation	P0
F3. Hindi/Hinglish support	🟡	Examples exist; no real corpus, ASR eval, or fallback strategy	P0
F4. Voice correction flow	❌	“No, this is onion not potato” needs explicit state machine	P1
F5. TTS model selection	🟡	HF candidates mapped; no pronunciation/listening benchmark	P1
F6. Wake-word / push-to-talk	🔵	Hands-free kitchen mode vs privacy risk not explored	P2
F7. Multi-speaker household audio	🔵	Speaker diarization/user attribution not explored	P3

G. Vision, Market Lens, Shelf/Fridge Understanding

Area	Status	Critical Gap	Priority
G1. Barcode + image scan surface	✅/🟡	Built/tested partially; real scan quality unknown	P0
G2. Product photo identification	🟡	VLM candidates exist; no fixed household/market image benchmark	P0
G3. Fridge/pantry scan	🔵	Huge value; needs zone model, confidence, and correction UX	P1
G4. Shelf/location snapshot memory	🔵	Same-shelf/same-item embeddings and last-seen memory not built	P2
G5. Freshness/ripeness detection	🔵	Valuable for produce; high false-claim risk	P2
G6. Segmentation/cropping	🟡	RMBG exists; ClipSeg/BiRefNet/RMBG-2.0 not benchmarked	P1
G7. Visual uncertainty UX	❌	Need review cards and “not sure” states; avoid overconfident wrong labels	P0
G8. AR/find-item mode	🔵	Interesting later; depends on spatial memory maturity	P3

H. Market Intelligence, Retail Sources, and Store Reliability

Area	Status	Critical Gap	Priority
H1. Swiggy data source	✅/🟡	Built snapshot loader; freshness and real coverage limits need product copy	P0
H2. Blinkit/Zepto/DMart adapters	🟡	Mentioned/built in docs/tests; need current verification and source reliability map	P1
H3. Kirana/manual quote memory	🔵	Critical India-local angle; no schema/workflow	P1
H4. Store/source reliability scoring	🔵	Price alone is insufficient; need freshness, substitutions, returns, punctuality	P1
H5. Community price commons	🔵	Strong moat; privacy/staleness/trust issues unsolved	P2
H6. Basket-level store choice	🔵	Compare total basket + travel + freshness, not item price only	P1
H7. Price anomaly detection	🟡	Basic price memory exists; no robust anomaly/freshness scoring	P1
H8. Scraping/API ethics	❌	Need official/user-provided/public/unsafe source classification	P1
H9. Delivery vs walk/drive tradeoff	🔵	Travel-worth-it score needs weather/time/carrying capacity	P2

I. Household Context Intelligence

Area	Status	Critical Gap	Priority
I1. Weather/trip context	🟡	Schema/docs exist; needs decision integration and validation	P1
I2. Calendar/event context	🔵	Guests, festivals, fasting, school/work schedules not modeled	P2
I3. Storage capacity context	🔵	Fridge/freezer/pantry capacity affects bulk-buy and freshness decisions	P2
I4. Health/diet/allergy context	🔵	Useful but safety-sensitive; no medical claims	P2
I5. Budget/pay-cycle context	🔵	Monthly stocking and payday cycles likely matter; unresearched	P2
I6. Mobility/accessibility context	🔵	Elder/caregiver and carrying capacity decisions unmodeled	P2
I7. Seasonality/festival demand	🔵	India-specific product moat; no dataset	P2
I8. Power-cut/monsoon resilience	🔵	Household ops vertical; freshness and emergency stock implications	P3

J. AI Pipeline, Model Ops, and Benchmarks

Area	Status	Critical Gap	Priority
J1. Model registry	✅/🟡	Exists; docs drift vs code must be continuously synced	P0
J2. HF pipeline model sweep	✅	Added 2026-06-09 exploration doc; still needs benchmarks	P1
J3. Command parser benchmark	❌	Must exist before changing planner defaults	P0
J4. Receipt OCR benchmark	🟡	Some evidence exists; needs fixed manifest and real labels	P0
J5. Vision benchmark	❌	Need product/photo benchmark for Market Lens/fridge scans	P0
J6. Voice benchmark	❌	Need audio fixtures for ASR/TTS choices	P0
J7. Alias/embedding benchmark	❌	Needed before semantic search promotion	P1
J8. Decision benchmark	❌	Need expected buy/skip/use-soon cases with inventory/source context	P0
J9. Cost/latency model routing	🟡	Local/cloud providers exist; routing by task complexity not fully documented	P1
J10. Fine-tuned parser LoRA	🟡	Candidate exists; dataset/training/eval not done	P1
J11. Model drift/error capture	🔵	Corrections should feed benchmark/fine-tune data	P1

K. Privacy, Safety, Trust, and Governance

Area	Status	Critical Gap	Priority
K1. Trace redaction	✅/🟡	Exists; needs image/receipt/audio extension	P0
K2. Public demo strict mode	🔵	Needed for hackathon/social sharing safety	P0
K3. Household data export/delete	🟡	Portability exists; privacy UX and deletion semantics need work	P1
K4. Encrypted backup	🔵	Important for trust; not built	P1
K5. Nutrition/health claim guardrails	❌	Nutrition fields exist; no safety language/policy	P1
K6. Auto-action governance	❌	Need autonomy levels, confirmation thresholds, never-automate list	P0
K7. Multi-user permissions	❌	Needed before shared household/caregiver modes	P1
K8. Receipt/image PII redaction	❌	Critical for public trace/dataset publication	P0
K9. Source freshness disclaimers	🟡	Some copy exists; should be uniform across all recommendations	P1

L. UX, Design, Accessibility, and Operator Workbench

Area	Status	Critical Gap	Priority
L1. Decision-first Today	✅/🟡	Implemented per docs; needs real usage validation	P0
L2. Review queue/workbench	❌	Missing central surface for uncertain signals	P0
L3. Market-mode mobile UX	❌	Gradio desktop-ish UI may fail in market; needs mobile/touch flow	P0
L4. Kitchen hands-free UX	🔵	Useful; needs voice/privacy tradeoff research	P2
L5. Accessibility / low vision	🟡	CSS/design notes exist; no real audit for elder/low-vision users	P1
L6. Empty-state onboarding	🔵	Setup burden is existential; needs sample data/import wizard	P1
L7. No-shame waste UX	🔵	Waste prevention can feel judgmental; needs tone exploration	P1
L8. Debug/operator view	🟡	Runtime diagnostics exist; per-decision provenance view missing	P1
L9. Shareable cards/reports	🟡	Image gen/card ideas exist; factual deterministic rendering policy needed	P2

M. Integrations and Platform Surface

Area	Status	Critical Gap	Priority
M1. WhatsApp import/export	🔵	High India fit; no implementation/research	P1
M2. Email receipt ingestion	🔵	Strong passive data capture; privacy/auth complexity	P2
M3. Browser extension for grocery carts	🔵	Strong online-shopping wedge; not researched	P2
M4. Mobile/PWA capture path	🟡	Mentioned; needs concrete architecture and test	P1
M5. Calendar/reminder integration	🔵	Use-soon and shopping reminders; not researched	P2
M6. Label printer / pantry labels	🔵	Nice household ops extension	P3
M7. Home Assistant/smart fridge/camera	🔵	Future ambient inventory path	P3
M8. Public product/barcode DBs	🟡	Barcode decode exists; product metadata source strategy missing	P1

N. Business Model, Distribution, and Growth

Area	Status	Critical Gap	Priority
N1. Hackathon positioning	🟡	Product story exists; needs final wedge and proof video	P0
N2. Product naming/positioning	🟡	ShopStack/ShopStock architecture clarified; tagline tests missing	P1
N3. Pricing/packaging	❌	No validation for free/local vs paid sync/market/caregiver plans	P2
N4. Community launch channels	🟡	Channels listed; no execution plan or asset map	P1
N5. Free tools / SEO	🔵	Many ideas; no priority or build plan	P2
N6. Apartment/community pilot	🔵	Strong distribution wedge; needs privacy/community research	P2
N7. Kirana/seller partnership	🔵	Potential B2B2C angle; needs validation	P3
N8. Data/dataset publishing story	🟡	Dataset ideas exist; no public/private split finalized	P1

O. Operations, QA, Docs, and Project Hygiene

Area	Status	Critical Gap	Priority
O1. Feature map / roadmap / decision records	✅/🟡	Exist; some drift visible, needs regular sync	P0
O2. Exploration maturity tracking	🔵	Need statuses/evidence paths like Travel Agent	P1
O3. Demo reproducibility	🟡	Demo script exists; final hackathon walkthrough checklist needed	P1
O4. Data catalog	❌	JSON/CSV/YAML sources need canonical catalog and validation	P1
O5. Benchmark results registry	🟡	Benchmark docs exist; model/task eval registry missing	P1
O6. CI/deployment verification	🟡	Docs mention gaps/drift; current status needs verification	P1
O7. Screenshot/artifact handling	🟡	Need disciplined Docs/review/assets/ policy	P2
O8. Public limitation/claims docs	🟡	Need clear limitations: stale prices, OCR errors, no medical guarantees	P0

P. New Research Areas Not Previously Covered Deeply

1. Household correction semantics 🔵 — amend/reverse/merge/split/overwrite event model for all inventory and receipt mistakes.
2. Strict public-demo safety system 🔵 — block unsafe trace/image/receipt export.
3. Source reliability and freshness scoring 🔵 — every external price/availability needs trust metadata.
4. Review queue as product core 🔵 — turn uncertain model outputs into a daily correction workbench.
5. Household autonomy gradient 🔵 — action risk classification and governance.
6. Real household command corpus 🔵 — Hinglish, English, Hindi, corrections, noisy-market utterances.
7. Indian receipt/packet label corpus 🔵 — kirana, supermarket, pharmacy, quick-commerce, bilingual, thermal, crumpled, low-light.
8. No-shame waste coaching 🔵 — behavior design for reducing waste without guilt.
9. Kirana/manual quote memory 🔵 — bridge offline retail and quick-commerce data.
10. Household context scoring 🔵 — weather, calendar, storage, budget, mobility, seasonality.
11. Community price commons 🔵 — privacy-preserving neighborhood price data.
12. Caregiver/elder household mode 🔵 — accessibility, permissions, remote family trust, never-run-out items.
13. Apartment/shared-flat mode 🔵 — split costs, ownership, accountability, shared vs personal shelves.
14. Passive ingestion moat 🔵 — email receipts, WhatsApp orders, browser carts, loyalty messages.
15. Model-error-to-benchmark loop 🔵 — one-click promotion of failures into evals.

---

17.1 Research Priority Recommendations

Tier 1 — Existential / Must Do for Credible Demo

#	Research Area	Why	Effort
1	Wedge validation: receipt-to-inventory vs Market Lens vs use-soon	Current scope is broad; need one irresistible demo path	2-4 days
2	Real household command corpus + parser benchmark	Voice/text is core; without eval, model choices are vibes	2-3 days
3	Real receipt + packet OCR benchmark	Receipt ingestion is likely the strongest passive wedge	3-5 days
4	Review queue contract and UX	Every imperfect model needs correction; this is core product safety	2-4 days
5	Autonomy/confirmation policy	Prevent unsafe mutation and overclaiming	1-2 days
6	Public demo strict privacy mode	Required before sharing traces/screenshots/receipts	1-2 days
7	Market Lens real-photo dogfood	Validate “Do I need this?” in actual shopping context	2-3 days
8	Limitations/claims doc	Must be honest about stale prices, OCR, nutrition, privacy	1 day

Tier 2 — Critical for Launch Quality

#	Research Area	Why	Effort
9	Item taxonomy + aliases dataset	Powers parser, search, receipts, embeddings, substitutions	3-5 days
10	Source reliability scoring	Price intelligence needs trust/freshness, not just numbers	2-3 days
11	Lot correction semantics	Real data will be messy; correction must be first-class	2-3 days
12	Mobile/market-mode UX	Desktop Gradio is not enough for shopping context	2-4 days
13	TTS/STT benchmark	Voice-first claim requires real audio evidence	2-3 days
14	Data catalog and validation	Data/config is product architecture	1-2 days
15	Encrypted backup/export/delete story	Trust and portability for private household data	2-4 days

Tier 3 — Strategic / Moat

#	Research Area	Why	Effort
16	Kirana/manual quote memory	India-local differentiation	1 week
17	Community price commons	Network/data moat, but trust-heavy	1-2 weeks
18	Caregiver/elder mode	High-value vertical with trust needs	1 week
19	Festival/event planning	Strong seasonal Indian household behavior	3-5 days
20	Browser/WhatsApp/email passive ingestion	Creates compounding memory with less manual entry	1-2 weeks
21	Household context scoring	Differentiates from plain inventory apps	1 week
22	Programmatic free tools/content	Acquisition engine after product wedge is clear	ongoing

---

17.2 Exploration Statistics Snapshot

Approximate status across the table above:

Metric	Approx count
Deep categories	15
Specific rows/status items	120+
Clearly built or strongly documented	15-20
Partial / needs validation	35-45
True gaps	35-45
New angles opened	40+
P0 existential items	20+

Takeaway: ShopStack has a lot built, but the main gap is real-world validation and correction infrastructure. The permanent product path is not “add more models”; it is household signal → normalized memory → safe decision bundle → review/confirmation → benchmark feedback loop.

---

18. Exhaustive Scenario, Persona, Edge-Case, and Benchmark Expansion (2026-06-09)

This section fills the remaining obvious exploration gaps: concrete scenarios, personas, lifecycle maps, failure catalogs, model-by-task benchmarks, data moat, commercial expansion, free tools, operations, and safety/regulatory surfaces.

18.1 Persona Matrix

Persona	Core job	Pain	ShopStack wedge	Special constraints
Primary household shopper	Buy what is needed, avoid duplicates	Mental inventory burden	“Do we need this?” + list + receipt reconcile	Speed, trust, low setup
Secondary family member	Check/add items occasionally	Doesn’t know where things are	Ask/search/find item	Simple UI, limited permissions
Domestic helper	Execute shopping/restocking tasks	Language, confirmation, accountability	Voice list, photo receipt, review queue	Hindi/Hinglish, no complex forms
Elder user	Know stock and avoid running out	Low vision, memory, mobility	Voice-first never-run-out mode	Accessibility, caregiver permissions
Caregiver/remote family	Monitor essentials for someone else	Can’t see what’s at home	Shared stock dashboard + alerts	Privacy/consent, low false alarms
Student/hostel user	Manage cheap staples	Budget, shared fridge chaos	Budget basket + shared shelf	Low cost, mobile-first
Shared-flat roommate	Split shared vs personal items	Disputes and freeloading	Ownership, consume log, split cost	Social friction, audit trail
Parent/home cook	Plan meals from stock	Waste and repeated shopping	Use-soon meals + festival/event planning	No-shame tone, dietary preferences
Budget optimizer	Spend less over time	Unit-price confusion	Price memory + high-price alerts	Accurate source freshness
Health-conscious shopper	Track nutrition/allergens	Label overload	Packet label parser + caution flags	No medical overclaims
Quick-commerce power user	Compare apps quickly	App switching, dynamic prices	Basket comparison + source memory	Stale data warnings
Kirana-first shopper	Buy locally/offline	No digital data	Manual quote/receipt memory	Offline-first, WhatsApp-friendly
Apartment/community admin	Coordinate bulk buys	Demand aggregation, fairness	Community basket + split logistics	Consent, settlement, governance
Kirana owner	Serve repeat households	Remember preferences/orders	Seller-side household memory	Separate product, B2B trust
Hackathon judge/builder	Understand small-model value	Demo clarity	Transparent model/pipeline evidence	Field notes, benchmark artifacts

18.2 Household Scenario Library

Before Shopping

• User asks “what should I buy today?” and expects low-stock, use-soon, and price-aware suggestions.
• User creates list from voice while cooking.
• User checks whether staples are enough for the week.
• User plans shopping around rain/heat/market day.
• User needs a budget basket under ₹500.
• User checks if Sunday market is worth the trip.
• User wants to avoid buying duplicate detergent/rice/oil.
• User plans groceries before guests arrive.
• User plans festival shopping with bulk sweets/oil/flour/dry fruits.
• User needs a “quick top-up” list after returning from travel.

During Shopping

• User scans tomatoes and asks if price is high.
• User scans a packet and asks whether it is already at home.
• User asks if the visible expiry date is too soon.
• User asks which pack size is cheaper by unit.
• User sees substitute brand and asks if household usually accepts it.
• User records a kirana quote by voice.
• User asks whether to walk to another store for cheaper produce.
• User corrects model: “no, this is coriander not spinach.”
• User cannot read small MRP/expiry text and needs OCR.
• User is in noisy market and wants one-sentence answer.

After Shopping

• User photographs receipt and wants inventory updated.
• User photographs all groceries on the counter.
• User marks purchased list items and adds prices.
• User splits receipt into inventory vs household expense only.
• User records store freshness rating.
• User reconciles duplicate lots after OCR error.
• User adds expiry dates from packet closeups.
• User moves items to fridge/pantry/bathroom cabinet.
• User shares weekly spend/waste summary.
• User exports redacted trace for demo/field notes.

At Home

• User asks where the dahi is.
• User asks what can be cooked without shopping.
• User asks what expires first.
• User consumes half a packet and updates quantity.
• User finds old item hidden behind shelf.
• User scans fridge before leaving for store.
• User wants lunchbox ideas from use-soon items.
• User asks if baby formula/pet food is enough.
• User gets reminder before milk/bread stockout.
• User wants to reduce waste without guilt.

Shared Household / Caregiver

• Roommate consumes shared milk; others need visibility.
• One user buys item another already bought.
• Helper buys groceries and sends receipt photo.
• Caregiver checks elder household essentials remotely.
• Parent asks kid to check fridge with photo.
• Apartment group coordinates bulk rice/oil order.
• Household member disputes an auto-applied consume event.
• User wants private personal shelf hidden from others.
• Remote family wants alerts only for critical items.
• Helper needs voice-only shopping confirmation.

18.3 Lifecycle Maps

Item Lifecycle

Need detected → list candidate → bought → receipt/price observed → stored → moved
→ opened → partially consumed → use-soon → consumed / wasted / donated → learned

State questions:

• What event created the item?
• Which source is trusted for quantity/price/expiry?
• Is this a new lot or duplicate?
• Who confirmed it?
• What correction history exists?
• What did the system learn for next time?

Shopping Decision Lifecycle

Signal → normalize → compare against inventory → enrich with price/source/context
→ classify → explain → confirm → apply → trace → benchmark failure if corrected

Receipt Lifecycle

Image/email/WhatsApp → preprocess → OCR → line grouping → structured extraction
→ source/price provenance → duplicate detection → review queue → inventory/price events

Model Failure Lifecycle

Bad output → user correction → capture raw input + expected output → redact → fixture
→ benchmark run → candidate prompt/model/config → keep/discard log

18.4 Edge-Case and Failure Catalog

Category	Edge cases to explore
OCR/receipts	crumpled receipt, thermal fading, mixed Hindi/English, right-aligned prices, discounts, taxes, delivery fees, refund lines, unreadable totals, duplicate receipt photo
Packet labels	MRP vs sale price, mfg vs expiry vs best-before, date format ambiguity, batch number mistaken as date, tiny font, curved packet, reflective packaging
Voice	negation missed, noisy market, mixed language, brand homophones, quantity omitted, correction utterance, multi-speaker command, child voice
Inventory	duplicate lot, partial consume, item moved without logging, stale quantity, opened vs unopened, household member disagreement, unit conversion error
Market price	stale app snapshot, unavailable item, combo pack, delivery fee hidden, quality lower despite cheaper price, store substitution, region mismatch
Vision	similar produce, occluded shelf, transparent containers, unlabeled jars, multiple items in one frame, overconfident wrong VLM label
Shared household	two users edit same item, private vs shared item, helper permissions, remote caregiver override, split-cost dispute
Privacy	receipt phone/address, UPI/card token, face in fridge photo, child voice/audio, household names in trace, location metadata in image
Safety	nutrition overclaim, medicine-like item advice, allergy false negative, spoiled food missed, auto-delete or auto-purchase risk
Offline/runtime	model missing, HF download fails, OCR timeout, DB locked, backup restore conflict, source adapter schema drift

18.5 Feature Universe Backlog

Cluster	Feature ideas
Capture	voice command, receipt photo, packet scan, fridge photo, shelf photo, barcode, email receipt, WhatsApp receipt/list, browser cart import
Normalize	item aliases, brand mapping, unit grammar, pack-size parser, expiry parser, store/source canonicalization, household location ontology
Decide	buy/skip/use-soon/watch/compare/confirm, budget basket, stockout prediction, waste risk, travel-worth-it, source reliability, substitution
Reconcile	review queue, merge/split lots, undo/reverse event, duplicate receipt detection, price-only observation, correction-to-fixture
Remember	price history, store quality, household preferences, recurring cadence, shelf locations, last-seen photo, field notes, model traces
Act	shopping list, reminders, label print, calendar reminder, family summary, helper handoff, caregiver alert, community bulk-buy request
Explain	why buy/skip, source freshness, confidence, fallback path, data provenance, no-medical-claim caveat, stale price warning
Share	redacted trace export, weekly digest, spend/waste report, public demo dataset, field notes, shareable card
Admin	model status, source adapter health, data catalog, backup/export/delete, autonomy settings, permissions, privacy guard

18.6 Model-by-Task Benchmark Matrix

Task	Candidate models/tools	Metric	Must-have fixture set
ASR command transcription	SenseVoice, Qwen3-ASR, Parakeet, Whisper, WhisperKit	WER + slot retention + latency	100 Hinglish/English/noisy commands
Intent classification	Qwen small, Gemma small, DistilBERT, rules	intent F1, confusion matrix	add/move/consume/find/price/check/correct
Slot extraction	parser LoRA, Qwen, token classifier, rules	exact item/qty/unit/location accuracy	household command corpus
Tool-call generation	Qwen3.5/4B, MiniCPM5, GGUF Llama/LFM	valid JSON, executable calls, no unsafe action	200 command/action cases
Receipt OCR	Tesseract, PaddleOCR, GLM-OCR, TrOCR, Donut	field accuracy, row grouping, latency	100 Indian receipts
Packet OCR	OCR + VLM	expiry/MRP/date disambiguation	100 packet labels
VLM item ID	MiniCPM-V, Qwen2.5-VL, BLIP/ViLT baselines	top-1/top-3, abstain quality	100 market/fridge photos
Object detection	DETR, YOLOS, RT-DETR, table/document detectors	box recall/precision	shelf/receipt/table crops
Segmentation	RMBG, BiRefNet, ClipSeg, SegFormer	mask IoU, visual usefulness	item card/shelf zone images
Embeddings	BGE-M3, Qwen3 Embedding, E5, Jina, MiniLM	top-k alias retrieval	1k alias/canonical pairs
Reranking	BGE reranker	top-1 after retrieve	hard alias negatives
Decision classifier	rules + model explanations	expected class accuracy + explanation faithfulness	inventory+price+context scenarios

• Current STT note: Qwen3-ASR-1.7B is now benchmarked on the 20-command Hinglish set with transliteration-aware scoring: 34.46% WER, 55.6% slot retention, 1.11s latency. It outputs Devanagari script, so the quality score only makes sense after Latin transliteration. | TTS | Kokoro, Qwen TTS, Indic Parler, CosyVoice, VibeVoice | pronunciation, MOS, latency | grocery terms + short replies | | Privacy redaction | regex, NER, OCR boxes, face detector | leak recall, false positives | receipts/images/traces/audio transcripts |

18.7 Data Moat and Dataset Strategy

Dataset	Public?	Why valuable	Risks
Household command corpus	Public synthetic + private real	Parser/fine-tune moat	Household language privacy
Indian item alias graph	Public	Core search/parser value	Regional bias/staleness
Receipt OCR corpus	Mostly private/redacted synthetic	Receipt-to-inventory benchmark	Phone/address/payment leakage
Packet label corpus	Public/private mix	Expiry/MRP extraction	Brand/copyright/photos
Price observation history	Private/aggregate public	Store/source intelligence	Stale/misleading prices
Store reliability observations	Private/aggregate public	Quality moat beyond price	Defamation/fairness concerns
Fridge/shelf images	Private only by default	Spatial memory/VLM eval	Faces/home privacy/location metadata
Correction traces	Redacted public subset	Model improvement and Field Notes	Raw private content leakage
Use-soon/waste outcomes	Private/aggregate	Waste coaching efficacy	Shame/sensitive household behavior
Community basket data	Consent-only aggregate	Group buying/network effects	Consent/governance/trust

18.8 Commercial and Expansion Map

Segment	Product	Why it could work	Research needed
Individual households	Free/local inventory + shopping memory	Broad wedge, privacy-friendly	Setup friction, retention
Families	Sync, shared permissions, reminders	Multi-user value	Permissions, pricing
Caregivers	Never-run-out essentials + remote alerts	High pain/trust	Safety, elder UX
Shared flats/hostels	Shared pantry + split costs	Clear social pain	Dispute avoidance
Apartment societies	Bulk-buy coordination	Network effects	Governance/payment logistics
Kirana stores	Repeat household memory / WhatsApp ordering	Offline commerce bridge	Seller workflows
Quick-commerce users	Basket comparison and price memory	Existing digital behavior	Source data access
Health/wellness users	Label/nutrition caution	High engagement	Claim boundaries
Hackathon/builders	Small-model multimodal case study	Demo/credibility	Field notes/benchmarks
Data/API product	Aggregated price/alias/source intelligence	Moat if consented	Privacy/legal/commercial demand

18.9 Programmatic Content and Free Tools

Tool/content	Audience	Data needed	Product tie-in
Unit-price calculator	Budget shoppers	pack sizes, units	Price memory
“Is this price high?” checker	Market shoppers	price history/snapshots	Market Lens
Receipt-to-spend summary	Families	OCR receipt	Receipt ingestion
Expiry label parser demo	Anyone scanning packets	OCR labels	Packet scan
Food waste estimator	Home cooks	inventory/use-soon	Waste coach
Festival shopping planner	Indian households	festival lists	Event planning
Rainy-day grocery checklist	Weather-aware shoppers	weather + staples	Context intelligence
Bulk-buy break-even calculator	Apartments/families	unit prices/storage	Community buying
Pantry audit checklist	New users	item taxonomy	Onboarding
Grocery alias explorer	Builders/judges	alias graph	Dataset story
“What can I cook?” mini-tool	Home cooks	inventory + recipes	Meal planning
Caregiver essentials checklist	Caregivers	essential categories	Elder mode

18.10 Regulatory, Safety, and Claims Expansion

Surface	Rule to research/document
Nutrition	Provide informational label extraction, not diet/medical advice.
Allergens	Warn only from explicit label/user data; never guarantee safety.
Medicines/pharmacy	Track stock/expiry only; no dosage/medical guidance.
Prices	Always show source and freshness; avoid “best price” guarantee if stale.
Community data	Consent, aggregation thresholds, removal rights, anti-abuse.
Receipts	Redact phone/address/payment IDs before sharing/export.
Household images	Strip EXIF/location; detect faces/private content before public use.
Child/elder data	Extra consent and conservative sharing defaults.
Auto-actions	No auto-purchase; no destructive delete without confirmation.
Public demo	Synthetic/redacted data by default; strict leakage guard.

18.11 Operations and Quality System

Operational system	Purpose
Data catalog	Every JSON/CSV/YAML/source snapshot has owner, schema, freshness, tests.
Benchmark registry	Every model/pipeline candidate has fixed eval, score, latency, keep/discard.
Source adapter monitor	Detect schema drift, stale snapshots, unit parsing failures.
Redaction review	Gate traces/images/receipts before public export.
Demo mode	Deterministic seeded data and walkthrough reset.
Backup/restore drill	Prove household memory can be exported/restored.
Failure fixture promotion	Corrections become regression tests.
Claims checklist	Validate UI/docs copy against actual capabilities.
Exploration maturity tracker	Each idea has status, evidence path, priority, and last verified date.
Dogfood log	Real household/market sessions with outcomes and failures.

18.12 More Open Questions

41. What exact 3-minute demo proves value fastest: receipt, market scan, or use-soon?
42. Which persona should the product optimize for first: primary shopper, helper, caregiver, or shared-flat user?
43. What is the minimum review queue that makes imperfect models useful rather than annoying?
44. Which household corrections should alter future model behavior automatically?
45. How much setup can users tolerate before seeing value?
46. What data should never leave the device, even with consent?
47. Can stale price/source uncertainty be explained in one short line?
48. Is the moat item aliases, receipt corpus, price memory, source reliability, or correction traces?
49. Should ShopStack sell sync/backup, intelligence, or community buying first?
50. What feature would a household still use after the hackathon novelty fades?

Last updated: 2026-06-09