--- title: Lost & Found Desk emoji: 🧾 colorFrom: indigo colorTo: yellow sdk: gradio sdk_version: 6.17.3 app_file: app.py python_version: 3.12.12 pinned: false tags: - track:backyard - sponsor:openbmb - sponsor:openai - sponsor:nvidia - achievement:offgrid - achievement:offbrand - achievement:sharing - achievement:fieldnotes --- # Lost & Found Desk - **Live Space:** https://huggingface.co/spaces/build-small-hackathon/lost-found-desk - **App URL:** https://build-small-hackathon-lost-found-desk.hf.space - **Demo video:** https://youtu.be/AsOM7K0tL-s - **GitHub:** https://github.com/JacobLinCool/Lost-n-Found-Desk - **Social post:** https://x.com/JacobLinCool/status/2066147773481951378 - **Team HF usernames:** [@JacobLinCool](https://huggingface.co/JacobLinCool) - **Article:** https://huggingface.co/spaces/build-small-hackathon/lost-found-desk/blob/main/docs/article.md - **Trace dataset:** https://huggingface.co/datasets/build-small-hackathon/lost-found-desk-codex-traces (official HF Agent Traces / Codex JSONL format) - **HF collection:** https://huggingface.co/collections/build-small-hackathon/lost-and-found-desk-6a2ec0551c48861e92dd8443 Lost & Found Desk is a **caption-first return desk** for conferences, event venues, gyms, coworking spaces, schools, and any front desk that accumulates forgotten bottles, badges, chargers, jackets, bags, keys, and accessories. It is **multi-event**: anyone can create an independent help desk for their own event and gets a short event code plus an auto-generated staff password. Each event has three **fully separated** surfaces β€” a public home page, a claimant-facing report flow, and a password-gated staff console β€” so claimants and staff never share a screen. Staff photograph **one item per photo**. MiniCPM-V writes a searchable caption. Claimants describe their lost item through a private, **language-aware** Claim Assistant (it replies in the claimant's own language β€” Traditional Chinese, English, …) and may optionally **upload a photo of their own lost item**, which MiniCPM-V captions to strengthen matching. MiniCPM5 compares the refined claim against the live inventory for staff review. Staff can post messages back into the claimant's thread, and the final handoff is always confirmed by staff offline. The app deliberately does **not**: - segment table photos; - run a separate OCR pipeline; - build an embedding index in the MVP; - expose item photos to claimants; - show ranked candidates to claimants; - let AI decide ownership. ## Default runtime The default is now the real model path: ```bash uv run python app.py ``` Equivalent explicit settings: ```bash LFD_MODEL_MODE=real LFD_DEVICE=auto uv run python app.py ``` `LFD_DEVICE=auto` resolves ZeroGPU/CUDA/MPS/CPU in one place. Mock mode is available only as an explicit path for tests and CPU-only UI review: ```bash LFD_MODEL_MODE=mock uv run python app.py ``` Real-model failures surface by default (`LFD_ALLOW_MOCK_FALLBACK=0`). Set `LFD_ALLOW_MOCK_FALLBACK=1` only when you deliberately want the rule-based mock path for local resilience testing. ## Multi-event model & roles The store is keyed by event; each event owns its own items, claims, and returns, and its staff password is stored only as a salted PBKDF2 hash. The three surfaces are routed client-side: ```text / Home β€” create an event, or enter an existing one /e/{event_id} Public claim β€” start or resume a lost-item report /e/{event_id}/c/{id} Public claim β€” a specific report (the resume link) /e/{event_id}/staff Staff console β€” password-gated (per event) ``` Creating an event returns its staff password **once**; it is never stored in plaintext. Claimants reach `/e/{event_id}` via a shared link/QR and get a unique resume link (`/e/{event_id}/c/{claim_id}`) to come back and check staff replies. ## Demo event On first boot a reproducible demo event is seeded so reviewers have data: ```text event code: demo (LFD_SEED_EVENT_ID) staff password: demo-pass (LFD_SEED_STAFF_PASSWORD) staff console: /e/demo/staff ``` Set `LFD_SEED_SAMPLE=0` to skip seeding, or change the seed values via env. ## Core workflow ```text one item photo -> MiniCPM-V caption -> item inventory -> Claim Assistant conversation -> MiniCPM5 prompt-packed inventory search -> staff-only match review -> safe draft message -> offline handoff log ``` ## Surfaces **Home** (`/`) β€” create an event (name + optional custom code), or enter an existing event as staff (code + password) or as a claimant (code). **Public claim** (`/e/{event_id}`) β€” claimant-only. A conversational, language- aware intake; optional photo upload of the claimant's own item; a running summary; a contact-info submit; a prominent "save this link" resume box; and any staff messages shown inline. No inventory photos, candidates, or ownership decisions are ever exposed here. **Staff console** (`/e/{event_id}/staff`) β€” password-gated tabs: 1. **Overview** β€” counts, recent items, open claims, the shareable public link. 2. **Add items** β€” one item, one photo, found location, optional note, generated caption. 3. **Claims (Inbox + Match)** β€” review claims (including claimant-uploaded photos), run candidate matching, draft + send a message to the claimant, mark returned after offline confirmation. 4. **Report** β€” returned items, safety counters, JSON export. ### Claim conversation (model-driven, inventory-guided) In real mode the English collecting-phase replies are **generated by MiniCPM5** β€” the assistant acknowledges what the claimant said and asks the next useful question in its own words. The model never sees the inventory; it only receives the dialogue plus a per-turn hint from the deterministic, **inventory-aware planner** (`conversation.plan_turn`). The planner runs on every turn regardless of mode and owns the bookkeeping: it looks at which unclaimed items the current description could match and works out whatever attribute best tells those candidates apart and that the claimant hasn't given yet β€” so the model's questions are **guided by the live inventory, never a fixed script**, while leaking nothing (a hint like "still missing: color" is generic). If the inventory has a black bottle and a blue bottle and the claimant only said "a bottle", color is the next question; once the description pins one item down (or no further detail would help), the report is declared ready. The matching summary always stays the claimant's own words (never model prose), and readiness/missing-info chips stay deterministic so the submit gate can't flap with model mood. Guardrails around the 1B model (each violation is surfaced; when the explicit fallback flag is enabled it is recorded as `outcome=fallback` in telemetry): - replies claiming inventory state ("the desk has nothing") are rejected; - verbatim repeats of an earlier reply are rejected; - inventory probes ("what do you have?") never reach the model β€” they get a fixed safe refusal plus the next question; - the ready turn's call to action stays product copy; - Traditional-Chinese conversations use the planner's native-quality phrasing end-to-end (MiniCPM5-1B's zh chat output proved unshippable: broken sentences and Simplified drift). In mock mode (CPU-only review, tests) the planner's own bilingual phrasing is the reply everywhere, so the workflow never stalls. It understands common English and Traditional-Chinese attribute words and tracks which attributes it has already asked about, so it never repeats in any language. Matching runs **continuously during the conversation** β€” candidates are stored on the claim so staff see them automatically, with no required button press (a "Re-run match" button re-runs a deeper match, e.g. after staff add new inventory). In real mode the live match uses the multilingual embedding space, so a Chinese description can retrieve an English-captioned item; mock/CPU uses the token matcher. Candidates are **stripped from the claimant-facing payload** β€” claimants never see candidates, scores, or item details. The intake chat only updates the *description-readiness* signal; it never sets the claim's lifecycle status. **Submitting** (which requires contact info) is the action that marks a claim ready for staff β€” so staff never see a "ready" claim they cannot contact, and a later chat turn can never demote a claim staff already matched. ### Scaling matching with embeddings (retrieve-then-rerank) Packing every caption into the matching prompt does not scale. When an event's unclaimed inventory exceeds `LFD_MATCH_TOPK` (default 10), matching first narrows to the top-N by embedding similarity to the claimant's description, then runs the LLM/ranker on that shortlist only: - At intake (caption time) each item is embedded with NVIDIA's multilingual vision-language model `nvidia/llama-nemotron-embed-vl-1b-v2` (photo + caption, 2048-d) and the vector is stored per-event (never shipped to the frontend). - At match time the claimant's description is embedded as a text query in the same space; the top-N items are kept. Because the space is multilingual and cross-modal, a Chinese description can retrieve an English-captioned item. - Small events (≀ `LFD_MATCH_TOPK`) are unaffected and pay no embedding cost. - Explicit mock mode uses a deterministic embedder for CPU/offline tests, and missing real embeddings self-heal on demand. ```bash LFD_MATCH_TOPK=10 # shortlist size / threshold for triggering retrieval LFD_RETRIEVAL=0 # disable embedding shortlisting entirely LFD_EMBED_MODEL=nvidia/llama-nemotron-embed-vl-1b-v2 ``` ### Claimant resume link is a secret A `claim_id` is a high-entropy (~144-bit) capability token: knowing the event code + claim id grants access to that claim's conversation, contact info, and photos. Treat the resume link (`/e/{event_id}/c/{claim_id}`) as a secret; do not post it publicly. A production deployment should add per-IP rate limiting on the public claim endpoints. ## Official runtime pass This repo was updated to match the current official docs as of 2026-06-14: | Layer | Choice in this repo | Why | | --- | --- | --- | | Gradio | `gradio==6.17.3` | Space metadata pins the same `sdk_version`. | | Gradio Server mode | `gradio.Server` + `@app.api()` | Gives FastAPI routes plus Gradio API queue/concurrency/ZeroGPU behavior. | | ZeroGPU | `spaces>=0.50.4`, `@spaces.GPU` via `maybe_zerogpu()` | GPU-dependent calls are registered with the official decorator; `spaces` activates the wrapper when HF sets `SPACES_ZERO_GPU=true`, and the real MiniCPM adapters are eagerly loaded at app startup to match the ZeroGPU model-loading recommendation. | | MiniCPM-V | `openbmb/MiniCPM-V-4.6` through `AutoProcessor` + `AutoModelForImageTextToText` | Matches the model card's Transformers path. | | MiniCPM5 | `openbmb/MiniCPM5-1B` through `AutoTokenizer` + `AutoModelForCausalLM` | Matches the model card's standard `LlamaForCausalLM` path. | | Frontend | Svelte 5 + Vite 8 + `@gradio/client` | The compiled Svelte app is committed into `static/` so the Space needs no Node build step. | | Pydantic | `>=2.11.10,<2.12.6` | Matches the current HF Spaces Gradio builder, which installs `gradio[mcp,oauth]==6.17.3` and caps Pydantic at `<=2.12.5`. | | PyTorch / torchvision | `torch>=2.8,<2.12`, `torchvision>=0.25,<0.27` | Matches the current HF Spaces builder cap of `torch<=2.11.0`; `torchvision 0.27.0` requires `torch==2.12.0`. | The built frontend uses normal REST routes for CRUD. Staff item photos are fetched as authenticated blobs from staff-only routes, so the public Claim Assistant never receives inventory photo URLs. When the Space reports ZeroGPU, the frontend switches model-heavy actions to Gradio API endpoints through `@gradio/client`, which is the official browser path for Server mode + ZeroGPU quota handling. ## Hackathon submission fit Lost & Found Desk targets **Backyard AI**: it is a practical, local-first workflow for a real front-desk problem. It uses only models below the 32B cap: | Model | Role | Parameter fit | | --- | --- | --- | | `openbmb/MiniCPM-V-4.6` | Staff and claimant photo captioning | Listed by the hackathon kit as about 1.3B. | | `openbmb/MiniCPM5-1B` | Claim intake, matching, safe staff messaging | 1B. | | `nvidia/llama-nemotron-embed-vl-1b-v2` | Multilingual vision-language retrieval shortlist | 1B-class embed model. | Requested official tags: - `track:backyard` - `sponsor:openbmb` β€” MiniCPM-V and MiniCPM5 are core to the experience. - `sponsor:openai` β€” Codex was used for final hardening, documentation, deployment prep, and Codex-attributed commits. - `sponsor:nvidia` β€” Nemotron Embed is used for multilingual cross-modal retrieval. - `achievement:offgrid` β€” the app does not depend on hosted model APIs. - `achievement:offbrand` β€” the UI is a compiled Svelte product surface served through `gradio.Server`, not stock Gradio components. - `achievement:sharing` β€” the public Codex trace dataset is on the Hub in the official HF Agent Traces / Codex JSONL format at https://huggingface.co/datasets/build-small-hackathon/lost-found-desk-codex-traces. - `achievement:fieldnotes` β€” the write-up and submission notes are in [`docs/article.md`](docs/article.md) and [`docs/submission-notes.md`](docs/submission-notes.md). The project also fits the prize-table spirit of **Tiny Titan** because every model is at or below roughly 4B parameters. The official README tag generator does not expose a `tiny` tag, so this is documented here instead of inventing a non-official tag. ## Install locally ```bash uv sync uv run python app.py ``` Open: ```text http://localhost:7860 ``` Mac MPS: ```bash PYTORCH_ENABLE_MPS_FALLBACK=1 LFD_DEVICE=mps uv run python app.py ``` Local CUDA: ```bash LFD_DEVICE=cuda uv run python app.py ``` Hugging Face ZeroGPU: 1. Create a Gradio SDK Space. 2. Select ZeroGPU hardware in Space settings. 3. Keep the repo default `LFD_MODEL_MODE=real` and `LFD_DEVICE=auto`. 4. Keep `LFD_ALLOW_MOCK_FALLBACK=0` for judge-facing validation. ## Frontend development The production Space serves the compiled Svelte frontend in `static/`. The frontend is written in **TypeScript** (Svelte 5 `