fix (llm): improve prompts for choosing outfits

README.md

@@ -13,62 +13,98 @@ hardware: cpu-basic
 short_description: AI wardrobe. catalog, combine and ask about your clothes
 ---
 
-# 👕 Wardrobe Us
+# 👕 Wardrobe AI
 
-**An AI-powered wardrobe assistant that helps you understand, organize, and make better use of the clothes you already own.**
+**Turn a physical wardrobe into a searchable, AI-powered catalog — and get outfit ideas from clothes you already own.**
 
 Built for the [Gradio × Hugging Face Build Small Hackathon](https://huggingface.co/build-small-hackathon) (June 2026).
 
+The original motivation: help someone with 200+ garments who forgets what they own, buys duplicates, and struggles to combine outfits every morning. Wardrobe AI is not a shopping app — it helps you *use* what you already have.
+
 ---
 
 ## What it does
 
-1. **Capture** — Upload photos of your clothes. AI detects individual garments, crops them, and extracts structured attributes (type, color, material, pattern, season, formality).
-2. **Catalog** — Browse your digital wardrobe with all extracted metadata. Search and filter by any attribute.
-3. **Combine** — Generate outfit combinations ranked by style rules. Optionally describe an occasion ("dinner on a terrace, summer") and the LLM re-ranks combinations for that context.
-4. **Ask** — Chat with your wardrobe. "What should I wear for a job interview?" gets answered based on what you actually own.
+| Step | Description |
+|------|-------------|
+| **Capture** | Upload photos of your clothes. A detector finds garments, you can adjust bounding boxes, and a VLM extracts structured attributes (type, color, material, pattern, season, formality, description). |
+| **Catalog** | Browse your digital wardrobe with images and metadata. Click any garment for a detail panel with full attributes. |
+| **Combine** | Generate top+bottom outfit combinations filtered by season and formality rules. Describe an occasion and the LLM re-ranks the best matches. Like/dislike outfits to build style preferences. |
+| **Ask** | Chat with your wardrobe in natural language. Answers reference your actual garments with images and descriptions. |
+
+All inference runs locally — no external APIs.
 
 ---
 
-## Tech Stack
+## Two frontends
+
+The app ships with two UIs sharing the same backend:
+
+| | **Custom UI** (default) | **Gradio Blocks** (`--default`) |
+|---|---|---|
+| Launch | `python app.py` | `python app.py --default` |
+| Stack | `gradio.Server` + Alpine.js + `@gradio/client` | Gradio 6.17 Blocks |
+| Language | English | Spanish |
+| Best for | End users — clean, minimal UX | Power users — full settings |
+| Manual crop editor | Annotorious v3 bounding-box editor | `gradio-image-annotation` |
+| Detection backend switch | — | Dropdown in settings |
+| Dataset load logs | Real-time log dock (streaming) | Markdown + gallery preview |
+| Ask tab | Garment chips with images in replies | Streaming chatbot |
+
+Both modes support sample dataset loading, outfit generation, and wardrobe chat.
+
+---
 
-| Component | Model / Library |
-|-----------|----------------|
-| Vision + Chat LLM | **Gemma 3 4B** (Q4_K_M GGUF) via `llama-cpp-python` |
-| Garment Detection | YOLOS-tiny (`transformers`) / YOLOv8n / GroundingDINO |
-| Runtime | llama.cpp — CPU on Space, GPU locally |
-| UI | Gradio 6.17 (`gr.Server` + Alpine.js default) |
+## Tech stack
+
+| Component | Choice |
+|-----------|--------|
+| VLM + Chat LLM | **Gemma 3 4B IT** (Q4_K_M GGUF) via `llama-cpp-python` |
+| Garment detection | **YOLOS-tiny** (default), YOLOv8n, or GroundingDINO — pluggable registry |
+| Runtime | llama.cpp — CPU on HF Spaces, CUDA locally |
+| UI | `gradio.Server` + Alpine.js (default) or Gradio Blocks |
 | Storage | Local filesystem or S3 (configurable) |
+| Catalog | `data/catalog.json` + `data/garments/*.jpg` |
+| Preferences | `data/outfits.json` (liked combinations) |
+
+**Total parameters: 4 billion** — fits Tiny Titan (≤4B) and runs on CPU Basic (16 GB RAM) with Q4_K_M quantization (~3 GB model).
 
-Total parameters: **4 billion** — Q4_K_M fits in 16 GB RAM on CPU Basic.
+The same Gemma 3 4B model handles vision extraction, outfit ranking, and chat. A singleton `_ModelManager` hot-swaps between vision (MTMD) and text-only modes.
 
 ---
 
-## Bonus Quests
+## Bonus quests
 
 | Badge | Status |
 |-------|--------|
-| 🔌 Off the Grid | All inference runs on the Space hardware. No external APIs. |
-| 🦙 Llama Champion | Model runs through llama.cpp runtime (`llama-cpp-python`). |
-| 🐜 Tiny Titan | Gemma 3 4B — well under the 4B threshold. |
-| 🎨 Off-Brand | Custom frontend via `gr.Server` + Alpine.js in `app.py`. |
+| 🔌 Off the Grid | All inference on Space hardware. No external APIs. |
+| 🦙 Llama Champion | Model runs through llama.cpp (`llama-cpp-python`). |
+| 🐜 Tiny Titan | Gemma 3 4B — under the 4B threshold. |
+| 🎨 Off-Brand | Custom frontend via `gr.Server` + Alpine.js. |
 | 📡 Sharing is Caring | Agent trace shared on the Hub. |
-| 📓 Field Notes | Build report documenting the process (`FIELD_NOTES.md`). |
+| 📓 Field Notes | Build report in `FIELD_NOTES.md`. |
 
 ---
 
-## How to Use
+## How to use
+
+### On Hugging Face Spaces
 
-### On HuggingFace Spaces
+Runs on **CPU Basic** (2 vCPU, 16 GB RAM). Set `HF_TOKEN` in Space Secrets before first use.
 
-The app runs on **CPU Basic** (2 vCPU, 16 GB RAM). All inference is CPU-only. On first use:
-1. Click **"Obtener Dataset"** to load a sample wardrobe from a HuggingFace dataset (takes ~15–45 minutes on CPU).
-2. Or upload your own clothes photos in the **Captura** tab (~30–90 s per garment).
-3. Explore combinations in **Combina** and ask questions in **Pregunta** (~5–15 s per response).
+1. **Load a sample wardrobe** — *Add Clothes* → *Load Dataset* (50 garments from a public HF dataset; ~15–45 min on CPU with live progress logs).
+2. **Or upload your own** — drag a flat-lay photo, review auto-detected boxes, click *Analyse* (~30–90 s per garment on CPU).
+3. **Get Dressed** — type an occasion, hit *Generate* (~5–15 s for LLM ranking).
+4. **Ask** — chat about outfits, care, or what you own (~5–15 s per response).
 
-Set `HF_TOKEN` in Space Secrets before first use (required for model and dataset downloads).
+**Sample datasets:**
 
-### Local Development (GPU accelerated)
+| Key | Dataset | Notes |
+|-----|---------|-------|
+| `second-hand` | `fnauman/fashion-second-hand-front-only-rgb` | Individual garments, no detection step |
+| `fashion-1k` | `Codatta/Fashion-1K` | Multi-garment photos, slower (needs detection) |
+
+### Local development (GPU accelerated)
 
 ```bash
 cd packages/wardrobe-us
@@ -82,63 +118,113 @@ pip install llama-cpp-python==0.3.28 \
   --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124 \
   --force-reinstall --no-deps
 
-# Custom minimal frontend (default — gr.Server + Alpine.js):
+# Custom minimal frontend (default):
 python app.py
 
-# Standard Gradio UI (full feature set):
+# Full Gradio Blocks UI:
 python app.py --default
 ```
 
-Requires a CUDA GPU with at least 8GB VRAM for GPU mode. Without CUDA, the app falls back to CPU inference automatically. Set `HF_TOKEN` in `.env` for model downloads.
+Requires a CUDA GPU with ≥8 GB VRAM for GPU mode. Without CUDA, inference falls back to CPU automatically. Copy `.env.example` to `.env` and set `HF_TOKEN`.
+
+**Pre-build a sample catalog offline** (optional):
 
-The default mode (`--ui`) uses `gradio.Server` with a vanilla HTML/CSS/JS + Alpine.js interface designed for non-technical users. Pass `--default` for the full Gradio Blocks UI with all advanced features (manual bounding box annotation, detection backend settings, etc.).
+```bash
+python scripts/build_sample_wardrobe.py --dataset second-hand --target 50
+```
 
 ---
 
 ## Architecture
 
 ```
-app.py                    # Unified entry point (--ui or --default)
+app.py                         # Entry point (--ui default | --default)
 src/
   ui/
-    index.html            # Custom frontend (Alpine.js + @gradio/client)
-    style.css             # Minimal CSS
-  model_loader.py         # GGUF singleton (Gemma 3 4B)
-  vision.py               # VLM attribute extraction pipeline
-  detector/               # Pluggable garment detection (YOLOS/YOLOv8/GroundingDINO)
-  catalog.py              # JSON catalog CRUD
-  combinations.py         # Outfit generation + LLM ranking
-  assistant.py            # Chat with wardrobe context
-  storage.py              # Local/S3 image storage
-  settings.py             # Runtime configuration
+    index.html                 # Custom frontend (Alpine.js + Annotorious)
+    style.css
+  model_loader.py              # GGUF singleton (Gemma 3 4B, n_ctx=4096)
+  vision.py                    # VLM attribute extraction pipeline
+  detector/                    # Pluggable garment detection
+    _registry.py               # @register("yolos") pattern
+    backends/                  # yolos | yolov8 | grounding_dino
+  catalog.py                   # JSON catalog CRUD
+  combinations.py              # Outfit generation + LLM ranking
+  assistant.py                 # Chat with wardrobe context
+  storage.py                   # Local / S3 image storage
+  settings.py                  # Runtime config (data/settings.json)
+data/
+  catalog.json                 # Garment metadata
+  garments/                    # Cropped garment images
+  outfits.json                 # Liked outfit preferences
+  _uploads/                    # Temp images during crop workflow
 ```
 
+### API endpoints (custom UI)
+
+Exposed via `gradio.Server` and consumed by `@gradio/client`:
+
+| Endpoint | Purpose |
+|----------|---------|
+| `prepare_image` | Save upload, auto-detect boxes → token + image URL for editor |
+| `analyze_boxes` | Crop user-confirmed boxes, VLM extract, add to catalog |
+| `add_photo` | One-shot upload + auto-detect + extract (no manual crop) |
+| `get_wardrobe` | Full catalog with cache-busted image URLs |
+| `get_combinations` | Generate + LLM-rank outfits (top 20 returned) |
+| `rate_outfit` | Save like/dislike preference |
+| `ask_question` | Natural-language wardrobe chat |
+| `load_dataset` | Stream dataset processing progress (generator) |
+
+Static mounts: `/garments` (catalog images), `/uploads` (temp crop images).
+
+### Outfit ranking
+
+1. Rule-based generation: all compatible top+bottom pairs (season + formality filters).
+2. LLM ranking: up to 20 diverse combinations sent to Gemma 3 4B with a compact prompt (fits `n_ctx=4096`). Remaining combos appended in original order.
+3. User likes feed back into future ranking prompts as style signals.
+
 ---
 
-## Environment Variables (Secrets)
+## Environment variables
 
 | Variable | Required | Description |
 |----------|----------|-------------|
-| `HF_TOKEN` | Yes | HuggingFace token for model/dataset downloads |
+| `HF_TOKEN` | Yes | Hugging Face token for model/dataset downloads |
 | `STORAGE_BACKEND` | No | `local` (default) or `s3` |
 | `S3_BUCKET_NAME` | If S3 | Bucket name |
 | `S3_ENDPOINT_URL` | If S3 | S3 endpoint |
 | `AWS_ACCESS_KEY_ID` | If S3 | AWS credentials |
 | `AWS_SECRET_ACCESS_KEY` | If S3 | AWS credentials |
 | `DETECTION_BACKEND` | No | `yolos` (default), `yolov8`, or `grounding_dino` |
+| `CUDA_VISIBLE_DEVICES` | No | GPU index (local only; forced to CPU on Spaces) |
+
+---
+
+## Performance notes
+
+| Task | CPU Basic (Space) | Local GPU |
+|------|-------------------|-----------|
+| First model download | ~2–3 min | ~2–3 min |
+| Garment extraction | ~30–90 s each | ~3–10 s each |
+| Dataset load (50 items) | ~15–45 min | ~5–15 min |
+| Outfit ranking | ~5–15 s | ~2–5 s |
+| Ask response | ~5–15 s | ~2–5 s |
+
+**Detection tips:** YOLOS-tiny works best on flat-lay photos. Hanger or worn-garment photos are harder — use the manual bounding-box editor as fallback.
+
+**VLM accuracy:** At 4B parameters, color and type labels are usually good but not perfect (e.g. navy vs black). Descriptions and structured JSON parsing with regex fallback help reliability.
 
 ---
 
-## Agent Trace (Sharing is Caring)
+## Agent trace (Sharing is Caring)
 
-The full agent trace (conversation history used to build this project) is published as a dataset on the Hub:
+The full development conversation is published as a dataset on the Hub:
 
 ```bash
-# To publish your agent trace:
 huggingface-cli upload-large-folder build-small-hackathon/wardrobe-us-agent-trace ./agent-trace --repo-type dataset
 ```
 
-The trace is a JSONL file containing the full development conversation — architecture decisions, debugging sessions, feature implementation, and deployment configuration.
+See also `FIELD_NOTES.md` for architecture decisions, what worked, and lessons learned.
 
 ---
 

data/outfits.json

@@ -1,72 +0,0 @@
-[
-  {
-    "id": "outfit_001",
-    "top": "garment_001",
-    "bottom": "garment_005",
-    "liked": false,
-    "timestamp": "2026-06-13T13:21:28.920888+00:00"
-  },
-  {
-    "id": "outfit_002",
-    "top": "garment_001",
-    "bottom": "garment_010",
-    "liked": false,
-    "timestamp": "2026-06-13T13:21:29.395598+00:00"
-  },
-  {
-    "id": "outfit_003",
-    "top": "garment_001",
-    "bottom": "garment_015",
-    "liked": false,
-    "timestamp": "2026-06-13T13:21:29.550680+00:00"
-  },
-  {
-    "id": "outfit_004",
-    "top": "garment_001",
-    "bottom": "garment_020",
-    "liked": false,
-    "timestamp": "2026-06-13T13:21:29.733320+00:00"
-  },
-  {
-    "id": "outfit_005",
-    "top": "garment_001",
-    "bottom": "garment_025",
-    "liked": false,
-    "timestamp": "2026-06-13T13:21:29.901984+00:00"
-  },
-  {
-    "id": "outfit_006",
-    "top": "garment_001",
-    "bottom": "garment_030",
-    "liked": false,
-    "timestamp": "2026-06-13T13:21:30.053074+00:00"
-  },
-  {
-    "id": "outfit_007",
-    "top": "garment_004",
-    "bottom": "garment_005",
-    "liked": false,
-    "timestamp": "2026-06-13T13:21:30.481244+00:00"
-  },
-  {
-    "id": "outfit_008",
-    "top": "garment_036",
-    "bottom": "garment_018",
-    "liked": false,
-    "timestamp": "2026-06-13T15:29:44.676129+00:00"
-  },
-  {
-    "id": "outfit_009",
-    "top": "garment_036",
-    "bottom": "garment_022",
-    "liked": false,
-    "timestamp": "2026-06-13T15:29:47.770911+00:00"
-  },
-  {
-    "id": "outfit_010",
-    "top": "garment_036",
-    "bottom": "garment_026",
-    "liked": false,
-    "timestamp": "2026-06-13T15:29:49.066766+00:00"
-  }
-]

src/combinations.py

@@ -35,36 +35,26 @@ FORMALITY_COMPAT = {
     "formal": {"smart-casual", "formal"},
 }
 
-RANKING_SYSTEM_PROMPT = """You are an experienced personal stylist helping someone dress from their own wardrobe.
-
-Your job is to rank outfit combinations (top + bottom) from best to worst. Apply these criteria in order of importance:
-
-1. Occasion fit — match the stated occasion, weather, and activity level.
-2. Color harmony — prefer complementary or neutral pairings; penalize clashing brights or competing patterns unless intentional.
-3. Formality coherence — top and bottom should feel like they belong together (no suit pants with a gym tank).
-4. Season appropriateness — fabrics and weights should suit the weather implied by the occasion.
-5. Pattern balance — if one piece is bold (stripes, plaid, floral), pair with a solid or subtle counterpart.
-6. Versatility — favour polished, wearable looks over gimmicky or repetitive pairings.
-
-Return ONLY a valid JSON array of outfit IDs, best first. No explanation, no markdown fences."""
-
-
-def _format_garment_line(garment: dict, role: str) -> str:
-    """Format a garment with structured attributes for the ranking prompt."""
-    desc = garment.get("description", "")
-    parts = [
-        f"{role}={garment.get('color', '?')} {garment.get('type', '?')}",
-        f"material={garment.get('material', '?')}",
-        f"pattern={garment.get('pattern', '?')}",
-        f"season={garment.get('season', '?')}",
-        f"formality={garment.get('formality', '?')}",
-    ]
-    if desc:
-        parts.append(f"note={desc[:100]}")
-    return ", ".join(parts)
+RANKING_SYSTEM_PROMPT = (
+    "You are a personal stylist. Rank outfit combinations (top + bottom) best-to-worst "
+    "for the given occasion. Prioritise: occasion fit, color harmony, formality match, "
+    "season, pattern balance. Return ONLY a JSON array of outfit IDs, best first."
+)
+
+# Max combos sent to the LLM — must fit in n_ctx=4096 alongside the response.
+MAX_RANKING_ITEMS = 20
+
+
+def _format_garment_line(garment: dict) -> str:
+    """Compact one-line garment summary for the ranking prompt."""
+    return (
+        f"{garment.get('color', '?')} {garment.get('type', '?')}"
+        f" ({garment.get('pattern', 'solid')}, {garment.get('season', 'all')},"
+        f" {garment.get('formality', 'casual')})"
+    )
 
 
-def _select_combos_for_ranking(combinations: list[dict], max_items: int = 40) -> list[dict]:
+def _select_combos_for_ranking(combinations: list[dict], max_items: int = MAX_RANKING_ITEMS) -> list[dict]:
     """Pick a diverse subset when the list is too large for the LLM context."""
     if len(combinations) <= max_items:
         return combinations
@@ -94,13 +84,13 @@ def _format_liked_hint() -> str:
     if not liked:
         return ""
 
-    lines = ["\nThe user previously liked these combinations (use as style signal):"]
-    for outfit in liked[:5]:
+    lines = ["\nUser liked (style signal):"]
+    for outfit in liked[:3]:
         top = outfit["top"]
         bottom = outfit["bottom"]
-        top_label = top.get("description") or f"{top.get('color')} {top.get('type')}"
-        bottom_label = bottom.get("description") or f"{bottom.get('color')} {bottom.get('type')}"
-        lines.append(f"- {top_label} + {bottom_label}")
+        lines.append(
+            f"- {_format_garment_line(top)} + {_format_garment_line(bottom)}"
+        )
     return "\n".join(lines)
 
 
@@ -207,7 +197,7 @@ def generate_combinations(
 def rank_combinations_prompt(
     combinations: list[dict],
     context: str = "",
-    max_items: int = 40,
+    max_items: int = MAX_RANKING_ITEMS,
 ) -> tuple[str, str]:
     """Build system + user prompts for the LLM to rank outfit combinations.
 
@@ -217,32 +207,24 @@ def rank_combinations_prompt(
         return "", ""
 
     subset = _select_combos_for_ranking(combinations, max_items=max_items)
-    occasion = context.strip() if context and context.strip() else (
-        "everyday wear — versatile, practical outfits suitable for most casual situations"
-    )
+    occasion = context.strip() if context and context.strip() else "everyday casual wear"
 
     user_lines = [
         f"Occasion: {occasion}",
-        "",
-        f"Rank these {len(subset)} outfit combinations from BEST to WORST for this occasion.",
-        "Each line is one outfit. Use the outfit ID exactly as shown.",
+        f"Rank these {len(subset)} outfits best-to-worst. Return JSON array of IDs only.",
         "",
     ]
 
     for combo in subset:
-        top_line = _format_garment_line(combo["top"], "top")
-        bottom_line = _format_garment_line(combo["bottom"], "bottom")
-        user_lines.append(f"- {combo['id']}: {top_line} | {bottom_line}")
+        top = _format_garment_line(combo["top"])
+        bottom = _format_garment_line(combo["bottom"])
+        user_lines.append(f"- {combo['id']}: {top} + {bottom}")
 
     liked_hint = _format_liked_hint()
     if liked_hint:
         user_lines.append(liked_hint)
 
-    user_lines.append("")
-    user_lines.append(
-        f"Return a JSON array of all {len(subset)} outfit IDs reordered best-to-worst, "
-        "e.g. [\"outfit_003\", \"outfit_001\", ...]"
-    )
+    user_lines.append(f'Return: ["outfit_XXX", ...] with all {len(subset)} IDs reordered.')
 
     return RANKING_SYSTEM_PROMPT, "\n".join(user_lines)
 
@@ -273,7 +255,7 @@ def rank_with_llm(combinations: list[dict], context: str = "") -> list[dict]:
             {"role": "system", "content": system_prompt},
             {"role": "user", "content": user_prompt},
         ],
-        max_tokens=1024,
+        max_tokens=512,
         temperature=0.2,
     )