Upload CAPTIONS_SETUP.md with huggingface_hub

CAPTIONS_SETUP.md

@@ -0,0 +1,152 @@
+# Qwen VL Captions Setup Guide
+
+## Overview
+This system uses **Alibaba Qwen VL Max** model to automatically caption cropped UI elements. The captions are added to each detected element in the output, making it easier to understand what each UI component is.
+
+## Setup Steps
+
+### 1. Get Alibaba Qwen API Key
+- Sign up at: https://dashscope.console.aliyun.com/
+- Navigate to API Keys section
+- Copy your API key (format: `sk-...`)
+
+### 2. Set Environment Variable
+
+**Local Development:**
+```bash
+export QWEN_API_KEY="sk-YOUR_API_KEY_HERE"
+```
+
+**Docker/HF Spaces:**
+Add to your `.env` file or Docker environment:
+```
+QWEN_API_KEY=sk-YOUR_API_KEY_HERE
+```
+
+### 3. Update app_hf_spaces_server.py
+The API key is automatically read from the environment. Make sure to set it before running.
+
+## Usage
+
+### With API Server
+Captions are automatically generated when you analyze an image:
+
+```bash
+python ui_element_client.py /path/to/image.png --api http://localhost:7860
+```
+
+Response includes `"caption"` field for each element:
+```json
+{
+  "elements": [
+    {
+      "element_id": "crop_0001",
+      "caption": "close button",
+      "x": 590,
+      "y": 21,
+      "confidence": 1.0
+    }
+  ]
+}
+```
+
+### Manual Captioning
+```bash
+python caption_cropped_images.py --dir /tmp/omoi_cropped_images --output crop_captions.json
+```
+
+## Output Format
+
+### JSON with Captions
+```json
+{
+  "element_id": "crop_0001",
+  "caption": "close button",
+  "x": 590,
+  "y": 21,
+  "confidence": 1.0,
+  "bbox": { ... },
+  "center": { ... }
+}
+```
+
+### CSV with Captions
+```csv
+Element_ID,Caption,X,Y,X1,Y1,X2,Y2,Width,Height,Confidence
+crop_0001,close button,590,21,580,5,600,35,20,30,1.0000
+crop_0002,search input,732,23,720,15,750,35,30,20,0.9950
+```
+
+## Configuration
+
+### Cache System
+- Captions are cached in `crop_captions.json`
+- Avoids re-captioning the same images
+- Use `--refresh` flag to force regeneration:
+```bash
+python caption_cropped_images.py --refresh
+```
+
+### Batch Processing
+- Concurrent workers: Default 10, adjust with `--workers`:
+```bash
+python caption_cropped_images.py --workers 20
+```
+
+## Pricing
+
+Alibaba Qwen VL pricing (as of April 2024):
+- **Qwen VL Max**: ~$0.01-0.02 per image
+- **Batch of 100 images**: ~$1-2
+- Check current pricing: https://dashscope.console.aliyun.com/pricing
+
+## Troubleshooting
+
+### "QWEN_API_KEY not set"
+```bash
+# Check if environment variable is set
+echo $QWEN_API_KEY
+
+# If not, set it
+export QWEN_API_KEY="sk-..."
+```
+
+### "API Rate Limit"
+Reduce concurrent workers:
+```bash
+python caption_cropped_images.py --workers 5
+```
+
+### "Captions module not available"
+Install dependencies:
+```bash
+pip install openai
+```
+
+## Fallback Behavior
+
+If Qwen API is not available:
+1. Captions are marked as "unknown"
+2. Output still works without captions
+3. No processing is blocked
+
+## Example Output with Captions
+
+```
+[API] Analyzing: screenshot.png
+[API] OmniParser: 152 elements detected
+[API] Template matching: 152 matches found
+[API] Adding captions: 152 images captioned
+[API] Results:
+   - crop_0001: "close button" @ (590, 21)
+   - crop_0002: "search input" @ (732, 23)
+   - crop_0003: "maximize button" @ (1136, 20)
+   ... and 149 more elements
+```
+
+## Next Steps
+
+1. Set your QWEN_API_KEY environment variable
+2. Deploy to HF Spaces with the updated code
+3. Captions will be automatically generated for all images
+4. Export results with captions in JSON/CSV format