CAPTIONING_EXPLAINED.md

""" How OmniParser Image Captioning Works

FLOW DIAGRAM:

1. YOLO Detection Phase ↓ [Full Screenshot] ↓ YOLO model detects UI elements → Returns bounding boxes ↓ filtered_boxes = [(x1, y1, x2, y2), ...] (normalized 0-1 coordinates)

2. OCR Phase (PaddleOCR) ↓ Extract text from screenshot ↓ ocr_text = ["Download Visual", "Code explanation", ...] ocr_bbox = [(x1, y1, x2, y2), ...]

3. CAPTION GENERATION Phase (Florence-2) ↓ FOR EACH detected UI element:

   Step A: Crop the UI element region ───────────────────────────────── Take bounding box coordinates and crop the image:

   xmin = int(bbox[0] * image_width) # e.g., 587 xmax = int(bbox[2] * image_width) # e.g., 764 ymin = int(bbox[1] * image_height) # e.g., 394 ymax = int(bbox[3] * image_height) # e.g., 441

   cropped = image[ymin:ymax, xmin:xmax, :]

   Then resize to 64x64: cropped = cv2.resize(cropped, (64, 64))

   Example: For "Select File" button: ┌─────────────────────────────┐ │ ┌──────────────────┐ │ │ │ CROPPED REGION │ │ │ │ (64x64 pixels) │ │ │ └──────────────────┘ │ └─────────────────────────────┘

   Step B: Prepare inputs for Florence model ────────────────────────────────────────── input_ids = processor.tokenize(prompt="") pixel_values = processor.process_image(cropped_64x64)

   inputs = { "input_ids": tensor([...]), # Tokenized prompt "pixel_values": tensor([...]) # Processed image }

   Step C: Generate caption using Florence-2 ────────────────────────────────────────── with torch.no_grad(): generated_ids = model.generate( input_ids=inputs["input_ids"], pixel_values=inputs["pixel_values"], max_new_tokens=20, num_beams=1, do_sample=False ) Step D: Decode output ──────────────────── caption_text = processor.decode(generated_ids, skip_special_tokens=True)

   Result: "Select File" or "File Selection Button" etc.

   4. Batch Processing (for speed) ────────────────────────────── Process multiple cropped images at once:

   GPU: batch_size = 128 (each batch ~30 seconds on high-end GPU) CPU: batch_size = 4 (each batch ~30 seconds on CPU - MUCH slower!)

   Example flow: ┌─────────────────────────────────────────┐ │ Detected 120 UI elements │ │ │ │ Batch 1: Elements 0-127 (128 images) │ → Process 30s │ Batch 2: Elements 128-255 (128 images) │ → Process 30s │ ... │ └─────────────────────────────────────────┘

4. Output - parsed_content_list ──────────────────────────── [ { 'type': 'icon', 'bbox': [0.43, 0.51, 0.56, 0.58], 'content': 'Select File', # ← Generated by Florence! 'source': 'box_caption_florence' }, { 'type': 'icon', 'bbox': [0.22, 0.34, 0.32, 0.36], 'content': 'JPG Converter', # ← Generated by Florence! 'source': 'box_caption_florence' }, ... ]

KEY DIFFERENCES: Florence vs OCR-only

With Florence Captions: ├─ Generates semantic descriptions: "Select File Button", "Download Link" ├─ Understands visual context: can identify buttons vs text vs icons ├─ Slower: ~30 seconds per batch on CPU, ~seconds on GPU └─ Result: High quality semantic labels

With OCR Text Only (no captions): ├─ Extracts visible text: "Select File", "Download Visual" ├─ Uses bbox intersection to match text to elements ├─ Fast: Done immediately (OCR already done in phase 2) └─ Result: Accurate for text-based elements, misses icons without text

Example: ───────── Button with text "Select File": With Florence: "File Selection UI Element" (semantic) With OCR only: "Select File" (exact text)

Icon without visible label: With Florence: "Download icon" or "Settings gear" With OCR only: No label (fallback to Icon N)

CODE PATHS:

✓ With captions (default in original code): Omniparser.init() ↓ self.caption_model_processor = get_caption_model_processor(...)

parse() ↓ get_som_labeled_img() ↓ if use_local_semantics and caption_model_processor is not None: get_parsed_content_icon() # ← Florence caption generation ↓ Crop each UI element ↓ model.generate() with Florence ↓ Return captions

✗ With captions disabled (what I changed): Omniparser.init() ↓ self.caption_model_processor = None # ← DISABLED

parse() ↓ get_som_labeled_img() ↓ if use_local_semantics and caption_model_processor is None: # ← Now takes this branch # Use OCR text matching with bbox intersection for bbox in filtered_boxes: Find intersecting OCR text Use as label

WHY SO SLOW ON CPU?

Florence-2 has ~7 billion parameters: • Model size: ~14GB • Batch of 4 images: Must do 7B matrix operations for each • Modern GPU: 100+ TFLOPS → can handle this • CPU: 0.1 TFLOPS → 1000x slower!

Result: Each 4-image batch takes ~30 seconds on CPU For 120 elements → ~30 batches → ~15 minutes total!

That's why I disabled it for testing. 😅 """