CAPTIONING_EXPLAINED.md

"""
How OmniParser Image Captioning Works
=====================================

FLOW DIAGRAM:
============

1. YOLO Detection Phase
   ↓
   [Full Screenshot] 
        ↓
   YOLO model detects UI elements → Returns bounding boxes
        ↓
   filtered_boxes = [(x1, y1, x2, y2), ...]  (normalized 0-1 coordinates)

2. OCR Phase (PaddleOCR)
   ↓
   Extract text from screenshot
        ↓
   ocr_text = ["Download Visual", "Code explanation", ...]
   ocr_bbox = [(x1, y1, x2, y2), ...]

3. CAPTION GENERATION Phase (Florence-2)
   ↓
   FOR EACH detected UI element:
   
      Step A: Crop the UI element region
      ─────────────────────────────────
      Take bounding box coordinates and crop the image:
      
      xmin = int(bbox[0] * image_width)    # e.g., 587
      xmax = int(bbox[2] * image_width)    # e.g., 764
      ymin = int(bbox[1] * image_height)   # e.g., 394
      ymax = int(bbox[3] * image_height)   # e.g., 441
      
      cropped = image[ymin:ymax, xmin:xmax, :]
      
      Then resize to 64x64:
      cropped = cv2.resize(cropped, (64, 64))
      
      Example: For "Select File" button:
      ┌─────────────────────────────┐
      │   ┌──────────────────┐       │
      │   │ CROPPED REGION   │       │
      │   │  (64x64 pixels)  │       │
      │   └──────────────────┘       │
      └─────────────────────────────┘
      
      Step B: Prepare inputs for Florence model
      ──────────────────────────────────────────
      input_ids = processor.tokenize(prompt="<CAPTION>")
      pixel_values = processor.process_image(cropped_64x64)
      
      inputs = {
          "input_ids": tensor([...]),          # Tokenized prompt
          "pixel_values": tensor([...])        # Processed image
      }
      
      Step C: Generate caption using Florence-2
      ──────────────────────────────────────────
      with torch.no_grad():
          generated_ids = model.generate(
              input_ids=inputs["input_ids"],
              pixel_values=inputs["pixel_values"],
              max_new_tokens=20,
              num_beams=1,
              do_sample=False
          )
      
      Step D: Decode output
      ────────────────────
      caption_text = processor.decode(generated_ids, skip_special_tokens=True)
      # Result: "Select File" or "File Selection Button" etc.
      
   4. Batch Processing (for speed)
   ──────────────────────────────
   Process multiple cropped images at once:
   
   GPU: batch_size = 128  (each batch ~30 seconds on high-end GPU)
   CPU: batch_size = 4    (each batch ~30 seconds on CPU - MUCH slower!)
   
   Example flow:
   ┌─────────────────────────────────────────┐
   │ Detected 120 UI elements                │
   │                                         │
   │ Batch 1: Elements 0-127 (128 images)   │ → Process 30s
   │ Batch 2: Elements 128-255 (128 images) │ → Process 30s
   │ ...                                     │
   └─────────────────────────────────────────┘
   
5. Output - parsed_content_list
   ────────────────────────────
   [
       {
           'type': 'icon',
           'bbox': [0.43, 0.51, 0.56, 0.58],
           'content': 'Select File',        # ← Generated by Florence!
           'source': 'box_caption_florence'
       },
       {
           'type': 'icon',
           'bbox': [0.22, 0.34, 0.32, 0.36],
           'content': 'JPG Converter',      # ← Generated by Florence!
           'source': 'box_caption_florence'
       },
       ...
   ]


KEY DIFFERENCES: Florence vs OCR-only
====================================

With Florence Captions:
├─ Generates semantic descriptions: "Select File Button", "Download Link"
├─ Understands visual context: can identify buttons vs text vs icons
├─ Slower: ~30 seconds per batch on CPU, ~seconds on GPU
└─ Result: High quality semantic labels

With OCR Text Only (no captions):
├─ Extracts visible text: "Select File", "Download Visual"
├─ Uses bbox intersection to match text to elements
├─ Fast: Done immediately (OCR already done in phase 2)
└─ Result: Accurate for text-based elements, misses icons without text

Example:
─────────
Button with text "Select File":
  With Florence: "File Selection UI Element" (semantic)
  With OCR only: "Select File" (exact text)

Icon without visible label:
  With Florence: "Download icon" or "Settings gear"
  With OCR only: No label (fallback to Icon N)


CODE PATHS:
===========

✓ With captions (default in original code):
  Omniparser.__init__()
    ↓
    self.caption_model_processor = get_caption_model_processor(...)
    
  parse()
    ↓
    get_som_labeled_img()
        ↓
        if use_local_semantics and caption_model_processor is not None:
            get_parsed_content_icon()  # ← Florence caption generation
                ↓
                Crop each UI element
                ↓
                model.generate() with Florence
                ↓
                Return captions


✗ With captions disabled (what I changed):
  Omniparser.__init__()
    ↓
    self.caption_model_processor = None  # ← DISABLED
    
  parse()
    ↓
    get_som_labeled_img()
        ↓
        if use_local_semantics and caption_model_processor is None:  # ← Now takes this branch
            # Use OCR text matching with bbox intersection
            for bbox in filtered_boxes:
                Find intersecting OCR text
                Use as label


WHY SO SLOW ON CPU?
===================

Florence-2 has ~7 billion parameters:
  • Model size: ~14GB
  • Batch of 4 images: Must do 7B matrix operations for each
  • Modern GPU: 100+ TFLOPS → can handle this
  • CPU: 0.1 TFLOPS → 1000x slower!
  
Result: Each 4-image batch takes ~30 seconds on CPU
For 120 elements → ~30 batches → ~15 minutes total!

That's why I disabled it for testing. 😅
"""