---
title: "Benchmark Proposal & Evaluation Framework"
subtitle: "Performance Evaluation of Generative Image Editing Models in Home Design"
author: "Generative AI Engineering Team"
date: today
format:
  pdf:
    toc: true
    number-sections: true
    colorlinks: true
    margin-left: 2cm
    margin-right: 2cm
    margin-top: 2cm
    margin-bottom: 2cm
    fig-pos: "H"
    geometry:
      - heightrounded
---

# Executive Summary

This proposal establishes a standardized, automated benchmark framework to evaluate generative image editing models (e.g., InstructPix2Pix, MagicBrush, Flux-Inpainting) within the **Home Design (Interior & Exterior)** domain.

Generative image editing lacks consistent benchmarking due to subjective quality standards and unstructured user prompts. This framework resolves these issues by introducing:

1. An automated dataset generator creating structured tasks across global and localized editing categories.
2. Dual-prompt testing (`simple_prompt` vs. `detailed_prompt`) to measure instruction-following limits.
3. Multi-dimensional quantitative metrics to evaluate visual realism, text adherence, and spatial grounding.

---

# Benchmark Dataset Architecture

The benchmark split evaluates models across two distinct categories of image editing tasks:

- **Region-Free Editing**: Focuses on global style shifts, relighting, seasonal changes, and room material transitions. This measures holistic semantic understanding without modifying spatial shapes.
- **Region-Guided Editing**: Focuses on localized object additions, removals, and replacements. The target editing region is highlighted using a red bounding box overlay (thickness = `6px`). This measures spatial adherence and local structural consistency.

# Benchmark Dataset Statistics

The generated benchmark dataset contains a total of **247 tasks** compiled across **163 unique home design images** (Interior and Exterior). The distribution of tasks is structured as follows:

| Editing Category | Task Type | Count |
| :--- | :--- | :---: |
| Region-Free | Time of Day Shift | 30 |
| Region-Free | Weather & Season Shift | 30 |
| Region-Free | Object Addition | 30 |
| Region-Free | Material Swap (Semantic) | 30 |
| Region-Free | Color Modification (Semantic) | 30 |
| Region-Free | Object Replacement | 10 |
| Region-Free | Relighting | 7 |
| Region-Free | Object Removal | 7 |
| Region-Free | **Total Region-Free** | **174** |
| Region-Guided | Local Instance Editing | 30 |
| Region-Guided | Object Removal | 23 |
| Region-Guided | Object Replacement | 20 |
| Region-Guided | **Total Region-Guided** | **73** |
| **Combined** | **Total Benchmark Tasks** | **247** |

---

# Benchmark Lifecycle: The 4 Core Stages

The benchmark operates through four main consecutive stages:

## Stage 1: Clean Image Synthesis (Data Generation)
Clean, high-quality images representing diverse interior (living rooms, kitchens, bedrooms) and exterior facades (yards, roofs, terraces) are generated using Text-to-Image models (Flux/Stable Diffusion) or curated from professional home design platforms. 

## Stage 2: Task Construction & Annotation
Tasks are automatically generated. Region-Free tasks represent global changes. For Region-Guided tasks, object grounding is run via Qwen 3.6 VL to find target boxes, a red outline is drawn on the clean image (thickness = `6px`), and instructions are generated referencing the box and target label.

## Stage 3: Model Inference Execution
The compiled task prompts are fed to target image editing models (e.g. InstructPix2Pix, MagicBrush, Flux-Inpainting) in batches, recording the edited outputs along with seed and parameters for reproducibility.

## Stage 4: VLM-Based Evaluation
An independent evaluator model (Qwen VLM 27B) evaluates the original image, edited output, and prompt. It outputs a score from 1 to 5 across specific design quality dimensions.

---

# Data Generation Flowchart

Below is the automated data pipeline representing Stage 1 and Stage 2:

![Benchmark Dataset Generation Flowchart](data_generation_flowchart.gv.png){width=80% fig-align="center"}

---

# Benchmark Task Pool Definition

The evaluation dataset comprises tasks drawn from two major editing methodologies:

## Region-Free Editing Tasks

Region-Free tasks modify the image globally or target semantic elements using only text instructions (the original clean image is used as input).

- **Relighting**: Altering room lighting mood (e.g., turning on lamps).
- **Time of Day Shift**: Shifting facades from day to night/sunset.
- **Weather & Season Shift**: Applying seasonal changes (e.g., adding snow).
- **Color Modification**: Editing surface/object colors (e.g., green sofa).
- **Material Swap**: Swapping textures (e.g., wood floor to concrete).
- **Style Propagation**: Matching style of one object to another.
- **Semantic Relocation**: Moving objects to a new text-specified position.

## Region-Guided Editing Tasks

Region-Guided tasks apply localized adjustments. The target region is outlined on the image canvas using a red bounding box, and the prompt refers to the highlighted area.

- **Object Replacement**: Swapping the object inside the red box for a different one.
- **Object Removal**: Erasing the boxed object and inpainting the background.
- **Object Addition**: Inserting a new object inside an empty highlighted region.
- **Local Instance Editing**: Modifying one instance among identical objects.
- **Quantity Modification**: Duplicating or changing count inside the red box.

---

# Quantitative Evaluation Strategy

To benchmark and rank image editing models, outputs are evaluated quantitatively across three core dimensions:

1. **Text Instruction Adherence**: Measured using CLIP-score to compute cosine similarity between the edited image and the input instruction. Models are compared on `simple_prompt` (direct commands) vs. `detailed_prompt` (multi-sentence specifications) to evaluate instruction capability.
2. **Spatial Grounding Accuracy (Region-Guided)**: Evaluated by measuring pixel change containment. We compute the intersection-over-union (IoU) of the pixel differences between the clean and edited images against the target bounding box region. High scores indicate edits were successfully confined within the red box.
3. **Visual Realism & Perceptual Quality**: Evaluated using **LPIPS** (Learned Perceptual Image Patch Similarity) and **PSNR** (Peak Signal-to-Noise Ratio) to assess structural fidelity, alongside pre-trained Aesthetic Predictors to score overall visual appeal.

---

# Representative Dataset Samples

Below are representative tasks from the generated benchmark validation set:

## Region-Free (Global Edit) Samples

### Sample A.1: Time of Day Shift
- **Task ID**: `29d73b1f-462b-4d02-813a-8b44e5a2e244`
- **Simple Prompt**: "Change the time of day from night to a sunny afternoon."
- **Detailed Prompt**: Transform the scene from a nighttime setting to a bright, sunny midday. Remove the artificial warm lighting from the pergola and eliminate the dark sky. Reveal the natural colors of the green velvet furniture and the yellow side tables under natural daylight. Enhance the visibility of the foliage on the pergola and the blue tiles of the swimming pool, creating a vibrant, high-contrast outdoor atmosphere.

::: {layout-nw}
![Original Clean Image](original_images/b14110c5-3937-4382-b97b-12107c96a2c9.png){width=50% fig-align="center"}
:::

\newpage

### Sample A.2: Object Replacement
- **Task ID**: `16978f59-729a-4fb3-8407-7b30ecff04f6`
- **Simple Prompt**: "Replace the green sofa with a modern white leather sectional."
- **Detailed Prompt**: Replace the existing bulky olive green sofa with a sleek, L-shaped white leather sectional. The new sofa should have clean lines, a smooth glossy texture, and chrome legs to create a sharp contrast with the rustic stone fireplace and natural wood coffee table, giving the room a contemporary and airy feel.

::: {layout-nw}
![Original Clean Image](original_images/eccc5f08-40a1-42bc-871b-e626c2e1376f.png){width=50% fig-align="center"}
:::

\newpage

## Region-Guided (Local Edit) Samples

### Sample B.1: Local Instance Editing
- **Task ID**: `f3760944-be35-4eef-8939-486953622fc8`
- **Bounding Box (xmin, ymin, xmax, ymax)**: `[311, 42, 1000, 850]`
- **Simple Prompt**: "Replace the pool wall in the red box with white marble."
- **Detailed Prompt**: Replace the black pool wall in the red box with a luxurious white marble finish. The marble should have subtle grey veining and a polished, glossy surface that reflects the blue water above. Ensure the new material blends seamlessly with the existing water and pebble textures.

::: {layout-ncol=2}
![Original Clean Image](original_images/7781a275-82f4-46c0-97b3-5c0d3299f0a2.png){width=45%}

![Annotated Image with Red Box](annotated_images/f3760944-be35-4eef-8939-486953622fc8_annotated.png){width=45%}
:::

\newpage

### Sample B.2: Local Instance Editing
- **Task ID**: `62fa3a0c-02f2-4f3d-b19c-72f68292f711`
- **Bounding Box (xmin, ymin, xmax, ymax)**: `[506, 539, 591, 770]`
- **Simple Prompt**: "Change the color of the bar stools inside the red box to black."
- **Detailed Prompt**: Change the material and color of the bar stools highlighted in the red box. Replace the yellow upholstery with textured black leather and update the legs to matte black metal to contrast with the marble island.

::: {layout-ncol=2}
![Original Clean Image](original_images/4dfbe8ce-9688-4a30-bec8-94b08475328c.png){width=45%}

![Annotated Image with Red Box](annotated_images/62fa3a0c-02f2-4f3d-b19c-72f68292f711_annotated.png){width=45%}
:::

\newpage

# Model Evaluation Leaderboard

The leaderboard below summarizes the benchmark results. These columns will be populated quantitatively as each model run is executed on the task pool:

| Model Reference | Region-Free CLIP (Simple) | Region-Free CLIP (Detailed) | Region-Guided CLIP-Score | Region-Guided Box-IoU | Aesthetic Score | LPIPS Score |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **InstructPix2Pix** | TBD | TBD | TBD | TBD | TBD | TBD |
| **MagicBrush** | TBD | TBD | TBD | TBD | TBD | TBD |
| **Flux-Inpainting** | TBD | TBD | TBD | TBD | TBD | TBD |

*Metrics definition:*
- **Region-Free CLIP**: Verifies global style compliance under varying instruction lengths.
- **Region-Guided Box-IoU**: Evaluates pixel modification boundaries relative to the target red box coordinates.
- **Aesthetic / LPIPS**: Measures output quality and distance from structural ground truth.
