deploy: 63a85616f5fc427cf1e1e7b425293131f2fce2b8

README.md

@@ -8,4 +8,170 @@ sdk_version: 4.36.1
 app_file: app.py
 pinned: false
 ---
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+
+# Layout Underlay Effectiveness
+
+## Description
+
+The Layout Underlay Effectiveness metric evaluates how well underlay (decoration/background) elements are placed in a layout. Underlay elements should be positioned underneath other layout elements to serve their intended purpose as background or decorative layers. This metric measures what proportion of underlay elements correctly have other elements placed on top of them.
+
+## What It Measures
+
+This metric computes:
+
+- **Valid underlay ratio**: Proportion of underlay elements that correctly have foreground elements on top
+- **Underlay functionality**: Whether decoration elements serve their background purpose
+- **Layer correctness**: How well the layout respects foreground/background hierarchy
+
+Higher scores indicate better underlay placement where decoration elements properly support foreground content.
+
+## Metric Details
+
+Two calculation modes:
+
+1. **Strict mode**: Underlay scores 1 if there's a non-underlay element completely inside it, 0 otherwise
+2. **Loose mode**: Calculates ratio of intersection area to underlay area (ai/a2)
+
+- Filters out text and other underlay elements when evaluating each underlay
+- From PosterLayout (Hsu et al., CVPR 2023) for poster design evaluation
+- Typical underlay class index: 3 (decoration/background)
+
+## Usage
+
+### Installation
+
+```bash
+pip install evaluate
+```
+
+### Basic Example
+
+```python
+import evaluate
+import numpy as np
+
+# Load the metric with canvas dimensions
+metric = evaluate.load(
+    "creative-graphic-design/layout-underlay-effectiveness",
+    canvas_width=360,
+    canvas_height=504,
+    text_label_index=1,
+    decoration_label_index=3
+)
+
+# Prepare data
+predictions = np.random.rand(1, 25, 4)  # normalized ltrb coordinates
+gold_labels = np.random.randint(0, 4, size=(1, 25))  # class labels
+score = metric.compute(predictions=predictions, gold_labels=gold_labels)
+print(score)
+```
+
+### Batch Processing Example
+
+```python
+import evaluate
+import numpy as np
+
+# Load the metric
+metric = evaluate.load(
+    "creative-graphic-design/layout-underlay-effectiveness",
+    canvas_width=360,
+    canvas_height=504,
+    text_label_index=1,
+    decoration_label_index=3
+)
+
+# Batch processing
+batch_size = 128
+predictions = np.random.rand(batch_size, 25, 4)
+gold_labels = np.random.randint(0, 4, size=(batch_size, 25))
+score = metric.compute(predictions=predictions, gold_labels=gold_labels)
+print(score)
+```
+
+## Parameters
+
+### Initialization Parameters
+
+- **canvas_width** (`int`, required): Width of the canvas in pixels
+- **canvas_height** (`int`, required): Height of the canvas in pixels
+- **text_label_index** (`int`, optional, default=1): Class index for text elements
+- **decoration_label_index** (`int`, optional, default=3): Class index for underlay/decoration elements
+
+### Computation Parameters
+
+- **predictions** (`list` of `lists` of `float`): Normalized bounding boxes in ltrb format (0.0 to 1.0)
+- **gold_labels** (`list` of `lists` of `int`): Class labels for each element (0 = padding)
+
+**Note**:
+
+- Elements with label == 0 are treated as padding
+- Text elements (text_label_index) and other underlays are excluded when checking each underlay
+- Very small elements (< 0.1% of canvas) are filtered out
+
+## Returns
+
+Returns a dictionary containing:
+
+- **underlay-effectiveness-strict** (`float`): Strict mode score (0.0 to 1.0)
+- **underlay-effectiveness-loose** (`float`): Loose mode score (0.0 to 1.0)
+
+## Interpretation
+
+- **Higher is better** (range: 0.0 to 1.0)
+- **Value of 1.0**: All underlay elements have foreground content on top (perfect)
+- **Value of 0.7-1.0**: Most underlays are effective
+- **Value of 0.5-0.7**: Some underlays are not effectively used
+- **Value < 0.5**: Many underlays lack foreground content (poor design)
+- **Value of 0.0**: No underlays have elements on top (problematic)
+
+### Mode Differences
+
+**Strict Mode**:
+
+- Binary scoring (0 or 1 per underlay)
+- Requires complete containment of a foreground element
+- More conservative evaluation
+
+**Loose Mode**:
+
+- Continuous scoring based on overlap ratio
+- Partial overlaps count proportionally
+- More lenient evaluation
+
+### Use Cases
+
+- **Poster/presentation layout evaluation**: Ensure decoration layers function properly
+- **Multi-layer designs**: Validate foreground/background hierarchy
+- **Content-aware generation**: Assess whether models understand layer relationships
+- **Design quality**: Identify layouts with ineffective underlay usage
+
+### Key Insights
+
+- **Purpose of underlay**: Should support foreground content, not exist independently
+- **Layout hierarchy matters**: Underlay effectiveness indicates proper layer understanding
+- **Design intent**: Some layouts intentionally have standalone decorative elements
+- **Context-specific**: Most relevant for poster, flyer, and presentation layouts
+
+## Citations
+
+```bibtex
+@inproceedings{hsu2023posterlayout,
+  title={Posterlayout: A new benchmark and approach for content-aware visual-textual presentation layout},
+  author={Hsu, Hsiao Yuan and He, Xiangteng and Peng, Yuxin and Kong, Hao and Zhang, Qing},
+  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
+  pages={6018--6026},
+  year={2023}
+}
+```
+
+## References
+
+- **Paper**: [PosterLayout (Hsu et al., CVPR 2023)](https://arxiv.org/abs/2303.15937)
+- **Reference Implementation**: [PosterLayout eval.py](https://github.com/PKU-ICST-MIPL/PosterLayout-CVPR2023/blob/main/eval.py)
+
+## Related Metrics
+
+- [Layout Overlay](../layout_overlay/): Measures overlap excluding underlay
+- [Layout Occlusion](../layout_occlusion/): Evaluates coverage of salient regions
+- [Layout Validity](../layout_validity/): Checks basic validity constraints

layout-underlay-effectiveness.py

@@ -4,10 +4,9 @@ import datasets as ds
 import evaluate
 import numpy as np
 import numpy.typing as npt
+from evaluate.utils.file_utils import add_start_docstrings
 
 _DESCRIPTION = r"""\
-Computes the non-flatness of regions that text elements are solely put on, referring to CGL-GAN.
-
 Computes the ratio of valid underlay elements to total underlay elements used in PosterLayout. Intuitively, underlay should be placed under other non-underlay elements.
 - strict: scoring the underlay as:
     - 1: there is a non-underlay element completely inside
@@ -16,7 +15,15 @@ Computes the ratio of valid underlay elements to total underlay elements used in
 """
 
 _KWARGS_DESCRIPTION = """\
-FIXME
+Args:
+    predictions (`list` of `lists` of `float`): A list of lists of floats representing normalized `ltrb`-format bounding boxes.
+    gold_labels (`list` of `lists` of `int`): A list of lists of integers representing class labels.
+
+Returns:
+    float: The ratio of valid underlay elements to total underlay elements.
+
+Examples:
+    FIXME
 """
 
 _CITATION = """\
@@ -30,6 +37,7 @@ _CITATION = """\
 """
 
 
+@add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
 class LayoutUnderlayEffectiveness(evaluate.Metric):
     def __init__(
         self,
@@ -106,7 +114,9 @@ class LayoutUnderlayEffectiveness(evaluate.Metric):
     def _compute_und_l(
         self, predictions: npt.NDArray[np.float64], gold_labels: npt.NDArray[np.int64]
     ) -> float:
-        metrics, avali = 0.0, 0
+        # metrics, avali = 0.0, 0
+        metrics = []
+        avali = 0
 
         for gold_label, prediction in zip(gold_labels, predictions):
             und = 0
@@ -127,9 +137,12 @@ class LayoutUnderlayEffectiveness(evaluate.Metric):
                     ios = self.metrics_inter_oneside(bb1, bb2)
                     max_ios = max(max_ios, ios)
                 und += max_ios
-            metrics += und / n1
+            # metrics += und / n1
+            metrics.append(und / n1)
 
-        return metrics / avali if avali > 0 else 0.0
+        # return metrics / avali if avali > 0 else 0.0
+        # return {"mean": np.mean(metrics), "std": np.std(metrics)}
+        return float(np.mean(metrics))
 
     def _compute_und_s(
         self, predictions: npt.NDArray[np.float64], gold_labels: npt.NDArray[np.int64]
@@ -145,7 +158,9 @@ class LayoutUnderlayEffectiveness(evaluate.Metric):
 
             return c1 and c2 and c3 and c4
 
-        metrics, avali = 0.0, 0
+        # metrics, avali = 0.0, 0
+        metrics = []
+        avali = 0
 
         for gold_label, prediction in zip(gold_labels, predictions):
             und = 0
@@ -165,9 +180,11 @@ class LayoutUnderlayEffectiveness(evaluate.Metric):
                     if is_contain(bb1, bb2):
                         und += 1
                         break
-            metrics += und / n1
+            # metrics += und / n1
+            metrics.append(und / n1)
 
-        return metrics / avali if avali > 0 else 0.0
+        # return metrics / avali if avali > 0 else 0.0
+        return float(np.mean(metrics))
 
     def _compute(
         self,

requirements.txt

@@ -1,89 +1,134 @@
-aiofiles==23.2.1 ; python_version >= "3.9" and python_version < "4.0"
-aiohttp==3.9.3 ; python_version >= "3.9" and python_version < "4.0"
-aiosignal==1.3.1 ; python_version >= "3.9" and python_version < "4.0"
-altair==5.2.0 ; python_version >= "3.9" and python_version < "4.0"
-annotated-types==0.6.0 ; python_version >= "3.9" and python_version < "4.0"
-anyio==4.2.0 ; python_version >= "3.9" and python_version < "4.0"
-arrow==1.3.0 ; python_version >= "3.9" and python_version < "4.0"
-async-timeout==4.0.3 ; python_version >= "3.9" and python_version < "3.11"
-attrs==23.2.0 ; python_version >= "3.9" and python_version < "4.0"
-binaryornot==0.4.4 ; python_version >= "3.9" and python_version < "4.0"
-certifi==2024.2.2 ; python_version >= "3.9" and python_version < "4.0"
-chardet==5.2.0 ; python_version >= "3.9" and python_version < "4.0"
-charset-normalizer==3.3.2 ; python_version >= "3.9" and python_version < "4.0"
-click==8.1.7 ; python_version >= "3.9" and python_version < "4.0"
-colorama==0.4.6 ; python_version >= "3.9" and python_version < "4.0"
-contourpy==1.2.0 ; python_version >= "3.9" and python_version < "4.0"
-cookiecutter==2.5.0 ; python_version >= "3.9" and python_version < "4.0"
-cycler==0.12.1 ; python_version >= "3.9" and python_version < "4.0"
-datasets==2.17.0 ; python_version >= "3.9" and python_version < "4.0"
-dill==0.3.8 ; python_version >= "3.9" and python_version < "4.0"
-evaluate[template]==0.4.1 ; python_version >= "3.9" and python_version < "4.0"
-exceptiongroup==1.2.0 ; python_version >= "3.9" and python_version < "3.11"
-fastapi==0.109.2 ; python_version >= "3.9" and python_version < "4.0"
-ffmpy==0.3.1 ; python_version >= "3.9" and python_version < "4.0"
-filelock==3.13.1 ; python_version >= "3.9" and python_version < "4.0"
-fonttools==4.48.1 ; python_version >= "3.9" and python_version < "4.0"
-frozenlist==1.4.1 ; python_version >= "3.9" and python_version < "4.0"
-fsspec==2023.10.0 ; python_version >= "3.9" and python_version < "4.0"
-fsspec[http]==2023.10.0 ; python_version >= "3.9" and python_version < "4.0"
-gradio-client==0.10.0 ; python_version >= "3.9" and python_version < "4.0"
-gradio==4.18.0 ; python_version >= "3.9" and python_version < "4.0"
-h11==0.14.0 ; python_version >= "3.9" and python_version < "4.0"
-httpcore==1.0.2 ; python_version >= "3.9" and python_version < "4.0"
-httpx==0.26.0 ; python_version >= "3.9" and python_version < "4.0"
-huggingface-hub==0.20.3 ; python_version >= "3.9" and python_version < "4.0"
-idna==3.6 ; python_version >= "3.9" and python_version < "4.0"
-importlib-resources==6.1.1 ; python_version >= "3.9" and python_version < "4.0"
-jinja2==3.1.3 ; python_version >= "3.9" and python_version < "4.0"
-jsonschema-specifications==2023.12.1 ; python_version >= "3.9" and python_version < "4.0"
-jsonschema==4.21.1 ; python_version >= "3.9" and python_version < "4.0"
-kiwisolver==1.4.5 ; python_version >= "3.9" and python_version < "4.0"
-markdown-it-py==3.0.0 ; python_version >= "3.9" and python_version < "4.0"
-markupsafe==2.1.5 ; python_version >= "3.9" and python_version < "4.0"
-matplotlib==3.8.2 ; python_version >= "3.9" and python_version < "4.0"
-mdurl==0.1.2 ; python_version >= "3.9" and python_version < "4.0"
-multidict==6.0.5 ; python_version >= "3.9" and python_version < "4.0"
-multiprocess==0.70.16 ; python_version >= "3.9" and python_version < "4.0"
-numpy==1.26.4 ; python_version >= "3.9" and python_version < "4.0"
-orjson==3.9.13 ; python_version >= "3.9" and python_version < "4.0"
-packaging==23.2 ; python_version >= "3.9" and python_version < "4.0"
-pandas==2.2.0 ; python_version >= "3.9" and python_version < "4.0"
-pillow==10.2.0 ; python_version >= "3.9" and python_version < "4.0"
-pyarrow-hotfix==0.6 ; python_version >= "3.9" and python_version < "4.0"
-pyarrow==15.0.0 ; python_version >= "3.9" and python_version < "4.0"
-pydantic-core==2.16.2 ; python_version >= "3.9" and python_version < "4.0"
-pydantic==2.6.1 ; python_version >= "3.9" and python_version < "4.0"
-pydub==0.25.1 ; python_version >= "3.9" and python_version < "4.0"
-pygments==2.17.2 ; python_version >= "3.9" and python_version < "4.0"
-pyparsing==3.1.1 ; python_version >= "3.9" and python_version < "4.0"
-python-dateutil==2.8.2 ; python_version >= "3.9" and python_version < "4.0"
-python-multipart==0.0.9 ; python_version >= "3.9" and python_version < "4.0"
-python-slugify==8.0.4 ; python_version >= "3.9" and python_version < "4.0"
-pytz==2024.1 ; python_version >= "3.9" and python_version < "4.0"
-pyyaml==6.0.1 ; python_version >= "3.9" and python_version < "4.0"
-referencing==0.33.0 ; python_version >= "3.9" and python_version < "4.0"
-requests==2.31.0 ; python_version >= "3.9" and python_version < "4.0"
-responses==0.18.0 ; python_version >= "3.9" and python_version < "4.0"
-rich==13.7.0 ; python_version >= "3.9" and python_version < "4.0"
-rpds-py==0.17.1 ; python_version >= "3.9" and python_version < "4.0"
-ruff==0.2.1 ; python_version >= "3.9" and python_version < "4.0"
-semantic-version==2.10.0 ; python_version >= "3.9" and python_version < "4.0"
-shellingham==1.5.4 ; python_version >= "3.9" and python_version < "4.0"
-six==1.16.0 ; python_version >= "3.9" and python_version < "4.0"
-sniffio==1.3.0 ; python_version >= "3.9" and python_version < "4.0"
-starlette==0.36.3 ; python_version >= "3.9" and python_version < "4.0"
-text-unidecode==1.3 ; python_version >= "3.9" and python_version < "4.0"
-tomlkit==0.12.0 ; python_version >= "3.9" and python_version < "4.0"
-toolz==0.12.1 ; python_version >= "3.9" and python_version < "4.0"
-tqdm==4.66.2 ; python_version >= "3.9" and python_version < "4.0"
-typer[all]==0.9.0 ; python_version >= "3.9" and python_version < "4.0"
-types-python-dateutil==2.8.19.20240106 ; python_version >= "3.9" and python_version < "4.0"
-typing-extensions==4.9.0 ; python_version >= "3.9" and python_version < "4.0"
-tzdata==2024.1 ; python_version >= "3.9" and python_version < "4.0"
-urllib3==2.2.0 ; python_version >= "3.9" and python_version < "4.0"
-uvicorn==0.27.1 ; python_version >= "3.9" and python_version < "4.0"
-websockets==11.0.3 ; python_version >= "3.9" and python_version < "4.0"
-xxhash==3.4.1 ; python_version >= "3.9" and python_version < "4.0"
-yarl==1.9.4 ; python_version >= "3.9" and python_version < "4.0"
-zipp==3.17.0 ; python_version >= "3.9" and python_version < "3.10"
+# This file was autogenerated by uv via the following command:
+#    uv export --package layout_underlay_effectiveness --no-dev --no-hashes --format requirements-txt
+aiohappyeyeballs==2.6.1
+    # via aiohttp
+aiohttp==3.13.2
+    # via fsspec
+aiosignal==1.4.0
+    # via aiohttp
+anyio==4.12.0
+    # via httpx
+attrs==25.4.0
+    # via aiohttp
+certifi==2025.11.12
+    # via
+    #   httpcore
+    #   httpx
+    #   requests
+charset-normalizer==3.4.4
+    # via requests
+click==8.3.1
+    # via typer-slim
+colorama==0.4.6 ; sys_platform == 'win32'
+    # via
+    #   click
+    #   tqdm
+datasets==4.4.2
+    # via evaluate
+dill==0.4.0
+    # via
+    #   datasets
+    #   evaluate
+    #   multiprocess
+evaluate==0.4.6
+    # via layout-underlay-effectiveness
+filelock==3.20.1
+    # via
+    #   datasets
+    #   huggingface-hub
+frozenlist==1.8.0
+    # via
+    #   aiohttp
+    #   aiosignal
+fsspec==2025.10.0
+    # via
+    #   datasets
+    #   evaluate
+    #   huggingface-hub
+h11==0.16.0
+    # via httpcore
+hf-xet==1.2.0 ; platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'
+    # via huggingface-hub
+httpcore==1.0.9
+    # via httpx
+httpx==0.28.1
+    # via
+    #   datasets
+    #   huggingface-hub
+huggingface-hub==1.2.3
+    # via
+    #   datasets
+    #   evaluate
+idna==3.11
+    # via
+    #   anyio
+    #   httpx
+    #   requests
+    #   yarl
+multidict==6.7.0
+    # via
+    #   aiohttp
+    #   yarl
+multiprocess==0.70.18
+    # via
+    #   datasets
+    #   evaluate
+numpy==2.2.6
+    # via
+    #   datasets
+    #   evaluate
+    #   pandas
+packaging==25.0
+    # via
+    #   datasets
+    #   evaluate
+    #   huggingface-hub
+pandas==2.3.3
+    # via
+    #   datasets
+    #   evaluate
+propcache==0.4.1
+    # via
+    #   aiohttp
+    #   yarl
+pyarrow==22.0.0
+    # via datasets
+python-dateutil==2.9.0.post0
+    # via pandas
+pytz==2025.2
+    # via pandas
+pyyaml==6.0.3
+    # via
+    #   datasets
+    #   huggingface-hub
+requests==2.32.5
+    # via
+    #   datasets
+    #   evaluate
+shellingham==1.5.4
+    # via huggingface-hub
+six==1.17.0
+    # via python-dateutil
+tqdm==4.67.1
+    # via
+    #   datasets
+    #   evaluate
+    #   huggingface-hub
+typer-slim==0.21.0
+    # via huggingface-hub
+typing-extensions==4.15.0
+    # via
+    #   aiosignal
+    #   anyio
+    #   huggingface-hub
+    #   typer-slim
+tzdata==2025.3
+    # via pandas
+urllib3==2.6.2
+    # via requests
+xxhash==3.6.0
+    # via
+    #   datasets
+    #   evaluate
+yarl==1.22.0
+    # via aiohttp