deploy: 63a85616f5fc427cf1e1e7b425293131f2fce2b8

README.md

@@ -0,0 +1,178 @@
+---
+title: Layout Generative Model Scores
+emoji: 📊
+colorFrom: gray
+colorTo: green
+sdk: gradio
+sdk_version: 4.36.1
+app_file: app.py
+pinned: false
+---
+
+# Layout Generative Model Scores
+
+## Description
+
+The Layout Generative Model Scores metric computes a comprehensive set of distribution-based metrics to evaluate generative models for layout generation. This metric compares feature distributions between real and generated layouts using state-of-the-art evaluation methods.
+
+## What It Measures
+
+This metric implements several widely-used generative model evaluation scores:
+
+1. **FID (Fréchet Inception Distance)**: Measures the distance between real and generated feature distributions
+2. **Precision**: Measures the proportion of generated samples that are realistic
+3. **Recall**: Measures the proportion of real samples covered by the generated distribution
+4. **Density**: Estimates the density of the generated distribution
+5. **Coverage**: Measures the diversity of generated samples
+
+These metrics provide a holistic view of both the quality (precision, FID) and diversity (recall, coverage) of generated layouts.
+
+## Metric Details
+
+- **FID**: Computed using Fréchet distance between Gaussian distributions fitted to real and fake features
+- **PRDC (Precision, Recall, Density, Coverage)**: Computed using k-nearest neighbors in feature space
+- Requires feature vectors extracted from layouts (typically using a pre-trained neural network)
+- All metrics are distribution-based, providing robust evaluation across large sample sets
+
+## Usage
+
+### Installation
+
+```bash
+pip install evaluate prdc pytorch-fid
+```
+
+### Basic Example
+
+```python
+import evaluate
+import numpy as np
+
+# Load the metric
+metric = evaluate.load("creative-graphic-design/layout-generative-model-scores")
+
+# Single processing
+feat_size = 256
+feats_real = np.random.rand(feat_size)
+feats_fake = np.random.rand(feat_size)
+metric.add(feats_real=feats_real, feats_fake=feats_fake)
+print(metric.compute())
+```
+
+### Batch Processing Example
+
+```python
+import evaluate
+import numpy as np
+
+# Load the metric
+metric = evaluate.load("creative-graphic-design/layout-generative-model-scores")
+
+# Batch processing (recommended for meaningful statistics)
+batch_size, feat_size = 512, 256
+feats_real = np.random.rand(batch_size, feat_size)
+feats_fake = np.random.rand(batch_size, feat_size)
+metric.add_batch(feats_real=feats_real, feats_fake=feats_fake)
+print(metric.compute())
+```
+
+## Parameters
+
+### Initialization Parameters
+
+- **nearest_k** (`int`, optional, default=5): Number of nearest neighbors to use for PRDC computation
+
+### Computation Parameters
+
+- **feats_real** (`list` of `list` of `float`): Feature vectors extracted from real layouts (shape: N × feature_dim)
+- **feats_fake** (`list` of `list` of `float`): Feature vectors extracted from generated/fake layouts (shape: N × feature_dim)
+
+**Note**: Features are typically extracted using a pre-trained neural network (e.g., layout encoder) that converts layouts into fixed-dimensional feature vectors.
+
+## Returns
+
+Returns a dictionary containing:
+
+- **fid** (`float`): Fréchet Inception Distance
+- **precision** (`float`): Precision score (0.0 to 1.0)
+- **recall** (`float`): Recall score (0.0 to 1.0)
+- **density** (`float`): Density score
+- **coverage** (`float`): Coverage score (0.0 to 1.0)
+
+## Interpretation
+
+### FID (Fréchet Inception Distance)
+
+- **Lower is better**: Measures similarity between real and generated distributions
+- **Value of 0**: Perfect match between distributions
+- **Typical range**: 0 to ∞ (in practice, usually 0-300 for layout tasks)
+- **Usage**: Primary metric for assessing overall generative quality
+
+### Precision
+
+- **Higher is better** (range: 0.0 to 1.0)
+- Measures what proportion of generated samples are realistic
+- High precision = most generated layouts look realistic
+- Low precision = many unrealistic generated layouts
+
+### Recall
+
+- **Higher is better** (range: 0.0 to 1.0)
+- Measures what proportion of real distribution is covered by generated samples
+- High recall = generated layouts cover the diversity of real layouts
+- Low recall = generated layouts miss parts of the real distribution (mode collapse)
+
+### Density
+
+- Estimates how densely generated samples are packed in feature space
+- Higher values indicate more samples in covered regions
+
+### Coverage
+
+- **Higher is better** (range: 0.0 to 1.0)
+- Measures the proportion of real samples within the generated distribution's support
+- Similar to recall but computed differently
+- High coverage = good diversity in generation
+
+### Trade-offs
+
+- **Precision vs Recall**: Common trade-off in generative models
+  - High precision, low recall: Safe but limited generation (mode dropping)
+  - Low precision, high recall: Diverse but potentially unrealistic generation
+  - Goal: Balance both for quality and diversity
+
+## Citations
+
+```bibtex
+@article{heusel2017gans,
+  title={Gans trained by a two time-scale update rule converge to a local nash equilibrium},
+  author={Heusel, Martin and Ramsauer, Hubert and Unterthiner, Thomas and Nessler, Bernhard and Hochreiter, Sepp},
+  journal={Advances in neural information processing systems},
+  volume={30},
+  year={2017}
+}
+
+@inproceedings{naeem2020reliable,
+  title={Reliable fidelity and diversity metrics for generative models},
+  author={Naeem, Muhammad Ferjad and Oh, Seong Joon and Uh, Youngjung and Choi, Yunjey and Yoo, Jaejun},
+  booktitle={International Conference on Machine Learning},
+  pages={7176--7185},
+  year={2020},
+  organization={PMLR}
+}
+```
+
+## References
+
+- **Paper**: [GANs Trained by a Two Time-Scale Update Rule (Heusel et al., NeurIPS 2017)](https://arxiv.org/abs/1706.08500)
+- **Paper**: [Reliable Fidelity and Diversity Metrics for Generative Models (Naeem et al., ICML 2020)](https://arxiv.org/abs/2002.09797)
+- **Reference Implementation (FID + PRDC)**: [layout-dm metric implementation](https://github.com/CyberAgentAILab/layout-dm/blob/main/src/trainer/trainer/helpers/metric.py#L37-L59)
+- **PRDC Library**: [generative-evaluation-prdc](https://github.com/clovaai/generative-evaluation-prdc)
+- **PyTorch FID**: [pytorch-fid](https://github.com/mseitzer/pytorch-fid)
+- **Hugging Face Space**: [creative-graphic-design/layout-generative-model-scores](https://huggingface.co/spaces/creative-graphic-design/layout-generative-model-scores)
+
+## Related Metrics
+
+- [Layout Average IoU](../layout_average_iou/): Measures element overlap
+- [Layout Maximum IoU](../layout_maximum_iou/): Compares layout similarity
+- [Layout Validity](../layout_validity/): Checks layout validity constraints

requirements.txt

@@ -1,113 +1,222 @@
-aiofiles==23.2.1 ; python_version >= "3.9" and python_version < "4.0"
-aiohttp==3.9.3 ; python_version >= "3.9" and python_version < "4.0"
-aiosignal==1.3.1 ; python_version >= "3.9" and python_version < "4.0"
-altair==5.2.0 ; python_version >= "3.9" and python_version < "4.0"
-annotated-types==0.6.0 ; python_version >= "3.9" and python_version < "4.0"
-anyio==4.2.0 ; python_version >= "3.9" and python_version < "4.0"
-arrow==1.3.0 ; python_version >= "3.9" and python_version < "4.0"
-async-timeout==4.0.3 ; python_version >= "3.9" and python_version < "3.11"
-attrs==23.2.0 ; python_version >= "3.9" and python_version < "4.0"
-binaryornot==0.4.4 ; python_version >= "3.9" and python_version < "4.0"
-certifi==2024.2.2 ; python_version >= "3.9" and python_version < "4.0"
-chardet==5.2.0 ; python_version >= "3.9" and python_version < "4.0"
-charset-normalizer==3.3.2 ; python_version >= "3.9" and python_version < "4.0"
-click==8.1.7 ; python_version >= "3.9" and python_version < "4.0"
-colorama==0.4.6 ; python_version >= "3.9" and python_version < "4.0"
-contourpy==1.2.0 ; python_version >= "3.9" and python_version < "4.0"
-cookiecutter==2.5.0 ; python_version >= "3.9" and python_version < "4.0"
-cycler==0.12.1 ; python_version >= "3.9" and python_version < "4.0"
-datasets==2.17.0 ; python_version >= "3.9" and python_version < "4.0"
-dill==0.3.8 ; python_version >= "3.9" and python_version < "4.0"
-evaluate[template]==0.4.1 ; python_version >= "3.9" and python_version < "4.0"
-exceptiongroup==1.2.0 ; python_version >= "3.9" and python_version < "3.11"
-fastapi==0.109.2 ; python_version >= "3.9" and python_version < "4.0"
-ffmpy==0.3.1 ; python_version >= "3.9" and python_version < "4.0"
-filelock==3.13.1 ; python_version >= "3.9" and python_version < "4.0"
-fonttools==4.48.1 ; python_version >= "3.9" and python_version < "4.0"
-frozenlist==1.4.1 ; python_version >= "3.9" and python_version < "4.0"
-fsspec==2023.10.0 ; python_version >= "3.9" and python_version < "4.0"
-fsspec[http]==2023.10.0 ; python_version >= "3.9" and python_version < "4.0"
-gradio-client==0.10.0 ; python_version >= "3.9" and python_version < "4.0"
-gradio==4.18.0 ; python_version >= "3.9" and python_version < "4.0"
-h11==0.14.0 ; python_version >= "3.9" and python_version < "4.0"
-httpcore==1.0.2 ; python_version >= "3.9" and python_version < "4.0"
-httpx==0.26.0 ; python_version >= "3.9" and python_version < "4.0"
-huggingface-hub==0.20.3 ; python_version >= "3.9" and python_version < "4.0"
-idna==3.6 ; python_version >= "3.9" and python_version < "4.0"
-importlib-resources==6.1.1 ; python_version >= "3.9" and python_version < "4.0"
-jinja2==3.1.3 ; python_version >= "3.9" and python_version < "4.0"
-joblib==1.3.2 ; python_version >= "3.9" and python_version < "4.0"
-jsonschema-specifications==2023.12.1 ; python_version >= "3.9" and python_version < "4.0"
-jsonschema==4.21.1 ; python_version >= "3.9" and python_version < "4.0"
-kiwisolver==1.4.5 ; python_version >= "3.9" and python_version < "4.0"
-markdown-it-py==3.0.0 ; python_version >= "3.9" and python_version < "4.0"
-markupsafe==2.1.5 ; python_version >= "3.9" and python_version < "4.0"
-matplotlib==3.8.2 ; python_version >= "3.9" and python_version < "4.0"
-mdurl==0.1.2 ; python_version >= "3.9" and python_version < "4.0"
-mpmath==1.3.0 ; python_version >= "3.9" and python_version < "4.0"
-multidict==6.0.5 ; python_version >= "3.9" and python_version < "4.0"
-multiprocess==0.70.16 ; python_version >= "3.9" and python_version < "4.0"
-networkx==3.2.1 ; python_version >= "3.9" and python_version < "4.0"
-numpy==1.26.4 ; python_version >= "3.9" and python_version < "4.0"
-nvidia-cublas-cu12==12.1.3.1 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.9" and python_version < "4.0"
-nvidia-cuda-cupti-cu12==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.9" and python_version < "4.0"
-nvidia-cuda-nvrtc-cu12==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.9" and python_version < "4.0"
-nvidia-cuda-runtime-cu12==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.9" and python_version < "4.0"
-nvidia-cudnn-cu12==8.9.2.26 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.9" and python_version < "4.0"
-nvidia-cufft-cu12==11.0.2.54 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.9" and python_version < "4.0"
-nvidia-curand-cu12==10.3.2.106 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.9" and python_version < "4.0"
-nvidia-cusolver-cu12==11.4.5.107 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.9" and python_version < "4.0"
-nvidia-cusparse-cu12==12.1.0.106 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.9" and python_version < "4.0"
-nvidia-nccl-cu12==2.19.3 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.9" and python_version < "4.0"
-nvidia-nvjitlink-cu12==12.3.101 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.9" and python_version < "4.0"
-nvidia-nvtx-cu12==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.9" and python_version < "4.0"
-orjson==3.9.13 ; python_version >= "3.9" and python_version < "4.0"
-packaging==23.2 ; python_version >= "3.9" and python_version < "4.0"
-pandas==2.2.0 ; python_version >= "3.9" and python_version < "4.0"
-pillow==10.2.0 ; python_version >= "3.9" and python_version < "4.0"
-prdc==0.2 ; python_version >= "3.9" and python_version < "4.0"
-pyarrow-hotfix==0.6 ; python_version >= "3.9" and python_version < "4.0"
-pyarrow==15.0.0 ; python_version >= "3.9" and python_version < "4.0"
-pydantic-core==2.16.2 ; python_version >= "3.9" and python_version < "4.0"
-pydantic==2.6.1 ; python_version >= "3.9" and python_version < "4.0"
-pydub==0.25.1 ; python_version >= "3.9" and python_version < "4.0"
-pygments==2.17.2 ; python_version >= "3.9" and python_version < "4.0"
-pyparsing==3.1.1 ; python_version >= "3.9" and python_version < "4.0"
-python-dateutil==2.8.2 ; python_version >= "3.9" and python_version < "4.0"
-python-multipart==0.0.9 ; python_version >= "3.9" and python_version < "4.0"
-python-slugify==8.0.4 ; python_version >= "3.9" and python_version < "4.0"
-pytorch-fid==0.3.0 ; python_version >= "3.9" and python_version < "4.0"
-pytz==2024.1 ; python_version >= "3.9" and python_version < "4.0"
-pyyaml==6.0.1 ; python_version >= "3.9" and python_version < "4.0"
-referencing==0.33.0 ; python_version >= "3.9" and python_version < "4.0"
-requests==2.31.0 ; python_version >= "3.9" and python_version < "4.0"
-responses==0.18.0 ; python_version >= "3.9" and python_version < "4.0"
-rich==13.7.0 ; python_version >= "3.9" and python_version < "4.0"
-rpds-py==0.17.1 ; python_version >= "3.9" and python_version < "4.0"
-ruff==0.2.1 ; python_version >= "3.9" and python_version < "4.0"
-scikit-learn==1.4.0 ; python_version >= "3.9" and python_version < "4.0"
-scipy==1.12.0 ; python_version >= "3.9" and python_version < "4.0"
-semantic-version==2.10.0 ; python_version >= "3.9" and python_version < "4.0"
-shellingham==1.5.4 ; python_version >= "3.9" and python_version < "4.0"
-six==1.16.0 ; python_version >= "3.9" and python_version < "4.0"
-sniffio==1.3.0 ; python_version >= "3.9" and python_version < "4.0"
-starlette==0.36.3 ; python_version >= "3.9" and python_version < "4.0"
-sympy==1.12 ; python_version >= "3.9" and python_version < "4.0"
-text-unidecode==1.3 ; python_version >= "3.9" and python_version < "4.0"
-threadpoolctl==3.2.0 ; python_version >= "3.9" and python_version < "4.0"
-tomlkit==0.12.0 ; python_version >= "3.9" and python_version < "4.0"
-toolz==0.12.1 ; python_version >= "3.9" and python_version < "4.0"
-torch==2.2.0 ; python_version >= "3.9" and python_version < "4.0"
-torchvision==0.17.0 ; python_version >= "3.9" and python_version < "4.0"
-tqdm==4.66.2 ; python_version >= "3.9" and python_version < "4.0"
-triton==2.2.0 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.9" and python_version < "4.0"
-typer[all]==0.9.0 ; python_version >= "3.9" and python_version < "4.0"
-types-python-dateutil==2.8.19.20240106 ; python_version >= "3.9" and python_version < "4.0"
-typing-extensions==4.9.0 ; python_version >= "3.9" and python_version < "4.0"
-tzdata==2024.1 ; python_version >= "3.9" and python_version < "4.0"
-urllib3==2.2.0 ; python_version >= "3.9" and python_version < "4.0"
-uvicorn==0.27.1 ; python_version >= "3.9" and python_version < "4.0"
-websockets==11.0.3 ; python_version >= "3.9" and python_version < "4.0"
-xxhash==3.4.1 ; python_version >= "3.9" and python_version < "4.0"
-yarl==1.9.4 ; python_version >= "3.9" and python_version < "4.0"
-zipp==3.17.0 ; python_version >= "3.9" and python_version < "3.10"
+# This file was autogenerated by uv via the following command:
+#    uv export --package layout_generative_model_scores --no-dev --no-hashes --format requirements-txt
+aiohappyeyeballs==2.6.1
+    # via aiohttp
+aiohttp==3.13.2
+    # via fsspec
+aiosignal==1.4.0
+    # via aiohttp
+anyio==4.12.0
+    # via httpx
+attrs==25.4.0
+    # via aiohttp
+certifi==2025.11.12
+    # via
+    #   httpcore
+    #   httpx
+    #   requests
+charset-normalizer==3.4.4
+    # via requests
+click==8.3.1
+    # via typer-slim
+colorama==0.4.6 ; sys_platform == 'win32'
+    # via
+    #   click
+    #   tqdm
+datasets==4.4.2
+    # via evaluate
+dill==0.4.0
+    # via
+    #   datasets
+    #   evaluate
+    #   multiprocess
+evaluate==0.4.6
+    # via layout-generative-model-scores
+filelock==3.20.1
+    # via
+    #   datasets
+    #   huggingface-hub
+    #   torch
+frozenlist==1.8.0
+    # via
+    #   aiohttp
+    #   aiosignal
+fsspec==2025.10.0
+    # via
+    #   datasets
+    #   evaluate
+    #   huggingface-hub
+    #   torch
+h11==0.16.0
+    # via httpcore
+hf-xet==1.2.0 ; platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'
+    # via huggingface-hub
+httpcore==1.0.9
+    # via httpx
+httpx==0.28.1
+    # via
+    #   datasets
+    #   huggingface-hub
+huggingface-hub==1.2.3
+    # via
+    #   datasets
+    #   evaluate
+idna==3.11
+    # via
+    #   anyio
+    #   httpx
+    #   requests
+    #   yarl
+jinja2==3.1.6
+    # via torch
+joblib==1.5.3
+    # via
+    #   prdc
+    #   scikit-learn
+markupsafe==3.0.3
+    # via jinja2
+mpmath==1.3.0
+    # via sympy
+multidict==6.7.0
+    # via
+    #   aiohttp
+    #   yarl
+multiprocess==0.70.18
+    # via
+    #   datasets
+    #   evaluate
+networkx==3.6.1
+    # via torch
+numpy==2.2.6
+    # via
+    #   datasets
+    #   evaluate
+    #   pandas
+    #   prdc
+    #   pytorch-fid
+    #   scikit-learn
+    #   scipy
+    #   torchvision
+nvidia-cublas-cu12==12.8.4.1 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via
+    #   nvidia-cudnn-cu12
+    #   nvidia-cusolver-cu12
+    #   torch
+nvidia-cuda-cupti-cu12==12.8.90 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via torch
+nvidia-cuda-nvrtc-cu12==12.8.93 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via torch
+nvidia-cuda-runtime-cu12==12.8.90 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via torch
+nvidia-cudnn-cu12==9.10.2.21 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via torch
+nvidia-cufft-cu12==11.3.3.83 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via torch
+nvidia-cufile-cu12==1.13.1.3 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via torch
+nvidia-curand-cu12==10.3.9.90 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via torch
+nvidia-cusolver-cu12==11.7.3.90 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via torch
+nvidia-cusparse-cu12==12.5.8.93 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via
+    #   nvidia-cusolver-cu12
+    #   torch
+nvidia-cusparselt-cu12==0.7.1 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via torch
+nvidia-nccl-cu12==2.27.5 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via torch
+nvidia-nvjitlink-cu12==12.8.93 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via
+    #   nvidia-cufft-cu12
+    #   nvidia-cusolver-cu12
+    #   nvidia-cusparse-cu12
+    #   torch
+nvidia-nvshmem-cu12==3.3.20 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via torch
+nvidia-nvtx-cu12==12.8.90 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via torch
+packaging==25.0
+    # via
+    #   datasets
+    #   evaluate
+    #   huggingface-hub
+pandas==2.3.3
+    # via
+    #   datasets
+    #   evaluate
+pillow==12.0.0
+    # via
+    #   pytorch-fid
+    #   torchvision
+prdc==0.2
+    # via layout-generative-model-scores
+propcache==0.4.1
+    # via
+    #   aiohttp
+    #   yarl
+pyarrow==22.0.0
+    # via datasets
+python-dateutil==2.9.0.post0
+    # via pandas
+pytorch-fid==0.3.0
+    # via layout-generative-model-scores
+pytz==2025.2
+    # via pandas
+pyyaml==6.0.3
+    # via
+    #   datasets
+    #   huggingface-hub
+requests==2.32.5
+    # via
+    #   datasets
+    #   evaluate
+scikit-learn==1.8.0
+    # via prdc
+scipy==1.16.3
+    # via
+    #   prdc
+    #   pytorch-fid
+    #   scikit-learn
+setuptools==80.9.0 ; python_full_version >= '3.12'
+    # via torch
+shellingham==1.5.4
+    # via huggingface-hub
+six==1.17.0
+    # via python-dateutil
+sympy==1.14.0
+    # via torch
+threadpoolctl==3.6.0
+    # via scikit-learn
+torch==2.9.1
+    # via
+    #   pytorch-fid
+    #   torchvision
+torchvision==0.24.1
+    # via pytorch-fid
+tqdm==4.67.1
+    # via
+    #   datasets
+    #   evaluate
+    #   huggingface-hub
+triton==3.5.1 ; platform_machine == 'x86_64' and sys_platform == 'linux'
+    # via torch
+typer-slim==0.21.0
+    # via huggingface-hub
+typing-extensions==4.15.0
+    # via
+    #   aiosignal
+    #   anyio
+    #   huggingface-hub
+    #   torch
+    #   typer-slim
+tzdata==2025.3
+    # via pandas
+urllib3==2.6.2
+    # via requests
+xxhash==3.6.0
+    # via
+    #   datasets
+    #   evaluate
+yarl==1.22.0
+    # via aiohttp