Add small-model benchmark and organize model card

.gitattributes

@@ -38,3 +38,4 @@ assets/clover-image-tiny-paired-contact-sheet.png filter=lfs diff=lfs merge=lfs
 assets/clover-image-tiny-banner.png filter=lfs diff=lfs merge=lfs -text
 examples/prompt-gallery/**/*.png filter=lfs diff=lfs merge=lfs -text
 examples/normal-to-lora/*.png filter=lfs diff=lfs merge=lfs -text
+benchmarks/**/*.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -61,6 +61,10 @@ Windows, or Linux.
 **323,384,964 denoiser parameters · about 1.67 GB · 4–100 inference steps ·
 PyTorch/Diffusers**
 
+Clover Image Tiny is intentionally a compact, local-first 512×512 model rather
+than a frontier-scale checkpoint. Its denoiser has 323,384,964 parameters; the
+small-model comparison below gives that scale practical runtime context.
+
 Clover Image Tiny is the public PyTorch/Diffusers checkpoint release behind
 these examples. Its output has a recognizable, playful
 **DALL·E mini-ish** character. That is a visual description, not a claim of
@@ -74,7 +78,24 @@ The demo exposes prompt, negative prompt, seed, guidance, dimensions,
 scheduler, and 4–100 conventional Diffusers inference steps. It creates one
 image per request and keeps the packaged safety checker enabled.
 
-## Examples
+## Contents
+
+1. [Examples](#1-examples)
+2. [Small-model benchmark](#2-small-model-benchmark)
+3. [iPhone and Core ML](#3-iphone-and-core-ml)
+4. [Run locally](#4-run-locally)
+5. [Generation controls](#5-generation-controls)
+6. [Hardware and operating systems](#6-hardware-and-operating-systems)
+7. [Python API](#7-python-api)
+8. [About this release](#8-about-this-release)
+9. [Quality and known behavior](#9-quality-and-known-behavior)
+10. [Safety](#10-safety)
+11. [Training lineage and data](#11-training-lineage-and-data)
+12. [Citation](#12-citation)
+13. [Licenses](#13-licenses)
+14. [Reproducibility and artifact identity](#14-reproducibility-and-artifact-identity)
+
+## 1. Examples
 
 ### Prompt gallery
 
@@ -112,7 +133,29 @@ seed 1469, and the same 50-step configuration:
 
 ![Clover Image Tiny local MPS library example](assets/clover-image-tiny-local-mps-library-seed-1469.png)
 
-## iPhone and Core ML
+## 2. Small-model benchmark
+
+Clover is compared with its pinned BK-SDM-Tiny-2M base and two public
+same-family references using 16 prompts, identical seeds, 512×512 output, 30
+DDIM steps, guidance 7.5, and a shared NVIDIA A10G runtime. The measurement is
+an engineering comparison, not a human-preference leaderboard.
+
+| Model | U-Net parameters | Mean latency | Peak CUDA | Mean CLIP cosine |
+|---|---:|---:|---:|---:|
+| [Clover Image Tiny](https://huggingface.co/neonforestmist/Clover-Image-Tiny) | 323.4M | 1.024 s | 2,233 MB | 0.3195 |
+| [BK-SDM-Tiny-2M](https://huggingface.co/nota-ai/bk-sdm-tiny-2m) | 323.4M | 1.027 s | 2,230 MB | 0.3246 |
+| [Segmind Tiny-SD](https://huggingface.co/segmind/tiny-sd) | 323.4M | 1.028 s | 1,649 MB | 0.3345 |
+| [BK-SDM-v2-Tiny](https://huggingface.co/nota-ai/bk-sdm-v2-tiny) | 326.8M | 0.957 s | 2,067 MB | 0.3303 |
+
+CLIP cosine is only a prompt-adherence proxy. It is not a human-quality score,
+FID, safety evaluation, or evidence that these models are interchangeable.
+The complete protocol, machine-readable results, and generated examples are in
+[`benchmarks/text-to-image/`](benchmarks/text-to-image/) and the
+[full benchmark report](benchmarks/text-to-image/results/clover-small-model-comparison-20260825/REPORT.md).
+
+![Four-prompt small-model comparison](benchmarks/text-to-image/results/clover-small-model-comparison-20260825/contact-sheet.png)
+
+## 3. iPhone and Core ML
 
 The SwiftUI app in [`Clover-iOS`](Clover-iOS/) follows Apple platform
 conventions and exposes prompt, negative prompt, steps, guidance, seed, image
@@ -137,7 +180,7 @@ own Core ML picker download:
 See [`COREML.md`](COREML.md) for conversion details and
 [`training/README.md`](training/README.md) for the pinned LoRA jobs.
 
-### Inpainting track
+### 3.1 Inpainting track
 
 The 9-channel SD 1.4-class inpainting adaptation is trained and packaged separately:
 [`neonforestmist/Clover-Image-Tiny-Inpaint`](https://huggingface.co/neonforestmist/Clover-Image-Tiny-Inpaint).
@@ -153,12 +196,12 @@ diverse free-form and object-like masks. It improved held-out masked MAE by
 96-pixel mask-context crop; the runtime composites through the exact mask so
 unmasked pixels remain unchanged.
 
-## Run locally
+## 4. Run locally
 
 Download once, then generate offline with the bundled runner. Python 3.11 and
 3.12 are supported.
 
-### macOS — Apple silicon
+### 4.1 macOS — Apple silicon
 
 ~~~bash
 mkdir clover-image-tiny-local
@@ -189,7 +232,7 @@ open clover-image-tiny.png
 
 Use `python3.11` instead if that is the installed supported Python.
 
-### Windows — PowerShell
+### 4.2 Windows — PowerShell
 
 ~~~powershell
 mkdir clover-image-tiny-local
@@ -221,7 +264,7 @@ Invoke-Item .\clover-image-tiny.png
 Use `py -3.11` if needed. With `--device auto`, the runner selects an
 available NVIDIA CUDA GPU and otherwise uses CPU.
 
-### Linux
+### 4.3 Linux
 
 ~~~bash
 mkdir clover-image-tiny-local
@@ -250,7 +293,7 @@ python model/examples/generate.py \
 uses CPU. After the first download, `--local-files-only` prevents network
 access during generation.
 
-## Generation controls
+## 5. Generation controls
 
 The command above is ready to copy. Change these flags to explore the model:
 
@@ -281,7 +324,7 @@ outputs are never overwritten.
 
 Run `python model/examples/generate.py --help` for the complete CLI reference.
 
-## Hardware and operating systems
+## 6. Hardware and operating systems
 
 | System | Automatic backend | Precision | Current evidence |
 |---|---|---|---|
@@ -301,7 +344,7 @@ image in 18.21 seconds with fp16 MPS. Its process-lifetime maximum RSS was
 631,341,056 bytes. This is a measured point, not a minimum-RAM claim. No Core
 ML package is required for the Python path.
 
-## Python API
+## 7. Python API
 
 ~~~python
 import torch
@@ -337,7 +380,7 @@ image.save("clover-image-tiny.png")
 Seeded generation is repeatable within the selected runtime. Different
 devices, dtypes, kernels, and dependency builds can produce different pixels.
 
-## About this release
+## 8. About this release
 
 Clover Image Tiny is a conventional knowledge-distillation checkpoint trained
 for 500 optimizer steps on an exact licensed 1,000-pair calibration set. The
@@ -356,7 +399,7 @@ reproducible Core ML conversion and iPhone app source. The downloadable Core ML
 artifacts and style adapters are versioned in the separate repositories linked
 above.
 
-## Quality and known behavior
+## 9. Quality and known behavior
 
 - The included gallery demonstrates recognizable subjects across colorful
   scenes, products, food, an animal, a landscape, and an interior.
@@ -368,7 +411,7 @@ above.
   controlled benchmark or broad human-preference study.
 - Resolution and batch size multiply memory use.
 
-## Safety
+## 10. Safety
 
 The upstream safety checker is packaged and enabled in both the supported
 runner and hosted demo. A flagged output may be returned as a black placeholder;
@@ -381,7 +424,7 @@ outputs before sharing them. Do not use the model for consequential decisions,
 identity claims, medical or legal conclusions, harassment, exploitation,
 illegal activity, or uses prohibited by CreativeML OpenRAIL-M.
 
-## Training lineage and data
+## 11. Training lineage and data
 
 - Clover fine-tuning data: exactly 1,000 accepted image-caption pairs from
   `Spawning/PD3M@2a5eb24a8dccf245acd8e56341761aee06da0bdf`
@@ -401,7 +444,7 @@ for all foundational pretraining is not available to this project.
 See `DATA_PROVENANCE.md` for the portable manifest identity and
 `MODEL_DATA_LICENSES.md` for the complete component ledger.
 
-## Citation
+## 12. Citation
 
 If Clover Image Tiny is useful in your work, please cite the model release:
 
@@ -414,7 +457,7 @@ If Clover Image Tiny is useful in your work, please cite the model release:
 }
 ```
 
-## Licenses
+## 13. Licenses
 
 The model weights are a derivative under **CreativeML OpenRAIL-M**. The example
 runner and packaging code are under **Apache-2.0**. Dataset and item-level terms
@@ -426,7 +469,7 @@ request for display in this public model repository. It is not benchmark
 evidence, its panel-generation provenance is not claimed, and this package
 does not grant a downstream reuse license for it.
 
-## Reproducibility and artifact identity
+## 14. Reproducibility and artifact identity
 
 | Field | Value |
 |---|---|

benchmark_text_to_image.py

@@ -0,0 +1,283 @@
+#!/usr/bin/env python3
+"""Run a reproducible small-model text-to-image comparison on Modal.
+
+The benchmark is intentionally modest: it compares prompt adherence and
+runtime behavior under one shared recipe. It is not a human-preference study,
+FID benchmark, or claim of overall image quality.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+import time
+from pathlib import Path
+
+import modal
+
+APP_NAME = "clover-image-tiny-text-to-image-benchmark"
+OUTPUT_VOLUME_NAME = "clover-image-tiny-text-to-image-benchmark-output"
+CACHE_VOLUME_NAME = "clover-image-tiny-text-to-image-benchmark-cache"
+OUTPUT_ROOT = Path("/outputs")
+CACHE_ROOT = Path("/cache")
+CLIP_MODEL_ID = "openai/clip-vit-base-patch32"
+
+MODELS = {
+    "clover": "neonforestmist/Clover-Image-Tiny",
+    "base_bk_sdm_tiny_2m": "nota-ai/bk-sdm-tiny-2m",
+    "segmind_tiny_sd": "segmind/tiny-sd",
+    "bk_sdm_v2_tiny": "nota-ai/bk-sdm-v2-tiny",
+}
+
+PROMPTS = [
+    "a glass of red wine",
+    "a tiny glass greenhouse glowing in a moonlit garden",
+    "daisies in a blue ceramic pot",
+    "snowy mountains under a cloudy sky",
+    "a desert with a big moon in the sky",
+    "a bouquet of blue flowers",
+    "a stained-glass window of a starry night",
+    "an origami heart on textured paper",
+    "an anime boy with light blue hair and eyes",
+    "a red bicycle beside a yellow cottage",
+    "a small blue cat beside a quiet garden stream",
+    "a bowl of colorful fruit on a wooden table",
+    "an astronaut riding a horse in space",
+    "a lighthouse during a storm",
+    "a cozy cabin in a snowy forest",
+    "a vintage train station at sunset",
+]
+
+SEEDS = [1337 + index for index in range(len(PROMPTS))]
+STEPS = 30
+GUIDANCE_SCALE = 7.5
+WIDTH = 512
+HEIGHT = 512
+
+image = (
+    modal.Image.debian_slim(python_version="3.11")
+    .pip_install(
+        "accelerate==1.14.0",
+        "diffusers==0.39.0",
+        "huggingface_hub==0.36.2",
+        "numpy==2.2.6",
+        "pillow==12.3.0",
+        "safetensors==0.8.0",
+        "torch==2.7.0",
+        "torchvision==0.22.0",
+        "transformers==4.57.6",
+    )
+)
+
+output_volume = modal.Volume.from_name(OUTPUT_VOLUME_NAME, create_if_missing=True)
+cache_volume = modal.Volume.from_name(CACHE_VOLUME_NAME, create_if_missing=True)
+app = modal.App(
+    APP_NAME,
+    image=image,
+    volumes={str(OUTPUT_ROOT): output_volume, str(CACHE_ROOT): cache_volume},
+)
+
+
+def _module_parameters(module: object) -> int:
+    if module is None or not hasattr(module, "parameters"):
+        return 0
+    return sum(parameter.numel() for parameter in module.parameters())
+
+
+def _load_pipeline(model_id: str):
+    import torch
+    from diffusers import DDIMScheduler, DiffusionPipeline
+
+    pipe = DiffusionPipeline.from_pretrained(
+        model_id,
+        torch_dtype=torch.float16,
+        cache_dir=str(CACHE_ROOT / "huggingface"),
+    )
+    # A single scheduler makes the comparison recipe explicit and portable.
+    pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
+    pipe = pipe.to("cuda")
+    pipe.set_progress_bar_config(disable=True)
+    return pipe
+
+
+@app.function(
+    gpu="A10G",
+    timeout=4 * 60 * 60,
+    cpu=8,
+    memory=32768,
+)
+def benchmark(*, output_name: str) -> str:
+    import torch
+    from PIL import Image
+    from transformers import CLIPModel, CLIPProcessor
+
+    output_dir = OUTPUT_ROOT / output_name
+    if output_dir.exists():
+        raise RuntimeError(f"Benchmark output already exists: {output_dir}")
+    output_dir.mkdir(parents=True)
+    (output_dir / "images").mkdir()
+    (output_dir / "prompts.json").write_text(
+        json.dumps(
+            {
+                "prompts": PROMPTS,
+                "seeds": SEEDS,
+                "recipe": {
+                    "scheduler": "DDIMScheduler",
+                    "steps": STEPS,
+                    "guidance_scale": GUIDANCE_SCALE,
+                    "width": WIDTH,
+                    "height": HEIGHT,
+                    "num_images_per_prompt": 1,
+                    "negative_prompt": "",
+                },
+            },
+            indent=2,
+        )
+        + "\n"
+    )
+
+    env = os.environ.copy()
+    env.update(
+        {
+            "HF_HOME": str(CACHE_ROOT / "huggingface"),
+            "HF_HUB_CACHE": str(CACHE_ROOT / "huggingface" / "hub"),
+            "TOKENIZERS_PARALLELISM": "false",
+            "PYTHONUNBUFFERED": "1",
+        }
+    )
+    os.environ.update(env)
+    all_records: dict[str, dict] = {}
+
+    for label, model_id in MODELS.items():
+        print(f"Loading {label}: {model_id}", flush=True)
+        pipe = _load_pipeline(model_id)
+        unet_parameters = _module_parameters(pipe.unet)
+        total_parameters = sum(
+            _module_parameters(module)
+            for module in (
+                getattr(pipe, "unet", None),
+                getattr(pipe, "text_encoder", None),
+                getattr(pipe, "vae", None),
+                getattr(pipe, "safety_checker", None),
+            )
+        )
+
+        # Warm up kernels before measuring. The warmup image is discarded.
+        warmup_generator = torch.Generator(device="cuda").manual_seed(7)
+        with torch.inference_mode():
+            pipe(
+                "a simple red apple",
+                num_inference_steps=4,
+                guidance_scale=GUIDANCE_SCALE,
+                height=HEIGHT,
+                width=WIDTH,
+                generator=warmup_generator,
+            )
+        torch.cuda.synchronize()
+        torch.cuda.reset_peak_memory_stats()
+
+        model_dir = output_dir / "images" / label
+        model_dir.mkdir()
+        records = []
+        for index, (prompt, seed) in enumerate(zip(PROMPTS, SEEDS, strict=True)):
+            generator = torch.Generator(device="cuda").manual_seed(seed)
+            start = time.perf_counter()
+            with torch.inference_mode():
+                result = pipe(
+                    prompt,
+                    negative_prompt="",
+                    num_inference_steps=STEPS,
+                    guidance_scale=GUIDANCE_SCALE,
+                    height=HEIGHT,
+                    width=WIDTH,
+                    generator=generator,
+                )
+            torch.cuda.synchronize()
+            elapsed = time.perf_counter() - start
+            filename = f"{index + 1:02d}.png"
+            result.images[0].save(model_dir / filename, format="PNG")
+            safety = getattr(result, "nsfw_content_detected", None)
+            records.append(
+                {
+                    "index": index,
+                    "prompt": prompt,
+                    "seed": seed,
+                    "filename": f"images/{label}/{filename}",
+                    "latency_seconds": elapsed,
+                    "nsfw_content_detected": safety[0] if isinstance(safety, list) else None,
+                }
+            )
+            print(f"{label} {index + 1}/{len(PROMPTS)} {elapsed:.3f}s", flush=True)
+
+        peak_memory_mb = torch.cuda.max_memory_allocated() / (1024 * 1024)
+        all_records[label] = {
+            "model_id": model_id,
+            "unet_parameters": unet_parameters,
+            "pipeline_parameters": total_parameters,
+            "mean_latency_seconds": sum(r["latency_seconds"] for r in records) / len(records),
+            "median_latency_seconds": sorted(r["latency_seconds"] for r in records)[len(records) // 2],
+            "peak_cuda_allocated_mb": peak_memory_mb,
+            "images": records,
+        }
+        del pipe
+        torch.cuda.empty_cache()
+
+    print("Scoring generated images with CLIP", flush=True)
+    processor = CLIPProcessor.from_pretrained(
+        CLIP_MODEL_ID,
+        cache_dir=str(CACHE_ROOT / "huggingface"),
+    )
+    clip_model = CLIPModel.from_pretrained(
+        CLIP_MODEL_ID,
+        torch_dtype=torch.float16,
+        cache_dir=str(CACHE_ROOT / "huggingface"),
+    ).to("cuda")
+    clip_model.eval()
+    text_inputs = processor(text=PROMPTS, return_tensors="pt", padding=True).to("cuda")
+    with torch.inference_mode():
+        text_features = clip_model.get_text_features(**text_inputs)
+        text_features = text_features / text_features.norm(dim=-1, keepdim=True)
+
+    for label, summary in all_records.items():
+        images = [Image.open(output_dir / record["filename"]).convert("RGB") for record in summary["images"]]
+        scores = []
+        for image_item, prompt in zip(images, PROMPTS, strict=True):
+            image_inputs = processor(images=image_item, return_tensors="pt").to("cuda")
+            with torch.inference_mode():
+                image_features = clip_model.get_image_features(**image_inputs)
+            image_features = image_features / image_features.norm(dim=-1, keepdim=True)
+            prompt_index = PROMPTS.index(prompt)
+            scores.append(float((image_features @ text_features[prompt_index : prompt_index + 1].T).item()))
+        summary["clip_prompt_cosine_mean"] = sum(scores) / len(scores)
+        summary["clip_prompt_cosine_median"] = sorted(scores)[len(scores) // 2]
+        summary["clip_prompt_cosine_scores"] = scores
+
+    report = {
+        "schema_version": 1,
+        "benchmark": "clover-image-tiny-small-model-comparison",
+        "status": "completed",
+        "hardware": "Modal NVIDIA A10G",
+        "public_identity_note": "The compute account identity is intentionally omitted.",
+        "protocol": {
+            "prompt_count": len(PROMPTS),
+            "resolution": f"{WIDTH}x{HEIGHT}",
+            "scheduler": "DDIMScheduler",
+            "steps": STEPS,
+            "guidance_scale": GUIDANCE_SCALE,
+            "negative_prompt": "",
+            "same_prompts_and_seeds": True,
+            "metric_note": "CLIP cosine is a prompt-adherence proxy, not a human-quality score or a broad benchmark.",
+        },
+        "models": all_records,
+    }
+    (output_dir / "results.json").write_text(json.dumps(report, indent=2) + "\n")
+    output_volume.commit()
+    cache_volume.commit()
+    return str(output_dir)
+
+
+@app.local_entrypoint()
+def main(output_name: str = f"clover-small-model-comparison-{int(time.time())}") -> None:
+    result = benchmark.remote(output_name=output_name)
+    print(f"Benchmark output: {result}")

benchmarks/text-to-image/README.md

@@ -0,0 +1,37 @@
+# Clover Image Tiny small-model comparison
+
+This benchmark compares Clover Image Tiny with its pinned BK-SDM-Tiny-2M base
+and two public Diffusers checkpoints in the same broad SD-tiny family:
+
+- `neonforestmist/Clover-Image-Tiny`
+- `nota-ai/bk-sdm-tiny-2m`
+- `segmind/tiny-sd`
+- `nota-ai/bk-sdm-v2-tiny`
+
+The comparison uses the same 16 prompts, seeds, 512×512 resolution, empty
+negative prompt, 30 DDIM steps, guidance scale 7.5, and one NVIDIA A10G
+runtime. It records U-Net and loaded-pipeline parameter counts, generation
+latency, peak CUDA allocation, and CLIP image/text cosine similarity.
+
+CLIP cosine is used only as a prompt-adherence proxy. It is not a human
+preference score, FID, a safety evaluation, or a claim that the models are
+identical in training data, scheduler defaults, or intended use.
+
+Run it with:
+
+```bash
+modal run benchmark_text_to_image.py
+```
+
+The Modal app writes a timestamped result directory to its private output
+volume. Download a completed run with:
+
+```bash
+modal volume get clover-image-tiny-text-to-image-benchmark-output \
+  clover-small-model-comparison-<timestamp> \
+  benchmarks/text-to-image/results
+```
+
+The public model card should include only the exported `results.json`, selected
+example images, the protocol, and the benchmark caveats. Compute-account
+identity is intentionally not part of the artifact.

benchmarks/text-to-image/results/clover-small-model-comparison-20260825/REPORT.md

@@ -0,0 +1,53 @@
+# Clover Image Tiny — small-model comparison
+
+This is a same-recipe engineering comparison, not a human-preference or
+state-of-the-art benchmark. It places Clover Image Tiny beside its pinned base
+and two public Diffusers checkpoints with the same broad SD-tiny U-Net shape.
+
+## Protocol
+
+- 16 fixed prompts and seeds `1337`–`1352`
+- 512 × 512 output, empty negative prompt
+- DDIM scheduler, 30 steps, guidance scale 7.5
+- One image per prompt on the same NVIDIA A10G runtime
+- CLIP image/text cosine similarity as a prompt-adherence proxy
+- Latency measured after one warm-up generation per model
+
+CLIP cosine is not a human-quality score, FID, a safety evaluation, or proof
+that one model is generally better. The models have different training data,
+text encoders, and original release goals.
+
+## Results
+
+| Model | U-Net parameters | Loaded pipeline parameters | Mean latency (s) | Peak CUDA (MB) | Mean CLIP cosine |
+|---|---:|---:|---:|---:|---:|
+| [Clover Image Tiny](https://huggingface.co/neonforestmist/Clover-Image-Tiny) | 323,384,964 | 834,080,895 | 1.024 | 2,233.0 | 0.3195 |
+| [BK-SDM-Tiny-2M](https://huggingface.co/nota-ai/bk-sdm-tiny-2m) | 323,384,964 | 834,080,895 | 1.027 | 2,230.0 | 0.3246 |
+| [Segmind Tiny-SD](https://huggingface.co/segmind/tiny-sd) | 323,384,964 | 530,099,307 | 1.028 | 1,648.9 | 0.3345 |
+| [BK-SDM-v2-Tiny](https://huggingface.co/nota-ai/bk-sdm-v2-tiny) | 326,825,604 | 750,867,307 | 0.957 | 2,067.1 | 0.3303 |
+
+The result is best read as scale and runtime context: Clover is a compact
+512×512 model with a 323.4M-parameter denoiser, and its scores are in the same
+range as these similarly sized references under this limited recipe. This is
+not evidence that Clover wins a broad quality contest, nor that a larger model
+would be unnecessary.
+
+## Visual examples
+
+![Four-prompt comparison across the four models](contact-sheet.png)
+
+The full generated set contains all 16 prompts for all four models under
+`images/`. The exact machine-readable report is [`results.json`](results.json)
+and the prompt/seed manifest is [`prompts.json`](prompts.json).
+
+## Reproduction
+
+From the Clover source repository:
+
+```bash
+modal run benchmark_text_to_image.py \
+  --output-name clover-small-model-comparison-<timestamp>
+```
+
+The benchmark intentionally omits the compute-account identity from its public
+report and artifacts.