Blame for README.md

---
title: DesignGym
emoji: 🎨
colorFrom: indigo
colorTo: pink
sdk: docker
app_port: 8000
base_path: /web
pinned: false
tags: [openenv, mcp, computer-use, graphic-design, crello]
short_description: Recreate real graphic designs via MCP, HTML or a real editor
---

# DesignGym

An [OpenEnv](https://github.com/openenv/OpenEnv) environment for rebuilding real graphic designs from
their parts. Each task is a [Crello](https://huggingface.co/datasets/cyberagent/crello) template taken apart
into its pre-rendered layers and its typed copy. The agent puts it back together on a canvas and is scored
against a render of the original.

The package, the CLI, the OpenEnv name and the `DESIGN_CANVAS_*` settings keep the id `design_canvas`
(`design-canvas`). Clients, datasets and traces already use it.

- **Two tasks.** `mode="reference"`: here is the target image, recreate it. `mode="description"`: here is a
  written brief naming every asset and line of copy, build it.
- **Three surfaces over one document.** `surface="tools"`: structured MCP edits. `surface="html"`: one page of
  restricted HTML. `surface="gui"`: mouse and keyboard on a real browser editor, seeing only screenshots.
- **One renderer, one reward.** Every surface edits the same document, and every score comes from the same
  skia renderer, so a GUI episode and a tool-call episode are directly comparable.

## Run it

```bash
design-canvas prepare             # build the task pack from Crello (~1.4 GB download, 30 s to convert)
design-canvas serve --port 8007   # OpenEnv server + editor + playground
```

- Editor: `http://127.0.0.1:8007/editor/` · Playground: `/web/` · MCP: `/mcp` · Task API: `/design_canvas/...`
- Or `docker build -t design-canvas . && docker run -p 8000:8000 design-canvas` (builds the pack into the image).

The pack is a small offline sample. To serve **the whole corpus** (every playable Crello design) from the
[FineEnvs/crello-bucket](https://huggingface.co/buckets/FineEnvs/crello-bucket) bucket, point the server at the
corpus manifest:

```bash
DESIGN_CANVAS_CORPUS=corpus-manifest.json design-canvas serve --port 8007
```

The catalog (every task's ground truth, brief and difficulty) is one SQLite file copied from the bucket at
startup, so listing tasks and `reset` never download images. A design's assets are fetched the first time an
episode uses it: its 100-design row group is read (over HTTP locally, from the read-only `/corpus` mount on the
Space), checked against the indexed record, and cached on disk under a byte budget (`DESIGN_CANVAS_CACHE`).
Set `DESIGN_CANVAS_SOURCE_ROOT` to a mounted or downloaded copy of the bucket to read it as local files.

```bash
design-canvas corpus build --out artifacts/corpus-index --workers 12   # convert + index every design
design-canvas corpus publish --out artifacts/corpus-index               # upload the catalog to the bucket
design-canvas evalset build --out ../../evalsets/crello-test-300.json   # freeze the evaluation set
design-canvas evalset export ../../evalsets/crello-test-300.json --out artifacts/evalpack  # offline copy
```

Computer-use data: record verified GUI trajectories, then write them as a Hub-ready `cua-v1` dataset
(`cua_format.py`: OpenAI-style messages with pixel tool calls, embedded screenshots, per-message `trainable`;
`windows()` derives SFT rows with the same screenshot window the Tinker RL adapter uses):

```bash
design-canvas export-cua --split train --out artifacts/cua --limit 200          # record (webp screenshots)
design-canvas export-hf --cua artifacts/cua/cua.jsonl --out artifacts/hf/cua    # cua-v1 Parquet + dataset card
```

Check a running server end to end (HTTP routes, target gating, expert replays scoring 1.0 on tools and HTML in both
modes, an empty submission scoring 0, concurrent episodes, and with `--gui` two verified recorder episodes; the client
must load the same task source as the server):

```bash
DESIGN_CANVAS_CORPUS=artifacts/corpus-index .venv/bin/python -m design_canvas.verify_server --server http://127.0.0.1:8007 --gui
```

Watch Claude work, live:

```bash
claude mcp add design-canvas -- design-canvas mcp
```

then ask it to "start a design task and recreate it", and open the `editor_url` that `start_task` returns.

For more throughput, `design-canvas serve --workers N` (or `DESIGN_CANVAS_WORKERS=N`) runs N server processes behind
one URL ([`server/router.py`](server/router.py)). Don't use uvicorn's own `--workers`: OpenEnv's client creates a
session over HTTP and then attaches a WebSocket to it, and the built-in editor's page calls back for its episode,
so both must reach the process that holds the state. The router keeps every session and episode on its worker.
Measured locally (`scripts/loadtest.py`, 4 workers, no errors):

- tools: about 2,900 episodes per minute, 3× one process;
- built-in GUI editor: 128 concurrent episodes at about 190 actions per second, 2× one process.

The router itself saturates one core at about 300–450 tool calls per second, so 4–6 workers is the useful range.

On the private Space [FineEnvs/DesignGym](https://huggingface.co/spaces/FineEnvs/DesignGym) (`scripts/deploy_space.py`: cpu-upgrade, 8 vCPU / 32 GB, 4 workers × 48 sessions, the full
corpus read over HTTP), tested from India, where the network round trip is about 0.22 s:

| Load | Result | Throughput | Step p50 / p95 |
|---|---|---|---|
| Tools, 192 concurrent | 575/576 ok | 643 episodes/min, 110 calls/s | 0.26 / 0.8 s |
| Built-in GUI editor, 128 concurrent | 256/256 ok | 75 actions/s | 0.46 / 1.3 s |

- The one failure arrived at the capacity edge, just before a closing session was freed.
- At capacity, cold resets take 7–10 s; each new design is fetched once.
- More than 192 sessions are refused cleanly ("Server at capacity") after every worker has been tried.
- OpenEnv's client has no option for auth headers. `scripts/loadtest.py --hf-token-env HF_TOKEN` shows the two
  wrappers a client needs to reach a private Space.

## Talk to it

```python
from openenv.core.mcp_client import MCPToolClient
from openenv.core.env_server.mcp_types import CallToolAction

with MCPToolClient(base_url="http://127.0.0.1:8007").sync() as env:
    obs = env.reset(split="test", index=3, mode="reference", surface="tools")
    print(obs.observation.metadata["instructions"])     # the task, as the model should see it
    tools = env.list_tools()                           # depends on the surface
    env.step(CallToolAction(tool_name="add_image", arguments={"asset_id": "a2", "x": 0, "y": 0, "width": 1080}))
    out = env.step(CallToolAction(tool_name="submit", arguments={}))
    print(out.reward, out.observation.metadata["grade"])
```

`reset` arguments: `split`, `index` or `task_id`, `mode` (`reference` | `description`), `surface`
(`tools` | `html` | `gui`), `skin` (`figma` | `canva` | `photoshop` | `slides` | `powerpoint` | `inkscape` editor layouts; random per `seed`;
or a real app, below),
`theme`, `file_dialog` (`mac` | `windows` | `linux`: the look of the file window images are opened from; random per `seed`),
`coordinates` (`pixels`, or `normalized` for a 0-1000 grid), `max_steps`,
`reward_mode` (`final`, or `dense` for per-edit score deltas that sum to the final score).

The Task API lists tasks without giving them away: `GET /design_canvas/splits`,
`POST /design_canvas/num_tasks`, `/task`, `/task_range`.

## Tools

| Surface | Tools |
|---|---|
| all | `get_task`, `view_reference`, `submit`, `start_task` (for MCP clients with no `reset`) |
| tools, html | `view_canvas`, `view_assets` (a labelled contact sheet), `list_fonts` |
| tools | `add_image`, `add_text`, `add_shape`, `update_element`, `delete_element`, `duplicate_element`, `reorder_element`, `align_elements`, `set_background`, `list_elements`, `undo`, `redo` |
| html | `write_html`, `read_html` |
| gui | `screenshot`, `left_click`, `double_click`, `triple_click`, `right_click`, `mouse_move`, `left_click_drag`, `scroll`, `type`, `key`, `wait` |

Images come back as MCP image content blocks, so a vision model sees pixels, not base64.

## Real apps

The GUI surface can also put the model in a real design app instead of the built-in editor: `skin="penpot"`
(Penpot in the env's browser), `"impress"` (LibreOffice Impress), `"inkscape-desktop"` or `"gimp"` (on a Linux
desktop). The tools, the prompt's task and the reward are the same. At `reset` the env sets the app up with a
blank canvas at the task's size and the task's assets loaded. On `submit` it reads the app's own document back
(Penpot's file, the slide, the SVG, the layers) as a design document and scores it like any other.

The client always talks to one env server, and that server runs the apps:

- **Locally.** Run Penpot with `realapps/penpot/docker-compose.yaml` and the desktop image from
  `realapps/desktop/`. The server finds them at `DESIGN_CANVAS_PENPOT_URL` and `DESIGN_CANVAS_DESKTOP_URLS`.
- **On Daytona.** Set `DESIGN_CANVAS_APPS=daytona` and `DAYTONA_API_KEY`. The server keeps its own pool of
  private sandboxes (`realapps/daytona.py`): one per running desktop episode (`DESIGN_CANVAS_DAYTONA_DESKTOPS`,
  default 2, built from a snapshot of the desktop image) and one Penpot sandbox shared by all Penpot episodes.
  It reaches them through their preview URLs with the access token in a header, and the browser may load that
  app and nothing else. Sandboxes are reused across server restarts and stop when idle.
  `python -m design_canvas.realapps.daytona up | status | down` starts, lists or deletes them.

A real-app episode's `reset` observation carries `app` (`{"skin", "name", "real": true}`), and its `editor_url` is
the app's live screen (`/api/episodes/{id}/screen.png`). `GET /api/apps` lists the apps and where they run
(`?check=1` also asks each endpoint or sandbox whether it is up). In the `/web` playground, pick a real app under
**Editor**: the Editor tab shows its screen, and a click on the screen is a click in the app. The Expert tab
writes the target into the app, and **Read the document from the app** shows what Submit will score.

Expert round trips (the ground truth written into the app and read back) score 1.000 in Penpot and Inkscape,
0.975 to 0.999 in Impress, and 0.953 to 0.991 in GIMP, which has no mixed-colour text runs.

## Reward

`0.60·structure + 0.30·visual + 0.05·order + 0.05·background` in reference mode. Structure matches elements
one-to-one (Hungarian) on content, geometry and typography; visual is multi-scale SSIM and colour layout against
`render(ground truth)`, normalised so an empty canvas scores 0. Description mode loosens geometry and adds
coverage. Replaying the ground truth scores exactly 1.000 on every task. Details:
[`core/score.py`](core/score.py) and the project README.

`DESIGN_CANVAS_REWARD=3` (or `grade(..., version=3)`) selects reward v3 for reference mode, calibrated against
a pairwise visual judge (Claude Opus 5.5, both orders) on frontier and Qwen attempts across all three surfaces:

- **Coarse-to-fine geometry.** Half the tight kernel, half a loose one (15% of the diagonal), scaled by √position,
  so a roughly placed element in the right region earns credit, but 1.0 still needs exact placement.
- **Typography that multiplies.** Font and colour scale the rest of the style match, and style scales placement.
- **Copy counts triple in structure.** Every unmatched submitted element counts against the score.
- **Gated pixel terms.** Visual, order and background count in proportion to min(1, structure/0.3), so colour
  patches and scattered dumps (~0.25 under v1) score ~0.

Results:

- **Validation:** agrees with the judge on 89.9% of 556 pairs vs 85.4% for v1 (+33/−8, p<0.001), and on 91.4%
  of Qwen pairs.
- **Held-out test tasks:** 88.4% vs 84.2% (p=0.005), including 271 GUI pairs.
- **Description mode:** unchanged.

## Layout

```
core/      document model, ops, fonts, skia renderer, reward, HTML profile, briefs, expert, Crello conversion
server/    OpenEnv environment, editor API + live sync, GUI driver (Playwright), Gradio playground, stdio MCP
realapps/  real apps behind the GUI surface: Penpot, the desktop agent (Impress, Inkscape, GIMP), Daytona pool
web/       the editor: vanilla JS, six layouts (Studio / Template / Pixel / Deck / Present / Vector), no build step
scripts/   parity.py (browser vs renderer vs Crello), gui_expert.py (scripted computer-use agent)
tests/     core, reward invariants over every task, OpenEnv contract, browser E2E, GUI agent
rollout.py real models on every surface (Anthropic, OpenAI, Hugging Face router)
```

Data: Crello is CDLA-Permissive-2.0; the templates belong to VistaCreate. The pack is built locally from the
dataset and not redistributed here.

## Served corpus

22,688 Crello designs (train 18,954, validation 1,817, test 1,917), read on demand from the [FineEnvs/crello-bucket](https://huggingface.co/buckets/FineEnvs/crello-bucket) bucket mounted at `/corpus`. Snapshot `aafdb0e86816e6af`.