| --- |
| title: DesignGym |
| emoji: 🎨 |
| colorFrom: indigo |
| colorTo: pink |
| sdk: docker |
| app_port: 8000 |
| base_path: /web |
| pinned: false |
| tags: [openenv, mcp, computer-use, graphic-design, crello] |
| short_description: Recreate real graphic designs via MCP, HTML or a real editor |
| --- |
| |
| # DesignGym |
|
|
| An [OpenEnv](https://github.com/openenv/OpenEnv) environment for rebuilding real graphic designs from |
| their parts. Each task is a [Crello](https://huggingface.co/datasets/cyberagent/crello) template taken apart |
| into its pre-rendered layers and its typed copy. The agent puts it back together on a canvas and is scored |
| against a render of the original. |
|
|
| The package, the CLI, the OpenEnv name and the `DESIGN_CANVAS_*` settings keep the id `design_canvas` |
| (`design-canvas`). Clients, datasets and traces already use it. |
|
|
| - **Two tasks.** `mode="reference"`: here is the target image, recreate it. `mode="description"`: here is a |
| written brief naming every asset and line of copy, build it. |
| - **Three surfaces over one document.** `surface="tools"`: structured MCP edits. `surface="html"`: one page of |
| restricted HTML. `surface="gui"`: mouse and keyboard on a real browser editor, seeing only screenshots. |
| - **One renderer, one reward.** Every surface edits the same document, and every score comes from the same |
| skia renderer, so a GUI episode and a tool-call episode are directly comparable. |
|
|
| ## Run it |
|
|
| ```bash |
| design-canvas prepare # build the task pack from Crello (~1.4 GB download, 30 s to convert) |
| design-canvas serve --port 8007 # OpenEnv server + editor + playground |
| ``` |
|
|
| - Editor: `http://127.0.0.1:8007/editor/` · Playground: `/web/` · MCP: `/mcp` · Task API: `/design_canvas/...` |
| - Or `docker build -t design-canvas . && docker run -p 8000:8000 design-canvas` (builds the pack into the image). |
|
|
| The pack is a small offline sample. To serve **the whole corpus** (every playable Crello design) from the |
| [FineEnvs/crello-bucket](https://huggingface.co/buckets/FineEnvs/crello-bucket) bucket, point the server at the |
| corpus manifest: |
|
|
| ```bash |
| DESIGN_CANVAS_CORPUS=corpus-manifest.json design-canvas serve --port 8007 |
| ``` |
|
|
| The catalog (every task's ground truth, brief and difficulty) is one SQLite file copied from the bucket at |
| startup, so listing tasks and `reset` never download images. A design's assets are fetched the first time an |
| episode uses it: its 100-design row group is read (over HTTP locally, from the read-only `/corpus` mount on the |
| Space), checked against the indexed record, and cached on disk under a byte budget (`DESIGN_CANVAS_CACHE`). |
| Set `DESIGN_CANVAS_SOURCE_ROOT` to a mounted or downloaded copy of the bucket to read it as local files. |
|
|
| ```bash |
| design-canvas corpus build --out artifacts/corpus-index --workers 12 # convert + index every design |
| design-canvas corpus publish --out artifacts/corpus-index # upload the catalog to the bucket |
| design-canvas evalset build --out ../../evalsets/crello-test-300.json # freeze the evaluation set |
| design-canvas evalset export ../../evalsets/crello-test-300.json --out artifacts/evalpack # offline copy |
| ``` |
|
|
| Computer-use data: record verified GUI trajectories, then write them as a Hub-ready `cua-v1` dataset |
| (`cua_format.py`: OpenAI-style messages with pixel tool calls, embedded screenshots, per-message `trainable`; |
| `windows()` derives SFT rows with the same screenshot window the Tinker RL adapter uses): |
|
|
| ```bash |
| design-canvas export-cua --split train --out artifacts/cua --limit 200 # record (webp screenshots) |
| design-canvas export-hf --cua artifacts/cua/cua.jsonl --out artifacts/hf/cua # cua-v1 Parquet + dataset card |
| ``` |
|
|
| Check a running server end to end (HTTP routes, target gating, expert replays scoring 1.0 on tools and HTML in both |
| modes, an empty submission scoring 0, concurrent episodes, and with `--gui` two verified recorder episodes; the client |
| must load the same task source as the server): |
|
|
| ```bash |
| DESIGN_CANVAS_CORPUS=artifacts/corpus-index .venv/bin/python -m design_canvas.verify_server --server http://127.0.0.1:8007 --gui |
| ``` |
|
|
| Watch Claude work, live: |
|
|
| ```bash |
| claude mcp add design-canvas -- design-canvas mcp |
| ``` |
|
|
| then ask it to "start a design task and recreate it", and open the `editor_url` that `start_task` returns. |
|
|
| For more throughput, `design-canvas serve --workers N` (or `DESIGN_CANVAS_WORKERS=N`) runs N server processes behind |
| one URL ([`server/router.py`](server/router.py)). Don't use uvicorn's own `--workers`: OpenEnv's client creates a |
| session over HTTP and then attaches a WebSocket to it, and the built-in editor's page calls back for its episode, |
| so both must reach the process that holds the state. The router keeps every session and episode on its worker. |
| Measured locally (`scripts/loadtest.py`, 4 workers, no errors): |
|
|
| - tools: about 2,900 episodes per minute, 3× one process; |
| - built-in GUI editor: 128 concurrent episodes at about 190 actions per second, 2× one process. |
|
|
| The router itself saturates one core at about 300–450 tool calls per second, so 4–6 workers is the useful range. |
|
|
| On the private Space [FineEnvs/DesignGym](https://huggingface.co/spaces/FineEnvs/DesignGym) (`scripts/deploy_space.py`: cpu-upgrade, 8 vCPU / 32 GB, 4 workers × 48 sessions, the full |
| corpus read over HTTP), tested from India, where the network round trip is about 0.22 s: |
|
|
| | Load | Result | Throughput | Step p50 / p95 | |
| |---|---|---|---| |
| | Tools, 192 concurrent | 575/576 ok | 643 episodes/min, 110 calls/s | 0.26 / 0.8 s | |
| | Built-in GUI editor, 128 concurrent | 256/256 ok | 75 actions/s | 0.46 / 1.3 s | |
|
|
| - The one failure arrived at the capacity edge, just before a closing session was freed. |
| - At capacity, cold resets take 7–10 s; each new design is fetched once. |
| - More than 192 sessions are refused cleanly ("Server at capacity") after every worker has been tried. |
| - OpenEnv's client has no option for auth headers. `scripts/loadtest.py --hf-token-env HF_TOKEN` shows the two |
| wrappers a client needs to reach a private Space. |
|
|
| ## Talk to it |
|
|
| ```python |
| from openenv.core.mcp_client import MCPToolClient |
| from openenv.core.env_server.mcp_types import CallToolAction |
| |
| with MCPToolClient(base_url="http://127.0.0.1:8007").sync() as env: |
| obs = env.reset(split="test", index=3, mode="reference", surface="tools") |
| print(obs.observation.metadata["instructions"]) # the task, as the model should see it |
| tools = env.list_tools() # depends on the surface |
| env.step(CallToolAction(tool_name="add_image", arguments={"asset_id": "a2", "x": 0, "y": 0, "width": 1080})) |
| out = env.step(CallToolAction(tool_name="submit", arguments={})) |
| print(out.reward, out.observation.metadata["grade"]) |
| ``` |
|
|
| `reset` arguments: `split`, `index` or `task_id`, `mode` (`reference` | `description`), `surface` |
| (`tools` | `html` | `gui`), `skin` (`figma` | `canva` | `photoshop` | `slides` | `powerpoint` | `inkscape` editor layouts; random per `seed`; |
| or a real app, below), |
| `theme`, `file_dialog` (`mac` | `windows` | `linux`: the look of the file window images are opened from; random per `seed`), |
| `coordinates` (`pixels`, or `normalized` for a 0-1000 grid), `max_steps`, |
| `reward_mode` (`final`, or `dense` for per-edit score deltas that sum to the final score). |
|
|
| The Task API lists tasks without giving them away: `GET /design_canvas/splits`, |
| `POST /design_canvas/num_tasks`, `/task`, `/task_range`. |
|
|
| ## Tools |
|
|
| | Surface | Tools | |
| |---|---| |
| | all | `get_task`, `view_reference`, `submit`, `start_task` (for MCP clients with no `reset`) | |
| | tools, html | `view_canvas`, `view_assets` (a labelled contact sheet), `list_fonts` | |
| | tools | `add_image`, `add_text`, `add_shape`, `update_element`, `delete_element`, `duplicate_element`, `reorder_element`, `align_elements`, `set_background`, `list_elements`, `undo`, `redo` | |
| | html | `write_html`, `read_html` | |
| | gui | `screenshot`, `left_click`, `double_click`, `triple_click`, `right_click`, `mouse_move`, `left_click_drag`, `scroll`, `type`, `key`, `wait` | |
|
|
| Images come back as MCP image content blocks, so a vision model sees pixels, not base64. |
|
|
| ## Real apps |
|
|
| The GUI surface can also put the model in a real design app instead of the built-in editor: `skin="penpot"` |
| (Penpot in the env's browser), `"impress"` (LibreOffice Impress), `"inkscape-desktop"` or `"gimp"` (on a Linux |
| desktop). The tools, the prompt's task and the reward are the same. At `reset` the env sets the app up with a |
| blank canvas at the task's size and the task's assets loaded. On `submit` it reads the app's own document back |
| (Penpot's file, the slide, the SVG, the layers) as a design document and scores it like any other. |
|
|
| The client always talks to one env server, and that server runs the apps: |
|
|
| - **Locally.** Run Penpot with `realapps/penpot/docker-compose.yaml` and the desktop image from |
| `realapps/desktop/`. The server finds them at `DESIGN_CANVAS_PENPOT_URL` and `DESIGN_CANVAS_DESKTOP_URLS`. |
| - **On Daytona.** Set `DESIGN_CANVAS_APPS=daytona` and `DAYTONA_API_KEY`. The server keeps its own pool of |
| private sandboxes (`realapps/daytona.py`): one per running desktop episode (`DESIGN_CANVAS_DAYTONA_DESKTOPS`, |
| default 2, built from a snapshot of the desktop image) and one Penpot sandbox shared by all Penpot episodes. |
| It reaches them through their preview URLs with the access token in a header, and the browser may load that |
| app and nothing else. Sandboxes are reused across server restarts and stop when idle. |
| `python -m design_canvas.realapps.daytona up | status | down` starts, lists or deletes them. |
|
|
| A real-app episode's `reset` observation carries `app` (`{"skin", "name", "real": true}`), and its `editor_url` is |
| the app's live screen (`/api/episodes/{id}/screen.png`). `GET /api/apps` lists the apps and where they run |
| (`?check=1` also asks each endpoint or sandbox whether it is up). In the `/web` playground, pick a real app under |
| **Editor**: the Editor tab shows its screen, and a click on the screen is a click in the app. The Expert tab |
| writes the target into the app, and **Read the document from the app** shows what Submit will score. |
|
|
| Expert round trips (the ground truth written into the app and read back) score 1.000 in Penpot and Inkscape, |
| 0.975 to 0.999 in Impress, and 0.953 to 0.991 in GIMP, which has no mixed-colour text runs. |
|
|
| ## Reward |
|
|
| `0.60·structure + 0.30·visual + 0.05·order + 0.05·background` in reference mode. Structure matches elements |
| one-to-one (Hungarian) on content, geometry and typography; visual is multi-scale SSIM and colour layout against |
| `render(ground truth)`, normalised so an empty canvas scores 0. Description mode loosens geometry and adds |
| coverage. Replaying the ground truth scores exactly 1.000 on every task. Details: |
| [`core/score.py`](core/score.py) and the project README. |
|
|
| `DESIGN_CANVAS_REWARD=3` (or `grade(..., version=3)`) selects reward v3 for reference mode, calibrated against |
| a pairwise visual judge (Claude Opus 5.5, both orders) on frontier and Qwen attempts across all three surfaces: |
|
|
| - **Coarse-to-fine geometry.** Half the tight kernel, half a loose one (15% of the diagonal), scaled by √position, |
| so a roughly placed element in the right region earns credit, but 1.0 still needs exact placement. |
| - **Typography that multiplies.** Font and colour scale the rest of the style match, and style scales placement. |
| - **Copy counts triple in structure.** Every unmatched submitted element counts against the score. |
| - **Gated pixel terms.** Visual, order and background count in proportion to min(1, structure/0.3), so colour |
| patches and scattered dumps (~0.25 under v1) score ~0. |
|
|
| Results: |
|
|
| - **Validation:** agrees with the judge on 89.9% of 556 pairs vs 85.4% for v1 (+33/−8, p<0.001), and on 91.4% |
| of Qwen pairs. |
| - **Held-out test tasks:** 88.4% vs 84.2% (p=0.005), including 271 GUI pairs. |
| - **Description mode:** unchanged. |
|
|
| ## Layout |
|
|
| ``` |
| core/ document model, ops, fonts, skia renderer, reward, HTML profile, briefs, expert, Crello conversion |
| server/ OpenEnv environment, editor API + live sync, GUI driver (Playwright), Gradio playground, stdio MCP |
| realapps/ real apps behind the GUI surface: Penpot, the desktop agent (Impress, Inkscape, GIMP), Daytona pool |
| web/ the editor: vanilla JS, six layouts (Studio / Template / Pixel / Deck / Present / Vector), no build step |
| scripts/ parity.py (browser vs renderer vs Crello), gui_expert.py (scripted computer-use agent) |
| tests/ core, reward invariants over every task, OpenEnv contract, browser E2E, GUI agent |
| rollout.py real models on every surface (Anthropic, OpenAI, Hugging Face router) |
| ``` |
|
|
| Data: Crello is CDLA-Permissive-2.0; the templates belong to VistaCreate. The pack is built locally from the |
| dataset and not redistributed here. |
|
|
| ## Served corpus |
|
|
| 22,688 Crello designs (train 18,954, validation 1,817, test 1,917), read on demand from the [FineEnvs/crello-bucket](https://huggingface.co/buckets/FineEnvs/crello-bucket) bucket mounted at `/corpus`. Snapshot `aafdb0e86816e6af`. |
|
|
