README.md

metadata

title: DesignGym
emoji: 🎨
colorFrom: indigo
colorTo: pink
sdk: docker
app_port: 8000
base_path: /web
pinned: false
tags:
  - openenv
  - mcp
  - computer-use
  - graphic-design
  - crello
short_description: Recreate real graphic designs via MCP, HTML or a real editor

DesignGym

An OpenEnv environment for rebuilding real graphic designs from their parts. Each task is a Crello template taken apart into its pre-rendered layers and its typed copy. The agent puts it back together on a canvas and is scored against a render of the original.

The package, the CLI, the OpenEnv name and the DESIGN_CANVAS_* settings keep the id design_canvas (design-canvas). Clients, datasets and traces already use it.

• Two tasks. mode="reference": here is the target image, recreate it. mode="description": here is a written brief naming every asset and line of copy, build it.
• Three surfaces over one document. surface="tools": structured MCP edits. surface="html": one page of restricted HTML. surface="gui": mouse and keyboard on a real browser editor, seeing only screenshots.
• One renderer, one reward. Every surface edits the same document, and every score comes from the same skia renderer, so a GUI episode and a tool-call episode are directly comparable.

Run it

design-canvas prepare             # build the task pack from Crello (~1.4 GB download, 30 s to convert)
design-canvas serve --port 8007   # OpenEnv server + editor + playground

• Editor: http://127.0.0.1:8007/editor/ · Playground: /web/ · MCP: /mcp · Task API: /design_canvas/...
• Or docker build -t design-canvas . && docker run -p 8000:8000 design-canvas (builds the pack into the image).

The pack is a small offline sample. To serve the whole corpus (every playable Crello design) from the FineEnvs/crello-bucket bucket, point the server at the corpus manifest:

DESIGN_CANVAS_CORPUS=corpus-manifest.json design-canvas serve --port 8007

The catalog (every task's ground truth, brief and difficulty) is one SQLite file copied from the bucket at startup, so listing tasks and reset never download images. A design's assets are fetched the first time an episode uses it: its 100-design row group is read (over HTTP locally, from the read-only /corpus mount on the Space), checked against the indexed record, and cached on disk under a byte budget (DESIGN_CANVAS_CACHE). Set DESIGN_CANVAS_SOURCE_ROOT to a mounted or downloaded copy of the bucket to read it as local files.

design-canvas corpus build --out artifacts/corpus-index --workers 12   # convert + index every design
design-canvas corpus publish --out artifacts/corpus-index               # upload the catalog to the bucket
design-canvas evalset build --out ../../evalsets/crello-test-300.json   # freeze the evaluation set
design-canvas evalset export ../../evalsets/crello-test-300.json --out artifacts/evalpack  # offline copy

Computer-use data: record verified GUI trajectories, then write them as a Hub-ready cua-v1 dataset (cua_format.py: OpenAI-style messages with pixel tool calls, embedded screenshots, per-message trainable; windows() derives SFT rows with the same screenshot window the Tinker RL adapter uses):

design-canvas export-cua --split train --out artifacts/cua --limit 200          # record (webp screenshots)
design-canvas export-hf --cua artifacts/cua/cua.jsonl --out artifacts/hf/cua    # cua-v1 Parquet + dataset card

Check a running server end to end (HTTP routes, target gating, expert replays scoring 1.0 on tools and HTML in both modes, an empty submission scoring 0, concurrent episodes, and with --gui two verified recorder episodes; the client must load the same task source as the server):

DESIGN_CANVAS_CORPUS=artifacts/corpus-index .venv/bin/python -m design_canvas.verify_server --server http://127.0.0.1:8007 --gui

Watch Claude work, live:

claude mcp add design-canvas -- design-canvas mcp

then ask it to "start a design task and recreate it", and open the editor_url that start_task returns.

For more throughput, design-canvas serve --workers N (or DESIGN_CANVAS_WORKERS=N) runs N server processes behind one URL (server/router.py). Don't use uvicorn's own --workers: OpenEnv's client creates a session over HTTP and then attaches a WebSocket to it, and the built-in editor's page calls back for its episode, so both must reach the process that holds the state. The router keeps every session and episode on its worker. Measured locally (scripts/loadtest.py, 4 workers, no errors):

• tools: about 2,900 episodes per minute, 3× one process;
• built-in GUI editor: 128 concurrent episodes at about 190 actions per second, 2× one process.

The router itself saturates one core at about 300–450 tool calls per second, so 4–6 workers is the useful range.

On the private Space FineEnvs/DesignGym (scripts/deploy_space.py: cpu-upgrade, 8 vCPU / 32 GB, 4 workers × 48 sessions, the full corpus read over HTTP), tested from India, where the network round trip is about 0.22 s:

Load	Result	Throughput	Step p50 / p95
Tools, 192 concurrent	575/576 ok	643 episodes/min, 110 calls/s	0.26 / 0.8 s
Built-in GUI editor, 128 concurrent	256/256 ok	75 actions/s	0.46 / 1.3 s

• The one failure arrived at the capacity edge, just before a closing session was freed.
• At capacity, cold resets take 7–10 s; each new design is fetched once.
• More than 192 sessions are refused cleanly ("Server at capacity") after every worker has been tried.
• OpenEnv's client has no option for auth headers. scripts/loadtest.py --hf-token-env HF_TOKEN shows the two wrappers a client needs to reach a private Space.

Talk to it

from openenv.core.mcp_client import MCPToolClient
from openenv.core.env_server.mcp_types import CallToolAction

with MCPToolClient(base_url="http://127.0.0.1:8007").sync() as env:
    obs = env.reset(split="test", index=3, mode="reference", surface="tools")
    print(obs.observation.metadata["instructions"])     # the task, as the model should see it
    tools = env.list_tools()                           # depends on the surface
    env.step(CallToolAction(tool_name="add_image", arguments={"asset_id": "a2", "x": 0, "y": 0, "width": 1080}))
    out = env.step(CallToolAction(tool_name="submit", arguments={}))
    print(out.reward, out.observation.metadata["grade"])

reset arguments: split, index or task_id, mode (reference | description), surface (tools | html | gui), skin (figma | canva | photoshop | slides | powerpoint | inkscape editor layouts; random per seed; or a real app, below), theme, file_dialog (mac | windows | linux: the look of the file window images are opened from; random per seed), coordinates (pixels, or normalized for a 0-1000 grid), max_steps, reward_mode (final, or dense for per-edit score deltas that sum to the final score).

The Task API lists tasks without giving them away: GET /design_canvas/splits, POST /design_canvas/num_tasks, /task, /task_range.

Tools

Surface	Tools
all	get_task, view_reference, submit, start_task (for MCP clients with no reset)
tools, html	view_canvas, view_assets (a labelled contact sheet), list_fonts
tools	add_image, add_text, add_shape, update_element, delete_element, duplicate_element, reorder_element, align_elements, set_background, list_elements, undo, redo
html	write_html, read_html
gui	screenshot, left_click, double_click, triple_click, right_click, mouse_move, left_click_drag, scroll, type, key, wait

Images come back as MCP image content blocks, so a vision model sees pixels, not base64.

Real apps

The GUI surface can also put the model in a real design app instead of the built-in editor: skin="penpot" (Penpot in the env's browser), "impress" (LibreOffice Impress), "inkscape-desktop" or "gimp" (on a Linux desktop). The tools, the prompt's task and the reward are the same. At reset the env sets the app up with a blank canvas at the task's size and the task's assets loaded. On submit it reads the app's own document back (Penpot's file, the slide, the SVG, the layers) as a design document and scores it like any other.

The client always talks to one env server, and that server runs the apps:

• Locally. Run Penpot with realapps/penpot/docker-compose.yaml and the desktop image from realapps/desktop/. The server finds them at DESIGN_CANVAS_PENPOT_URL and DESIGN_CANVAS_DESKTOP_URLS.
• On Daytona. Set DESIGN_CANVAS_APPS=daytona and DAYTONA_API_KEY. The server keeps its own pool of private sandboxes (realapps/daytona.py): one per running desktop episode (DESIGN_CANVAS_DAYTONA_DESKTOPS, default 2, built from a snapshot of the desktop image) and one Penpot sandbox shared by all Penpot episodes. It reaches them through their preview URLs with the access token in a header, and the browser may load that app and nothing else. Sandboxes are reused across server restarts and stop when idle. python -m design_canvas.realapps.daytona up | status | down starts, lists or deletes them.

A real-app episode's reset observation carries app ({"skin", "name", "real": true}), and its editor_url is the app's live screen (/api/episodes/{id}/screen.png). GET /api/apps lists the apps and where they run (?check=1 also asks each endpoint or sandbox whether it is up). In the /web playground, pick a real app under Editor: the Editor tab shows its screen, and a click on the screen is a click in the app. The Expert tab writes the target into the app, and Read the document from the app shows what Submit will score.

Expert round trips (the ground truth written into the app and read back) score 1.000 in Penpot and Inkscape, 0.975 to 0.999 in Impress, and 0.953 to 0.991 in GIMP, which has no mixed-colour text runs.

Reward

0.60·structure + 0.30·visual + 0.05·order + 0.05·background in reference mode. Structure matches elements one-to-one (Hungarian) on content, geometry and typography; visual is multi-scale SSIM and colour layout against render(ground truth), normalised so an empty canvas scores 0. Description mode loosens geometry and adds coverage. Replaying the ground truth scores exactly 1.000 on every task. Details: core/score.py and the project README.

DESIGN_CANVAS_REWARD=3 (or grade(..., version=3)) selects reward v3 for reference mode, calibrated against a pairwise visual judge (Claude Opus 5.5, both orders) on frontier and Qwen attempts across all three surfaces:

• Coarse-to-fine geometry. Half the tight kernel, half a loose one (15% of the diagonal), scaled by √position, so a roughly placed element in the right region earns credit, but 1.0 still needs exact placement.
• Typography that multiplies. Font and colour scale the rest of the style match, and style scales placement.
• Copy counts triple in structure. Every unmatched submitted element counts against the score.
• Gated pixel terms. Visual, order and background count in proportion to min(1, structure/0.3), so colour patches and scattered dumps (~0.25 under v1) score ~0.

Results:

• Validation: agrees with the judge on 89.9% of 556 pairs vs 85.4% for v1 (+33/−8, p<0.001), and on 91.4% of Qwen pairs.
• Held-out test tasks: 88.4% vs 84.2% (p=0.005), including 271 GUI pairs.
• Description mode: unchanged.

Layout

core/      document model, ops, fonts, skia renderer, reward, HTML profile, briefs, expert, Crello conversion
server/    OpenEnv environment, editor API + live sync, GUI driver (Playwright), Gradio playground, stdio MCP
realapps/  real apps behind the GUI surface: Penpot, the desktop agent (Impress, Inkscape, GIMP), Daytona pool
web/       the editor: vanilla JS, six layouts (Studio / Template / Pixel / Deck / Present / Vector), no build step
scripts/   parity.py (browser vs renderer vs Crello), gui_expert.py (scripted computer-use agent)
tests/     core, reward invariants over every task, OpenEnv contract, browser E2E, GUI agent
rollout.py real models on every surface (Anthropic, OpenAI, Hugging Face router)

Data: Crello is CDLA-Permissive-2.0; the templates belong to VistaCreate. The pack is built locally from the dataset and not redistributed here.

Served corpus

22,688 Crello designs (train 18,954, validation 1,817, test 1,917), read on demand from the FineEnvs/crello-bucket bucket mounted at /corpus. Snapshot aafdb0e86816e6af.