DesignGym / Blog.md
yashvyasop's picture
Upload folder using huggingface_hub
17163ae verified

๐ŸŽจ DesignGym 2.0 โ€” Teaching an LLM to Think Like a Designer

What if a machine didn't just generate a layout โ€” but learned how to improve one?


The Big Idea

Imagine a blank canvas. On it, you can place anything โ€” a headline, a hero image, a call-to-action, a sofa, a warehouse shelf, a hospital bed, a circuit component. The problem of arrangement is universal.

DesignGym 2.0 asks a radical question:

Can we train an LLM to reason about space โ€” iteratively, purposefully, the way a designer does?

Not a one-shot generator. Not a template-picker. An agent that looks at a layout, diagnoses what's wrong, chooses a meaningful edit, and gets better over time through feedback.

That's what this project is. And it's bigger than posters.

DesignGym architecture Figure 1 โ€” End-to-end architecture: the OpenEnv-compliant DesignGym environment, the heuristic planner that bootstraps SFT data, the SFT adapter that locks in the action interface, and the GRPO trainer that learns design preference from verifiable reward.


๐ŸŒ The Vision: A Canvas Is a Universal Problem

Here's what hit me building this: graphic design is just one instance of a much deeper problem.

Every time humans arrange objects under constraints to serve a goal, they are solving a layout problem:

  • ๐Ÿ›‹๏ธ Interior Design โ€” furniture in a room, flow, light, balance
  • ๐Ÿญ Warehouse Planning โ€” shelf placement, picking efficiency, safety zones
  • ๐Ÿฅ Hospital Floor Plans โ€” patient flow, emergency access, quiet zones
  • ๐Ÿ’ป UI Dashboards โ€” information hierarchy, click efficiency, accessibility
  • ๐Ÿ–ฅ๏ธ Chip Placement (VLSI) โ€” routing, thermal zones (a billion-dollar optimization problem)
  • ๐Ÿซ Classroom Seating โ€” learning groups, teacher sightlines, collaborative zones
  • ๐Ÿช Retail Layouts โ€” foot traffic, product discovery, impulse zones
  • ๐Ÿ“ฐ Editorial Design โ€” reading order, visual weight, column balance
  • ๐Ÿ™๏ธ Urban Zoning โ€” density, transit access, green space allocation

The abstract structure is always the same:

objects+constraints+goalsโ†’optimized arrangement\text{objects} + \text{constraints} + \text{goals} \rightarrow \text{optimized arrangement}

DesignGym proves this framework works for graphic design. The architecture is a foundation for any domain where arrangement matters.


๐Ÿ”— Project Links

Resource Link
๐ŸŒ Environment (HF Space) DesignGym Environment Server
๐Ÿ’ป GitHub Repo canboyedits/DesignGym
๐Ÿง  SFT Trained Adapter designgym2-sft-qwen05-lora
๐Ÿ““ GRPO Training Notebook grpo_train_colab.ipynb
๐Ÿ““ SFT Training Notebook SFT_training_script.ipynb
๐Ÿ““ Evaluation Notebook evaluate_base_vs_sft.ipynb
๐Ÿ“Š Training Logs HF Training Job

๐Ÿค” Why Not Just Generate the Layout?

Because that's not how it works in the real world.

A good designer doesn't conjure the final poster from thin air. They start somewhere, step back, ask "what's wrong here?", make a targeted edit, look again, and repeat. The process is:

bad layout โ†’ structural choice โ†’ placement โ†’ refinement โ†’ polish โ†’ final design

DesignGym 2.0 formalizes that process as a reinforcement learning environment. The agent learns the workflow of design, not just the output.

This matters because layout improvement is an optimization problem, not a generation problem. And RL is the right tool for optimization.


๐Ÿ”ฌ Round 1 โ†’ Round 2: What Changed?

Area DesignGym Round 1 DesignGym 2.0
Core goal Optimize layouts with structured actions Learn the process of design over multiple phases
Task style Short-horizon layout refinement Long-horizon planning + instruction following
Agent behavior Local edit selection Structure โ†’ placement โ†’ refinement โ†’ polish
Reward Geometry + aesthetic deltas Delta-aware, instruction-aware, phase-aware
Learning Environment-ready foundation SFT + GRPO training pipeline
Evaluation Layout quality + validity Base vs heuristic vs SFT vs GRPO comparison

Round 1 answered: Can layout design be a real RL environment?

Round 2 answers: Can an LLM learn to act like a designer?


๐Ÿงฉ The Environment: How It Works

State

Each design is a canvas of normalized elements. Every element $b_i$ has geometry:

bi=(xi,โ€…โ€Šyi,โ€…โ€Šwi,โ€…โ€Šhi),where all valuesโˆˆ[0,1]b_i = (x_i,\; y_i,\; w_i,\; h_i), \quad \text{where all values} \in [0,1]

A layout is:

L={b1,b2,โ€ฆ,bn}L = \{b_1, b_2, \dots, b_n\}

The state given to the agent includes: task ID, step count, current score, best score so far, instruction score, phase score, metric vector, weakest metrics, focus elements, action history, full geometry, and current phase. The agent can reason not just about what the layout looks like but where it is in the design process.

The RL loop:

stโ€…โ€Šโ†’โ€…โ€Šatโ€…โ€Šโ€…โ€Šst+1,โ€…โ€Šrts_t \;\xrightarrow{\;a_t\;}\; s_{t+1},\; r_t

Actions

The action space spans both pixel-level and structural edits โ€” because real design isn't just moving boxes:

Primitive actions: move, resize, align, distribute, snap, swap_z

Structural actions: apply_template, promote, reflow_group, anchor_to_region, finalize

For example:

  • "Make the title louder" โ†’ promote
  • "Clean up the spacing" โ†’ distribute or reflow_group
  • "Move the CTA to the footer" โ†’ anchor_to_region
  • "Choose a stronger layout" โ†’ apply_template

Hard Constraints

The feasible layout space is:

F={LโˆฃL satisfies margin, size, ratio, and region constraints}\mathcal{F} = \{L \mid L \text{ satisfies margin, size, ratio, and region constraints}\}

Only $L \in \mathcal{F}$ are accepted. Invalid actions are rejected โ€” forcing the agent to learn legal creativity.


๐ŸŽฏ Solving Aesthetics: The Math of "Good Design"

This is where DesignGym gets serious. Rather than asking "does it look good?" (subjective), we ask "what makes it measurably good?" (computable).

My research into computational aesthetics โ€” from Birkhoff's 1933 aesthetic measure $M = O/C$ to modern Visual Moment Equilibrium models โ€” confirmed that most design quality signals are mathematically expressible. DesignGym implements seven of them:

The Composite Score

U(L)=โˆ‘k=1Kฮปkโ€‰gk(L),gk(L)โˆˆ[0,1],โ€…โ€Šโˆ‘kฮปk=1U(L) = \sum_{k=1}^{K} \lambda_k \, g_k(L), \quad g_k(L) \in [0,1],\; \sum_k \lambda_k = 1

Each $g_k$ is an independent, interpretable aesthetic signal. Together they give a holistic quality score $U(L) \in [0,1]$.


1. Overlap โ€” Do elements collide?

goverlap(L)=expโกโ€‰โฃ(โˆ’โˆ‘i<jI(bi,bj)โˆ‘iai+ฯต),ai=wihig_{\text{overlap}}(L) = \exp\!\left(-\frac{\sum_{i<j} \mathrm{I}(b_i,b_j)}{\sum_i a_i + \epsilon}\right), \quad a_i = w_i h_i

Exponential penalty for any intersection. Clean layouts score near 1.


2. Alignment โ€” Do things line up intentionally?

galign(L)=1โˆฃQโˆฃโˆ‘qโˆˆQexpโกโ€‰โฃ(โˆ’dqฯ„align)g_{\text{align}}(L) = \frac{1}{|\mathcal{Q}|} \sum_{q \in \mathcal{Q}} \exp\!\left(-\frac{d_q}{\tau_{\text{align}}}\right)

$d_q$ is the distance from element anchors (left, center, right edges) to the nearest alignment guide. Lower distance โ†’ higher score.


3. Spacing โ€” Is the rhythm intentional?

Coefficient of variation on gaps $\Delta$:

CV(ฮ”)=ฯƒ(ฮ”)ฮผ(ฮ”)+ฯต,gspacing(L)=expโกโ€‰โฃ(โˆ’CV(ฮ”)ฯ„space)\mathrm{CV}(\Delta) = \frac{\sigma(\Delta)}{\mu(\Delta) + \epsilon}, \qquad g_{\text{spacing}}(L) = \exp\!\left(-\frac{\mathrm{CV}(\Delta)}{\tau_{\text{space}}}\right)

Low variation = consistent rhythm = high score. This captures the difference between accidental spacing and deliberate structure.


4. Balance โ€” Does visual weight feel stable?

Each element carries visual mass $m_i = a_i \cdot p_i$ (area ร— semantic importance). The center of mass:

cx=โˆ‘imixi(c)โˆ‘imi,cy=โˆ‘imiyi(c)โˆ‘imic_x = \frac{\sum_i m_i x_i^{(c)}}{\sum_i m_i}, \qquad c_y = \frac{\sum_i m_i y_i^{(c)}}{\sum_i m_i}

Balance rewards layouts centered around the canvas midpoint:

gbalance(L)=expโกโ€‰โฃ(โˆ’(cxโˆ’0.5)2+(cyโˆ’0.5)2ฯ„bal)g_{\text{balance}}(L) = \exp\!\left(-\frac{\sqrt{(c_x - 0.5)^2 + (c_y - 0.5)^2}}{\tau_{\text{bal}}}\right)

Inspired by the Visual Moment Equilibrium model โ€” treating layout as a physics problem where visual torques must cancel. This achieves Pearson $r = 0.942$ correlation with human perception benchmarks.


5. Hierarchy โ€” Do big things matter more?

Visual salience per element:

ฮถi=ฮฑlogโก(ai+ฯต)โˆ’ฮฒyi+ฮณfi+ฮดzi\zeta_i = \alpha \log(a_i + \epsilon) - \beta y_i + \gamma f_i + \delta z_i

Hierarchy is measured by rank agreement with semantic importance $p_i$ via Spearman correlation:

ghier(L)=1+ฯS(p,โ€‰ฮถ)2g_{\text{hier}}(L) = \frac{1 + \rho_S(p,\, \zeta)}{2}

If the biggest element is also the most important, the layout looks right because it is right.


6. Reading Order โ€” Does the eye flow correctly?

Given required reading relations $\mathcal{R} = {(i,j)}$:

gread(L)=1โˆฃRโˆฃโˆ‘(i,j)โˆˆR1{iโ‰บj}g_{\text{read}}(L) = \frac{1}{|\mathcal{R}|} \sum_{(i,j)\in\mathcal{R}} \mathbf{1}\{i \prec j\}

$i \prec j$ means element $i$ appears before $j$ in the intended scan path (F-pattern, Z-pattern, etc.).


7. Instruction Fit โ€” Does the layout follow the brief?

Tasks have semantic placement goals (CTA in lower-right, masthead at top, etc.). Given target region center $r_i$ and element center $c_i$:

gintent(L)=1nโˆ‘iexpโกโ€‰โฃ(โˆ’โˆฅciโˆ’riโˆฅฯ„intent)g_{\text{intent}}(L) = \frac{1}{n} \sum_i \exp\!\left(-\frac{\|c_i - r_i\|}{\tau_{\text{intent}}}\right)

This is the bridge from pure geometry to design intention โ€” an element can be perfectly aligned yet still be in the wrong place.


๐Ÿ† The Reward Function: Teaching Taste

This is the most important design decision in the whole project. A flat reward teaches nothing.

After SFT, nearly every action is valid JSON โ€” so rewarding "valid JSON" gives GRPO a flat signal with no learning gradient. The reward had to be richer, more informative, and process-aware.

Delta-Sensitive Shaping

Let $S_t$, $I_t$, $P_t$, $B_t$ be layout score, instruction score, phase score, and best-so-far score at step $t$:

ฮ”St=Stโˆ’Stโˆ’1,ฮ”It=Itโˆ’Itโˆ’1,ฮ”Bt=maxโก(0,โ€‰Btโˆ’Btโˆ’1)\Delta S_t = S_t - S_{t-1}, \qquad \Delta I_t = I_t - I_{t-1}, \qquad \Delta B_t = \max(0,\, B_t - B_{t-1})

With a clipped scaling function that makes small improvements legible:

ฯ•(ฮ”,ฯ„)=clipโ€‰โฃ(ฮ”ฯ„,โ€…โ€Šโˆ’1,โ€…โ€Š1)\phi(\Delta, \tau) = \mathrm{clip}\!\left(\frac{\Delta}{\tau},\; -1,\; 1\right)

The full shaped reward:

rt=clipโ€‰โฃ(0.03โŸbase+0.35โ€‰ฯ•(ฮ”St,โ€…โ€Š0.015)โŸlayout quality+0.35โ€‰ฯ•(ฮ”It,โ€…โ€Š0.020)โŸinstruction fit+0.15โ€‰ฯ•(ฮ”Pt,โ€…โ€Š0.025)โŸphase progress+0.10โ€‰ฯ•(ฮ”Bt,โ€…โ€Š0.015)โŸbest score+0.05โ€‰renvโŸenv signal+bprocessโˆ’ฯ€t,โ€…โ€Šโˆ’1,โ€…โ€Š1)r_t = \mathrm{clip}\!\Big(\underbrace{0.03}_{\text{base}} + \underbrace{0.35\,\phi(\Delta S_t,\; 0.015)}_{\text{layout quality}} + \underbrace{0.35\,\phi(\Delta I_t,\; 0.020)}_{\text{instruction fit}} + \underbrace{0.15\,\phi(\Delta P_t,\; 0.025)}_{\text{phase progress}} + \underbrace{0.10\,\phi(\Delta B_t,\; 0.015)}_{\text{best score}} + \underbrace{0.05\,r_{\text{env}}}_{\text{env signal}} + b_{\text{process}} - \pi_t,\; -1,\; 1\Big)

Where $b_{\text{process}}$ is a phase-aware bonus and $\pi_t$ is a penalty for invalid, useless, or premature actions.

Process-Aware Reward: Diagnosing Before Acting

The reward checks not just whether the layout improved, but whether the agent chose the right kind of action for the current weakness:

Weak Metric Preferred Actions
Overlap / crowding move, resize, reflow_group
Spacing / rhythm distribute, align, reflow_group
Hierarchy promote, resize, apply_template
Instruction fit anchor_to_region, promote
Early structure phase apply_template
Late polish phase align, snap, finalize

This teaches the agent to diagnose โ†’ choose relevantly โ†’ improve โ€” not just make random valid edits.

Early Finalize Penalty

The agent must not give up early. finalize is only rewarded if:

Stโ‰ฅ0.82andItโ‰ฅ0.72S_t \geq 0.82 \quad \text{and} \quad I_t \geq 0.72

Otherwise $\pi_{\text{finalize}} > 0$. This enforces long-horizon behavior and prevents the agent from prematurely declaring victory.


๐ŸŽ“ Training Pipeline: From Language to Design Policy

Base Model  โ†’  Heuristic Planner  โ†’  SFT Model  โ†’  GRPO Model

Heuristic planner output Figure 2 โ€” The heuristic planner solving an episode end-to-end. It diagnoses the weakest metric, picks a phase-appropriate action, applies it, and re-scores โ€” generating the (state โ†’ action) pairs we use as SFT training data. This is how the agent gets a warm start without human labels.

Why SFT First?

The base model (Qwen 0.5B) understands design language but cannot speak the environment's action format. It says things like:

"Move the title higher and make the CTA stronger."

Semantically meaningful โ€” but not executable. The environment needs:

{
  "action_type": "anchor_to_region",
  "element_id": "cta",
  "region": "lower_right"
}

SFT teaches the model the interface. The goal is not creativity โ€” it's action correctness.

Result: valid_json_rate: 0% โ†’ valid_json_rate: 100%. That's not a fine-tune. That's a capability phase transition.

SFT training metrics Figure 3 โ€” SFT training evidence on Qwen-0.5B: loss converges cleanly, validity rates jump to 100%, and the action distribution learned by the model mirrors the heuristic teacher. After SFT the model can act in DesignGym; before SFT it cannot.

Why GRPO After?

Once the model can act, GRPO teaches it which valid action is better.

GRPO uses environment reward directly โ€” no static labels, no human annotation on every trajectory. The loop:

  1. Sample multiple candidate actions
  2. Execute them in DesignGym
  3. Score each with the verifier
  4. Increase probability of higher-reward actions
  5. Decrease probability of lower-reward actions

This works because DesignGym has verifiable rewards โ€” the environment is the oracle. No reward model needed.

The learning story:

Language Modelโ€…โ€Šโ†’โ€…โ€ŠSFTโ€…โ€Šโ€…โ€ŠAction Modelโ€…โ€Šโ†’โ€…โ€ŠGRPOโ€…โ€Šโ€…โ€ŠDesign Policy\text{Language Model} \;\xrightarrow{\;\text{SFT}\;}\; \text{Action Model} \;\xrightarrow{\;\text{GRPO}\;}\; \text{Design Policy}


๐Ÿ“Š Results: Evidence of Learning

Policy Final Score Instruction Score Valid JSON Early Finalize
Base Qwen 0.5B 0.6948 0.5360 0% 100%
SFT Qwen 0.5B 0.7101 0.6263 100% 0%
SFT @ Best-of-4 0.7057 0.6672 100% 0%
GRPO Qwen 0.5B 0.6717 0.5483 98% 67%
GRPO @ Best-of-4 โ˜… 0.6781 0.5817 100% 17%

โ˜… = shipped headline configuration. Baseโ†’Final pipeline gain on instruction score: +8.5%, on valid JSON: 0% โ†’ 100%, on premature finalizes: down 83%.

The most important result is the jump from 0% to 100% valid actions. That's the model crossing from "knows about design" to "can act in a design environment."

GRPO then learns which valid actions are better โ€” a harder problem that continues to improve with more training. The infrastructure and reward signal are proven.

Finetuned output


๐Ÿ› ๏ธ Tasks Supported

poster_basic_v1 โ€” Headline hierarchy, hero image placement, CTA placement, clean spacing, reading order.

editorial_cover_v1 โ€” Masthead preservation, headline stack, visual balance, editorial hierarchy.

dense_flyer_v1 โ€” Crowded layout repair, support group reflow, spacing under density, caption alignment.

Three very different design problems. A model that improves across all three has learned general layout reasoning, not template memorization.


๐Ÿ”ฎ How Big Can This Get?

The framework underlying DesignGym isn't about posters. It's about this general structure:

U(L)=ฮป1โ€‰gefficiency+ฮป2โ€‰gsafety+ฮป3โ€‰gaccessibility+ฮป4โ€‰gaestheticU(L) = \lambda_1\, g_{\text{efficiency}} + \lambda_2\, g_{\text{safety}} + \lambda_3\, g_{\text{accessibility}} + \lambda_4\, g_{\text{aesthetic}}

Swap the metrics. Swap the domain. The agent still learns to optimize arrangement through sequential decision-making.

Interior Design: empty room โ†’ zoning โ†’ furniture placement โ†’ spacing โ†’ style polish

VLSI Chip Placement: component placement โ†’ routing โ†’ thermal zones โ†’ signal integrity

Warehouse Logistics: shelf placement โ†’ picking paths โ†’ safety zones โ†’ density optimization

Hospital Layout: patient flow โ†’ emergency access โ†’ staff circulation โ†’ quiet zones

UI Design: information hierarchy โ†’ click paths โ†’ accessibility โ†’ responsive stability

The same RL loop. A different reward function. The canvas is always the problem, and the canvas is everywhere.


๐Ÿ—๏ธ Architecture

.
โ”œโ”€โ”€ client.py                      # OpenEnv client interface
โ”œโ”€โ”€ inference.py                   # Policy inference runner
โ”œโ”€โ”€ models.py                      # Typed action/observation models
โ”œโ”€โ”€ notebooks/
โ”‚   โ”œโ”€โ”€ grpo_train_colab.ipynb    # GRPO training โ† start here
โ”‚   โ”œโ”€โ”€ SFT_training_script.ipynb # SFT training
โ”‚   โ””โ”€โ”€ evaluate_base_vs_sft_designgym2.ipynb
โ”œโ”€โ”€ training/
โ”‚   โ”œโ”€โ”€ grpo_train.py              # GRPO training script
โ”‚   โ”œโ”€โ”€ generate_sft_data.py       # SFT data generation
โ”‚   โ””โ”€โ”€ run_grpo.sh                # Smoke test runner
โ”œโ”€โ”€ server/
โ”‚   โ”œโ”€โ”€ app.py                     # FastAPI wrapper
โ”‚   โ”œโ”€โ”€ DesignGym_environment.py   # Core environment logic
โ”‚   โ”œโ”€โ”€ rewards.py                 # Reward functions
โ”‚   โ”œโ”€โ”€ phases.py                  # Phase-aware logic
โ”‚   โ””โ”€โ”€ requirements.txt
โ”œโ”€โ”€ data/sft/                      # SFT train/eval data
โ”œโ”€โ”€ assets/                        # Plots and diagrams
โ”œโ”€โ”€ web/                           # Demo web UI
โ”œโ”€โ”€ Dockerfile
โ”œโ”€โ”€ pyproject.toml
โ”œโ”€โ”€ openenv.yaml                   # OpenEnv manifest
โ””โ”€โ”€ README.md

โšก OpenEnv Compatibility

DesignGym is fully OpenEnv-compatible: reset(), step(action), typed models, FastAPI server, Docker + HF Space deployment, deterministic seeded episodes.


๐Ÿš€ Running It

Install:

pip install -e .
openenv validate
python server/app.py

Docker:

docker build -t designgym-env .
docker run --rm -p 8000:8000 designgym-env

Inference:

HF_TOKEN=your_token \
API_BASE_URL=https://router.huggingface.co/v1 \
MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct:scaleway \
DESIGNGYM_TASK=poster_basic_v1 \
python inference.py

๐Ÿงช What's Next

  • Image-Aware Reward โ€” Saliency maps, OCR regions, protected focal zones
  • Preference-Trained Reward Model โ€” Learn from human pairwise layout comparisons
  • Style-Conditioned Design โ€” Luxury, editorial, minimal, corporate as goal inputs
  • Interior DesignGym โ€” Same framework extended to room/furniture layout
  • Multi-Agent Critique โ€” Designer agent proposes โ†’ Critic agent diagnoses โ†’ Designer improves
  • Multi-Page Layout โ€” Magazines, slide decks, reports, product catalogs

๐Ÿ’ก The Bottom Line

Design is usually treated as subjective and unteachable by machine.

DesignGym proves that's not true. Many aspects of layout quality are computable, verifiable, and learnable. And because the underlying framework is general, the same approach extends to any domain where arrangement matters.

The final vision: models that don't just generate designs โ€” but learn how to improve them, step by step, the way a skilled designer does.

A good designer does not just produce a layout.
A good designer repeatedly diagnoses, edits, compares, and improves.

DesignGym 2.0 teaches that process to a machine.


Built for the OpenEnv Hackathon ยท India 2026
Math rendered with KaTeX / MathJax