README.md

---
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
- designcoder
- ui-generation
- front-end
- html
- css
- javascript
- code-generation
- full-sft
---

# DesignCoder

Checkpoint collection for **DesignCoder**, a family of full-parameter SFT models for UI design
research and end-to-end HTML/CSS/JavaScript implementation.

Each subfolder in this repository is a self-contained, directly loadable checkpoint.

## Naming convention

```
designcoder_{basemodel}_{size}_{optimizer}_bs{global_batch}[_{extra_axes}]_step{global_step}
```

- `basemodel` / `size`: base model family and parameter scale
- `optimizer`: `muon` or `adamw`
- `bs`: global batch size (`per_device × grad_accum × world_size`)
- `extra_axes`: any hyper-parameter that deviates from the default recipe, e.g. `wd0.05`
  (weight decay, default 0.0), `ep20` (epochs, default 2), or `data41287` (dataset revision)
- `step`: trainer `global_step` of the exported weights

## Dataset revisions

Checkpoints in this repository come from two different dataset revisions. **Scores and loss
values are only comparable within the same revision.**

| Tag | Samples | Used by |
|---|---:|---|
| *(untagged)* `data37865` | 37,865 | `*_step1900`, `*_step3800` |
| `data41287` | 41,287 | `*_data41287_step200`, `*_data41287_step400` |

## Checkpoints

| Subfolder | Base model | Optimizer | LR | Global batch | Dataset | Step | bench-200 V5.9.2 (n=200) | Notes |
|---|---|---|---|---|---|---|---|---|
| `designcoder_qwen3.5_4b_muon_bs32_step1900` | Qwen3.5-4B | Muon | 1e-5 | 32 | 37,865 | 1900 | – | smallest of the first release |
| `designcoder_qwen3.5_9b_muon_bs16_step3800` | Qwen3.5-9B | Muon | 1e-5 | 16 | 37,865 | 3800 | – | optimizer ablation (Muon arm) |
| `designcoder_qwen3.5_9b_adamw_bs16_step3800` | Qwen3.5-9B | AdamW | 2e-5 | 16 | 37,865 | 3800 | – | optimizer ablation (AdamW arm) |
| `designcoder_qwen3.6_27b_adamw_bs32_step1900` | Qwen3.6-27B | AdamW | 1e-5 | 32 | 37,865 | 1900 | – | largest of the first release |
| `designcoder_qwen3.5_4b_adamw_bs256_data41287_step200` | Qwen3.5-4B | AdamW | 2e-5 | 256 | 41,287 | 200 | 80.06 | best 4B / AdamW |
| `designcoder_qwen3.5_4b_muon_bs256_data41287_step200` | Qwen3.5-4B | Muon | 2e-5 | 256 | 41,287 | 200 | 71.34 | best 4B / Muon |
| `designcoder_qwen3.5_9b_adamw_bs256_data41287_step200` | Qwen3.5-9B | AdamW | 2e-5 | 256 | 41,287 | 200 | **82.14** | best 9B |
| `designcoder_qwen3.8_27b_adamw_bs128_data41287_step400` | Qwen3.8-27B | AdamW | 1e-5 | 128 | 41,287 | 400 | **87.04** | strongest checkpoint in the collection |

All four `data41287` scores are **final full-benchmark runs: 200/200 rollouts, 200/200
screenshot captures, 200/200 judge evaluations** per model, scored with the **complete
V5.9.2 rubric set (all three families)** — no subsetting, no omitted rubric family.

## Benchmark

`bench-200` is the frozen 200-case DesignCoder benchmark (100 Track A landing, 40 Track A
dashboard, 30 Track B landing, 30 Track B dashboard; Track A cases specify a style, Track B
cases are style-free).

**Rubric composition.** Scoring uses three independent rubric families. Only the first varies
per case; the other two are fixed for every case of a given surface.

| Family | Scope | Size | Scale |
|---|---|---|---|
| **Frozen** | per case | 23–25 checks/case, 4,983 total (184 cases carry 25, 15 carry 24, 1 carries 23) | binary 0/1 |
| **Prompt Fit & Product** | fixed per surface | 5 rubrics | 0/1/2 |
| **Static** | fixed per surface | landing 27 `d_*` + 8 `q_*`; dashboard 25 `d_*` + 9 `q_*` | `d_*` 2/0/N-A, `q_*` 0/1/2/N-A |

Frozen checks are distributed across six dimensions: Components 1,517 (30.4%), Layout 842
(16.9%), Aesthetics 782 (15.7%), Typography 642 (12.9%), Alignment 616 (12.4%), Assets 584
(11.7%). Every check is `check_with=screenshot`.

`overall_score` (0–100) is the unweighted mean of top-level slots: Prompt Fit (1 slot), each
active Static dimension (1 slot each), and the whole Frozen family (1 slot). Aggregation is
performed by the reference collector `DesignEvaluator-Skill/scripts/collect_unified.py`, not
by a reimplementation. Judge: `gpt-5.6-sol` (vision) with structured JSON output.

### Full-run results (n=200 per model)

| Model | **Overall (V5.9.2)** | Prompt Fit | Static | Frozen | Landing | Dashboard | Track A | Track B | Render fails |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 27B AdamW step400 | **87.04** | 84.10 | 87.31 | 88.52 | 89.65 | 82.19 | 86.56 | 88.16 | 0/200 |
| 9B AdamW step200 | **82.14** | 78.35 | 82.20 | 85.41 | 86.96 | 73.19 | 82.71 | 80.82 | 1/200 |
| 4B AdamW step200 | **80.06** | 75.25 | 80.40 | 83.27 | 85.70 | 69.59 | 80.61 | 78.78 | 1/200 |
| 4B Muon step200 | **71.34** | 64.60 | 70.96 | 79.79 | 78.48 | 58.06 | 72.05 | 69.68 | 3/200 |

Scores are monotone in scale. Five of the six pairwise differences are significant (paired
bootstrap 10k-resample 95% CI excludes 0 and Wilcoxon p < 4e-7); **9B vs 4B AdamW is not
significant** (mean diff +2.08, CI [-0.03, 4.18], p = 0.065). See
[`eval/significance_tests_v592.json`](./blob/main/eval/significance_tests_v592.json).

Two observations that only the full rubric set exposes:

- **Dashboards are the bottleneck, and they degrade faster than landings.** The 27B loses 7.5
  points moving from landing to dashboard; the 4B Muon loses 20.4. The Static family's
  chart (`d_data_*`) and workflow (`d_work_*`) checks catch empty or non-functional charts
  that the Frozen checks largely miss.
- **The Frozen family alone compresses the ranking.** Across the four models Frozen spans only
  8.7 points (88.52 → 79.79) while Prompt Fit spans 19.5 and Static spans 16.4. Reporting
  Frozen-heavy scores therefore understates the gap between scales.

### Evaluation artifacts (`eval/`)

| File | Content |
|---|---|
| [`eval/benchmark_summary_v592.csv`](./blob/main/eval/benchmark_summary_v592.csv) | per-model aggregates: overall, three family scores, Track/Surface splits, Static and Frozen dimensions |
| [`eval/benchmark_per_case_v592.csv`](./blob/main/eval/benchmark_per_case_v592.csv) | long-form per-case scores (overall + three families) for all 4 models × 200 cases |
| [`eval/significance_tests_v592.json`](./blob/main/eval/significance_tests_v592.json) | paired bootstrap (10k resamples) + Wilcoxon signed-rank for all 6 model pairs |
| [`eval/rubric_stats.json`](./blob/main/eval/rubric_stats.json) | composition of all three rubric families and the aggregation rule |
| [`eval/reports.html`](./blob/main/eval/reports.html) | self-contained interactive HTML report: model comparison, dimension heatmap, score distributions, per-case tables |

### Checkpoint selection

The `data41287` checkpoints were selected by **running the benchmark**, not by taking the
lowest training loss. In all four runs the best checkpoint sits at roughly 75% of training,
and loss kept improving while benchmark scores fell. The table below shows the **8-case
selection subset** (used only to rank checkpoints, not comparable to the final full-run
numbers in the tables above):

| Run | Step | Train loss | subset bench (n=8, Frozen+Prompt-Fit only) | final V5.9.2 (n=200) |
|---|---:|---:|---:|---:|
| 4B AdamW | 200 | 0.2696 | 84.22 | **80.06** |
| 4B AdamW | 266 | 0.2682 | 68.35 | – |
| 4B Muon | 200 | 0.3339 | 83.36 | **71.34** |
| 4B Muon | 266 | 0.3349 | 81.27 | – |
| 9B AdamW | 200 | 0.2518 | 84.40 | **82.14** |
| 9B AdamW | 266 | 0.2504 | lowest of the three | – |
| 27B AdamW | 400 | 0.2067 | 91.19 | **87.04** |
| 27B AdamW | 530 | 0.2059 | 86.37 | – |

The 4B AdamW pair is the clearest example: loss improved from 0.2696 to 0.2682 while the
subset score collapsed from 84.22 to 68.35. **Do not pick checkpoints from this family by
loss.**

The two score columns are not comparable: the selection subset used 8 cases and only two of
the three rubric families, and it overestimates by 4–12 points, with the largest error on the
weakest model. It is reliable enough to rank checkpoints within a run, which is all it was
used for — every number reported elsewhere in this card is the full 200-case V5.9.2 score.

## Shared training setup

- Objective: full-parameter supervised fine-tuning (no LoRA / adapters)
- Dataset: `designcoder_sft_v2_train` in ShareGPT format (see revision table above)
- Chat template: `qwen3_5` with thinking enabled
- Context length: 32,768
- Sequence packing: enabled, with neat packing (no cross-sample attention)
- LR schedule: cosine, warmup ratio 0.1

## Usage

```python
from transformers import AutoModelForCausalLM, AutoProcessor

repo = "xingxm/DesignCoder"
subfolder = "designcoder_qwen3.8_27b_adamw_bs128_data41287_step400"

model = AutoModelForCausalLM.from_pretrained(repo, subfolder=subfolder, dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained(repo, subfolder=subfolder)
```

To download a single checkpoint only:

```bash
hf download xingxm/DesignCoder --include "designcoder_qwen3.8_27b_adamw_bs128_data41287_step400/*" --local-dir ./DesignCoder
```

### Inference contract

These models are trained as tool-using agents, not single-turn generators. A case runs
`design_search` → (`websearch`, landing only) → a final answer containing exactly three code
blocks in the order `html`, `css`, `js`. Reproduce the system prompts and tool observation
format from `examples/designcoder/runtime/infer_designcoder.py`; prompting with a bare
instruction and no tool turns does not match the training distribution and will score far
below the numbers above.

## Provenance

Each subfolder additionally ships `trainer_state.json` / `trainer_log.jsonl` (and
`training_loss.png` where available) so that the loss curve and exact step schedule of the run
can be recovered from the checkpoint itself.