amaljoe88 commited on
Commit
cf6c0e0
Β·
0 Parent(s):

deploy: sync 712e5bc -> HF

Browse files
This view is limited to 50 files because it contains too many changes. Β  See raw diff
Files changed (50) hide show
  1. .claude/commands/job.md +55 -0
  2. .claude/commands/rsync-server.md +9 -0
  3. .github/workflows/deploy-hf.yml +57 -0
  4. .gitignore +18 -0
  5. CLAUDE.md +494 -0
  6. Dockerfile +26 -0
  7. README.md +103 -0
  8. benchmark.py +251 -0
  9. data/easy/0.html +15 -0
  10. data/easy/1.html +20 -0
  11. data/easy/2.html +23 -0
  12. data/easy/3.html +23 -0
  13. data/easy/4.html +29 -0
  14. data/hard/0.html +39 -0
  15. data/hard/1.html +51 -0
  16. data/hard/2.html +50 -0
  17. data/hard/3.html +55 -0
  18. data/hard/4.html +37 -0
  19. data/medium/0.html +41 -0
  20. data/medium/1.html +38 -0
  21. data/medium/2.html +34 -0
  22. data/medium/3.html +42 -0
  23. data/medium/4.html +36 -0
  24. data/tests/0/expected_scores.json +9 -0
  25. data/tests/0/meta.json +5 -0
  26. data/tests/0/reference.html +15 -0
  27. data/tests/0/variants/bad_colors.html +15 -0
  28. data/tests/0/variants/blank.html +1 -0
  29. data/tests/0/variants/half_styled.html +15 -0
  30. data/tests/0/variants/minor_diff.html +15 -0
  31. data/tests/0/variants/no_layout.html +15 -0
  32. data/tests/0/variants/no_style.html +15 -0
  33. data/tests/0/variants/perfect.html +15 -0
  34. data/tests/1/expected_scores.json +9 -0
  35. data/tests/1/meta.json +5 -0
  36. data/tests/1/reference.html +20 -0
  37. data/tests/1/variants/bad_colors.html +20 -0
  38. data/tests/1/variants/blank.html +1 -0
  39. data/tests/1/variants/half_styled.html +20 -0
  40. data/tests/1/variants/minor_diff.html +20 -0
  41. data/tests/1/variants/no_layout.html +20 -0
  42. data/tests/1/variants/no_style.html +20 -0
  43. data/tests/1/variants/perfect.html +20 -0
  44. data/tests/10/expected_scores.json +9 -0
  45. data/tests/10/meta.json +5 -0
  46. data/tests/10/reference.html +39 -0
  47. data/tests/10/variants/bad_colors.html +39 -0
  48. data/tests/10/variants/blank.html +1 -0
  49. data/tests/10/variants/half_styled.html +39 -0
  50. data/tests/10/variants/minor_diff.html +39 -0
.claude/commands/job.md ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Open a tmux window named `job:<window>` inside apptainer with the qwen35 mamba env activated.
2
+
3
+ Usage:
4
+ `/job <window-name>` β€” open window, drop into shell
5
+ `/job <window-name>: <command>` β€” open window and run command immediately
6
+
7
+ Examples:
8
+ `/job work`
9
+ `/job train`
10
+ `/job train-alt: python train.py --phase alternate --episodes-per-phase 20`
11
+
12
+ `$ARGUMENTS` is parsed as `<window>` or `<window>: <command>`.
13
+ Window name defaults to "work" if omitted.
14
+
15
+ ```bash
16
+ TMUX_BIN="$HOME/.local/bin/tmux"
17
+ SESSION="job"
18
+ ENV_PATH="/dev/shm/qwen35"
19
+
20
+ # Parse arguments: "window-name: command to run" OR just "window-name"
21
+ ARGS="$ARGUMENTS"
22
+ if [[ "$ARGS" == *": "* ]]; then
23
+ WINDOW="${ARGS%%: *}"
24
+ COMMAND="${ARGS#*: }"
25
+ else
26
+ WINDOW="${ARGS:-work}"
27
+ COMMAND=""
28
+ fi
29
+
30
+ # Create session if it doesn't exist
31
+ $TMUX_BIN has-session -t "$SESSION" 2>/dev/null || $TMUX_BIN new-session -d -s "$SESSION" -x 220 -y 50
32
+
33
+ # Reuse existing window or create new one
34
+ IS_NEW=false
35
+ if $TMUX_BIN list-windows -t "$SESSION" -F "#{window_name}" 2>/dev/null | grep -qx "$WINDOW"; then
36
+ echo "Reusing existing window: $SESSION:$WINDOW"
37
+ else
38
+ IS_NEW=true
39
+ $TMUX_BIN new-window -t "$SESSION" -n "$WINDOW"
40
+ INIT_CMD="apptainer exec --nv ~/apptainer-images/cuda-custom-amal_latest.sif bash --rcfile <(echo 'source ~/.bashrc; export MAMBA_EXE=\$HOME/.local/bin/micromamba; export MAMBA_ROOT_PREFIX=\$HOME/micromamba; eval \"\$(\$MAMBA_EXE shell hook --shell bash --root-prefix \$MAMBA_ROOT_PREFIX 2>/dev/null)\"; micromamba activate ${ENV_PATH}; export LD_PRELOAD=${ENV_PATH}/lib/libstdc++.so.6; cd ~/workspace/vision-coder-openenv; echo \"[job] apptainer ready | env: ${ENV_PATH} | session: ${SESSION}:${WINDOW}\"')"
41
+ $TMUX_BIN send-keys -t "$SESSION:$WINDOW" "$INIT_CMD" Enter
42
+ echo "Created window $SESSION:$WINDOW β€” apptainer + qwen35 env starting"
43
+ fi
44
+
45
+ # Send command if provided (wait for apptainer to boot on new windows)
46
+ if [[ -n "$COMMAND" ]]; then
47
+ if $IS_NEW; then
48
+ sleep 7
49
+ fi
50
+ $TMUX_BIN send-keys -t "$SESSION:$WINDOW" "$COMMAND" Enter
51
+ echo "Sent to $SESSION:$WINDOW: $COMMAND"
52
+ fi
53
+
54
+ echo "Attach with: ~/.local/bin/tmux attach -t $SESSION"
55
+ ```
.claude/commands/rsync-server.md ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ Pull the latest code from the remote server into the local workspace.
2
+
3
+ Use $ARGUMENTS as the SSH host if provided, otherwise default to "param".
4
+
5
+ Run exactly this bash command:
6
+
7
+ rsync -avz --delete --progress --exclude='.venv' --exclude='__pycache__' --exclude='*.pyc' --exclude='.git' --exclude='checkpoints' --exclude='*.output' ${ARGUMENTS:-param}:~/workspace/vision-coder-openenv/ /Users/amaljoe/Desktop/Workspace/AI/vision-coder-openenv/
8
+
9
+ After it completes, print a one-line summary: how many files were transferred and total bytes received. If rsync fails, show the error.
.github/workflows/deploy-hf.yml ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ name: Deploy to HF Space
2
+
3
+ on:
4
+ push:
5
+ branches: [main]
6
+
7
+ jobs:
8
+ deploy:
9
+ name: Deploy to HF Space
10
+ runs-on: ubuntu-latest
11
+ steps:
12
+ - uses: actions/checkout@v4
13
+
14
+ - name: Install huggingface_hub
15
+ run: pip install huggingface_hub -q
16
+
17
+ - name: Upload to HF Space
18
+ env:
19
+ HF_TOKEN: ${{ secrets.HF_TOKEN }}
20
+ run: |
21
+ python3 - <<'EOF'
22
+ from huggingface_hub import HfApi
23
+ import os, pathlib, tempfile
24
+
25
+ api = HfApi(token=os.environ["HF_TOKEN"])
26
+ repo_id = "amaljoe88/vision-coder-openenv"
27
+ sha = os.environ.get("GITHUB_SHA", "")[:8]
28
+ raw_base = "https://raw.githubusercontent.com/amaljoe/vision-coder-openenv/main/"
29
+
30
+ # Upload everything except blog.md and binary PNGs.
31
+ # delete_patterns removes HF files that no longer exist in GitHub.
32
+ api.upload_folder(
33
+ folder_path=".",
34
+ repo_id=repo_id,
35
+ repo_type="space",
36
+ ignore_patterns=["assets/*.png", "*.png", "__pycache__", "*.pyc",
37
+ ".git", "checkpoints/", "*.log", "blog.md"],
38
+ delete_patterns=["*.py", "*.md", "*.sh", "*.yaml", "*.yml",
39
+ "*.txt", "*.toml", "*.lock", "*.json", "*.jsonl"],
40
+ commit_message=f"deploy: sync {sha} from GitHub Actions",
41
+ )
42
+
43
+ # Upload blog.md with local asset paths replaced by GitHub raw URLs
44
+ content = pathlib.Path("blog.md").read_text()
45
+ content = content.replace("assets/", raw_base + "assets/")
46
+ tmp = pathlib.Path(tempfile.mktemp(suffix=".md"))
47
+ tmp.write_text(content)
48
+ api.upload_file(
49
+ path_or_fileobj=str(tmp),
50
+ path_in_repo="blog.md",
51
+ repo_id=repo_id,
52
+ repo_type="space",
53
+ commit_message=f"deploy: blog.md with GitHub raw URLs ({sha})",
54
+ )
55
+ tmp.unlink()
56
+ print(f"Deployed to https://huggingface.co/spaces/{repo_id}")
57
+ EOF
.gitignore ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ __pycache__/
2
+ *.pyc
3
+ *.pyo
4
+ *.egg-info/
5
+ .env
6
+ debug_renders/
7
+ outputs/
8
+ .claude/*
9
+ !.claude/commands/
10
+ .venv/
11
+ venv/
12
+ dist/
13
+ build/
14
+ # Test case renders (gitignored, regenerated with --render)
15
+ data/tests/*/renders/
16
+ data/tests/last_run_report.json
17
+ # Bundled dataset (tracked intentionally β€” use git add -f data/)
18
+
CLAUDE.md ADDED
@@ -0,0 +1,494 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # VisionCoder OpenEnv β€” Claude Code Guide
2
+
3
+ ## Project
4
+ Screenshot-to-HTML RL environment for the Scaler x Meta PyTorch Hackathon.
5
+ OpenEnv-compatible HTTP API: `reset()` / `step()` / `render()` / `state()`.
6
+
7
+ **Round 1** (backup branch `round1`): single-step, single-agent inference.
8
+ **Round 2** (current `main`): multi-step iterative environment + multi-agent (Developer + Critic) + RL training.
9
+
10
+ ## Package structure
11
+ - `src/` β€” maps to `openenv` package (`client.py`, `models.py`, `agents.py`, `prompts.py`, `inference.py`, `train.py`, `dataset.py`)
12
+ - `src/server/` β€” maps to `openenv.server` (`app.py`, `environment.py`)
13
+ - `src/server/rewards/` β€” maps to `openenv.server.rewards` (one file per reward function)
14
+ - `data/` β€” bundled synthetic samples (5 per difficulty, ~40KB each)
15
+ - `data/tests/` β€” reward stability test cases (0-14; committed HTML + expected scores; renders gitignored)
16
+ - `tests/test_rewards.py` β€” unified test suite (unit + stability + correlation tests)
17
+
18
+ ## Running on rmgpu006 (cluster)
19
+
20
+ ### Step 1 β€” start vLLM (tmux session: `vllm`)
21
+ ```bash
22
+ ~/.local/bin/tmux new-session -s vllm
23
+ # inside that session:
24
+ apptainer exec --nv ~/apptainer-images/cuda-custom-amal_latest.sif bash -c \
25
+ 'export LD_PRELOAD=/dev/shm/qwen35/lib/libstdc++.so.6;
26
+ /dev/shm/qwen35/bin/python -m vllm.entrypoints.openai.api_server \
27
+ --model ~/models/Qwen3.5-2B --served-model-name qwen35 \
28
+ --tensor-parallel-size 2 --port 8001 --host 0.0.0.0 \
29
+ --max-model-len 65536 --enable-auto-tool-choice --tool-call-parser hermes' \
30
+ 2>&1 | tee ~/vllm_qwen35.log
31
+ ```
32
+ **`--enable-auto-tool-choice --tool-call-parser hermes` is mandatory** β€” without it every Developer call fails with 400 Bad Request and falls back to FALLBACK_HTML.
33
+
34
+ ### Step 2 β€” start env server (tmux session: `openenv`, no apptainer needed)
35
+ ```bash
36
+ ~/.local/bin/tmux new-session -s openenv
37
+ # inside that session:
38
+ export PLAYWRIGHT_BROWSERS_PATH=~/playwright-browsers
39
+ cd ~/workspace/vision-coder-openenv
40
+ /dev/shm/qwen35/bin/python -m uvicorn openenv.server.app:app --host 127.0.0.1 --port 18080
41
+ ```
42
+
43
+ ### Step 3 β€” run inference (same openenv session or a new window)
44
+ ```bash
45
+ export API_BASE_URL=http://localhost:8001/v1
46
+ export MODEL_NAME=qwen35
47
+ export HF_TOKEN=sk-local
48
+ export MAX_STEPS=2
49
+ export PLAYWRIGHT_BROWSERS_PATH=~/playwright-browsers
50
+ cd ~/workspace/vision-coder-openenv
51
+ /dev/shm/qwen35/bin/python inference.py
52
+ ```
53
+
54
+ ### First-time setup
55
+ ```bash
56
+ # Install package (once per env build)
57
+ /dev/shm/qwen35/bin/pip install -e .
58
+
59
+ # Download model (once, needs proxy sourced)
60
+ source ~/proxy-setup/scripts/proxy_env.sh
61
+ /dev/shm/qwen35/bin/python -c "
62
+ from huggingface_hub import snapshot_download
63
+ snapshot_download('Qwen/Qwen3.5-2B', local_dir='$HOME/models/Qwen3.5-2B')
64
+ "
65
+ ```
66
+
67
+ ## Running locally (generic)
68
+ ```bash
69
+ pip install -e .
70
+ uvicorn openenv.server.app:app --host 0.0.0.0 --port 7860
71
+ ```
72
+
73
+ ## Running inference (HF router)
74
+ ```bash
75
+ export API_BASE_URL=https://router.huggingface.co/v1
76
+ export MODEL_NAME=Qwen/Qwen3.5-35B-A3B
77
+ export HF_TOKEN=hf_...
78
+ python inference.py
79
+ ```
80
+
81
+ ## Environment variables
82
+ - `API_BASE_URL` β€” OpenAI-compatible LLM endpoint (required)
83
+ - `MODEL_NAME` β€” vision-capable model ID (required); Developer and Critic share this endpoint
84
+ - `HF_TOKEN` β€” Hugging Face token / API key for LLM calls (primary auth key)
85
+ - `MAX_STEPS` β€” max developer turns per episode (default: 5)
86
+ - `INFERENCE_SERVER_PORT` β€” env server port (default: 18080)
87
+ - `DEBUG` β€” set to `1` to enable full episode debug logging. Creates `outputs/<run_id>/` with:
88
+ - `<difficulty>.md` β€” per-episode markdown log (reference image, all step renders, HTML, rewards, critic text)
89
+ - `images/` β€” PNGs saved separately (reference, each step's rendered output, critic comparison views)
90
+ - Run: `DEBUG=1 API_BASE_URL=... MODEL_NAME=... /dev/shm/qwen35/bin/python inference.py`
91
+ - `ONE_SHOT` β€” removed. Zero-shot outperforms few-shot (mean 0.679 vs 0.653) on this model; few-shot was causing early termination from hallucinated items and content contamination from example pages.
92
+
93
+ ## Python version
94
+ Use `python3.13` locally (pip maps to python3.13 here, not `python3`).
95
+ On rmgpu006: use `/dev/shm/qwen35/bin/python`.
96
+
97
+ ---
98
+
99
+ ## Reward function
100
+
101
+ Composite of 8 sub-rewards, weighted and normalised to [0, 1]:
102
+
103
+ | Reward | Weight | What it measures |
104
+ |---|---|---|
105
+ | `format` | 0.5 | Has ` ```html ` fence + `<!DOCTYPE html>` |
106
+ | `validity` | 0.5 | Structural completeness (`html`/`head`/`body`, diverse tags) |
107
+ | `structural` | 0.5 | Tag-sequence similarity + inline-style property coverage |
108
+ | `text_block` | 3.0 | Hungarian-matched text block IoU + text similarity |
109
+ | `position` | 1.0 | Hungarian-matched centroid distance |
110
+ | `color` | 1.5 | Spatial CIEDE2000 on reference non-white pixels |
111
+ | `clip` | 2.5 | CLIP ViT-B/32 cosine similarity, renormalised (threshold 0.65) |
112
+ | `ssim` | 1.5 | Pixel-level SSIM (skimage, 320Γ—240 RGB) β€” near-perfect zone sensitivity |
113
+
114
+ **Weight sum = 11.0.** `format`/`validity`/`structural` reduced (saturate early); `color`/`clip`/`ssim` boosted for continuous near-perfect discrimination.
115
+
116
+ **Content multiplier:** applied to the weighted total. If reference has content but prediction is nearly blank (< 0.5% non-white pixels at 32Γ—32), multiplier scales linearly from 0 to 1. Ensures blank predictions score 0.0 even if individual sub-rewards are nonzero.
117
+
118
+ **CLIP renormalisation:** raw cosine ≀ 0.65 β†’ score 0; 1.0 β†’ 1.0. Makes blank pages (raw ~0.45) and unstyled pages (raw ~0.76) meaningfully separated.
119
+
120
+ **Observed scores on 15 test cases (averages):**
121
+ ```
122
+ perfect 0.977 minor_diff 0.883 bad_colors 0.740
123
+ half_styled 0.524 no_layout 0.469 no_style 0.393 blank 0.000
124
+ ```
125
+ Global Spearman ρ vs canonical targets = 0.955 (15/15 PASS). Gaps improved: perfectβ†’minor_diff +58%, minor_diffβ†’bad_colors +51% vs old weights.
126
+
127
+ ---
128
+
129
+ ## Reward stability tests
130
+
131
+ Test suite at `tests/test_rewards.py`. Test data in `data/tests/<num>/` (0-14, mapping easy/0-4, medium/0-4, hard/0-4).
132
+
133
+ Each case has:
134
+ - `reference.html`, `variants/*.html` (7 quality levels) β€” committed
135
+ - `expected_scores.json` β€” per-case baseline scores β€” committed
136
+ - `renders/` β€” PNG renders + block JSONs β€” **gitignored**, auto-generated
137
+
138
+ ```bash
139
+ # First run on a new machine (needs apptainer for Playwright on rmgpu006)
140
+ apptainer exec ~/apptainer-images/cuda-custom-amal_latest.sif bash -c \
141
+ 'export PLAYWRIGHT_BROWSERS_PATH=~/playwright-browsers
142
+ /dev/shm/qwen35/bin/python tests/test_rewards.py --render'
143
+
144
+ # Score only (fast, uses cached renders β€” works outside apptainer)
145
+ /dev/shm/qwen35/bin/python tests/test_rewards.py
146
+
147
+ # Re-render specific cases
148
+ /dev/shm/qwen35/bin/python tests/test_rewards.py --render --cases 0,1,5
149
+
150
+ # After changing reward functions, lock in new baseline
151
+ /dev/shm/qwen35/bin/python tests/test_rewards.py --update-expected
152
+
153
+ # As pytest (unit tests only, no Playwright needed)
154
+ /dev/shm/qwen35/bin/python -m pytest tests/test_rewards.py -v -m "not integration"
155
+ ```
156
+
157
+ Pass criteria: Spearman ρ β‰₯ 0.80 per case, global ρ β‰₯ 0.85, blank ≀ 0.05, perfect β‰₯ 0.80.
158
+
159
+ ---
160
+
161
+ ## Round 2 architecture
162
+
163
+ ### Models
164
+ | Role | Inference (eval) | Training |
165
+ |---|---|---|
166
+ | Developer | `Qwen/Qwen3.5-35B-A3B` via HF router | `Qwen/Qwen3.5-2B` with LoRA (rank=16) |
167
+ | Critic | `Qwen/Qwen3.5-35B-A3B` via HF router | shared 2B base |
168
+
169
+ - Qwen3.5 is unified vision+text β€” no separate VL variant needed
170
+ - 2B fits 2Γ—A100 with LoRA; training completes in ~2h for 20 episodes Γ— 4 rollouts
171
+ - Run 2 uses `--resume-from checkpoints/run2/developer_final` for continued improvement
172
+
173
+ ### Environment changes (server/)
174
+ - `reset()` returns task + `session_id`; session state lives in-memory per episode
175
+ - `step(html, session_id)` β†’ `{reward, render_low (base64), render_full (base64), done}`
176
+ - `render(html)` β†’ `{image (base64)}` β€” renders only, no reward (used by Developer tool call)
177
+ - `done=true` when max steps reached
178
+
179
+ ### Multi-agent inference loop (inference.py)
180
+ ```
181
+ for each task (episode):
182
+ state = env.reset() # β†’ session_id, reference_image
183
+ code, critique, render_prev = "", None, None
184
+
185
+ for step i in range(MAX_STEPS):
186
+ # Developer: fast mode, render() tool available
187
+ # Input: reference_image (low-res) + code + critique
188
+ # Calls render(new_html) mid-generation to self-check
189
+ code = developer.generate(ref_image, code, critique)
190
+
191
+ result = env.step(code, session_id) # β†’ reward, render_low, render_full, done
192
+ log_step(i, code, result.reward, result.done)
193
+ if result.done: break
194
+
195
+ # Critic: thinking mode on
196
+ # Input: reference_image (full-res) + render_{i-1} + critique_{i-1} + render_i
197
+ critique = critic.review(ref_image, render_prev, critique, result.render_full)
198
+ if "DONE" in critique: break
199
+
200
+ render_prev = result.render_full
201
+
202
+ log_end(...)
203
+ ```
204
+
205
+ ### RL training (train.py)
206
+
207
+ **Reward function (per turn t in trajectory):**
208
+ ```
209
+ R_total(t) = R_terminal + Ξ» Β· Ξ£(r_s - r_{s-1} for s = t..n)
210
+
211
+ R_terminal = environment score at final step n ← main signal
212
+ r_s - r_{s-1} = per-step improvement delta ← shaped signal
213
+ Ξ» = 0.2 ← keeps shaped signal subordinate
214
+ ```
215
+
216
+ - `R_terminal` propagates backward to all turns (solves long-horizon credit assignment)
217
+ - Shaped reward gives additional gradient signal at early turns without dominating
218
+ - Both Developer and Critic tokens receive this advantage; Critic's shaped reward
219
+ is the improvement delta from step i+1 onward (first step after its critique)
220
+
221
+ **Training algorithm: full-episode GRPO**
222
+ ```
223
+ for each task:
224
+ sample K full trajectories Ο„_1..Ο„_K (different temperatures/seeds)
225
+ score each trajectory: R_terminal_k + shaped deltas
226
+ compute group-relative advantage: A_t = (G_t - mean_k) / std_k
227
+ update: βˆ‡ log Ο€(a_t | s_t) Β· A_t for all tokens in trajectory
228
+ ```
229
+
230
+ **Alternating training schedule:**
231
+ ```
232
+ Phase A (N episodes): Train Developer (LoRA), freeze Critic
233
+ Phase B (N episodes): Train Critic (LoRA, thinking on), freeze Developer
234
+ Repeat until convergence
235
+ ```
236
+
237
+ ### inference.py output format (unchanged from Round 1)
238
+ ```
239
+ [START] task=<difficulty> env=vision-coder model=<model>
240
+ [STEP] step=<n> action=<html_preview> reward=<0.00> done=<true|false> error=<msg|null>
241
+ [END] success=<true|false> steps=<n> score=<0.000> rewards=<r1,...>
242
+ ```
243
+
244
+ ---
245
+
246
+ ## Git remotes
247
+ Two remotes must both be kept in sync:
248
+ - `origin` β†’ GitHub (`https://github.com/amaljoe/vision-coder-openenv`)
249
+ - `hf` β†’ HuggingFace Spaces (`https://huggingface.co/spaces/amaljoe88/vision-coder-openenv`)
250
+
251
+ **A `post-push` hook auto-pushes to `hf` whenever you push to `origin`.** Never push only to origin and forget HF β€” that's what caused multiple Phase 2 failures (HF Space was 10 commits behind for most of the day).
252
+
253
+ To deploy manually use the `/deploy` skill which pushes both, waits for build, and health-checks `/reset`.
254
+
255
+ ---
256
+
257
+ ## Submission workflow
258
+
259
+ ### 1. Push code
260
+ ```bash
261
+ git push origin main # post-push hook auto-syncs to hf
262
+ ```
263
+
264
+ ### 2. Verify HF Space is live
265
+ ```bash
266
+ curl -s "https://huggingface.co/api/spaces/amaljoe88/vision-coder-openenv" \
267
+ | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['runtime']['stage'], d['sha'][:8])"
268
+ ```
269
+ Wait for `stage: RUNNING`. Docker build with CLIP takes 5–10 minutes.
270
+
271
+ ### 3. Health check
272
+ ```bash
273
+ curl -X POST "https://amaljoe88-vision-coder-openenv.hf.space/reset?difficulty=easy" -o /dev/null -w "%{http_code}"
274
+ # Must return 200
275
+ ```
276
+
277
+ ### 4. Submit via dashboard
278
+ Only the team lead submits at: `https://www.scaler.com/school-of-technology/meta-pytorch-hackathon/dashboard`
279
+ - GitHub URL: `https://github.com/amaljoe/vision-coder-openenv`
280
+ - HF Space URL: `https://huggingface.co/spaces/amaljoe88/vision-coder-openenv`
281
+
282
+ ### 5. Check submission status
283
+ Use the `/check-hackathon-status` skill or:
284
+ ```bash
285
+ curl -s "https://www.scaler.com/meta-pytorch-hackathon/api/v1/submissions/status?round=1" \
286
+ -H "Authorization: Bearer BAh7B0kiDHVzZXJfaWQGOgZFVGkCcp1JIg5pc3N1ZWRfYXQGOwBUbCsHLfLDaQ==--bf98a96719f439decefc69bb1a42fe0d28fa29f82fb956d8546f326225af310d" \
287
+ | python3 -m json.tool
288
+ ```
289
+ Check `validation.agentic_evaluation.status` β€” target is `"success"` with all 5 steps `"pass"`.
290
+
291
+ ---
292
+
293
+ ## How evaluation works
294
+
295
+ The evaluator does NOT use the HF Space to run inference. The flow is:
296
+ 1. **HF Space ping** β€” pre-submission only: `POST /reset` must return 200
297
+ 2. **Agentic eval**:
298
+ - Clones GitHub repo to `/tmp/workspace/`
299
+ - `docker build` from `Dockerfile`
300
+ - Runs `inference.py` inside Docker with `HF_TOKEN`, `API_BASE_URL`, `MODEL_NAME` set
301
+ - Parses `[START]`/`[STEP]`/`[END]` stdout
302
+ - Validates task scores
303
+
304
+ Eval pipeline: `docker_build` β†’ `inference` β†’ `parse_output` β†’ `task_validation` β†’ `llm_check`
305
+
306
+ ---
307
+
308
+ ## inference.py requirements (critical)
309
+
310
+ Rules enforced by evaluator parser:
311
+ - `action=` must be a plain string β€” **no `!r` repr quoting** (caused one failure)
312
+ - `done` and `success` are lowercase: `true`/`false`
313
+ - `error` is `null` if none
314
+ - `[END]` must always be emitted β€” put it in a `finally` block
315
+ - Exit code must be 0 β€” wrap everything in try/except
316
+ - LLM call must be inside try/except with a fallback HTML so it never crashes
317
+
318
+ Auth variable priority (evaluator sets `HF_TOKEN`, not `OPENAI_API_KEY`):
319
+ ```python
320
+ API_KEY = os.getenv("HF_TOKEN") or os.getenv("API_KEY") or os.getenv("OPENAI_API_KEY") or "sk-placeholder"
321
+ ```
322
+
323
+ ---
324
+
325
+ ## Challenges faced and lessons learned
326
+
327
+ ### 1. HF Space drifted 10 commits behind
328
+ **What happened:** `git push origin main` only updates GitHub. HF Spaces has a separate `hf` remote that needs an explicit `git push hf main`. We pushed all code fixes to GitHub but forgot to push to HF, so the Space ran month-old code for the entire evaluation window.
329
+
330
+ **Fix:** Added `.git/hooks/post-push` that auto-pushes to `hf` whenever `origin` is pushed.
331
+
332
+ ---
333
+
334
+ ### 2. Auth failure β€” wrong API key env var
335
+ **What happened:** Old `inference.py` read `OPENAI_API_KEY` but the evaluator only sets `HF_TOKEN`. This caused every LLM call to use `"sk-placeholder"` β†’ 401 auth error β†’ crash.
336
+
337
+ **Fix:** Read `HF_TOKEN` first, fall back through `API_KEY` β†’ `OPENAI_API_KEY` β†’ placeholder.
338
+
339
+ ---
340
+
341
+ ### 3. Timing race β€” 3-minute window between commit and submission
342
+ **What happened:** Commit `4dc307c` (inference.py rewrite with auth fix) was pushed at 11:49 AM. User submitted at 11:52 AM β€” only 3 minutes later. The evaluator cloned the repo before the push fully propagated and ran the older version without proper try/except. The submission failed despite the fix being "live".
343
+
344
+ **Lesson:** After pushing a critical fix, wait at least 5–10 minutes before resubmitting to avoid the evaluator picking up stale code.
345
+
346
+ ---
347
+
348
+ ### 4. `!r` repr quoting on action field broke output parsing
349
+ **What happened:** `log_step` used `f"action={action!r}"` which wraps the HTML string in Python quotes: `action='<!DOCTYPE...'`. The evaluator parser expected `action=<!DOCTYPE...` (no quotes). Found by comparing against the sample inference script.
350
+
351
+ **Fix:** Changed `{action_summary!r}` to `{action_summary}`.
352
+
353
+ ---
354
+
355
+ ### 5. transformers v5 API change for CLIP
356
+ **What happened:** `model.get_image_features()` in transformers v5 returns a dataclass (`BaseModelOutputWithPooling`), not a raw tensor. Code that did `features / features.norm()` crashed with `'BaseModelOutputWithPooling' object has no attribute 'norm'`.
357
+
358
+ **Fix:**
359
+ ```python
360
+ out = model.get_image_features(pixel_values=pv)
361
+ features = out.pooler_output if hasattr(out, "pooler_output") else out
362
+ ```
363
+
364
+ ---
365
+
366
+ ### 6. Docker build downloads ~1.2GB β€” fragile on slow networks
367
+ The Dockerfile downloads: `torch` CPU (~600MB), CLIP model weights (~600MB), Playwright Chromium (~400MB). If any download fails or times out during `docker build`, the whole eval fails at `docker_build` step.
368
+
369
+ **Mitigation:** Keep these RUN layers cached by not changing requirements.txt or the download commands unnecessarily.
370
+
371
+ ---
372
+
373
+ ### 7. 4 Playwright browser launches per step (performance)
374
+ **What happened:** `text_block_reward`, `position_reward`, `color_reward`, and `clip_visual_reward` each launched Playwright independently β€” 4–6 browser sessions per `step()` call.
375
+
376
+ **Fix:** Render the predicted HTML once in `environment.py`, pass the `PIL.Image` as `pred_image` parameter to both `color_reward` and `clip_visual_reward` to skip duplicate renders.
377
+
378
+ ---
379
+
380
+ ### 9. `color_reward` false positive on white-background pages
381
+ **What happened:** Original implementation sampled non-white pixels independently from each image, then compared them. A blank white prediction vs a mostly-white reference (e.g. `#f0f2f5` login form) both sampled near-white pixels β†’ CIEDE2000 β‰ˆ 0 β†’ score 0.80–0.96 (false high).
382
+
383
+ **Fix:** Spatial comparison β€” resize both images to 128Γ—128, compute per-pixel CIEDE2000, average only over positions where the **reference** is non-white. Blank prediction at those positions gets correct high Ξ”E. Falls back to full mean when reference is nearly all white (< 2% non-white).
384
+
385
+ ---
386
+
387
+ ### 10. `structural_reward` trivially inflated for inline-style HTML
388
+ **What happened:** All 15 reference HTMLs use inline `style=""` attributes, not CSS classes. The CSS class overlap term always returned 1.0 (neither ref nor pred had classes), making blank pages score 0.50 on structural instead of ~0.25.
389
+
390
+ **Fix:** When ref has no CSS classes, fall back to inline style **property name** coverage: `len(pred_props ∩ ref_props) / len(ref_props)`. Blank page has 1–2 style props vs 15–25 in ref β†’ score 0.08–0.25. Perfect match (same HTML) β†’ same props β†’ 1.0.
391
+
392
+ ---
393
+
394
+ ### 11. `validity_reward` too generous on blank pages
395
+ **What happened:** `_MIN_DIVERSE_TAGS` was 5. A blank page with 4 tags (`html`, `head`, `title`, `body`) scored 4/5 = 0.80 on diversity, giving total validity β‰ˆ 0.90.
396
+
397
+ **Fix:** Raised to 8. Blank page now scores 4/8 = 0.50 on diversity β†’ total validity β‰ˆ 0.75.
398
+
399
+ ---
400
+
401
+ ### 12. Per-step GRPO loses long-horizon signal (Round 2 lesson)
402
+ **What happened (design trap):** Applying GRPO independently at each step means Dev_0 only sees `r_0` β€” the final reward never flows back. Early turns get misguided signal regardless of episode outcome.
403
+
404
+ **Fix:** Full-episode GRPO β€” sample K complete trajectories, apply group-relative advantage to all tokens uniformly. Augment with shaped improvement-delta reward (Ξ»=0.2) for early-turn credit assignment. See `train.py`.
405
+
406
+ ---
407
+
408
+ ### 13. Critic DONE-too-early collapses GRPO variance
409
+ **What happened:** The training CRITIC_TRAIN_SYSTEM said "Output DONE if closely matches" (too permissive). The Critic said DONE after step 1 on medium/hard tasks regardless of quality. With all 4 rollouts at 1 step and similar rewards, group std β‰ˆ 0 β†’ advantages β‰ˆ 0 β†’ no gradient signal.
410
+
411
+ **Fix:** Stricter prompt: "Output DONE only if >90% visual similarity. If ANY section is missing/wrong, list it β€” do NOT output DONE." Fixes variance collapse for run 2.
412
+
413
+ ---
414
+
415
+ ### 14. MAX_NEW_TOKENS=1024 caused truncated HTML β†’ artificially low training rewards
416
+ **What happened:** Complex HTML pages need 1500–2500 tokens. With MAX_NEW_TOKENS=1024, the model generated truncated HTML missing closing tags and lower sections. Reward was dominated by fmt/validity only (clip=0 because truncated HTML renders blank/broken).
417
+
418
+ **Fix:** Increased to 2048 in src/train.py. Also helps that 2B model's training rewards (0.23–0.35) were far below vLLM inference rewards (0.6+) β€” gap partly explained by truncation.
419
+
420
+ ---
421
+
422
+ ### 15. GRPO breakthrough emerges at ep=16 (easy difficulty)
423
+ **What happened:** After 15 episodes of noisy rewards (0.23–0.35 for easy), ep=16 easy jumped to 0.496 mean reward. Individual rollouts reached 0.82 with clip=0.95 (raw cosine ~0.98). The model learned to generate HTML with high visual similarity.
424
+
425
+ **Why it happened:** GRPO with variance in rollouts β€” when 1 of 4 rollouts achieves clip=0.90+ while others get 0.00, the group advantage strongly reinforces the high-scoring generation strategy. This is the critical moment when GRPO starts working.
426
+
427
+ **Observation:** Medium and hard tasks didn't break through in run 1 due to Critic early-termination. Run 2 (with fixed CRITIC_TRAIN_SYSTEM) should show similar breakthrough for all difficulties.
428
+
429
+ ---
430
+
431
+ ### 16. eval_lora.py β€” comparing trained vs base without vLLM
432
+ **Command (runs after training, outside apptainer):**
433
+ ```bash
434
+ export PLAYWRIGHT_BROWSERS_PATH=~/playwright-browsers
435
+ /dev/shm/qwen35/bin/python eval_lora.py \
436
+ --lora-path checkpoints/run2/developer_final \
437
+ --model ~/models/Qwen3.5-2B \
438
+ --episodes 2
439
+ ```
440
+ Outputs blog-ready markdown table + saves `checkpoints/eval_results.json`.
441
+
442
+ **Run 1 results** (20/20 episodes, `checkpoints/run2/developer_final`, 2 ep/difficulty, temperature=0.3):
443
+ | Difficulty | Base 2B | Trained 2B | Delta |
444
+ |---|---|---|---|
445
+ | easy | 0.924 | **0.961** | +0.037 |
446
+ | medium | 0.937 | **0.955** | +0.018 |
447
+ | hard | 0.919 | **0.952** | +0.034 |
448
+ | **mean** | 0.927 | **0.956** | **+3.2%** |
449
+
450
+ > Evaluated on bundled training samples (in-distribution). Relative delta is the meaningful signal.
451
+
452
+ **Command to restart vLLM with trained LoRA** (in tmux `vllm` session, after killing old process):
453
+ ```bash
454
+ LORA_PATH=checkpoints/run2/developer_final LORA_NAME=qwen35-trained bash scripts/vllm.sh 2b
455
+ # Then use MODEL_NAME=qwen35-trained in inference.py for trained model
456
+ ```
457
+
458
+ **Command to start run 2 (resume from run 1 LoRA):**
459
+ ```bash
460
+ apptainer exec --nv ~/apptainer-images/cuda-custom-amal_latest.sif bash -c '
461
+ export LD_PRELOAD=/dev/shm/qwen35/lib/libstdc++.so.6
462
+ /dev/shm/qwen35/bin/python train.py \
463
+ --phase developer \
464
+ --episodes 10 \
465
+ --k-rollouts 4 \
466
+ --model ~/models/Qwen3.5-2B \
467
+ --checkpoint-dir checkpoints/run3 \
468
+ --resume-from checkpoints/run2/developer_final
469
+ ' 2>&1 | tee checkpoints/train_run3.log
470
+ ```
471
+
472
+ ---
473
+
474
+ ## Self-Update Protocol
475
+
476
+ **This file is a living document. Claude must keep it current.**
477
+
478
+ Whenever you discover something not already recorded here β€” a new flag, env var, command, lesson, bug, or architectural decision β€” add it immediately. Do not wait to be asked.
479
+
480
+ ### What to update and where
481
+
482
+ | Discovery | Where to add |
483
+ |---|---|
484
+ | New `--flag` for vLLM / uvicorn / inference | Relevant step in "Running on rmgpu006" |
485
+ | New environment variable | "Environment variables" section |
486
+ | New bug or production incident | "Challenges faced and lessons learned" (next numbered entry) |
487
+ | New architectural decision | Relevant subsection under "Round 2 architecture" |
488
+ | Change to submission / eval behavior | "Submission workflow" or "How evaluation works" |
489
+ | New debug flag or mode | "Environment variables" section with a note on what it enables |
490
+
491
+ ### Rules
492
+ - Write lessons in past tense under "Challenges faced" β€” **What happened**, **Fix** format.
493
+ - Keep commands copy-pasteable and tested; update them if they change.
494
+ - After updating this file, commit it: `git add CLAUDE.md && git commit -m "docs: update CLAUDE.md β€” <one-line summary>"`
Dockerfile ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ FROM python:3.11-slim
2
+
3
+ WORKDIR /app
4
+
5
+ # Install pip dependencies first (layer cache)
6
+ COPY requirements.txt .
7
+ RUN pip install --no-cache-dir -r requirements.txt
8
+
9
+ # Install torch CPU-only (avoids ~2 GB of GPU kernels)
10
+ RUN pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/cpu
11
+
12
+ # Pre-download CLIP model so first request isn't slow
13
+ RUN python -c "from transformers import CLIPModel, CLIPProcessor; CLIPModel.from_pretrained('openai/clip-vit-base-patch32'); CLIPProcessor.from_pretrained('openai/clip-vit-base-patch32'); print('CLIP ready')"
14
+
15
+ # Let Playwright install Chromium + all its own system dependencies
16
+ RUN playwright install chromium --with-deps
17
+
18
+ # Copy source and install package
19
+ COPY . .
20
+ RUN pip install --no-cache-dir -e .
21
+
22
+ EXPOSE 7860
23
+
24
+ ENV PORT=7860
25
+
26
+ CMD ["sh", "-c", "uvicorn openenv.server.app:app --host 0.0.0.0 --port ${PORT}"]
README.md ADDED
@@ -0,0 +1,103 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: Vision Coder OpenEnv
3
+ emoji: πŸ–ΌοΈ
4
+ colorFrom: blue
5
+ colorTo: purple
6
+ sdk: docker
7
+ app_port: 7860
8
+ pinned: false
9
+ models:
10
+ - Qwen/Qwen3.5-2B
11
+ tags:
12
+ - reinforcement-learning
13
+ - html-generation
14
+ - computer-vision
15
+ - grpo
16
+ short_description: RL environment for screenshot-to-HTML generation
17
+ ---
18
+
19
+ # VisionCoder OpenEnv
20
+
21
+ An RL environment for screenshot-to-HTML generation. An agent receives a UI screenshot and iteratively refines HTML until the rendered output visually matches the reference.
22
+
23
+ **[Live environment](https://huggingface.co/spaces/amaljoe88/vision-coder-openenv) Β· [Blog](https://github.com/amaljoe/vision-coder-openenv/blob/main/blog.md) Β· [Interactive demo](https://amaljoe.github.io/vision-coder-openenv/)**
24
+
25
+ ---
26
+
27
+ ## The Problem
28
+
29
+ A single LLM call can produce structurally valid HTML that looks nothing like the reference β€” the model has no way to see its own rendered output and no feedback loop to improve. We frame this as RL: generate HTML, render in a real browser, compute visual reward, iterate.
30
+
31
+ ---
32
+
33
+ ## How It Works
34
+
35
+ ### API
36
+
37
+ ```
38
+ POST /reset?difficulty=easy|medium|hard β†’ { session_id, screenshot_b64 }
39
+ POST /step { html, session_id } β†’ { reward, render_low, render_full, done }
40
+ POST /render { html } β†’ { image_b64 }
41
+ ```
42
+
43
+ Episodes run for up to 5 steps. Every submission is rendered by Playwright (headless Chromium) at `320Γ—240` (low-res preview) and `640Γ—480` (full-res, used for reward and Critic).
44
+
45
+ ### Two-Agent Loop
46
+
47
+ A **Developer** generates HTML; a **Critic** compares the render against the reference and produces selector-specific CSS fix instructions.
48
+
49
+ ![Dual-agent architecture](https://raw.githubusercontent.com/amaljoe/vision-coder-openenv/main/assets/dual-agent-architecture.png)
50
+
51
+ The Critic compresses ~5,000 tokens of visual+code context into ~200 tokens of actionable fix instructions the Developer can apply directly.
52
+
53
+ ### Reward Function
54
+
55
+ 8 sub-rewards weighted by discriminativeness (weight sum = 11.0):
56
+
57
+ | Signal | Weight |
58
+ |---|---|
59
+ | `format` + `validity` + `structural` | 0.5 each β€” saturate early |
60
+ | `position` | 1.0 |
61
+ | `color` + `ssim` | 1.5 each |
62
+ | `clip` | 2.5 |
63
+ | `text_block` | **3.0** |
64
+
65
+ A content multiplier forces blank renders to score 0.0 regardless of sub-reward values.
66
+ **Spearman ρ = 0.955** vs human quality rankings across 15 test cases.
67
+
68
+ The reward test suite runs across all 15 cases. Browse all renders, scores, and per-sub-reward breakdowns in the **[interactive demo](https://amaljoe.github.io/vision-coder-openenv/)**.
69
+
70
+ ---
71
+
72
+ ## Results
73
+
74
+ Training Qwen3.5-2B with full-episode GRPO for 20 episodes on 2Γ— A100 80GB (~2h):
75
+
76
+ ![Training Curve](https://raw.githubusercontent.com/amaljoe/vision-coder-openenv/main/assets/training_curve.png)
77
+
78
+ | Difficulty | Base | Trained | Delta |
79
+ |---|---|---|---|
80
+ | easy | 0.629 | 0.634 | +0.005 |
81
+ | medium | 0.488 | 0.634 | +0.146 |
82
+ | hard | 0.346 | 0.564 | +0.218 |
83
+ | **mean** | **0.488** | **0.611** | **+25.2%** |
84
+
85
+ Hard tasks improve the most β€” complex layouts have the most to gain from the Critic's structured feedback.
86
+
87
+ ![Episode reward progression](https://raw.githubusercontent.com/amaljoe/vision-coder-openenv/main/assets/episode_progression.png)
88
+
89
+ ---
90
+
91
+ ## Quickstart
92
+
93
+ ```bash
94
+ pip install -e .
95
+ uvicorn openenv.server.app:app --host 0.0.0.0 --port 7860
96
+ ```
97
+
98
+ ```bash
99
+ export API_BASE_URL=https://router.huggingface.co/v1
100
+ export MODEL_NAME=Qwen/Qwen3.5-35B-A3B
101
+ export HF_TOKEN=hf_...
102
+ python inference.py
103
+ ```
benchmark.py ADDED
@@ -0,0 +1,251 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Benchmark three inference approaches on 15 examples (5 per difficulty).
2
+
3
+ Approaches:
4
+ A β€” Multi-agent: Developer (full-res ref + Critic TODO) + Critic (full-res ref + renders)
5
+ B β€” Long-horizon Developer: full-res ref + all previous renders + all previous HTML, no Critic
6
+ C β€” Short-horizon Developer: full-res ref + only last render + only last HTML, no Critic
7
+
8
+ Usage:
9
+ export API_BASE_URL=http://localhost:8001/v1
10
+ export MODEL_NAME=qwen35
11
+ export HF_TOKEN=sk-local
12
+ export MAX_STEPS=5
13
+ export PLAYWRIGHT_BROWSERS_PATH=~/playwright-browsers
14
+ cd ~/workspace/vision-coder-openenv
15
+ /dev/shm/qwen35/bin/python benchmark.py 2>&1 | tee benchmark_results.txt
16
+ """
17
+ from __future__ import annotations
18
+
19
+ import json
20
+ import os
21
+ import sys
22
+ import threading
23
+ import time
24
+ import urllib.request
25
+ from datetime import datetime
26
+ from pathlib import Path
27
+ from typing import Dict, List, Tuple
28
+
29
+ import uvicorn
30
+
31
+ API_KEY = os.getenv("HF_TOKEN") or os.getenv("API_KEY") or os.getenv("OPENAI_API_KEY") or "sk-placeholder"
32
+ API_BASE_URL = os.getenv("API_BASE_URL") or "https://router.huggingface.co/v1"
33
+ MODEL_NAME = os.getenv("MODEL_NAME") or "Qwen/Qwen3.5-35B-A3B"
34
+ SERVER_PORT = int(os.environ.get("INFERENCE_SERVER_PORT", "18080"))
35
+ SERVER_URL = f"http://127.0.0.1:{SERVER_PORT}"
36
+ MAX_STEPS = int(os.environ.get("MAX_STEPS", "5"))
37
+ NUM_EPISODES = int(os.environ.get("NUM_EPISODES", "5")) # per difficulty
38
+ DIFFICULTIES = ["easy", "medium", "hard"]
39
+
40
+ # Partial results written here after every episode so a crash loses nothing
41
+ _RUN_ID = datetime.now().strftime("%Y%m%d_%H%M%S")
42
+ PARTIAL_PATH = Path(f"benchmark_partial_{_RUN_ID}.json")
43
+ _partial: dict = {} # keyed by "A/easy/1", etc.
44
+
45
+
46
+ def _flush_partial() -> None:
47
+ PARTIAL_PATH.write_text(json.dumps(_partial, indent=2))
48
+
49
+
50
+ def _start_server() -> None:
51
+ from openenv.server.app import app
52
+ config = uvicorn.Config(app, host="127.0.0.1", port=SERVER_PORT, log_level="error")
53
+ uvicorn.Server(config).run()
54
+
55
+
56
+ def _wait_for_server(timeout: float = 120.0) -> None:
57
+ deadline = time.time() + timeout
58
+ while time.time() < deadline:
59
+ try:
60
+ urllib.request.urlopen(f"{SERVER_URL}/health", timeout=2)
61
+ return
62
+ except Exception:
63
+ time.sleep(1.0)
64
+ raise RuntimeError(f"Env server did not start within {timeout}s")
65
+
66
+
67
+ def _run_approach(
68
+ label: str,
69
+ approach_id: str,
70
+ env_client,
71
+ config,
72
+ ) -> Tuple[List[float], float]:
73
+ """Run approach on all examples. Returns (rewards_per_episode, total_wall_time)."""
74
+ from openenv.agents import run_episode, run_episode_long_dev, run_episode_short_dev, run_episode_d
75
+
76
+ runners = {
77
+ "A": run_episode,
78
+ "B": run_episode_long_dev,
79
+ "C": run_episode_short_dev,
80
+ "D": run_episode_d,
81
+ }
82
+ runner = runners[approach_id]
83
+
84
+ # Reset dataset indices so every approach sees the same samples
85
+ env_client.post("/reset_dataset")
86
+
87
+ all_rewards: List[float] = []
88
+ approach_start = time.time()
89
+
90
+ for difficulty in DIFFICULTIES:
91
+ for ep in range(1, NUM_EPISODES + 1):
92
+ ep_start = time.time()
93
+ final_reward = 0.0
94
+ all_step_rewards: List[float] = []
95
+
96
+ try:
97
+ resp = env_client.post("/reset", params={"difficulty": difficulty})
98
+ resp.raise_for_status()
99
+ obs = resp.json()
100
+ session_id = obs["session_id"]
101
+ ref_b64 = obs["screenshot_b64"]
102
+
103
+ results = runner(
104
+ env_client, config, session_id, ref_b64,
105
+ dbg=None, on_step=None,
106
+ )
107
+ if results:
108
+ final_reward = results[-1].reward
109
+ all_step_rewards = [r.reward for r in results]
110
+
111
+ except Exception as exc:
112
+ print(f"[BENCH] ERROR approach={approach_id} difficulty={difficulty} ep={ep}: {exc}", flush=True)
113
+
114
+ ep_time = time.time() - ep_start
115
+ all_rewards.append(final_reward)
116
+
117
+ # Save final render as PNG
118
+ if results and results[-1].render_full_b64:
119
+ import base64 as _b64
120
+ renders_dir = Path("benchmark_renders") / _RUN_ID / approach_id
121
+ renders_dir.mkdir(parents=True, exist_ok=True)
122
+ png_path = renders_dir / f"{difficulty}_ep{ep}_r{final_reward:.3f}.png"
123
+ png_path.write_bytes(_b64.b64decode(results[-1].render_full_b64))
124
+
125
+ # Persist after every episode
126
+ key = f"{approach_id}/{difficulty}/{ep}"
127
+ _partial[key] = {
128
+ "approach": approach_id,
129
+ "difficulty": difficulty,
130
+ "episode": ep,
131
+ "final_reward": final_reward,
132
+ "step_rewards": all_step_rewards,
133
+ "time_s": round(ep_time, 2),
134
+ }
135
+ _flush_partial()
136
+
137
+ print(
138
+ f"[BENCH] approach={approach_id} difficulty={difficulty} ep={ep} "
139
+ f"reward={final_reward:.4f} time={ep_time:.1f}s",
140
+ flush=True,
141
+ )
142
+
143
+ total_time = time.time() - approach_start
144
+ return all_rewards, total_time
145
+
146
+
147
+ def _print_table(results: Dict[str, Tuple[str, List[float], float]]) -> None:
148
+ """Print per-approach summary and per-difficulty breakdown."""
149
+ print("\n" + "=" * 70, flush=True)
150
+ print("BENCHMARK RESULTS", flush=True)
151
+ print("=" * 70, flush=True)
152
+
153
+ header = f"{'Approach':<32} {'Mean Rwd':>9} {'Total Rwd':>10} {'Time':>8}"
154
+ print(header, flush=True)
155
+ print("-" * 70, flush=True)
156
+
157
+ for approach_id, (label, rewards, total_time) in results.items():
158
+ mean_r = sum(rewards) / len(rewards) if rewards else 0.0
159
+ total_r = sum(rewards)
160
+ print(
161
+ f" {label:<30} {mean_r:>9.4f} {total_r:>10.4f} {total_time:>7.1f}s",
162
+ flush=True,
163
+ )
164
+
165
+ print("\nPer-difficulty breakdown (mean reward):", flush=True)
166
+ print(f"{'Approach':<32} {'easy':>8} {'medium':>8} {'hard':>8}", flush=True)
167
+ print("-" * 70, flush=True)
168
+
169
+ for approach_id, (label, rewards, _) in results.items():
170
+ # rewards: [easyΓ—5, mediumΓ—5, hardΓ—5]
171
+ easy_r = sum(rewards[0:5]) / 5 if len(rewards) >= 5 else 0.0
172
+ medium_r = sum(rewards[5:10]) / 5 if len(rewards) >= 10 else 0.0
173
+ hard_r = sum(rewards[10:15]) / 5 if len(rewards) >= 15 else 0.0
174
+ print(
175
+ f" {label:<30} {easy_r:>8.4f} {medium_r:>8.4f} {hard_r:>8.4f}",
176
+ flush=True,
177
+ )
178
+
179
+ print("=" * 70, flush=True)
180
+
181
+
182
+ def main() -> None:
183
+ import httpx
184
+ from openenv.agents import AgentConfig
185
+
186
+ # Start env server only if not already running
187
+ try:
188
+ urllib.request.urlopen(f"{SERVER_URL}/health", timeout=2)
189
+ print("Environment server already running β€” skipping startup.", flush=True)
190
+ except Exception:
191
+ t = threading.Thread(target=_start_server, daemon=True)
192
+ t.start()
193
+ print("Waiting for environment server …", flush=True)
194
+ try:
195
+ _wait_for_server()
196
+ except RuntimeError as exc:
197
+ print(f"[BENCH] Server startup failed: {exc}", flush=True)
198
+ sys.exit(1)
199
+ print("Server ready.", flush=True)
200
+
201
+ config = AgentConfig(
202
+ api_key=API_KEY,
203
+ api_base=API_BASE_URL,
204
+ model=MODEL_NAME,
205
+ max_steps=MAX_STEPS,
206
+ )
207
+ env_client = httpx.Client(base_url=SERVER_URL, timeout=180.0)
208
+
209
+ _only = os.getenv("ONLY_APPROACHES", "").upper().split(",") if os.getenv("ONLY_APPROACHES") else None
210
+ APPROACHES = [
211
+ (aid, lbl) for aid, lbl in [
212
+ ("A", "A: Multi-agent (Dev+Critic)"),
213
+ ("B", "B: Long-horizon Developer"),
214
+ ("C", "C: Short-horizon Developer"),
215
+ ("D", "D: LongDev(low-res)+SimpleCritic"),
216
+ ] if _only is None or aid in _only
217
+ ]
218
+
219
+ print(f"[BENCH] Partial results β†’ {PARTIAL_PATH} (flushed after every episode)", flush=True)
220
+
221
+ results: Dict[str, Tuple[str, List[float], float]] = {}
222
+ for approach_id, label in APPROACHES:
223
+ print(f"\n{'='*60}", flush=True)
224
+ print(f"Running approach {label} ({NUM_EPISODES} eps Γ— {len(DIFFICULTIES)} difficulties = {NUM_EPISODES * len(DIFFICULTIES)} episodes)", flush=True)
225
+ print(f"{'='*60}", flush=True)
226
+ rewards, total_time = _run_approach(label, approach_id, env_client, config)
227
+ results[approach_id] = (label, rewards, total_time)
228
+ mean_r = sum(rewards) / len(rewards) if rewards else 0.0
229
+ print(f"[BENCH] Approach {approach_id} done β€” mean={mean_r:.4f} time={total_time:.1f}s", flush=True)
230
+
231
+ env_client.close()
232
+ _print_table(results)
233
+
234
+ # Write final summary JSON
235
+ summary_path = Path(f"benchmark_summary_{_RUN_ID}.json")
236
+ summary = {
237
+ approach_id: {
238
+ "label": label,
239
+ "mean_reward": round(sum(rw) / len(rw), 6) if rw else 0.0,
240
+ "total_reward": round(sum(rw), 6),
241
+ "total_time_s": round(tt, 2),
242
+ "rewards": rw,
243
+ }
244
+ for approach_id, (label, rw, tt) in results.items()
245
+ }
246
+ summary_path.write_text(json.dumps(summary, indent=2))
247
+ print(f"[BENCH] Final summary β†’ {summary_path}", flush=True)
248
+
249
+
250
+ if __name__ == "__main__":
251
+ main()
data/easy/0.html ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Login</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#f0f2f5;display:flex;align-items:center;justify-content:center;min-height:100vh;">
5
+ <div style="background:#fff;padding:40px;border-radius:8px;box-shadow:0 2px 12px rgba(0,0,0,.1);width:320px;">
6
+ <h2 style="margin:0 0 24px;text-align:center;color:#1a1a1a;">Sign In</h2>
7
+ <label style="display:block;margin-bottom:6px;font-size:14px;color:#555;">Email</label>
8
+ <input type="email" placeholder="you@example.com" style="width:100%;box-sizing:border-box;padding:10px 12px;border:1px solid #ddd;border-radius:6px;font-size:14px;margin-bottom:16px;">
9
+ <label style="display:block;margin-bottom:6px;font-size:14px;color:#555;">Password</label>
10
+ <input type="password" placeholder="β€’β€’β€’β€’β€’β€’β€’β€’" style="width:100%;box-sizing:border-box;padding:10px 12px;border:1px solid #ddd;border-radius:6px;font-size:14px;margin-bottom:24px;">
11
+ <button style="width:100%;padding:12px;background:#4f46e5;color:#fff;border:none;border-radius:6px;font-size:15px;cursor:pointer;">Sign In</button>
12
+ <p style="text-align:center;margin-top:16px;font-size:13px;color:#888;">Don't have an account? <a href="#" style="color:#4f46e5;">Sign up</a></p>
13
+ </div>
14
+ </body>
15
+ </html>
data/easy/1.html ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Blog Post</title></head>
4
+ <body style="margin:0;font-family:Georgia,serif;background:#fff;color:#222;max-width:680px;margin:0 auto;padding:40px 20px;">
5
+ <p style="font-size:13px;color:#888;margin-bottom:8px;">TECHNOLOGY Β· 5 MIN READ</p>
6
+ <h1 style="font-size:32px;line-height:1.3;margin:0 0 16px;">The Rise of AI-Assisted Development</h1>
7
+ <div style="display:flex;align-items:center;gap:12px;margin-bottom:32px;">
8
+ <div style="width:40px;height:40px;border-radius:50%;background:#4f46e5;display:flex;align-items:center;justify-content:center;color:#fff;font-weight:bold;">AJ</div>
9
+ <div>
10
+ <p style="margin:0;font-size:14px;font-weight:bold;color:#333;">Amal Joe</p>
11
+ <p style="margin:0;font-size:13px;color:#888;">April 8, 2026</p>
12
+ </div>
13
+ </div>
14
+ <div style="background:#f8f8f8;border-left:4px solid #4f46e5;padding:16px 20px;margin-bottom:28px;border-radius:0 6px 6px 0;">
15
+ <p style="margin:0;font-style:italic;color:#555;">"AI is not replacing developers β€” it's giving them superpowers."</p>
16
+ </div>
17
+ <p style="line-height:1.8;margin-bottom:20px;">Modern development has shifted dramatically with the introduction of large language models capable of understanding and generating code. Teams that once spent days on boilerplate now ship features in hours.</p>
18
+ <p style="line-height:1.8;">The key insight is that AI excels at pattern completion, while humans excel at problem definition. Together, the combination is unbeatable.</p>
19
+ </body>
20
+ </html>
data/easy/2.html ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Contact</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#fff;color:#222;max-width:560px;margin:0 auto;padding:60px 20px;">
5
+ <h1 style="font-size:28px;margin:0 0 8px;">Get in Touch</h1>
6
+ <p style="color:#666;margin:0 0 32px;">We'll get back to you within 24 hours.</p>
7
+ <div style="display:flex;gap:16px;margin-bottom:20px;">
8
+ <div style="flex:1;">
9
+ <label style="display:block;font-size:13px;font-weight:600;margin-bottom:6px;color:#444;">First Name</label>
10
+ <input style="width:100%;box-sizing:border-box;padding:10px 12px;border:1px solid #e0e0e0;border-radius:6px;font-size:14px;">
11
+ </div>
12
+ <div style="flex:1;">
13
+ <label style="display:block;font-size:13px;font-weight:600;margin-bottom:6px;color:#444;">Last Name</label>
14
+ <input style="width:100%;box-sizing:border-box;padding:10px 12px;border:1px solid #e0e0e0;border-radius:6px;font-size:14px;">
15
+ </div>
16
+ </div>
17
+ <label style="display:block;font-size:13px;font-weight:600;margin-bottom:6px;color:#444;">Email</label>
18
+ <input type="email" style="width:100%;box-sizing:border-box;padding:10px 12px;border:1px solid #e0e0e0;border-radius:6px;font-size:14px;margin-bottom:20px;">
19
+ <label style="display:block;font-size:13px;font-weight:600;margin-bottom:6px;color:#444;">Message</label>
20
+ <textarea rows="5" style="width:100%;box-sizing:border-box;padding:10px 12px;border:1px solid #e0e0e0;border-radius:6px;font-size:14px;resize:vertical;margin-bottom:24px;"></textarea>
21
+ <button style="padding:12px 28px;background:#111;color:#fff;border:none;border-radius:6px;font-size:15px;cursor:pointer;">Send Message</button>
22
+ </body>
23
+ </html>
data/easy/3.html ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Product</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#fff;color:#111;max-width:800px;margin:0 auto;padding:40px 20px;">
5
+ <div style="display:flex;gap:40px;align-items:flex-start;">
6
+ <div style="flex:1;background:#f5f5f5;border-radius:12px;aspect-ratio:1;display:flex;align-items:center;justify-content:center;">
7
+ <svg width="80" height="80" viewBox="0 0 24 24" fill="none" stroke="#999" stroke-width="1.5"><rect x="2" y="3" width="20" height="14" rx="2"/><line x1="8" y1="21" x2="16" y2="21"/><line x1="12" y1="17" x2="12" y2="21"/></svg>
8
+ </div>
9
+ <div style="flex:1;">
10
+ <p style="margin:0 0 8px;font-size:13px;color:#999;letter-spacing:.05em;">ELECTRONICS</p>
11
+ <h1 style="font-size:26px;margin:0 0 12px;">UltraView Monitor 27"</h1>
12
+ <div style="display:flex;align-items:center;gap:8px;margin-bottom:16px;">
13
+ <span style="color:#f59e0b;font-size:16px;">β˜…β˜…β˜…β˜…β˜†</span>
14
+ <span style="font-size:13px;color:#888;">4.2 (318 reviews)</span>
15
+ </div>
16
+ <p style="font-size:28px;font-weight:700;margin:0 0 20px;">$349<span style="font-size:16px;color:#888;font-weight:400;text-decoration:line-through;margin-left:10px;">$429</span></p>
17
+ <p style="line-height:1.7;color:#555;font-size:14px;margin-bottom:24px;">4K IPS display with 144Hz refresh rate, HDR400 support, and USB-C connectivity. Perfect for professionals and gamers alike.</p>
18
+ <button style="width:100%;padding:14px;background:#4f46e5;color:#fff;border:none;border-radius:8px;font-size:15px;font-weight:600;cursor:pointer;margin-bottom:12px;">Add to Cart</button>
19
+ <button style="width:100%;padding:14px;background:#fff;color:#111;border:1px solid #ddd;border-radius:8px;font-size:15px;cursor:pointer;">Save for Later</button>
20
+ </div>
21
+ </div>
22
+ </body>
23
+ </html>
data/easy/4.html ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>About</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#fff;color:#222;">
5
+ <div style="background:#0f172a;color:#fff;padding:60px 40px;text-align:center;">
6
+ <div style="width:80px;height:80px;border-radius:50%;background:#4f46e5;margin:0 auto 20px;display:flex;align-items:center;justify-content:center;font-size:28px;font-weight:700;">N</div>
7
+ <h1 style="font-size:28px;margin:0 0 10px;">Nova Technologies</h1>
8
+ <p style="color:#94a3b8;max-width:500px;margin:0 auto;">Building the next generation of developer tools since 2020.</p>
9
+ </div>
10
+ <div style="max-width:700px;margin:0 auto;padding:60px 20px;">
11
+ <h2 style="font-size:22px;margin:0 0 16px;">Our Mission</h2>
12
+ <p style="line-height:1.8;color:#555;margin-bottom:40px;">We believe every developer deserves tools that amplify their creativity, not constrain it. Our products are built with a deep empathy for the craft of software engineering.</p>
13
+ <div style="display:grid;grid-template-columns:repeat(3,1fr);gap:24px;">
14
+ <div style="text-align:center;padding:24px;background:#f8fafc;border-radius:10px;">
15
+ <p style="font-size:28px;font-weight:700;color:#4f46e5;margin:0 0 6px;">50K+</p>
16
+ <p style="font-size:13px;color:#888;margin:0;">Developers</p>
17
+ </div>
18
+ <div style="text-align:center;padding:24px;background:#f8fafc;border-radius:10px;">
19
+ <p style="font-size:28px;font-weight:700;color:#4f46e5;margin:0 0 6px;">120+</p>
20
+ <p style="font-size:13px;color:#888;margin:0;">Countries</p>
21
+ </div>
22
+ <div style="text-align:center;padding:24px;background:#f8fafc;border-radius:10px;">
23
+ <p style="font-size:28px;font-weight:700;color:#4f46e5;margin:0 0 6px;">99.9%</p>
24
+ <p style="font-size:13px;color:#888;margin:0;">Uptime</p>
25
+ </div>
26
+ </div>
27
+ </div>
28
+ </body>
29
+ </html>
data/hard/0.html ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Dashboard</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#0f172a;color:#e2e8f0;display:flex;min-height:100vh;">
5
+ <aside style="width:200px;background:#1e293b;padding:24px 0;flex-shrink:0;">
6
+ <div style="padding:0 20px 24px;font-size:18px;font-weight:700;color:#fff;border-bottom:1px solid #334155;">⚑ Pulsar</div>
7
+ <nav style="padding:16px 0;">
8
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#fff;background:#334155;text-decoration:none;margin-bottom:2px;">πŸ“Š Overview</a>
9
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#94a3b8;text-decoration:none;margin-bottom:2px;">πŸš€ Deployments</a>
10
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#94a3b8;text-decoration:none;margin-bottom:2px;">πŸ“ Projects</a>
11
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#94a3b8;text-decoration:none;margin-bottom:2px;">πŸ‘₯ Team</a>
12
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#94a3b8;text-decoration:none;">βš™οΈ Settings</a>
13
+ </nav>
14
+ </aside>
15
+ <main style="flex:1;padding:32px;">
16
+ <div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:28px;">
17
+ <h1 style="margin:0;font-size:22px;">Overview</h1>
18
+ <button style="padding:9px 18px;background:#4f46e5;color:#fff;border:none;border-radius:6px;font-size:13px;cursor:pointer;">+ New Project</button>
19
+ </div>
20
+ <div style="display:grid;grid-template-columns:repeat(4,1fr);gap:16px;margin-bottom:28px;">
21
+ <div style="background:#1e293b;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#94a3b8;">DEPLOYMENTS</p><p style="margin:0;font-size:28px;font-weight:700;">1,284</p><p style="margin:4px 0 0;font-size:12px;color:#34d399;">↑ 12% this week</p></div>
22
+ <div style="background:#1e293b;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#94a3b8;">SUCCESS RATE</p><p style="margin:0;font-size:28px;font-weight:700;">99.2%</p><p style="margin:4px 0 0;font-size:12px;color:#34d399;">↑ 0.3% this week</p></div>
23
+ <div style="background:#1e293b;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#94a3b8;">AVG BUILD TIME</p><p style="margin:0;font-size:28px;font-weight:700;">28s</p><p style="margin:4px 0 0;font-size:12px;color:#fb923c;">↓ 3s slower</p></div>
24
+ <div style="background:#1e293b;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#94a3b8;">ACTIVE PROJECTS</p><p style="margin:0;font-size:28px;font-weight:700;">47</p><p style="margin:4px 0 0;font-size:12px;color:#34d399;">↑ 3 new</p></div>
25
+ </div>
26
+ <div style="background:#1e293b;border-radius:10px;padding:24px;">
27
+ <h2 style="margin:0 0 20px;font-size:16px;">Recent Deployments</h2>
28
+ <table style="width:100%;border-collapse:collapse;font-size:13px;">
29
+ <thead><tr style="color:#64748b;border-bottom:1px solid #334155;"><th style="text-align:left;padding:8px 0;">Project</th><th style="text-align:left;padding:8px 0;">Branch</th><th style="text-align:left;padding:8px 0;">Status</th><th style="text-align:left;padding:8px 0;">Time</th></tr></thead>
30
+ <tbody>
31
+ <tr style="border-bottom:1px solid #1e293b;"><td style="padding:12px 0;color:#e2e8f0;">api-service</td><td style="padding:12px 0;color:#94a3b8;">main</td><td style="padding:12px 0;"><span style="background:#dcfce7;color:#16a34a;padding:3px 10px;border-radius:20px;font-size:11px;">βœ“ Success</span></td><td style="padding:12px 0;color:#94a3b8;">2 min ago</td></tr>
32
+ <tr style="border-bottom:1px solid #1e293b;"><td style="padding:12px 0;color:#e2e8f0;">web-frontend</td><td style="padding:12px 0;color:#94a3b8;">feat/auth</td><td style="padding:12px 0;"><span style="background:#fef3c7;color:#d97706;padding:3px 10px;border-radius:20px;font-size:11px;">⟳ Building</span></td><td style="padding:12px 0;color:#94a3b8;">5 min ago</td></tr>
33
+ <tr><td style="padding:12px 0;color:#e2e8f0;">ml-pipeline</td><td style="padding:12px 0;color:#94a3b8;">main</td><td style="padding:12px 0;"><span style="background:#fee2e2;color:#dc2626;padding:3px 10px;border-radius:20px;font-size:11px;">βœ— Failed</span></td><td style="padding:12px 0;color:#94a3b8;">18 min ago</td></tr>
34
+ </tbody>
35
+ </table>
36
+ </div>
37
+ </main>
38
+ </body>
39
+ </html>
data/hard/1.html ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>E-commerce</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#fff;color:#111;">
5
+ <nav style="background:#111;color:#fff;padding:14px 32px;display:flex;align-items:center;justify-content:space-between;">
6
+ <span style="font-weight:700;font-size:20px;letter-spacing:-.5px;">SHOP<span style="color:#f59e0b;">.</span></span>
7
+ <div style="display:flex;gap:28px;font-size:14px;color:#ccc;">
8
+ <a href="#" style="color:#fff;text-decoration:none;font-weight:500;">New</a>
9
+ <a href="#" style="color:#ccc;text-decoration:none;">Men</a>
10
+ <a href="#" style="color:#ccc;text-decoration:none;">Women</a>
11
+ <a href="#" style="color:#ccc;text-decoration:none;">Sale</a>
12
+ </div>
13
+ <div style="display:flex;gap:16px;align-items:center;">
14
+ <span style="font-size:14px;color:#ccc;">πŸ”</span>
15
+ <span style="font-size:14px;color:#ccc;">πŸ›’ 3</span>
16
+ </div>
17
+ </nav>
18
+ <div style="display:flex;max-width:1200px;margin:0 auto;padding:32px 20px;gap:32px;">
19
+ <aside style="width:200px;flex-shrink:0;">
20
+ <h3 style="font-size:13px;font-weight:700;text-transform:uppercase;letter-spacing:.08em;margin:0 0 16px;color:#888;">Category</h3>
21
+ <div style="font-size:14px;line-height:2.4;border-bottom:1px solid #eee;margin-bottom:20px;padding-bottom:16px;">
22
+ <p style="margin:0;cursor:pointer;font-weight:600;color:#111;">All (128)</p>
23
+ <p style="margin:0;cursor:pointer;color:#555;">T-Shirts (34)</p>
24
+ <p style="margin:0;cursor:pointer;color:#555;">Hoodies (28)</p>
25
+ <p style="margin:0;cursor:pointer;color:#555;">Jackets (22)</p>
26
+ <p style="margin:0;cursor:pointer;color:#555;">Pants (44)</p>
27
+ </div>
28
+ <h3 style="font-size:13px;font-weight:700;text-transform:uppercase;letter-spacing:.08em;margin:0 0 16px;color:#888;">Price</h3>
29
+ <div style="font-size:14px;line-height:2.4;">
30
+ <p style="margin:0;cursor:pointer;color:#555;">Under $50</p>
31
+ <p style="margin:0;cursor:pointer;color:#555;">$50 – $100</p>
32
+ <p style="margin:0;cursor:pointer;font-weight:600;color:#111;">$100 – $200</p>
33
+ <p style="margin:0;cursor:pointer;color:#555;">Over $200</p>
34
+ </div>
35
+ </aside>
36
+ <div style="flex:1;">
37
+ <div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:24px;">
38
+ <p style="margin:0;font-size:14px;color:#888;">128 products</p>
39
+ <select style="padding:8px 12px;border:1px solid #e5e7eb;border-radius:6px;font-size:14px;">
40
+ <option>Best selling</option><option>Price: Low to High</option><option>Price: High to Low</option>
41
+ </select>
42
+ </div>
43
+ <div style="display:grid;grid-template-columns:repeat(3,1fr);gap:20px;">
44
+ <div><div style="background:#f5f5f5;border-radius:8px;aspect-ratio:3/4;margin-bottom:12px;display:flex;align-items:center;justify-content:center;font-size:40px;">πŸ‘•</div><p style="margin:0 0 4px;font-size:14px;font-weight:500;">Essential Tee</p><p style="margin:0;font-size:14px;color:#888;">$38</p></div>
45
+ <div><div style="background:#e8f4fd;border-radius:8px;aspect-ratio:3/4;margin-bottom:12px;display:flex;align-items:center;justify-content:center;font-size:40px;position:relative;"><span style="position:absolute;top:10px;left:10px;background:#ef4444;color:#fff;font-size:11px;padding:3px 8px;border-radius:4px;">SALE</span>πŸ§₯</div><p style="margin:0 0 4px;font-size:14px;font-weight:500;">Puffer Jacket</p><p style="margin:0;font-size:14px;"><span style="color:#ef4444;font-weight:600;">$89</span> <span style="color:#bbb;text-decoration:line-through;font-size:13px;">$149</span></p></div>
46
+ <div><div style="background:#fdf5e8;border-radius:8px;aspect-ratio:3/4;margin-bottom:12px;display:flex;align-items:center;justify-content:center;font-size:40px;">πŸ‘”</div><p style="margin:0 0 4px;font-size:14px;font-weight:500;">Oxford Shirt</p><p style="margin:0;font-size:14px;color:#888;">$75</p></div>
47
+ </div>
48
+ </div>
49
+ </div>
50
+ </body>
51
+ </html>
data/hard/2.html ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Analytics</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#f8fafc;color:#1e293b;">
5
+ <nav style="background:#fff;border-bottom:1px solid #e2e8f0;padding:14px 28px;display:flex;align-items:center;justify-content:space-between;">
6
+ <div style="display:flex;align-items:center;gap:24px;">
7
+ <span style="font-weight:700;font-size:17px;color:#4f46e5;">Analytics</span>
8
+ <div style="display:flex;gap:4px;background:#f1f5f9;border-radius:6px;padding:3px;">
9
+ <button style="padding:5px 14px;background:#fff;border:none;border-radius:4px;font-size:13px;cursor:pointer;box-shadow:0 1px 3px rgba(0,0,0,.1);">7d</button>
10
+ <button style="padding:5px 14px;background:transparent;border:none;font-size:13px;cursor:pointer;color:#64748b;">30d</button>
11
+ <button style="padding:5px 14px;background:transparent;border:none;font-size:13px;cursor:pointer;color:#64748b;">90d</button>
12
+ </div>
13
+ </div>
14
+ <button style="padding:8px 16px;background:#4f46e5;color:#fff;border:none;border-radius:6px;font-size:13px;cursor:pointer;">Export CSV</button>
15
+ </nav>
16
+ <div style="padding:28px;">
17
+ <div style="display:grid;grid-template-columns:repeat(4,1fr);gap:16px;margin-bottom:24px;">
18
+ <div style="background:#fff;border:1px solid #e2e8f0;border-radius:10px;padding:20px;"><p style="margin:0 0 6px;font-size:12px;font-weight:600;color:#64748b;text-transform:uppercase;">Total Revenue</p><p style="margin:0;font-size:26px;font-weight:700;">$84,231</p><p style="margin:6px 0 0;font-size:12px;color:#16a34a;">↑ 18.2% vs last period</p></div>
19
+ <div style="background:#fff;border:1px solid #e2e8f0;border-radius:10px;padding:20px;"><p style="margin:0 0 6px;font-size:12px;font-weight:600;color:#64748b;text-transform:uppercase;">Visitors</p><p style="margin:0;font-size:26px;font-weight:700;">24,891</p><p style="margin:6px 0 0;font-size:12px;color:#16a34a;">↑ 7.4%</p></div>
20
+ <div style="background:#fff;border:1px solid #e2e8f0;border-radius:10px;padding:20px;"><p style="margin:0 0 6px;font-size:12px;font-weight:600;color:#64748b;text-transform:uppercase;">Conversion</p><p style="margin:0;font-size:26px;font-weight:700;">3.6%</p><p style="margin:6px 0 0;font-size:12px;color:#ef4444;">↓ 0.4%</p></div>
21
+ <div style="background:#fff;border:1px solid #e2e8f0;border-radius:10px;padding:20px;"><p style="margin:0 0 6px;font-size:12px;font-weight:600;color:#64748b;text-transform:uppercase;">Avg Order</p><p style="margin:0;font-size:26px;font-weight:700;">$94.30</p><p style="margin:6px 0 0;font-size:12px;color:#16a34a;">↑ 10.1%</p></div>
22
+ </div>
23
+ <div style="display:grid;grid-template-columns:2fr 1fr;gap:16px;">
24
+ <div style="background:#fff;border:1px solid #e2e8f0;border-radius:10px;padding:24px;">
25
+ <h3 style="margin:0 0 20px;font-size:15px;">Revenue Over Time</h3>
26
+ <div style="display:flex;align-items:flex-end;gap:8px;height:120px;">
27
+ <div style="flex:1;background:#ede9fe;border-radius:4px 4px 0 0;height:40%;"></div>
28
+ <div style="flex:1;background:#ede9fe;border-radius:4px 4px 0 0;height:55%;"></div>
29
+ <div style="flex:1;background:#ede9fe;border-radius:4px 4px 0 0;height:48%;"></div>
30
+ <div style="flex:1;background:#ede9fe;border-radius:4px 4px 0 0;height:72%;"></div>
31
+ <div style="flex:1;background:#ede9fe;border-radius:4px 4px 0 0;height:60%;"></div>
32
+ <div style="flex:1;background:#4f46e5;border-radius:4px 4px 0 0;height:85%;"></div>
33
+ <div style="flex:1;background:#4f46e5;border-radius:4px 4px 0 0;height:100%;"></div>
34
+ </div>
35
+ <div style="display:flex;gap:8px;margin-top:8px;font-size:11px;color:#94a3b8;">
36
+ <span style="flex:1;text-align:center;">Mon</span><span style="flex:1;text-align:center;">Tue</span><span style="flex:1;text-align:center;">Wed</span><span style="flex:1;text-align:center;">Thu</span><span style="flex:1;text-align:center;">Fri</span><span style="flex:1;text-align:center;">Sat</span><span style="flex:1;text-align:center;">Sun</span>
37
+ </div>
38
+ </div>
39
+ <div style="background:#fff;border:1px solid #e2e8f0;border-radius:10px;padding:24px;">
40
+ <h3 style="margin:0 0 16px;font-size:15px;">Top Sources</h3>
41
+ <div style="font-size:13px;">
42
+ <div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:12px;"><span>Google</span><div style="display:flex;align-items:center;gap:8px;"><div style="width:80px;height:6px;background:#e2e8f0;border-radius:3px;overflow:hidden;"><div style="width:62%;height:100%;background:#4f46e5;border-radius:3px;"></div></div><span style="color:#888;font-size:12px;">62%</span></div></div>
43
+ <div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:12px;"><span>Direct</span><div style="display:flex;align-items:center;gap:8px;"><div style="width:80px;height:6px;background:#e2e8f0;border-radius:3px;overflow:hidden;"><div style="width:22%;height:100%;background:#818cf8;border-radius:3px;"></div></div><span style="color:#888;font-size:12px;">22%</span></div></div>
44
+ <div style="display:flex;justify-content:space-between;align-items:center;"><span>Social</span><div style="display:flex;align-items:center;gap:8px;"><div style="width:80px;height:6px;background:#e2e8f0;border-radius:3px;overflow:hidden;"><div style="width:16%;height:100%;background:#c4b5fd;border-radius:3px;"></div></div><span style="color:#888;font-size:12px;">16%</span></div></div>
45
+ </div>
46
+ </div>
47
+ </div>
48
+ </div>
49
+ </body>
50
+ </html>
data/hard/3.html ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Docs</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#fff;color:#1e293b;display:flex;flex-direction:column;min-height:100vh;">
5
+ <nav style="border-bottom:1px solid #e2e8f0;padding:14px 28px;display:flex;align-items:center;justify-content:space-between;background:#fff;position:sticky;top:0;z-index:10;">
6
+ <div style="display:flex;align-items:center;gap:32px;">
7
+ <span style="font-weight:700;font-size:17px;color:#4f46e5;">Docs</span>
8
+ <div style="display:flex;gap:20px;font-size:14px;color:#64748b;">
9
+ <a href="#" style="color:#4f46e5;text-decoration:none;font-weight:500;">Guide</a>
10
+ <a href="#" style="color:#64748b;text-decoration:none;">API Reference</a>
11
+ <a href="#" style="color:#64748b;text-decoration:none;">Examples</a>
12
+ </div>
13
+ </div>
14
+ <div style="background:#f1f5f9;border-radius:6px;padding:7px 14px;font-size:13px;color:#94a3b8;width:200px;">πŸ” Search docs...</div>
15
+ </nav>
16
+ <div style="display:flex;flex:1;max-width:1100px;margin:0 auto;padding:32px 20px;gap:40px;width:100%;box-sizing:border-box;">
17
+ <aside style="width:200px;flex-shrink:0;">
18
+ <p style="font-size:11px;font-weight:700;text-transform:uppercase;letter-spacing:.08em;color:#94a3b8;margin:0 0 12px;">Getting Started</p>
19
+ <nav style="font-size:14px;margin-bottom:24px;">
20
+ <a style="display:block;padding:5px 0;color:#4f46e5;font-weight:600;text-decoration:none;">Introduction</a>
21
+ <a style="display:block;padding:5px 0;color:#64748b;text-decoration:none;">Installation</a>
22
+ <a style="display:block;padding:5px 0;color:#64748b;text-decoration:none;">Quick Start</a>
23
+ </nav>
24
+ <p style="font-size:11px;font-weight:700;text-transform:uppercase;letter-spacing:.08em;color:#94a3b8;margin:0 0 12px;">Core Concepts</p>
25
+ <nav style="font-size:14px;margin-bottom:24px;">
26
+ <a style="display:block;padding:5px 0;color:#64748b;text-decoration:none;">Environments</a>
27
+ <a style="display:block;padding:5px 0;color:#64748b;text-decoration:none;">Actions & Obs</a>
28
+ <a style="display:block;padding:5px 0;color:#64748b;text-decoration:none;">Rewards</a>
29
+ </nav>
30
+ <p style="font-size:11px;font-weight:700;text-transform:uppercase;letter-spacing:.08em;color:#94a3b8;margin:0 0 12px;">API</p>
31
+ <nav style="font-size:14px;">
32
+ <a style="display:block;padding:5px 0;color:#64748b;text-decoration:none;">POST /reset</a>
33
+ <a style="display:block;padding:5px 0;color:#64748b;text-decoration:none;">POST /step</a>
34
+ <a style="display:block;padding:5px 0;color:#64748b;text-decoration:none;">GET /state</a>
35
+ </nav>
36
+ </aside>
37
+ <main style="flex:1;min-width:0;">
38
+ <div style="display:flex;gap:8px;font-size:12px;color:#94a3b8;margin-bottom:24px;">
39
+ <span>Docs</span><span>β€Ί</span><span>Getting Started</span><span>β€Ί</span><span style="color:#4f46e5;">Introduction</span>
40
+ </div>
41
+ <h1 style="font-size:30px;margin:0 0 12px;">Introduction</h1>
42
+ <p style="color:#64748b;font-size:15px;margin:0 0 28px;line-height:1.7;">OpenEnv is a standard HTTP interface for building reinforcement learning environments. It provides a simple REST API that any agent can interact with, regardless of programming language.</p>
43
+ <div style="background:#f8fafc;border:1px solid #e2e8f0;border-radius:8px;padding:20px;margin-bottom:28px;">
44
+ <p style="margin:0 0 10px;font-size:13px;font-weight:600;color:#64748b;">INSTALLATION</p>
45
+ <code style="font-family:monospace;font-size:13px;color:#4f46e5;">pip install openenv-core</code>
46
+ </div>
47
+ <h2 style="font-size:20px;margin:0 0 14px;">How it works</h2>
48
+ <p style="color:#475569;line-height:1.8;margin-bottom:16px;">Each environment exposes three endpoints: <code style="background:#f1f5f9;padding:2px 6px;border-radius:4px;font-size:13px;">/reset</code> to start an episode, <code style="background:#f1f5f9;padding:2px 6px;border-radius:4px;font-size:13px;">/step</code> to submit an action, and <code style="background:#f1f5f9;padding:2px 6px;border-radius:4px;font-size:13px;">/state</code> to inspect current state.</p>
49
+ <div style="display:flex;gap:12px;margin-top:32px;font-size:14px;">
50
+ <button style="padding:10px 20px;background:#4f46e5;color:#fff;border:none;border-radius:6px;cursor:pointer;">Next: Installation β†’</button>
51
+ </div>
52
+ </main>
53
+ </div>
54
+ </body>
55
+ </html>
data/hard/4.html ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Kanban</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#f1f5f9;color:#1e293b;min-height:100vh;">
5
+ <nav style="background:#fff;border-bottom:1px solid #e2e8f0;padding:14px 24px;display:flex;align-items:center;justify-content:space-between;">
6
+ <div style="display:flex;align-items:center;gap:16px;">
7
+ <span style="font-weight:700;font-size:16px;">πŸ—‚ Sprint 24</span>
8
+ <span style="font-size:13px;color:#94a3b8;">Apr 1 – Apr 14</span>
9
+ </div>
10
+ <div style="display:flex;align-items:center;gap:10px;">
11
+ <div style="display:flex;"><div style="width:28px;height:28px;border-radius:50%;background:#4f46e5;color:#fff;font-size:11px;display:flex;align-items:center;justify-content:center;border:2px solid #fff;">A</div><div style="width:28px;height:28px;border-radius:50%;background:#34d399;color:#fff;font-size:11px;display:flex;align-items:center;justify-content:center;border:2px solid #fff;margin-left:-8px;">M</div><div style="width:28px;height:28px;border-radius:50%;background:#fb923c;color:#fff;font-size:11px;display:flex;align-items:center;justify-content:center;border:2px solid #fff;margin-left:-8px;">P</div></div>
12
+ <button style="padding:7px 14px;background:#4f46e5;color:#fff;border:none;border-radius:6px;font-size:13px;cursor:pointer;">+ Add Task</button>
13
+ </div>
14
+ </nav>
15
+ <div style="display:flex;gap:16px;padding:24px;overflow-x:auto;">
16
+ <div style="min-width:260px;background:#e2e8f0;border-radius:10px;padding:16px;">
17
+ <div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:14px;"><h3 style="margin:0;font-size:13px;font-weight:700;text-transform:uppercase;letter-spacing:.05em;color:#64748b;">Backlog</h3><span style="background:#94a3b8;color:#fff;font-size:11px;padding:2px 8px;border-radius:10px;">5</span></div>
18
+ <div style="background:#fff;border-radius:8px;padding:14px;margin-bottom:10px;box-shadow:0 1px 3px rgba(0,0,0,.06);"><p style="margin:0 0 8px;font-size:13px;font-weight:500;">Redesign onboarding flow</p><div style="display:flex;justify-content:space-between;align-items:center;"><span style="background:#ede9fe;color:#7c3aed;font-size:11px;padding:2px 8px;border-radius:4px;">Design</span><span style="font-size:11px;color:#94a3b8;">P</span></div></div>
19
+ <div style="background:#fff;border-radius:8px;padding:14px;margin-bottom:10px;box-shadow:0 1px 3px rgba(0,0,0,.06);"><p style="margin:0 0 8px;font-size:13px;font-weight:500;">Add dark mode support</p><div style="display:flex;justify-content:space-between;align-items:center;"><span style="background:#dcfce7;color:#16a34a;font-size:11px;padding:2px 8px;border-radius:4px;">Frontend</span><span style="font-size:11px;color:#94a3b8;">A</span></div></div>
20
+ </div>
21
+ <div style="min-width:260px;background:#e2e8f0;border-radius:10px;padding:16px;">
22
+ <div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:14px;"><h3 style="margin:0;font-size:13px;font-weight:700;text-transform:uppercase;letter-spacing:.05em;color:#64748b;">In Progress</h3><span style="background:#f59e0b;color:#fff;font-size:11px;padding:2px 8px;border-radius:10px;">3</span></div>
23
+ <div style="background:#fff;border-radius:8px;padding:14px;margin-bottom:10px;box-shadow:0 1px 3px rgba(0,0,0,.06);border-left:3px solid #4f46e5;"><p style="margin:0 0 8px;font-size:13px;font-weight:500;">Implement OAuth 2.0 login</p><div style="display:flex;justify-content:space-between;align-items:center;"><span style="background:#fee2e2;color:#dc2626;font-size:11px;padding:2px 8px;border-radius:4px;">πŸ”΄ High</span><span style="font-size:11px;color:#94a3b8;">M</span></div></div>
24
+ <div style="background:#fff;border-radius:8px;padding:14px;margin-bottom:10px;box-shadow:0 1px 3px rgba(0,0,0,.06);border-left:3px solid #4f46e5;"><p style="margin:0 0 8px;font-size:13px;font-weight:500;">Write API documentation</p><div style="display:flex;justify-content:space-between;align-items:center;"><span style="background:#fef3c7;color:#d97706;font-size:11px;padding:2px 8px;border-radius:4px;">Docs</span><span style="font-size:11px;color:#94a3b8;">A</span></div></div>
25
+ </div>
26
+ <div style="min-width:260px;background:#e2e8f0;border-radius:10px;padding:16px;">
27
+ <div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:14px;"><h3 style="margin:0;font-size:13px;font-weight:700;text-transform:uppercase;letter-spacing:.05em;color:#64748b;">Review</h3><span style="background:#6366f1;color:#fff;font-size:11px;padding:2px 8px;border-radius:10px;">2</span></div>
28
+ <div style="background:#fff;border-radius:8px;padding:14px;margin-bottom:10px;box-shadow:0 1px 3px rgba(0,0,0,.06);"><p style="margin:0 0 8px;font-size:13px;font-weight:500;">Fix payment webhook bug</p><div style="display:flex;justify-content:space-between;align-items:center;"><span style="background:#fee2e2;color:#dc2626;font-size:11px;padding:2px 8px;border-radius:4px;">πŸ”΄ Critical</span><span style="font-size:11px;color:#94a3b8;">P</span></div></div>
29
+ </div>
30
+ <div style="min-width:260px;background:#e2e8f0;border-radius:10px;padding:16px;">
31
+ <div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:14px;"><h3 style="margin:0;font-size:13px;font-weight:700;text-transform:uppercase;letter-spacing:.05em;color:#64748b;">Done</h3><span style="background:#16a34a;color:#fff;font-size:11px;padding:2px 8px;border-radius:10px;">8</span></div>
32
+ <div style="background:#fff;border-radius:8px;padding:14px;margin-bottom:10px;box-shadow:0 1px 3px rgba(0,0,0,.06);opacity:.7;"><p style="margin:0 0 8px;font-size:13px;font-weight:500;text-decoration:line-through;color:#94a3b8;">Setup CI/CD pipeline</p><div style="display:flex;justify-content:space-between;align-items:center;"><span style="background:#dcfce7;color:#16a34a;font-size:11px;padding:2px 8px;border-radius:4px;">DevOps</span><span style="font-size:11px;color:#94a3b8;">M</span></div></div>
33
+ <div style="background:#fff;border-radius:8px;padding:14px;box-shadow:0 1px 3px rgba(0,0,0,.06);opacity:.7;"><p style="margin:0 0 8px;font-size:13px;font-weight:500;text-decoration:line-through;color:#94a3b8;">Database schema migration</p><div style="display:flex;justify-content:space-between;align-items:center;"><span style="background:#f1f5f9;color:#64748b;font-size:11px;padding:2px 8px;border-radius:4px;">Backend</span><span style="font-size:11px;color:#94a3b8;">A</span></div></div>
34
+ </div>
35
+ </div>
36
+ </body>
37
+ </html>
data/medium/0.html ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>SaaS Landing</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#fff;color:#111;">
5
+ <nav style="display:flex;align-items:center;justify-content:space-between;padding:16px 40px;border-bottom:1px solid #f0f0f0;">
6
+ <span style="font-size:20px;font-weight:700;color:#4f46e5;">Streamline</span>
7
+ <div style="display:flex;gap:28px;font-size:14px;color:#555;">
8
+ <a href="#" style="text-decoration:none;color:#555;">Features</a>
9
+ <a href="#" style="text-decoration:none;color:#555;">Pricing</a>
10
+ <a href="#" style="text-decoration:none;color:#555;">Docs</a>
11
+ </div>
12
+ <button style="padding:9px 20px;background:#4f46e5;color:#fff;border:none;border-radius:6px;font-size:14px;cursor:pointer;">Get Started</button>
13
+ </nav>
14
+ <div style="text-align:center;padding:80px 20px 60px;background:linear-gradient(135deg,#f0f0ff 0%,#fff 100%);">
15
+ <span style="background:#ede9fe;color:#4f46e5;font-size:12px;font-weight:600;padding:4px 12px;border-radius:20px;letter-spacing:.05em;">NOW IN BETA</span>
16
+ <h1 style="font-size:48px;font-weight:800;margin:20px 0 16px;line-height:1.2;">Ship faster,<br>break nothing.</h1>
17
+ <p style="font-size:18px;color:#666;max-width:480px;margin:0 auto 32px;line-height:1.6;">The all-in-one platform for teams that care about quality. CI/CD, monitoring, and deployment in one place.</p>
18
+ <div style="display:flex;gap:12px;justify-content:center;">
19
+ <button style="padding:14px 28px;background:#4f46e5;color:#fff;border:none;border-radius:8px;font-size:15px;font-weight:600;cursor:pointer;">Start for free</button>
20
+ <button style="padding:14px 28px;background:#fff;color:#111;border:1px solid #ddd;border-radius:8px;font-size:15px;cursor:pointer;">Watch demo β†’</button>
21
+ </div>
22
+ </div>
23
+ <div style="display:grid;grid-template-columns:repeat(3,1fr);gap:24px;max-width:900px;margin:0 auto;padding:60px 20px;">
24
+ <div style="padding:28px;border:1px solid #f0f0f0;border-radius:12px;">
25
+ <div style="width:44px;height:44px;background:#ede9fe;border-radius:10px;margin-bottom:16px;display:flex;align-items:center;justify-content:center;font-size:22px;">⚑</div>
26
+ <h3 style="margin:0 0 10px;font-size:17px;">Fast Deploys</h3>
27
+ <p style="margin:0;font-size:14px;color:#666;line-height:1.6;">Push to production in under 30 seconds with zero downtime.</p>
28
+ </div>
29
+ <div style="padding:28px;border:1px solid #f0f0f0;border-radius:12px;">
30
+ <div style="width:44px;height:44px;background:#dcfce7;border-radius:10px;margin-bottom:16px;display:flex;align-items:center;justify-content:center;font-size:22px;">πŸ”’</div>
31
+ <h3 style="margin:0 0 10px;font-size:17px;">Secure by Default</h3>
32
+ <p style="margin:0;font-size:14px;color:#666;line-height:1.6;">SOC2 compliant. Secrets management and audit logs built in.</p>
33
+ </div>
34
+ <div style="padding:28px;border:1px solid #f0f0f0;border-radius:12px;">
35
+ <div style="width:44px;height:44px;background:#fef3c7;border-radius:10px;margin-bottom:16px;display:flex;align-items:center;justify-content:center;font-size:22px;">πŸ“Š</div>
36
+ <h3 style="margin:0 0 10px;font-size:17px;">Live Monitoring</h3>
37
+ <p style="margin:0;font-size:14px;color:#666;line-height:1.6;">Real-time metrics, error tracking, and alerting out of the box.</p>
38
+ </div>
39
+ </div>
40
+ </body>
41
+ </html>
data/medium/1.html ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Pricing</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#f9fafb;color:#111;padding:60px 20px;">
5
+ <h1 style="text-align:center;font-size:36px;margin:0 0 12px;">Simple, transparent pricing</h1>
6
+ <p style="text-align:center;color:#666;font-size:16px;margin:0 0 48px;">Start free. Scale as you grow.</p>
7
+ <div style="display:flex;gap:24px;max-width:900px;margin:0 auto;align-items:flex-start;">
8
+ <div style="flex:1;background:#fff;border:1px solid #e5e7eb;border-radius:12px;padding:32px;">
9
+ <h2 style="font-size:18px;margin:0 0 8px;">Hobby</h2>
10
+ <p style="font-size:13px;color:#888;margin:0 0 20px;">For personal projects</p>
11
+ <p style="font-size:36px;font-weight:700;margin:0 0 24px;">$0<span style="font-size:14px;font-weight:400;color:#888;">/mo</span></p>
12
+ <ul style="list-style:none;padding:0;margin:0 0 28px;font-size:14px;color:#555;line-height:2;">
13
+ <li>βœ“ 3 projects</li><li>βœ“ 1 GB storage</li><li>βœ“ Community support</li>
14
+ </ul>
15
+ <button style="width:100%;padding:12px;background:#f3f4f6;color:#111;border:none;border-radius:8px;font-size:14px;cursor:pointer;">Get started</button>
16
+ </div>
17
+ <div style="flex:1;background:#4f46e5;border-radius:12px;padding:32px;color:#fff;position:relative;">
18
+ <span style="position:absolute;top:-12px;left:50%;transform:translateX(-50%);background:#fbbf24;color:#000;font-size:11px;font-weight:700;padding:4px 12px;border-radius:20px;">MOST POPULAR</span>
19
+ <h2 style="font-size:18px;margin:0 0 8px;">Pro</h2>
20
+ <p style="font-size:13px;color:#c7d2fe;margin:0 0 20px;">For growing teams</p>
21
+ <p style="font-size:36px;font-weight:700;margin:0 0 24px;">$29<span style="font-size:14px;font-weight:400;color:#c7d2fe;">/mo</span></p>
22
+ <ul style="list-style:none;padding:0;margin:0 0 28px;font-size:14px;color:#e0e7ff;line-height:2;">
23
+ <li>βœ“ Unlimited projects</li><li>βœ“ 50 GB storage</li><li>βœ“ Priority support</li><li>βœ“ Team collaboration</li>
24
+ </ul>
25
+ <button style="width:100%;padding:12px;background:#fff;color:#4f46e5;border:none;border-radius:8px;font-size:14px;font-weight:600;cursor:pointer;">Start free trial</button>
26
+ </div>
27
+ <div style="flex:1;background:#fff;border:1px solid #e5e7eb;border-radius:12px;padding:32px;">
28
+ <h2 style="font-size:18px;margin:0 0 8px;">Enterprise</h2>
29
+ <p style="font-size:13px;color:#888;margin:0 0 20px;">For large organisations</p>
30
+ <p style="font-size:36px;font-weight:700;margin:0 0 24px;">Custom</p>
31
+ <ul style="list-style:none;padding:0;margin:0 0 28px;font-size:14px;color:#555;line-height:2;">
32
+ <li>βœ“ Unlimited everything</li><li>βœ“ SSO & SAML</li><li>βœ“ SLA guarantee</li><li>βœ“ Dedicated support</li>
33
+ </ul>
34
+ <button style="width:100%;padding:12px;background:#f3f4f6;color:#111;border:none;border-radius:8px;font-size:14px;cursor:pointer;">Contact sales</button>
35
+ </div>
36
+ </div>
37
+ </body>
38
+ </html>
data/medium/2.html ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Team</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#fff;color:#111;padding:60px 20px;">
5
+ <h1 style="text-align:center;font-size:34px;margin:0 0 12px;">Meet the Team</h1>
6
+ <p style="text-align:center;color:#888;margin:0 0 48px;">The people building the future of development tooling.</p>
7
+ <div style="display:grid;grid-template-columns:repeat(4,1fr);gap:24px;max-width:960px;margin:0 auto;">
8
+ <div style="text-align:center;">
9
+ <div style="width:80px;height:80px;border-radius:50%;background:#818cf8;margin:0 auto 14px;display:flex;align-items:center;justify-content:center;color:#fff;font-size:24px;font-weight:700;">S</div>
10
+ <p style="font-weight:600;margin:0 0 4px;">Sarah Chen</p>
11
+ <p style="font-size:13px;color:#888;margin:0 0 10px;">CEO & Co-founder</p>
12
+ <p style="font-size:13px;color:#555;line-height:1.5;">Previously at Google Brain. PhD in ML from Stanford.</p>
13
+ </div>
14
+ <div style="text-align:center;">
15
+ <div style="width:80px;height:80px;border-radius:50%;background:#34d399;margin:0 auto 14px;display:flex;align-items:center;justify-content:center;color:#fff;font-size:24px;font-weight:700;">M</div>
16
+ <p style="font-weight:600;margin:0 0 4px;">Marcus Lee</p>
17
+ <p style="font-size:13px;color:#888;margin:0 0 10px;">CTO & Co-founder</p>
18
+ <p style="font-size:13px;color:#555;line-height:1.5;">Built infra at Stripe. Open source contributor.</p>
19
+ </div>
20
+ <div style="text-align:center;">
21
+ <div style="width:80px;height:80px;border-radius:50%;background:#fb923c;margin:0 auto 14px;display:flex;align-items:center;justify-content:center;color:#fff;font-size:24px;font-weight:700;">P</div>
22
+ <p style="font-weight:600;margin:0 0 4px;">Priya Nair</p>
23
+ <p style="font-size:13px;color:#888;margin:0 0 10px;">Head of Design</p>
24
+ <p style="font-size:13px;color:#555;line-height:1.5;">Design systems expert. Ex-Figma and Notion.</p>
25
+ </div>
26
+ <div style="text-align:center;">
27
+ <div style="width:80px;height:80px;border-radius:50%;background:#f472b6;margin:0 auto 14px;display:flex;align-items:center;justify-content:center;color:#fff;font-size:24px;font-weight:700;">J</div>
28
+ <p style="font-weight:600;margin:0 0 4px;">Jake Torres</p>
29
+ <p style="font-size:13px;color:#888;margin:0 0 10px;">Lead Engineer</p>
30
+ <p style="font-size:13px;color:#555;line-height:1.5;">Full-stack dev. Rust and TypeScript enthusiast.</p>
31
+ </div>
32
+ </div>
33
+ </body>
34
+ </html>
data/medium/3.html ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Article</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#fff;color:#111;">
5
+ <nav style="display:flex;align-items:center;justify-content:space-between;padding:14px 32px;border-bottom:1px solid #eee;">
6
+ <span style="font-weight:700;font-size:18px;">DevDigest</span>
7
+ <div style="display:flex;gap:24px;font-size:14px;color:#555;">
8
+ <a href="#" style="text-decoration:none;color:#555;">Articles</a>
9
+ <a href="#" style="text-decoration:none;color:#555;">Newsletter</a>
10
+ <a href="#" style="text-decoration:none;color:#555;">Jobs</a>
11
+ </div>
12
+ </nav>
13
+ <div style="display:flex;max-width:1000px;margin:40px auto;padding:0 20px;gap:48px;">
14
+ <main style="flex:2;">
15
+ <span style="background:#fee2e2;color:#dc2626;font-size:11px;font-weight:700;padding:3px 10px;border-radius:4px;">RUST</span>
16
+ <h1 style="font-size:28px;line-height:1.3;margin:12px 0 20px;">Why Rust is Winning the Systems Programming War</h1>
17
+ <p style="font-size:14px;color:#888;margin-bottom:24px;">By <strong style="color:#555;">Alex Kim</strong> Β· 8 min read Β· Apr 8, 2026</p>
18
+ <p style="line-height:1.8;color:#333;margin-bottom:18px;">Memory safety without garbage collection was once considered impossible. Rust has proved the skeptics wrong, and major tech companies are taking notice.</p>
19
+ <p style="line-height:1.8;color:#333;margin-bottom:18px;">The Linux kernel, Android, and the Windows kernel have all started incorporating Rust. This isn't a trend β€” it's a fundamental shift in how critical infrastructure is built.</p>
20
+ <h2 style="font-size:20px;margin:32px 0 12px;">The Ownership Model</h2>
21
+ <p style="line-height:1.8;color:#333;">Rust's ownership system enforces memory safety at compile time. Every value has exactly one owner, and the compiler ensures that references never outlive their data.</p>
22
+ </main>
23
+ <aside style="flex:1;">
24
+ <div style="background:#f9fafb;border-radius:10px;padding:20px;margin-bottom:24px;">
25
+ <h3 style="margin:0 0 14px;font-size:15px;">Trending</h3>
26
+ <div style="font-size:13px;color:#555;line-height:2.2;border-top:1px solid #eee;padding-top:12px;">
27
+ <p style="margin:0;">1. Go vs Rust in 2026</p>
28
+ <p style="margin:0;">2. The React Server Components Guide</p>
29
+ <p style="margin:0;">3. Postgres vs SQLite at scale</p>
30
+ <p style="margin:0;">4. Building your own LLM</p>
31
+ </div>
32
+ </div>
33
+ <div style="background:#ede9fe;border-radius:10px;padding:20px;">
34
+ <h3 style="margin:0 0 8px;font-size:15px;color:#4f46e5;">Newsletter</h3>
35
+ <p style="font-size:13px;color:#555;margin:0 0 12px;">Weekly digest of the best dev articles.</p>
36
+ <input placeholder="your@email.com" style="width:100%;box-sizing:border-box;padding:8px 12px;border:1px solid #c4b5fd;border-radius:6px;font-size:13px;margin-bottom:8px;">
37
+ <button style="width:100%;padding:8px;background:#4f46e5;color:#fff;border:none;border-radius:6px;font-size:13px;cursor:pointer;">Subscribe</button>
38
+ </div>
39
+ </aside>
40
+ </div>
41
+ </body>
42
+ </html>
data/medium/4.html ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Settings</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#f9fafb;color:#111;display:flex;min-height:100vh;">
5
+ <aside style="width:220px;background:#fff;border-right:1px solid #e5e7eb;padding:24px 0;">
6
+ <div style="padding:0 20px 24px;border-bottom:1px solid #e5e7eb;">
7
+ <div style="display:flex;align-items:center;gap:10px;">
8
+ <div style="width:36px;height:36px;border-radius:50%;background:#4f46e5;color:#fff;display:flex;align-items:center;justify-content:center;font-weight:700;">A</div>
9
+ <div><p style="margin:0;font-size:13px;font-weight:600;">Amal Joe</p><p style="margin:0;font-size:11px;color:#888;">Pro Plan</p></div>
10
+ </div>
11
+ </div>
12
+ <nav style="padding:12px 0;">
13
+ <a style="display:block;padding:10px 20px;font-size:14px;color:#4f46e5;background:#ede9fe;font-weight:600;text-decoration:none;">Profile</a>
14
+ <a style="display:block;padding:10px 20px;font-size:14px;color:#555;text-decoration:none;">Security</a>
15
+ <a style="display:block;padding:10px 20px;font-size:14px;color:#555;text-decoration:none;">Notifications</a>
16
+ <a style="display:block;padding:10px 20px;font-size:14px;color:#555;text-decoration:none;">Billing</a>
17
+ <a style="display:block;padding:10px 20px;font-size:14px;color:#555;text-decoration:none;">Integrations</a>
18
+ </nav>
19
+ </aside>
20
+ <main style="flex:1;padding:40px;">
21
+ <h1 style="font-size:22px;margin:0 0 6px;">Profile Settings</h1>
22
+ <p style="color:#888;font-size:14px;margin:0 0 32px;">Manage your personal information.</p>
23
+ <div style="background:#fff;border:1px solid #e5e7eb;border-radius:10px;padding:28px;max-width:560px;">
24
+ <div style="display:flex;gap:16px;margin-bottom:24px;">
25
+ <div style="flex:1;"><label style="font-size:12px;font-weight:600;color:#444;display:block;margin-bottom:6px;">First Name</label><input value="Amal" style="width:100%;box-sizing:border-box;padding:9px 12px;border:1px solid #e5e7eb;border-radius:6px;font-size:14px;"></div>
26
+ <div style="flex:1;"><label style="font-size:12px;font-weight:600;color:#444;display:block;margin-bottom:6px;">Last Name</label><input value="Joe" style="width:100%;box-sizing:border-box;padding:9px 12px;border:1px solid #e5e7eb;border-radius:6px;font-size:14px;"></div>
27
+ </div>
28
+ <label style="font-size:12px;font-weight:600;color:#444;display:block;margin-bottom:6px;">Email</label>
29
+ <input value="amaljoe88@gmail.com" style="width:100%;box-sizing:border-box;padding:9px 12px;border:1px solid #e5e7eb;border-radius:6px;font-size:14px;margin-bottom:24px;">
30
+ <label style="font-size:12px;font-weight:600;color:#444;display:block;margin-bottom:6px;">Bio</label>
31
+ <textarea rows="3" style="width:100%;box-sizing:border-box;padding:9px 12px;border:1px solid #e5e7eb;border-radius:6px;font-size:14px;margin-bottom:24px;resize:vertical;">Building AI tools for developers.</textarea>
32
+ <button style="padding:10px 24px;background:#4f46e5;color:#fff;border:none;border-radius:6px;font-size:14px;cursor:pointer;">Save Changes</button>
33
+ </div>
34
+ </main>
35
+ </body>
36
+ </html>
data/tests/0/expected_scores.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "perfect": 0.9773,
3
+ "minor_diff": 0.8837,
4
+ "bad_colors": 0.7253,
5
+ "half_styled": 0.4663,
6
+ "no_layout": 0.5222,
7
+ "no_style": 0.4424,
8
+ "blank": 0.0
9
+ }
data/tests/0/meta.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "source": "easy/0",
3
+ "difficulty": "easy",
4
+ "idx": 0
5
+ }
data/tests/0/reference.html ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Login</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#f0f2f5;display:flex;align-items:center;justify-content:center;min-height:100vh;">
5
+ <div style="background:#fff;padding:40px;border-radius:8px;box-shadow:0 2px 12px rgba(0,0,0,.1);width:320px;">
6
+ <h2 style="margin:0 0 24px;text-align:center;color:#1a1a1a;">Sign In</h2>
7
+ <label style="display:block;margin-bottom:6px;font-size:14px;color:#555;">Email</label>
8
+ <input type="email" placeholder="you@example.com" style="width:100%;box-sizing:border-box;padding:10px 12px;border:1px solid #ddd;border-radius:6px;font-size:14px;margin-bottom:16px;">
9
+ <label style="display:block;margin-bottom:6px;font-size:14px;color:#555;">Password</label>
10
+ <input type="password" placeholder="β€’β€’β€’β€’β€’β€’β€’β€’" style="width:100%;box-sizing:border-box;padding:10px 12px;border:1px solid #ddd;border-radius:6px;font-size:14px;margin-bottom:24px;">
11
+ <button style="width:100%;padding:12px;background:#4f46e5;color:#fff;border:none;border-radius:6px;font-size:15px;cursor:pointer;">Sign In</button>
12
+ <p style="text-align:center;margin-top:16px;font-size:13px;color:#888;">Don't have an account? <a href="#" style="color:#4f46e5;">Sign up</a></p>
13
+ </div>
14
+ </body>
15
+ </html>
data/tests/0/variants/bad_colors.html ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Login</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#0f0d0a;display:flex;align-items:center;justify-content:center;min-height:100vh;">
5
+ <div style="background:#fff;padding:40px;border-radius:8px;box-shadow:0 2px 12px rgba(0,0,0,.1);width:320px;">
6
+ <h2 style="margin:0 0 24px;text-align:center;color:#e5e5e5;">Sign In</h2>
7
+ <label style="display:block;margin-bottom:6px;font-size:14px;color:#555;">Email</label>
8
+ <input type="email" placeholder="you@example.com" style="width:100%;box-sizing:border-box;padding:10px 12px;border:1px solid #ddd;border-radius:6px;font-size:14px;margin-bottom:16px;">
9
+ <label style="display:block;margin-bottom:6px;font-size:14px;color:#555;">Password</label>
10
+ <input type="password" placeholder="β€’β€’β€’β€’β€’β€’β€’β€’" style="width:100%;box-sizing:border-box;padding:10px 12px;border:1px solid #ddd;border-radius:6px;font-size:14px;margin-bottom:24px;">
11
+ <button style="width:100%;padding:12px;background:#b0b91a;color:#fff;border:none;border-radius:6px;font-size:15px;cursor:pointer;">Sign In</button>
12
+ <p style="text-align:center;margin-top:16px;font-size:13px;color:#888;">Don't have an account? <a href="#" style="color:#b0b91a;">Sign up</a></p>
13
+ </div>
14
+ </body>
15
+ </html>
data/tests/0/variants/blank.html ADDED
@@ -0,0 +1 @@
 
 
1
+ <!DOCTYPE html><html><head><title>Page</title></head><body style="background:#fff;"></body></html>
data/tests/0/variants/half_styled.html ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Login</title></head>
4
+ <body style="margin:0; background:#f0f2f5; align-items:center; min-height:100vh">
5
+ <div style="background:#fff; border-radius:8px; width:320px">
6
+ <h2 style="margin:0 0 24px; color:#1a1a1a">Sign In</h2>
7
+ <label style="display:block; font-size:14px">Email</label>
8
+ <input type="email" placeholder="you@example.com" style="width:100%; padding:10px 12px; border-radius:6px; margin-bottom:16px">
9
+ <label style="display:block; font-size:14px">Password</label>
10
+ <input type="password" placeholder="β€’β€’β€’β€’β€’β€’β€’β€’" style="width:100%; padding:10px 12px; border-radius:6px; margin-bottom:24px">
11
+ <button style="width:100%; background:#4f46e5; border:none; font-size:15px">Sign In</button>
12
+ <p style="text-align:center; font-size:13px">Don't have an account? <a href="#" style="color:#4f46e5">Sign up</a></p>
13
+ </div>
14
+ </body>
15
+ </html>
data/tests/0/variants/minor_diff.html ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Login</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#888888;display:flex;align-items:center;justify-content:center;min-height:100vh;">
5
+ <div style="background:#fff;padding:40px;border-radius:8px;box-shadow:0 2px 12px rgba(0,0,0,.1);width:320px;">
6
+ <h2 style="margin:0 0 24px;text-align:center;color:#1a1a1a;">Sign In</h2>
7
+ <label style="display:block;margin-bottom:6px;font-size:10px;color:#555;">Email</label>
8
+ <input type="email" placeholder="you@example.com" style="width:100%;box-sizing:border-box;padding:10px 12px;border:1px solid #ddd;border-radius:6px;font-size:10px;margin-bottom:16px;">
9
+ <label style="display:block;margin-bottom:6px;font-size:14px;color:#555;">Password</label>
10
+ <input type="password" placeholder="β€’β€’β€’β€’β€’β€’β€’β€’" style="width:100%;box-sizing:border-box;padding:10px 12px;border:1px solid #ddd;border-radius:6px;font-size:14px;margin-bottom:24px;">
11
+ <button style="width:100%;padding:12px;background:#4f46e5;color:#fff;border:none;border-radius:6px;font-size:15px;cursor:pointer;">Sign In</button>
12
+ <p style="text-align:center;margin-top:16px;font-size:13px;color:#888;">Don't have an account? <a href="#" style="color:#4f46e5;">Sign up</a></p>
13
+ </div>
14
+ </body>
15
+ </html>
data/tests/0/variants/no_layout.html ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Login</title></head>
4
+ <body style="font-family:sans-serif; background:#f0f2f5">
5
+ <div style="background:#fff">
6
+ <h2 style="text-align:center; color:#1a1a1a">Sign In</h2>
7
+ <label style="font-size:14px; color:#555">Email</label>
8
+ <input type="email" placeholder="you@example.com" style="border:1px solid #ddd; font-size:14px">
9
+ <label style="font-size:14px; color:#555">Password</label>
10
+ <input type="password" placeholder="β€’β€’β€’β€’β€’β€’β€’β€’" style="border:1px solid #ddd; font-size:14px">
11
+ <button style="background:#4f46e5; color:#fff; border:none; font-size:15px; cursor:pointer">Sign In</button>
12
+ <p style="text-align:center; font-size:13px; color:#888">Don't have an account? <a href="#" style="color:#4f46e5">Sign up</a></p>
13
+ </div>
14
+ </body>
15
+ </html>
data/tests/0/variants/no_style.html ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Login</title></head>
4
+ <body>
5
+ <div>
6
+ <h2>Sign In</h2>
7
+ <label>Email</label>
8
+ <input type="email" placeholder="you@example.com">
9
+ <label>Password</label>
10
+ <input type="password" placeholder="β€’β€’β€’β€’β€’β€’β€’β€’">
11
+ <button>Sign In</button>
12
+ <p>Don't have an account? <a href="#">Sign up</a></p>
13
+ </div>
14
+ </body>
15
+ </html>
data/tests/0/variants/perfect.html ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Login</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#f0f2f5;display:flex;align-items:center;justify-content:center;min-height:100vh;">
5
+ <div style="background:#fff;padding:40px;border-radius:8px;box-shadow:0 2px 12px rgba(0,0,0,.1);width:320px;">
6
+ <h2 style="margin:0 0 24px;text-align:center;color:#1a1a1a;">Sign In</h2>
7
+ <label style="display:block;margin-bottom:6px;font-size:14px;color:#555;">Email</label>
8
+ <input type="email" placeholder="you@example.com" style="width:100%;box-sizing:border-box;padding:10px 12px;border:1px solid #ddd;border-radius:6px;font-size:14px;margin-bottom:16px;">
9
+ <label style="display:block;margin-bottom:6px;font-size:14px;color:#555;">Password</label>
10
+ <input type="password" placeholder="β€’β€’β€’β€’β€’β€’β€’β€’" style="width:100%;box-sizing:border-box;padding:10px 12px;border:1px solid #ddd;border-radius:6px;font-size:14px;margin-bottom:24px;">
11
+ <button style="width:100%;padding:12px;background:#4f46e5;color:#fff;border:none;border-radius:6px;font-size:15px;cursor:pointer;">Sign In</button>
12
+ <p style="text-align:center;margin-top:16px;font-size:13px;color:#888;">Don't have an account? <a href="#" style="color:#4f46e5;">Sign up</a></p>
13
+ </div>
14
+ </body>
15
+ </html>
data/tests/1/expected_scores.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "perfect": 0.9773,
3
+ "minor_diff": 0.7689,
4
+ "bad_colors": 0.9176,
5
+ "half_styled": 0.7297,
6
+ "no_layout": 0.5567,
7
+ "no_style": 0.543,
8
+ "blank": 0.0
9
+ }
data/tests/1/meta.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "source": "easy/1",
3
+ "difficulty": "easy",
4
+ "idx": 1
5
+ }
data/tests/1/reference.html ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Blog Post</title></head>
4
+ <body style="margin:0;font-family:Georgia,serif;background:#fff;color:#222;max-width:680px;margin:0 auto;padding:40px 20px;">
5
+ <p style="font-size:13px;color:#888;margin-bottom:8px;">TECHNOLOGY Β· 5 MIN READ</p>
6
+ <h1 style="font-size:32px;line-height:1.3;margin:0 0 16px;">The Rise of AI-Assisted Development</h1>
7
+ <div style="display:flex;align-items:center;gap:12px;margin-bottom:32px;">
8
+ <div style="width:40px;height:40px;border-radius:50%;background:#4f46e5;display:flex;align-items:center;justify-content:center;color:#fff;font-weight:bold;">AJ</div>
9
+ <div>
10
+ <p style="margin:0;font-size:14px;font-weight:bold;color:#333;">Amal Joe</p>
11
+ <p style="margin:0;font-size:13px;color:#888;">April 8, 2026</p>
12
+ </div>
13
+ </div>
14
+ <div style="background:#f8f8f8;border-left:4px solid #4f46e5;padding:16px 20px;margin-bottom:28px;border-radius:0 6px 6px 0;">
15
+ <p style="margin:0;font-style:italic;color:#555;">"AI is not replacing developers β€” it's giving them superpowers."</p>
16
+ </div>
17
+ <p style="line-height:1.8;margin-bottom:20px;">Modern development has shifted dramatically with the introduction of large language models capable of understanding and generating code. Teams that once spent days on boilerplate now ship features in hours.</p>
18
+ <p style="line-height:1.8;">The key insight is that AI excels at pattern completion, while humans excel at problem definition. Together, the combination is unbeatable.</p>
19
+ </body>
20
+ </html>
data/tests/1/variants/bad_colors.html ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Blog Post</title></head>
4
+ <body style="margin:0;font-family:Georgia,serif;background:#fff;color:#222;max-width:680px;margin:0 auto;padding:40px 20px;">
5
+ <p style="font-size:13px;color:#888;margin-bottom:8px;">TECHNOLOGY Β· 5 MIN READ</p>
6
+ <h1 style="font-size:32px;line-height:1.3;margin:0 0 16px;">The Rise of AI-Assisted Development</h1>
7
+ <div style="display:flex;align-items:center;gap:12px;margin-bottom:32px;">
8
+ <div style="width:40px;height:40px;border-radius:50%;background:#b0b91a;display:flex;align-items:center;justify-content:center;color:#fff;font-weight:bold;">AJ</div>
9
+ <div>
10
+ <p style="margin:0;font-size:14px;font-weight:bold;color:#333;">Amal Joe</p>
11
+ <p style="margin:0;font-size:13px;color:#888;">April 8, 2026</p>
12
+ </div>
13
+ </div>
14
+ <div style="background:#070707;border-left:4px solid #b0b91a;padding:16px 20px;margin-bottom:28px;border-radius:0 6px 6px 0;">
15
+ <p style="margin:0;font-style:italic;color:#555;">"AI is not replacing developers β€” it's giving them superpowers."</p>
16
+ </div>
17
+ <p style="line-height:1.8;margin-bottom:20px;">Modern development has shifted dramatically with the introduction of large language models capable of understanding and generating code. Teams that once spent days on boilerplate now ship features in hours.</p>
18
+ <p style="line-height:1.8;">The key insight is that AI excels at pattern completion, while humans excel at problem definition. Together, the combination is unbeatable.</p>
19
+ </body>
20
+ </html>
data/tests/1/variants/blank.html ADDED
@@ -0,0 +1 @@
 
 
1
+ <!DOCTYPE html><html><head><title>Page</title></head><body style="background:#fff;"></body></html>
data/tests/1/variants/half_styled.html ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Blog Post</title></head>
4
+ <body style="margin:0; background:#fff; max-width:680px; padding:40px 20px">
5
+ <p style="font-size:13px; margin-bottom:8px">TECHNOLOGY Β· 5 MIN READ</p>
6
+ <h1 style="font-size:32px; margin:0 0 16px">The Rise of AI-Assisted Development</h1>
7
+ <div style="display:flex; gap:12px">
8
+ <div style="width:40px; border-radius:50%; display:flex; justify-content:center; font-weight:bold">AJ</div>
9
+ <div>
10
+ <p style="margin:0; font-weight:bold">Amal Joe</p>
11
+ <p style="margin:0; color:#888">April 8, 2026</p>
12
+ </div>
13
+ </div>
14
+ <div style="background:#f8f8f8; padding:16px 20px; border-radius:0 6px 6px 0">
15
+ <p style="margin:0; color:#555">"AI is not replacing developers β€” it's giving them superpowers."</p>
16
+ </div>
17
+ <p style="line-height:1.8">Modern development has shifted dramatically with the introduction of large language models capable of understanding and generating code. Teams that once spent days on boilerplate now ship features in hours.</p>
18
+ <p style="line-height:1.8">The key insight is that AI excels at pattern completion, while humans excel at problem definition. Together, the combination is unbeatable.</p>
19
+ </body>
20
+ </html>
data/tests/1/variants/minor_diff.html ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Blog Post</title></head>
4
+ <body style="margin:0;font-family:Georgia,serif;background:#fff;color:#222;max-width:680px;margin:0 auto;padding:40px 20px;">
5
+ <p style="font-size:9px;color:#888;margin-bottom:8px;">TECHNOLOGY Β· 5 MIN READ</p>
6
+ <h1 style="font-size:28px;line-height:1.3;margin:0 0 16px;">The Rise of AI-Assisted Development</h1>
7
+ <div style="display:flex;align-items:center;gap:12px;margin-bottom:32px;">
8
+ <div style="width:40px;height:40px;border-radius:50%;background:#888888;display:flex;align-items:center;justify-content:center;color:#fff;font-weight:bold;">AJ</div>
9
+ <div>
10
+ <p style="margin:0;font-size:14px;font-weight:bold;color:#333;">Amal Joe</p>
11
+ <p style="margin:0;font-size:13px;color:#888;">April 8, 2026</p>
12
+ </div>
13
+ </div>
14
+ <div style="background:#f8f8f8;border-left:4px solid #4f46e5;padding:16px 20px;margin-bottom:28px;border-radius:0 6px 6px 0;">
15
+ <p style="margin:0;font-style:italic;color:#555;">"AI is not replacing developers β€” it's giving them superpowers."</p>
16
+ </div>
17
+ <p style="line-height:1.8;margin-bottom:20px;">Modern development has shifted dramatically with the introduction of large language models capable of understanding and generating code. Teams that once spent days on boilerplate now ship features in hours.</p>
18
+ <p style="line-height:1.8;">The key insight is that AI excels at pattern completion, while humans excel at problem definition. Together, the combination is unbeatable.</p>
19
+ </body>
20
+ </html>
data/tests/1/variants/no_layout.html ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Blog Post</title></head>
4
+ <body style="font-family:Georgia,serif; background:#fff; color:#222; max-width:680px">
5
+ <p style="font-size:13px; color:#888">TECHNOLOGY Β· 5 MIN READ</p>
6
+ <h1 style="font-size:32px; line-height:1.3">The Rise of AI-Assisted Development</h1>
7
+ <div style="gap:12px">
8
+ <div style="background:#4f46e5; color:#fff; font-weight:bold">AJ</div>
9
+ <div>
10
+ <p style="font-size:14px; font-weight:bold; color:#333">Amal Joe</p>
11
+ <p style="font-size:13px; color:#888">April 8, 2026</p>
12
+ </div>
13
+ </div>
14
+ <div style="background:#f8f8f8; border-left:4px solid #4f46e5">
15
+ <p style="font-style:italic; color:#555">"AI is not replacing developers β€” it's giving them superpowers."</p>
16
+ </div>
17
+ <p style="line-height:1.8">Modern development has shifted dramatically with the introduction of large language models capable of understanding and generating code. Teams that once spent days on boilerplate now ship features in hours.</p>
18
+ <p style="line-height:1.8">The key insight is that AI excels at pattern completion, while humans excel at problem definition. Together, the combination is unbeatable.</p>
19
+ </body>
20
+ </html>
data/tests/1/variants/no_style.html ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Blog Post</title></head>
4
+ <body>
5
+ <p>TECHNOLOGY Β· 5 MIN READ</p>
6
+ <h1>The Rise of AI-Assisted Development</h1>
7
+ <div>
8
+ <div>AJ</div>
9
+ <div>
10
+ <p>Amal Joe</p>
11
+ <p>April 8, 2026</p>
12
+ </div>
13
+ </div>
14
+ <div>
15
+ <p>"AI is not replacing developers β€” it's giving them superpowers."</p>
16
+ </div>
17
+ <p>Modern development has shifted dramatically with the introduction of large language models capable of understanding and generating code. Teams that once spent days on boilerplate now ship features in hours.</p>
18
+ <p>The key insight is that AI excels at pattern completion, while humans excel at problem definition. Together, the combination is unbeatable.</p>
19
+ </body>
20
+ </html>
data/tests/1/variants/perfect.html ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Blog Post</title></head>
4
+ <body style="margin:0;font-family:Georgia,serif;background:#fff;color:#222;max-width:680px;margin:0 auto;padding:40px 20px;">
5
+ <p style="font-size:13px;color:#888;margin-bottom:8px;">TECHNOLOGY Β· 5 MIN READ</p>
6
+ <h1 style="font-size:32px;line-height:1.3;margin:0 0 16px;">The Rise of AI-Assisted Development</h1>
7
+ <div style="display:flex;align-items:center;gap:12px;margin-bottom:32px;">
8
+ <div style="width:40px;height:40px;border-radius:50%;background:#4f46e5;display:flex;align-items:center;justify-content:center;color:#fff;font-weight:bold;">AJ</div>
9
+ <div>
10
+ <p style="margin:0;font-size:14px;font-weight:bold;color:#333;">Amal Joe</p>
11
+ <p style="margin:0;font-size:13px;color:#888;">April 8, 2026</p>
12
+ </div>
13
+ </div>
14
+ <div style="background:#f8f8f8;border-left:4px solid #4f46e5;padding:16px 20px;margin-bottom:28px;border-radius:0 6px 6px 0;">
15
+ <p style="margin:0;font-style:italic;color:#555;">"AI is not replacing developers β€” it's giving them superpowers."</p>
16
+ </div>
17
+ <p style="line-height:1.8;margin-bottom:20px;">Modern development has shifted dramatically with the introduction of large language models capable of understanding and generating code. Teams that once spent days on boilerplate now ship features in hours.</p>
18
+ <p style="line-height:1.8;">The key insight is that AI excels at pattern completion, while humans excel at problem definition. Together, the combination is unbeatable.</p>
19
+ </body>
20
+ </html>
data/tests/10/expected_scores.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "perfect": 0.9773,
3
+ "minor_diff": 0.8818,
4
+ "bad_colors": 0.6161,
5
+ "half_styled": 0.5619,
6
+ "no_layout": 0.4915,
7
+ "no_style": 0.2505,
8
+ "blank": 0.0
9
+ }
data/tests/10/meta.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "source": "hard/0",
3
+ "difficulty": "hard",
4
+ "idx": 0
5
+ }
data/tests/10/reference.html ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Dashboard</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#0f172a;color:#e2e8f0;display:flex;min-height:100vh;">
5
+ <aside style="width:200px;background:#1e293b;padding:24px 0;flex-shrink:0;">
6
+ <div style="padding:0 20px 24px;font-size:18px;font-weight:700;color:#fff;border-bottom:1px solid #334155;">⚑ Pulsar</div>
7
+ <nav style="padding:16px 0;">
8
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#fff;background:#334155;text-decoration:none;margin-bottom:2px;">πŸ“Š Overview</a>
9
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#94a3b8;text-decoration:none;margin-bottom:2px;">πŸš€ Deployments</a>
10
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#94a3b8;text-decoration:none;margin-bottom:2px;">πŸ“ Projects</a>
11
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#94a3b8;text-decoration:none;margin-bottom:2px;">πŸ‘₯ Team</a>
12
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#94a3b8;text-decoration:none;">βš™οΈ Settings</a>
13
+ </nav>
14
+ </aside>
15
+ <main style="flex:1;padding:32px;">
16
+ <div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:28px;">
17
+ <h1 style="margin:0;font-size:22px;">Overview</h1>
18
+ <button style="padding:9px 18px;background:#4f46e5;color:#fff;border:none;border-radius:6px;font-size:13px;cursor:pointer;">+ New Project</button>
19
+ </div>
20
+ <div style="display:grid;grid-template-columns:repeat(4,1fr);gap:16px;margin-bottom:28px;">
21
+ <div style="background:#1e293b;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#94a3b8;">DEPLOYMENTS</p><p style="margin:0;font-size:28px;font-weight:700;">1,284</p><p style="margin:4px 0 0;font-size:12px;color:#34d399;">↑ 12% this week</p></div>
22
+ <div style="background:#1e293b;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#94a3b8;">SUCCESS RATE</p><p style="margin:0;font-size:28px;font-weight:700;">99.2%</p><p style="margin:4px 0 0;font-size:12px;color:#34d399;">↑ 0.3% this week</p></div>
23
+ <div style="background:#1e293b;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#94a3b8;">AVG BUILD TIME</p><p style="margin:0;font-size:28px;font-weight:700;">28s</p><p style="margin:4px 0 0;font-size:12px;color:#fb923c;">↓ 3s slower</p></div>
24
+ <div style="background:#1e293b;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#94a3b8;">ACTIVE PROJECTS</p><p style="margin:0;font-size:28px;font-weight:700;">47</p><p style="margin:4px 0 0;font-size:12px;color:#34d399;">↑ 3 new</p></div>
25
+ </div>
26
+ <div style="background:#1e293b;border-radius:10px;padding:24px;">
27
+ <h2 style="margin:0 0 20px;font-size:16px;">Recent Deployments</h2>
28
+ <table style="width:100%;border-collapse:collapse;font-size:13px;">
29
+ <thead><tr style="color:#64748b;border-bottom:1px solid #334155;"><th style="text-align:left;padding:8px 0;">Project</th><th style="text-align:left;padding:8px 0;">Branch</th><th style="text-align:left;padding:8px 0;">Status</th><th style="text-align:left;padding:8px 0;">Time</th></tr></thead>
30
+ <tbody>
31
+ <tr style="border-bottom:1px solid #1e293b;"><td style="padding:12px 0;color:#e2e8f0;">api-service</td><td style="padding:12px 0;color:#94a3b8;">main</td><td style="padding:12px 0;"><span style="background:#dcfce7;color:#16a34a;padding:3px 10px;border-radius:20px;font-size:11px;">βœ“ Success</span></td><td style="padding:12px 0;color:#94a3b8;">2 min ago</td></tr>
32
+ <tr style="border-bottom:1px solid #1e293b;"><td style="padding:12px 0;color:#e2e8f0;">web-frontend</td><td style="padding:12px 0;color:#94a3b8;">feat/auth</td><td style="padding:12px 0;"><span style="background:#fef3c7;color:#d97706;padding:3px 10px;border-radius:20px;font-size:11px;">⟳ Building</span></td><td style="padding:12px 0;color:#94a3b8;">5 min ago</td></tr>
33
+ <tr><td style="padding:12px 0;color:#e2e8f0;">ml-pipeline</td><td style="padding:12px 0;color:#94a3b8;">main</td><td style="padding:12px 0;"><span style="background:#fee2e2;color:#dc2626;padding:3px 10px;border-radius:20px;font-size:11px;">βœ— Failed</span></td><td style="padding:12px 0;color:#94a3b8;">18 min ago</td></tr>
34
+ </tbody>
35
+ </table>
36
+ </div>
37
+ </main>
38
+ </body>
39
+ </html>
data/tests/10/variants/bad_colors.html ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Dashboard</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#f0e8d5;color:#1d170f;display:flex;min-height:100vh;">
5
+ <aside style="width:200px;background:#e1d6c4;padding:24px 0;flex-shrink:0;">
6
+ <div style="padding:0 20px 24px;font-size:18px;font-weight:700;color:#fff;border-bottom:1px solid #ccbeaa;">⚑ Pulsar</div>
7
+ <nav style="padding:16px 0;">
8
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#fff;background:#ccbeaa;text-decoration:none;margin-bottom:2px;">πŸ“Š Overview</a>
9
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#6b5c47;text-decoration:none;margin-bottom:2px;">πŸš€ Deployments</a>
10
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#6b5c47;text-decoration:none;margin-bottom:2px;">πŸ“ Projects</a>
11
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#6b5c47;text-decoration:none;margin-bottom:2px;">πŸ‘₯ Team</a>
12
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#6b5c47;text-decoration:none;">βš™οΈ Settings</a>
13
+ </nav>
14
+ </aside>
15
+ <main style="flex:1;padding:32px;">
16
+ <div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:28px;">
17
+ <h1 style="margin:0;font-size:22px;">Overview</h1>
18
+ <button style="padding:9px 18px;background:#b0b91a;color:#fff;border:none;border-radius:6px;font-size:13px;cursor:pointer;">+ New Project</button>
19
+ </div>
20
+ <div style="display:grid;grid-template-columns:repeat(4,1fr);gap:16px;margin-bottom:28px;">
21
+ <div style="background:#e1d6c4;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#6b5c47;">DEPLOYMENTS</p><p style="margin:0;font-size:28px;font-weight:700;">1,284</p><p style="margin:4px 0 0;font-size:12px;color:#cb2c66;">↑ 12% this week</p></div>
22
+ <div style="background:#e1d6c4;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#6b5c47;">SUCCESS RATE</p><p style="margin:0;font-size:28px;font-weight:700;">99.2%</p><p style="margin:4px 0 0;font-size:12px;color:#cb2c66;">↑ 0.3% this week</p></div>
23
+ <div style="background:#e1d6c4;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#6b5c47;">AVG BUILD TIME</p><p style="margin:0;font-size:28px;font-weight:700;">28s</p><p style="margin:4px 0 0;font-size:12px;color:#046dc3;">↓ 3s slower</p></div>
24
+ <div style="background:#e1d6c4;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#6b5c47;">ACTIVE PROJECTS</p><p style="margin:0;font-size:28px;font-weight:700;">47</p><p style="margin:4px 0 0;font-size:12px;color:#cb2c66;">↑ 3 new</p></div>
25
+ </div>
26
+ <div style="background:#e1d6c4;border-radius:10px;padding:24px;">
27
+ <h2 style="margin:0 0 20px;font-size:16px;">Recent Deployments</h2>
28
+ <table style="width:100%;border-collapse:collapse;font-size:13px;">
29
+ <thead><tr style="color:#9b8b74;border-bottom:1px solid #ccbeaa;"><th style="text-align:left;padding:8px 0;">Project</th><th style="text-align:left;padding:8px 0;">Branch</th><th style="text-align:left;padding:8px 0;">Status</th><th style="text-align:left;padding:8px 0;">Time</th></tr></thead>
30
+ <tbody>
31
+ <tr style="border-bottom:1px solid #e1d6c4;"><td style="padding:12px 0;color:#1d170f;">api-service</td><td style="padding:12px 0;color:#6b5c47;">main</td><td style="padding:12px 0;"><span style="background:#230318;color:#e95cb5;padding:3px 10px;border-radius:20px;font-size:11px;">βœ“ Success</span></td><td style="padding:12px 0;color:#6b5c47;">2 min ago</td></tr>
32
+ <tr style="border-bottom:1px solid #e1d6c4;"><td style="padding:12px 0;color:#1d170f;">web-frontend</td><td style="padding:12px 0;color:#6b5c47;">feat/auth</td><td style="padding:12px 0;"><span style="background:#010c38;color:#2688f9;padding:3px 10px;border-radius:20px;font-size:11px;">⟳ Building</span></td><td style="padding:12px 0;color:#6b5c47;">5 min ago</td></tr>
33
+ <tr><td style="padding:12px 0;color:#1d170f;">ml-pipeline</td><td style="padding:12px 0;color:#6b5c47;">main</td><td style="padding:12px 0;"><span style="background:#011d1d;color:#23d9d9;padding:3px 10px;border-radius:20px;font-size:11px;">βœ— Failed</span></td><td style="padding:12px 0;color:#6b5c47;">18 min ago</td></tr>
34
+ </tbody>
35
+ </table>
36
+ </div>
37
+ </main>
38
+ </body>
39
+ </html>
data/tests/10/variants/blank.html ADDED
@@ -0,0 +1 @@
 
 
1
+ <!DOCTYPE html><html><head><title>Page</title></head><body style="background:#fff;"></body></html>
data/tests/10/variants/half_styled.html ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Dashboard</title></head>
4
+ <body style="margin:0; background:#0f172a; display:flex">
5
+ <aside style="width:200px; padding:24px 0">
6
+ <div style="padding:0 20px 24px; font-weight:700; border-bottom:1px solid #334155">⚑ Pulsar</div>
7
+ <nav style="padding:16px 0">
8
+ <a style="display:flex; gap:10px; font-size:13px; background:#334155; margin-bottom:2px">πŸ“Š Overview</a>
9
+ <a style="display:flex; gap:10px; font-size:13px; text-decoration:none">πŸš€ Deployments</a>
10
+ <a style="display:flex; gap:10px; font-size:13px; text-decoration:none">πŸ“ Projects</a>
11
+ <a style="display:flex; gap:10px; font-size:13px; text-decoration:none">πŸ‘₯ Team</a>
12
+ <a style="display:flex; gap:10px; font-size:13px; text-decoration:none">βš™οΈ Settings</a>
13
+ </nav>
14
+ </aside>
15
+ <main style="flex:1">
16
+ <div style="display:flex; align-items:center">
17
+ <h1 style="margin:0">Overview</h1>
18
+ <button style="padding:9px 18px; color:#fff; border-radius:6px; cursor:pointer">+ New Project</button>
19
+ </div>
20
+ <div style="display:grid; gap:16px">
21
+ <div style="background:#1e293b; padding:20px"><p style="margin:0 0 8px; color:#94a3b8">DEPLOYMENTS</p><p style="margin:0; font-weight:700">1,284</p><p style="margin:4px 0 0; color:#34d399">↑ 12% this week</p></div>
22
+ <div style="background:#1e293b; padding:20px"><p style="margin:0 0 8px; color:#94a3b8">SUCCESS RATE</p><p style="margin:0; font-weight:700">99.2%</p><p style="margin:4px 0 0; color:#34d399">↑ 0.3% this week</p></div>
23
+ <div style="background:#1e293b; padding:20px"><p style="margin:0 0 8px; color:#94a3b8">AVG BUILD TIME</p><p style="margin:0; font-weight:700">28s</p><p style="margin:4px 0 0; color:#fb923c">↓ 3s slower</p></div>
24
+ <div style="background:#1e293b; padding:20px"><p style="margin:0 0 8px; color:#94a3b8">ACTIVE PROJECTS</p><p style="margin:0; font-weight:700">47</p><p style="margin:4px 0 0; color:#34d399">↑ 3 new</p></div>
25
+ </div>
26
+ <div style="background:#1e293b; padding:24px">
27
+ <h2 style="margin:0 0 20px">Recent Deployments</h2>
28
+ <table style="width:100%; font-size:13px">
29
+ <thead><tr style="color:#64748b"><th style="text-align:left">Project</th><th style="text-align:left">Branch</th><th style="text-align:left">Status</th><th style="text-align:left">Time</th></tr></thead>
30
+ <tbody>
31
+ <tr style="border-bottom:1px solid #1e293b"><td style="padding:12px 0">api-service</td><td style="padding:12px 0">main</td><td style="padding:12px 0"><span style="background:#dcfce7; padding:3px 10px; font-size:11px">βœ“ Success</span></td><td style="padding:12px 0">2 min ago</td></tr>
32
+ <tr style="border-bottom:1px solid #1e293b"><td style="padding:12px 0">web-frontend</td><td style="padding:12px 0">feat/auth</td><td style="padding:12px 0"><span style="background:#fef3c7; padding:3px 10px; font-size:11px">⟳ Building</span></td><td style="padding:12px 0">5 min ago</td></tr>
33
+ <tr><td style="padding:12px 0">ml-pipeline</td><td style="padding:12px 0">main</td><td style="padding:12px 0"><span style="background:#fee2e2; padding:3px 10px; font-size:11px">βœ— Failed</span></td><td style="padding:12px 0">18 min ago</td></tr>
34
+ </tbody>
35
+ </table>
36
+ </div>
37
+ </main>
38
+ </body>
39
+ </html>
data/tests/10/variants/minor_diff.html ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!DOCTYPE html>
2
+ <html lang="en">
3
+ <head><meta charset="UTF-8"><title>Dashboard</title></head>
4
+ <body style="margin:0;font-family:sans-serif;background:#888888;color:#e2e8f0;display:flex;min-height:100vh;">
5
+ <aside style="width:200px;background:#1e293b;padding:24px 0;flex-shrink:0;">
6
+ <div style="padding:0 20px 24px;font-size:14px;font-weight:700;color:#fff;border-bottom:1px solid #334155;">⚑ Pulsar</div>
7
+ <nav style="padding:16px 0;">
8
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:9px;color:#fff;background:#334155;text-decoration:none;margin-bottom:2px;">πŸ“Š Overview</a>
9
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#94a3b8;text-decoration:none;margin-bottom:2px;">πŸš€ Deployments</a>
10
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#94a3b8;text-decoration:none;margin-bottom:2px;">πŸ“ Projects</a>
11
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#94a3b8;text-decoration:none;margin-bottom:2px;">πŸ‘₯ Team</a>
12
+ <a style="display:flex;align-items:center;gap:10px;padding:10px 20px;font-size:13px;color:#94a3b8;text-decoration:none;">βš™οΈ Settings</a>
13
+ </nav>
14
+ </aside>
15
+ <main style="flex:1;padding:32px;">
16
+ <div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:28px;">
17
+ <h1 style="margin:0;font-size:22px;">Overview</h1>
18
+ <button style="padding:9px 18px;background:#4f46e5;color:#fff;border:none;border-radius:6px;font-size:13px;cursor:pointer;">+ New Project</button>
19
+ </div>
20
+ <div style="display:grid;grid-template-columns:repeat(4,1fr);gap:16px;margin-bottom:28px;">
21
+ <div style="background:#1e293b;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#94a3b8;">DEPLOYMENTS</p><p style="margin:0;font-size:28px;font-weight:700;">1,284</p><p style="margin:4px 0 0;font-size:12px;color:#34d399;">↑ 12% this week</p></div>
22
+ <div style="background:#1e293b;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#94a3b8;">SUCCESS RATE</p><p style="margin:0;font-size:28px;font-weight:700;">99.2%</p><p style="margin:4px 0 0;font-size:12px;color:#34d399;">↑ 0.3% this week</p></div>
23
+ <div style="background:#1e293b;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#94a3b8;">AVG BUILD TIME</p><p style="margin:0;font-size:28px;font-weight:700;">28s</p><p style="margin:4px 0 0;font-size:12px;color:#fb923c;">↓ 3s slower</p></div>
24
+ <div style="background:#1e293b;border-radius:10px;padding:20px;"><p style="margin:0 0 8px;font-size:12px;color:#94a3b8;">ACTIVE PROJECTS</p><p style="margin:0;font-size:28px;font-weight:700;">47</p><p style="margin:4px 0 0;font-size:12px;color:#34d399;">↑ 3 new</p></div>
25
+ </div>
26
+ <div style="background:#1e293b;border-radius:10px;padding:24px;">
27
+ <h2 style="margin:0 0 20px;font-size:16px;">Recent Deployments</h2>
28
+ <table style="width:100%;border-collapse:collapse;font-size:13px;">
29
+ <thead><tr style="color:#64748b;border-bottom:1px solid #334155;"><th style="text-align:left;padding:8px 0;">Project</th><th style="text-align:left;padding:8px 0;">Branch</th><th style="text-align:left;padding:8px 0;">Status</th><th style="text-align:left;padding:8px 0;">Time</th></tr></thead>
30
+ <tbody>
31
+ <tr style="border-bottom:1px solid #1e293b;"><td style="padding:12px 0;color:#e2e8f0;">api-service</td><td style="padding:12px 0;color:#94a3b8;">main</td><td style="padding:12px 0;"><span style="background:#dcfce7;color:#16a34a;padding:3px 10px;border-radius:20px;font-size:11px;">βœ“ Success</span></td><td style="padding:12px 0;color:#94a3b8;">2 min ago</td></tr>
32
+ <tr style="border-bottom:1px solid #1e293b;"><td style="padding:12px 0;color:#e2e8f0;">web-frontend</td><td style="padding:12px 0;color:#94a3b8;">feat/auth</td><td style="padding:12px 0;"><span style="background:#fef3c7;color:#d97706;padding:3px 10px;border-radius:20px;font-size:11px;">⟳ Building</span></td><td style="padding:12px 0;color:#94a3b8;">5 min ago</td></tr>
33
+ <tr><td style="padding:12px 0;color:#e2e8f0;">ml-pipeline</td><td style="padding:12px 0;color:#94a3b8;">main</td><td style="padding:12px 0;"><span style="background:#fee2e2;color:#dc2626;padding:3px 10px;border-radius:20px;font-size:11px;">βœ— Failed</span></td><td style="padding:12px 0;color:#94a3b8;">18 min ago</td></tr>
34
+ </tbody>
35
+ </table>
36
+ </div>
37
+ </main>
38
+ </body>
39
+ </html>