README.md

metadata

title: Watercolour Environment Server
emoji: 🎨
colorFrom: pink
colorTo: indigo
sdk: docker
pinned: false
app_port: 8000
base_path: /web
tags:
  - openenv

Hardware note. This Space ships paused on the free tier so it costs nothing to host. To train against it, set the hardware to cpu-upgrade first (Settings → Hardware): on cpu-basic a render takes longer than the 90-second deadline and every rollout fails. Set HF_TOKEN in the Space secrets so the pairwise judge can call Inference Providers. Or duplicate it into your own account and do the same there.

Watercolour Environment

The model is asked to paint something and answers with a p5.brush sketch. The environment runs the sketch in a headless browser, and a vision judge is asked which of two paintings is the better watercolour: the submission, or a reference from a hand-authored pool. That verdict is the reward.

The model never sees the painting. What is being measured is whether it can hold a composition in its head and emit code for it blind.

Why comparative judging

Absolute rating drifts. "Score this watercolour 0 to 10" moves between runs and between judge models, and calibrating it against a policy that is itself moving is a project of its own. Asking which of two paintings is better is a question a vision model answers stably, and it is the shape of signal GRPO already normalises within a group.

Every comparison runs in both presentation orders, which is not a refinement. Measured across two judges (Qwen2.5-VL-72B and Qwen3-VL-30B-A3B) on pairs with a real quality difference, the verdict was correct and order-invariant eight times out of eight. On a pair of two equally poor paintings, both judges picked whichever image came first, every time. The position bias appears exactly when the judge has no real preference, so running both orders turns that failure into a visible tie instead of an invented winner.

Scoring per reference: won both orders is 1.0, lost both is 0.0, a split is 0.5. The episode reward is the mean over the references the judge resolved.

The 30B judge matched the 72B on every pair measured, so the small one is the default.

The system prompt is an allowlist, not a reference

reset() returns a system_prompt alongside the task, and it has to be used. Without any prompt at all, no model tested up to 30B emitted a single real brush.* call: Qwen3-4B, Qwen2.5-Coder-7B and Qwen3-Coder-30B-A3B all fell back silently to bare p5 primitives, which the gate rejects.

The prompt is a closed list of nineteen methods with no API documentation and no worked example, which is the shape Narreddi's write-up converged on after 200 GEPA iterations. Their finding is that a 400-line p5.brush reference produced "confident, well-formatted code that invented APIs that did not exist", and that long reference documentation makes models hallucinate APIs while a short opinionated allowlist constrains output better than the spec does.

The measurements here are consistent with that. With a full 25-method reference in context a 4B emitted ten to thirteen real calls and no bare primitives, which looked like success, and then three of four sketches died at runtime on API misuse: an invented brush.noLoop, a painting call before any brush.set, and the method name wash passed as a brush name.

The contents are derived, not guessed. Narreddi published three sketches from the trained model, and the nineteen methods are the union of what those sketches call, with every signature copied from their call sites and every method checked against the vendored bundle. The write-up says eight; their own tweet says the write-up is out of date.

Deriving it caught three defects that guessing had introduced:

• A first version allowed polygon and spline, used in none of their sketches, and omitted beginShape/vertex/endShape, used in all three. Petals are curves: without the custom-shape path the model paints triangles where the task wants petals, and the difference is visible in the render.
• It omitted hatch, hatchStyle and noHatch. Frames pulled from the video Narreddi posted show dense hatching carrying half the visual language of the finished paintings.
• The skeleton was wrong three ways at once. Their sketches call angleMode(DEGREES), they translate(-width / 2, -height / 2) so coordinates run 0 to 600 from the top left rather than -300 to 300 from the centre, and they call noLoop() inside setup rather than at the end of draw.

That last one also closed a latent flake in the renderer. Readiness used to be "setup finished and the loop has stopped", and a sketch that calls noLoop() in setup satisfies both the instant setup returns, before draw has painted anything. It now waits for a frame as well, so the race cannot capture a blank canvas and report it as a zero reward.

The frames also gave two composition facts the prompt now states outright: their paintings fill the canvas, median coverage around 0.3 with several above 0.8, and fourteen of thirty-eight sit on a dark ground. The reference paintings shipped here run 0.02 to 0.35 on pale paper only, so the pool and the target are not yet in the same regime.

Scoring

The weights come from the final rubric in Narreddi's write-up, reached after a first attempt with nine separate signals plateaued.

Layer	Weight	Cost	What it does
gate	0.05	free	Source inspection and render. Rejects no sketch, truncated source, a non-WEBGL canvas, bare p5 primitives, external image loads, text labels, blank canvases.
length	0.05	free	Binary check that the sketch is neither a stub nor a runaway.
judge	0.60	two vision calls per reference	Comparative scoring against the pool, both orders.
HPSv3	0.30	not implemented	A human preference model.

Two deliberate differences from the original, both of which change what the numbers mean.

HPSv3 is absent, so the maximum reward here is 0.70, not 1.0. It is an infrastructure decision rather than a missing function: MizzenAI/HPSv3 is a 7B Qwen2.5-VL fine-tune weighing 16.6 GB, served by its own hpsv3 package rather than a standard endpoint, and on no inference provider. It does not fit beside a 4B LoRA run on a 24 GB card, so it needs a bigger training flavour or a second GPU service. A tiered reference pool is the cheaper thing to try first, because a pool with a floor gives a small policy something to beat without any of that. The weights are left unnormalised because renormalising would silently rebalance the terms that remain, and GRPO standardises within a group anyway. That term is also the one most likely to matter: it scores a painting on its own rather than against a reference, so it is what gives a policy something to climb before it can beat anything in the pool. Suspect its absence first if the reward sits flat at the floor.

The gate stays absolute for the dishonest cases. Awarding 0.05 for compiling is their structure and worth keeping, because it is the only term a policy can earn on day one and so the only reward variance a group has before the judge engages. But a sketch that loaded somebody else's painting or wrote the answer as text scores zero outright rather than collecting the compile credit.

The length band is calibrated to this setup rather than copied. Their target was around 3,000 tokens, reflecting a 35B that wrote 13,500-token sketches and had to be compressed; measured here a 4B writes 234 to 293 tokens and the hand-authored reference paintings run 202 to 315, so their number would score zero for everything including the references. The band rejects only the degenerate ends, which means with a small model this term is close to constant and contributes almost nothing to a group.

Two gate checks are about honesty rather than competence. bare_primitives catches painting with plain p5 shapes, which scores on composition while dodging the medium. external_access catches loading somebody else's painting, which is the only way to score well without painting at all.

A sketch that throws inside draw() is admitted, not rejected. It painted something before it died, and how good that something is belongs to the judge. A sketch that paints everything in setup() and defines no draw() is admitted too: it renders the same picture.

Quick start

from envs.watercolour_env import WatercolourAction, WatercolourEnv

with WatercolourEnv(base_url="http://localhost:8000") as env:
    observation = env.reset().observation
    reply = my_model(observation.system_prompt, observation.prompt)
    result = env.step(WatercolourAction(response=reply))
    print(result.reward, result.observation.feedback)

Raise the client timeout. EnvClient defaults to message_timeout_s=60.0, and a rich sketch takes longer than that to paint on modest hardware, so the default cuts off legitimate renders as if the environment had hung:

WatercolourEnv(base_url=..., message_timeout_s=300.0)

Run the server locally. Note that uv run alone is not enough despite what openenv validate reports: the environment needs a real Chromium, which arrives with the image and not with uv sync. Use the container, or install the browser first with playwright install chromium.

PYTHONPATH=src:envs uv run uvicorn envs.watercolour_env.server.app:app --port 8000

The judge needs HF_TOKEN. Without one the environment still runs, scores on gate admission alone, and sets judged=false on every observation so a harness can tell "painted badly" from "nobody looked at it". WATERCOLOUR_DISABLE_JUDGE=1 forces that mode even when a token is present.

Rendering

Chromium through Playwright, because the submission is untrusted JavaScript and p5.brush needs a real WEBGL context, which in a container means software GL through SwiftShader.

Four details here are load-bearing, and each one was found by watching a plausible-looking blank canvas come back:

• p5.brush 2.x requires p5.js 2.x. Against p5 1.x the canvas is created and background() paints, so the page looks healthy, while every brush.* call draws nothing with no error anywhere.
• A WEBGL canvas loses its back buffer before a page screenshot is taken. The pixels have to be read inside the page with toDataURL().
• A sketch that throws inside draw() never stops looping, because it never reaches its own noLoop(). It is cut off, and cut off early once it has both an error and a frame, so the long deadline does not make broken submissions expensive.
• Readiness is polled with explicit evaluate calls, not Playwright's wait_for_function, whose requestAnimationFrame polling is throttled to a standstill on a backgrounded page. That one silently reported every render as a timeout.
• The browser owns a private event loop on a background thread. A Playwright browser is bound to the loop that launched it, and the synchronous step() wraps step_async in its own asyncio.run, so a second step() on the same environment would reach the browser from a foreign loop and hang there forever.

Render time tracks paint, not source length, and then tracks how much CPU there is. Measured on the same reference sketch:

Where	Time
laptop, native	8.2s
local container	16.1s
free-tier Space, 2 shared vCPUs	64.4s

A plain sketch from a 4B model finishes in 250ms and a whole episode in about half a second. The deadline is 90s to clear the slowest of those, and it is a soft bound checked between polls, so a legitimately slow sketch can overshoot. That is deliberate: the case the deadline exists for is a sketch that loops forever without erroring, and those poll fast and get cut off on time.

A free-tier Space is fine for trying the environment and too slow to train against. At a minute per rich render, a GRPO run wants the container next to the trainer or an upgraded Space.

The browser is started once and each submission gets a fresh page. Reusing a page breaks p5's global-mode setup on the second sketch.

The reference pool is the reward function

server/reference_pool/ ships six hand-authored paintings so the environment runs and the reward has a defined target from the first episode. They are a seed pool, not a curated corpus, and replacing them with hand-rated paintings is the main open piece of work here. Six sketches by one author is a narrow definition of "good watercolour".

The PNGs ship rather than the sources, because p5.brush uses randomness and re-rendering a source would silently move the target between runs. The sources are kept in reference_pool/sources/ for provenance.

Generalising to another medium

Alex Wang, one of the three people on Narreddi's project, posted a set of animals "created using the same mechanism": deer, elephant, rabbit, giraffe, cat, snail, tortoise and bee, each in side view on white paper, one colour family per image, built from dense directional hatching that reads as woven texture. Not a watercolour at all.

That makes the seam visible, and it is worth writing down before anyone needs it. Everything here is medium-agnostic except four things, and three of them are strings:

Piece	Where it lives	Generic?
Render, readiness, browser pool	render.py	yes
Gate, anti-cheat, paint floor	gate.py, sketch_source.py	yes
Pairwise mechanics, both orders, tie handling	pairwise_judge.py	yes
Rubric weights and composition	scoring.py, rubric.py	yes
Pool build and rating tools	examples/watercolour_pool_*.py	yes
Subject	tasks.py	no
Composition guidance in the prompt	prompt.py	no
What the judge is told to value	pairwise_judge.py	no
The reference pool	reference_pool/	no

The third one is the one that hides. The judge is told to ignore what a painting depicts, so it looks subject-agnostic, and it is: what it is not is medium-agnostic. Its prompt asks for "soft pigment bleeds, layered translucent washes, varied edge softness", which is the wrong question to ask about hatched animals. Swapping the subject without swapping that would score a good hatching against a criterion it cannot satisfy.

The four live in four files because nothing ever needed them together. A second medium is the thing that would justify collecting them, and until there is one, collecting them would be an abstraction with a single implementation.

The pool is the expensive piece in any medium. Narreddi hand-rated 1,664 generations down to 581; the tools here do the generating and the packing, and the rating is the part that stays human because the ratings are the reward function.

Prior art

The task comes from Surya Narreddi's write-up on RL'ing Qwen to paint with code, which trains a 35B model on the same loop: sketch, render, judge against a hand-rated pool, GRPO. This environment is an independent implementation of that idea on OpenEnv, aimed at a small model, with the both-orders judging rule added from the measurements above.