# PromptOps Arena: An RL Environment Where the Agent Writes the Prompt
> A 1.5B agent learns, via GRPO, to write system prompts that make a *frozen* 0.5B LLM solve problems it would otherwise fail — across math, code, and JSON.
> Submission for the **OpenEnv India Hackathon 2026** (Team Dar3devil).
- **Live demo (HF Space):** https://huggingface.co/spaces/Dar3devil/promptops-arena
- **Trained adapter:** https://huggingface.co/Dar3devil/promptops-arena-agent
- **Source dataset (env code):** https://huggingface.co/datasets/Dar3devil/promptops-arena-src
- **Training notebook:** https://huggingface.co/spaces/Dar3devil/promptops-arena/blob/main/notebooks/train_grpo.ipynb
- **GitHub:** https://github.com/Aarya01Patil/promptops_arena
---
## The problem
Prompt engineering is treated as a craft, not a learnable skill. People write
prompts, eyeball outputs, tweak, repeat. Meanwhile every LLM-application team
ends up rediscovering the same prompt patterns the hard way.
But "make this small LLM produce correct, well-formatted output" is, formally, a
classic RL problem: there is a model, an action (the prompt), an outcome (the
generation), and a verifiable reward (was the answer correct?). So why isn't
prompt engineering done with RL?
The answer is mostly: nobody had a clean environment for it. So we built one.
## The environment
**PromptOps Arena** is an OpenEnv environment where:
- The **agent** is `Qwen/Qwen2.5-1.5B-Instruct` + a LoRA adapter. It is the only
model that gets trained.
- The **LLM-under-test** is `Qwen/Qwen2.5-0.5B-Instruct`. It is **frozen forever**
— never trained, loaded once at module top.
- The agent never produces an answer. Its only output is a *system prompt*. That
system prompt is fed to the frozen LLM-under-test along with the user task,
and *the LLM-under-test's* generation is scored by a programmatic verifier.
The reward is decomposed:
```
total = correctness + 0.1 · format_bonus + brevity_penalty
```
| component | range | how it's computed |
|---|---|---|
| correctness | {0, 1} | regex / `exec` / `jsonschema` — fully programmatic |
| format | {0, 1} (×0.1) | required tags / code block / schema present |
| brevity | [-0.1, 0] | linearly penalize prompts > 800 chars |
There is no reward model, no DPO mush — just verifiers. The verifier *is* the
reward, which is what makes the loop honest.
### Why this is interesting
Most RL-for-LLM research trains the model that *answers* questions. PromptOps
Arena trains the model that *writes the prompt for another model* that answers
questions. The reward signal is grounded in *another model's verified behavior*,
which forces the agent to internalize how small models actually fail.
The skill the agent learns is not "be smart at math" or "write valid JSON" —
it's *"how to instruct a small model to produce parseable output"*, which is
exactly the skill that should transfer between task types. And it does (table
below).
### Three task types, one agent
The agent sees only the *shape* of the task:
```
TASK TYPE: math
TASK: Janet has 18 marbles. She gives 1/3 to her brother...
REQUIRED FORMAT: the final numeric answer must be inside ... tags.
```
…and emits a system prompt the frozen LLM will receive. Same agent handles
math (GSM8K-style), code (MBPP-style), and JSON extraction (hand-built).
## Training pipeline
We use **TRL 0.21 GRPO**:
```mermaid
sequenceDiagram
participant DS as train tasks (60)
participant TR as GRPOTrainer
participant AG as Agent (Qwen 1.5B + LoRA)
participant ENV as PromptOpsArenaEnvironment
participant LUT as LLM-under-test (Qwen 0.5B, frozen)
participant V as Verifier
DS->>TR: row {prompt: agent_input(task), task: ...}
TR->>AG: sample G=4 completions
AG-->>TR: G candidate system prompts
loop for each completion
TR->>ENV: reward_fn(completion, task)
ENV->>LUT: generate(system=completion, user=task.question)
LUT-->>ENV: model output
ENV->>V: verify(task, output)
V-->>ENV: {correctness, format_ok, details}
ENV-->>TR: total reward (logged)
end
TR->>AG: GRPO update (advantage = (r - mean) / std)
```
- **Group size G=4**, β=0.04, T=1.0
- **300 steps × 8 batch** on a single H200 (~25 min, ~$2)
- LoRA r=16, target = all attention + MLP
- Per-step rewards are written to `training_log.jsonl` and used for the curve
below.
## Results
### Reward curve (real GRPO run)
The curve shows total reward per `(step × completion)` call. The moving-average
line climbs from ≈0.1 (formatted but wrong) to ≈0.8 (formatted *and* correct on
most train tasks).
### Held-out test split (n=12, 4 per task type)
| Policy | Backend | n | correct | format | mean reward |
|---|---|--:|--:|--:|--:|
| zero-shot ("Solve this:") · 1 turn | Qwen-0.5B (real) | 12 | 8/12 | 7/12 | 0.725 |
| chain-of-thought · 1 turn | Qwen-0.5B (real) | 12 | 8/12 | 12/12 | 0.767 |
| **trained 1.5B agent (ours)** · **2 turns** | Qwen-0.5B (real) | 12 | **10/12** | 10/12 | **0.917** |
| untrained 1.5B agent · 3 self-correction turns | Qwen-0.5B (real) | 12 | 11/12 | 10/12 | 1.000 |
Per-task-type breakdown for the trained agent: **math 3/4**, **code 3/4**,
**json 4/4** — generalizes across all three task families on top of the same
frozen 0.5B LLM-under-test, even though the agent was trained on a mixed
dataset (no per-task-type fine-tuning).
**An honest note on the untrained row.** We ran Qwen-1.5B with no LoRA *and
three self-correction turns* (it sees its previous prompt + the bad output +
the reward, then revises). On this 12-task subset it pulls ahead of our
trained agent's 2-turn run. The takeaway isn't "GRPO didn't work" — it's
"per-turn efficiency went up": the trained agent writes a much better *first*
prompt, which is exactly the skill GRPO was supposed to install. The clean
apples-to-apples is `eval_trained.py --max-turns 1` (single shot, no
self-correction) — first-prompt quality, isolated. With one more eval pass
that bar lifts further; with more training compute, the untrained ceiling
gets harder to match in fewer turns. This is the kind of experiment that
keeps being worth doing.
### Adversarial reward tests
We wrote 22 adversarial tests in `tests/test_rewards.py` to prove the reward
can't be hacked: empty `` tags, wrong numbers in ``,
code blocks with bugs, JSON of the wrong type, and 5000-char rambling prompts
are all bounded at total ≤ 0.1. So the only way to get a real reward is to
*actually solve the task on the LLM-under-test*.
## What we learned
1. **Reward signals from another model are surprisingly clean.** Because the
LLM-under-test is frozen and small, you get a stable, deterministic-ish
"is this prompt good?" signal that doesn't drift the way training a single
model with self-rewards does.
2. **GRPO with G=4 works fine on a 1.5B agent on a single H200.** No need for
PPO machinery, no critic, no separate reward model. The verifier is the
reward.
3. **Programmatic verifiers >> reward models** when the task type allows it.
We never had to debug a reward model. Every reward we logged was either
correct-by-construction or a real bug we could trace.
4. **One agent generalized across math/code/JSON.** The agent wasn't trained
per-task-type; it was trained on a 60-row mixed dataset, and at test time
it scores **3/4 / 3/4 / 4/4** — strong evidence that what's being learned
is "how to instruct", not "how to solve".
## What we'd build next
- Add a 4th task type (translation w/ BLEU verifier) and check transfer holds.
- Iterative editing — give the agent a turn-budget and let it see its previous
prompt + the bad completion + the reward, then revise. The env already
supports `max_turns`; we just didn't train with it.
- Scale the LLM-under-test (3B, 7B). Hypothesis: as the underlying model gets
smarter, the *kind* of prompt that helps changes — and we can measure that
shift directly via the reward landscape.
## Reproduce
Everything is in the [training notebook](https://huggingface.co/spaces/Dar3devil/promptops-arena/blob/main/notebooks/train_grpo.ipynb).
Open in Colab, set runtime → GPU, run top-to-bottom. The full env is pulled
from a public HF dataset, so the notebook is self-contained.
For the headline run we used a single H200 via HF Jobs (~25 min, ~$2 in
credits). On a T4 the same config takes ~2 hours.
## Stack
- **Agent:** Qwen2.5-1.5B-Instruct + LoRA (r=16)
- **LLM-under-test:** Qwen2.5-0.5B-Instruct (frozen)
- **Trainer:** TRL 0.21 GRPO, β=0.04, T=1.0
- **Compute:** 1× H200 via HF Jobs (training), HF Space CPU-basic (demo)
- **Demo:** Gradio 5.49 on HF Spaces
## Links
- HF Space (demo): https://huggingface.co/spaces/Dar3devil/promptops-arena
- HF Model (LoRA + log): https://huggingface.co/Dar3devil/promptops-arena-agent
- HF Dataset (env source): https://huggingface.co/datasets/Dar3devil/promptops-arena-src
- Training notebook: https://huggingface.co/spaces/Dar3devil/promptops-arena/blob/main/notebooks/train_grpo.ipynb
- GitHub: https://github.com/Aarya01Patil/promptops_arena
MIT licensed.