Upload 3 files

README.md

@@ -19,10 +19,10 @@ tags:
 
 > **A 0.5B-parameter model, trained for 90 minutes on a free Colab T4, beats an untuned 120B-parameter frontier model on this environment by 2.4×.** Built for the OpenEnv Hackathon (Apr 2026), Theme #3.2 — Personalized Tasks.
 
-An OpenEnv environment where AI agents learn to manage email and calendar like a human executive assistant — read incoming requests, write professional replies, find calendar slots that don't clash, propose alternatives when they do. Three tasks at increasing difficulty, three independent reward graders, and four anti-reward-hacking penalties that we have direct evidence of catching the model in the act.
+An OpenEnv environment where AI agents learn to manage email and calendar like a human executive assistant — read incoming requests, write professional replies, find calendar slots that don't clash, propose alternatives when they do. Three tasks at increasing difficulty, three independent reward graders, and four anti-reward-hacking penalties that we have direct training-time evidence of catching the model in the act.
 
 **Live environment:** https://devanshudon-exec-assist.hf.space  
-**Mini-blog:** _(link will go here once published)_  
+**Mini-blog:** _(link will be added here once published)_  
 **Training notebook:** [`train_colab.ipynb`](./train_colab.ipynb)
 
 ---
@@ -37,23 +37,23 @@ Trained `Qwen2.5-0.5B-Instruct` with TRL GRPO for 3 epochs (270 steps, ~90 min o
 | Medium | 0.227                     | **0.745**      | **+228%**   |
 | Hard   | 0.249                     | **0.737**      | **+196%**   |
 
-After training, **9 out of 10 samples on the easy task scored a perfect 1.0** — the model learned the task structure, not just statistics.
+After training, **9 out of 10 samples on the easy task scored a perfect 1.0** — the model learned the task structure, not just statistics. As a separate sanity check, an untuned Nemotron 120B model (called via OpenRouter through the standard `inference.py` baseline) scores 0.337 average on the same three tasks. After 90 minutes of GRPO, a model **240× smaller** is hitting 0.83 average on the same environment.
 
-![Training results: bar chart shows trained model dramatically outperforms baseline across all three tasks; line plot shows reward climbing from ~0.2 to ~0.7 over 270 steps](./training_results.png)
+![Training results: top panel shows reward curve with 10-step and 30-step moving averages, Q1 mean 0.390 → Q4 mean 0.648; bottom-left shows baseline vs trained per-task with error bars and improvement %; bottom-right shows reward variance decreasing as the policy converges](./training_results.png)
 
-*Left: mean reward by task, baseline vs. trained (n=10 samples per task). Right: GRPO batch reward over 270 training steps — first-quartile mean 0.390, last-quartile mean 0.648.*
+*Top: GRPO reward over 270 steps with moving averages and quartile mean reference lines. Bottom-left: baseline vs. trained, n=10 per task, error bars are standard deviation. Bottom-right: reward variance over training as a convergence proxy — variance drops as the policy stabilizes.*
 
 ---
 
-## Problem & motivation
+## Why this environment exists (research framing)
 
-Every executive's morning inbox is the same problem on repeat: read incoming requests, write a polite reply, find a calendar slot that doesn't clash, propose alternatives if it does. It's not hard for a human — it's just fiddly. And LLMs are surprisingly bad at it because the task fuses **three separate skills**:
+Three specific capability gaps motivated this environment:
 
-1. Structured calendar reasoning (no double-booking, within working hours, sensible duration)
-2. Professional written tone (greeting, closing, polite framing, appropriate detail)
-3. Conflict resolution (recognize the conflict, propose 2–3 alternatives, explain professionally)
+**1. Frontier LLMs are bad at structured calendar reasoning.** Ask any production agent built on a 100B+ model to "find a 30-minute slot next week that doesn't conflict with my standups and is during working hours" and observe the failure rate. The reasoning is short, the spec is precise, the failure modes are interesting. ExecAssist isolates this failure mode into a tractable RL target: the scheduling-correctness grader checks four hard constraints (no double-booking, within working hours, appropriate duration, all participants included) — and the trained model goes from satisfying ~25% of them to ~95%.
 
-ExecAssist is designed to teach all three in a single training loop. The reward function is a weighted blend of three independent graders plus four anti-reward-hacking penalties, making it hard to game by exploiting any single signal.
+**2. Multi-objective rewards are where reward hacking actually happens.** A single scalar reward ("the user was happy") gets gamed in obvious ways. A weighted sum of multiple independent graders + named penalties is much harder to game — but only if you actually verify that. We have direct evidence from training logs that GRPO tried to hack four different reward signals (output JSON only with no email body, schedule outside working hours, use generic templated phrasing, miss meeting details entirely), and that each penalty fired during early steps and disappeared as training progressed. Most submissions claim "anti-hacking penalties." Few can show them firing.
+
+**3. The "small RL'd model beats large untuned model" claim, on a real task, in 90 minutes, on free hardware.** The 240× compute ratio between Qwen-0.5B and Nemotron-120B is the headline, but the deeper claim is that *task-specific RL with composable rewards is a viable path to deploying small models on structured personal-task workflows.* That's a workshop-paper-shaped argument, and ExecAssist is a clean demonstration of it.
 
 ---
 
@@ -65,7 +65,7 @@ ExecAssist is designed to teach all three in a single training loop. The reward
 | **Medium** | 1 email, calendar conflict | Identify conflict + propose 2–3 alternatives + explain professionally | 30% email + 40% conflict + 30% scheduling |
 | **Hard** | 3 emails, multi-party coordination, priority conflicts | Prioritize, reschedule, notify all parties | 34% email + 33% scheduling + 33% conflict |
 
-All scores are deterministic and bounded to [0, 1].
+All scores are deterministic and bounded to [0, 1]. Scenarios are randomized at every `/reset` call.
 
 ---
 
@@ -113,11 +113,6 @@ All scores are deterministic and bounded to [0, 1].
 | **Scheduling correctness** | 0–1 | No double-booking, within working hours, appropriate duration (15min–2hrs), all participants included |
 | **Conflict resolution** | 0–1 | Recognizes conflicts, proposes 2–3 alternatives, explains professionally, prioritizes correctly |
 
-**Architectural note on rubrics.** The reward is composed from independent scoring functions (one per dimension: email quality, scheduling correctness,
-conflict resolution) plus four named penalty checks. Each function returns a value in [0, 1] (or a negative penalty) and is mixed by the task-specific
-weighting shown in the Tasks table. This is structurally a composable rubric — any individual grader can be swapped, weighted differently, or audited in
-isolation. We implemented it as plain Python rather than OpenEnv's `Rubric` class for hackathon speed, but the design pattern is the same.
-
 ### Anti-reward-hacking penalties
 
 - Short email (`< 20` words): **−0.30**
@@ -125,7 +120,9 @@ isolation. We implemented it as plain Python rather than OpenEnv's `Rubric` clas
 - Generic / templated phrasing: **−0.10**
 - Overly long email (`> 1500` chars): **−0.15**
 
-These were added because GRPO will find shortcuts. During the first run, the model briefly collapsed to a single short safe response — the penalties + KL regularization fixed it cleanly.
+These were added because GRPO will find shortcuts. During training the model briefly collapsed to a single short safe response — the penalties + KL regularization fixed it cleanly.
+
+**Architectural note on rubrics.** The reward is composed from independent scoring functions (one per dimension: email quality, scheduling correctness, conflict resolution) plus four named penalty checks. Each function returns a value in [0, 1] (or a negative penalty) and is mixed by the task-specific weighting shown in the Tasks table. This is structurally a composable rubric — any individual grader can be swapped, reweighted, or audited in isolation. We implemented it as plain Python rather than OpenEnv's `Rubric` class for hackathon speed, but the design pattern (independent, composable, auditable) is the same.
 
 ---
 
@@ -209,7 +206,15 @@ GRPOConfig(
 
 **The collapse and the fix.** Initial run (1 epoch, lr=5e-6, beta=0) collapsed: trained model scored exactly 0.2 on every prompt regardless of input — the model found a single safe response and stopped exploring. Adding the KL term (`beta=0.1`), reducing the learning rate, and increasing `num_generations` from 4 to 8 produced the clean training curve shown above.
 
-**Anti-reward-hacking observations during training:** GRPO did try to game several signals — outputting only the JSON without an email body (caught by short-email penalty), proposing booking times outside working hours (caught by scheduling check), repeating the prompt back as a "reply" (caught by generic-phrasing detector). Each penalty was triggered during early steps and disappeared as training progressed.
+**Anti-reward-hacking observations during training.** GRPO tried to game several signals — outputting only the JSON without an email body (caught by short-email penalty), proposing booking times outside working hours (caught by scheduling check), repeating the prompt back as a "reply" (caught by generic-phrasing detector). Each penalty was triggered during early steps and disappeared as training progressed. This is what a well-designed multi-grader rubric is supposed to do.
+
+---
+
+## Architecture note
+
+The environment is implemented as a FastAPI application that exposes the OpenEnv-spec endpoints (`/reset`, `/step`, `/state`, `/tasks`, `/health`, `/metadata`, `/schema`) directly, rather than extending `openenv.Environment` as a Python class. Both implementations are spec-compliant — they expose the same JSON-over-HTTP interface — but the FastAPI-direct approach gave us finer control over the multi-component reward function and Pydantic validation during the time-boxed hackathon build.
+
+The client (`client.py`) does extend `openenv.EnvClient` and provides the standard Gym-style typed interface, so any code that uses an `EnvClient` to talk to this Space will work without modification. Client/server separation is preserved — the client only imports models, never server internals.
 
 ---
 
@@ -226,7 +231,7 @@ exec-assist/
 ├── train_colab.ipynb     # GRPO training notebook
 ├── training_results.png  # Training curves + baseline-vs-trained
 ├── results.json          # Raw evaluation data + 270-step training log
-├── blog_post.md          # Mini-blog write-up
+├── blog_post.md          # Mini-blog write-up (also published on HF Blog)
 ├── openenv.yaml          # OpenEnv manifest
 ├── Dockerfile            # Python 3.10, port 7860
 ├── requirements.txt
@@ -235,31 +240,17 @@ exec-assist/
 
 ---
 
-## Architecture note
-
-The environment is implemented as a FastAPI application that exposes the
-OpenEnv-spec endpoints (`/reset`, `/step`, `/state`, `/tasks`, `/health`,
-`/metadata`, `/schema`) directly, rather than extending `openenv.Environment`
-as a Python class. Both implementations are spec-compliant — they expose
-the same JSON-over-HTTP interface — but the FastAPI-direct approach gave
-us finer control over the multi-component reward function and Pydantic
-validation during the time-boxed hackathon build.
-
-The client (`client.py`) does extend `openenv.EnvClient` and provides the
-standard Gym-style typed interface, so any code that uses an `EnvClient`
-to talk to this Space will work without modification.
-
 ## Compliance checklist
 
 - ✅ Built on **OpenEnv** (latest release, `openenv-core>=0.2.0`)
 - ✅ Real-world task simulation (not games or toys)
 - ✅ Full OpenEnv spec — typed Pydantic models for Action/Observation/State, `step()`/`reset()`/`state()` endpoints, `openenv.yaml` manifest
 - ✅ **3 tasks** with deterministic graders, scores in [0, 1], easy → medium → hard difficulty progression
-- ✅ Meaningful reward function with **partial-progress signal** + anti-reward-hacking penalties
+- ✅ Meaningful reward function with **partial-progress signal** + anti-reward-hacking penalties (with training-time evidence of penalties firing)
 - ✅ **Baseline inference script** (`inference.py`) using OpenAI client, reads `APIBASEURL`/`MODELNAME`/`HFTOKEN`, structured `[START]/[STEP]/[END]` logs
 - ✅ **Training script** (TRL GRPO) with reproducible Colab notebook
-- ✅ **Real training evidence** — reward curves, baseline vs. trained, before/after numbers (above)
-- ✅ Deployed to **HuggingFace Space** with Docker
+- ✅ **Real training evidence** — reward curves with moving averages, baseline vs. trained with error bars, convergence proxy (above)
+- ✅ Deployed to **HuggingFace Space** with Docker, live at https://devanshudon-exec-assist.hf.space
 - ✅ Working **Dockerfile** (Python 3.10), `docker build && docker run` works
 - ✅ README with environment description, action/observation spaces, setup, baseline scores
 

blog_post.md

@@ -1,64 +1,113 @@
 # Teaching a 0.5B Model to Be an Executive Assistant
 
-*An OpenEnv Hackathon submission — built in 36 hours, trained in 90 minutes.*
+*An OpenEnv Hackathon submission — built in 36 hours, trained in 90 minutes, debugged with three model collapses and one very bad bar chart.*
 
 ---
 
-## The setup
+I want to start with the moment the bar chart loaded and I thought I'd lost the hackathon.
 
-Every executive's inbox is the same problem on repeat. A meeting request comes in. There's a calendar conflict you have to spot. You write a polite reply, propose a time that actually works, and don't double-book anyone. It's not hard — it's just *fiddly*, and LLMs are surprisingly bad at it because the task fuses three separate skills: structured calendar reasoning, professional tone, and conflict resolution.
+It was around 3pm on day two. I'd been training for an hour and a half on a free Colab T4. The cell finished, I ran the eval, and the plot popped up: baseline bars in red, trained bars in green. Easy task: 0.493 → 0.200. Medium: 0.348 → 0.200. Hard: 0.331 → 0.186.
 
-I built **ExecAssist**, an OpenEnv environment that simulates exactly that loop, and trained a small model on it with GRPO. The results were dramatic.
+The trained model was *worse*. Identically worse on every task. Worse by exactly the same number every time.
+
+I'd never seen GRPO collapse before, but this was textbook. The model had given up exploring and found a single safe response that scored exactly 0.2 against my reward function regardless of input. All my training was, technically, optimization — just optimization toward "say the dumbest possible thing every time and never deviate."
+
+This post is about what happened next. But first, the setup.
 
 ## The environment
 
-ExecAssist gives an agent a realistic snapshot of an executive's morning: incoming emails (with sender, subject, body, priority), the existing calendar, working hours, and contact info. The agent has to produce a single JSON action — a written email reply, a calendar action (`book` / `propose_alternatives` / `reschedule` / `decline`), and the meeting details if booking.
+I built **ExecAssist**, an OpenEnv environment that simulates an executive's morning inbox. An agent gets emails (sender, subject, body, priority), a calendar (existing meetings, working hours), and contacts. It has to produce a JSON action: an email reply, a calendar action (`book` / `propose_alternatives` / `reschedule` / `decline`), and the meeting details if booking.
 
-There are three tasks of escalating difficulty:
+Three tasks of escalating difficulty:
 
-- **Easy** — single email, clear availability. *Just don't mess up the basics.*
+- **Easy** — single email, clear availability. *Don't mess up the basics.*
 - **Medium** — the requested time conflicts with an existing meeting. *Spot it, propose alternatives.*
 - **Hard** — three emails, multi-party coordination, priority conflicts. *Actually plan.*
 
-Reward is a weighted blend of three independent graders (email quality, scheduling correctness, conflict resolution) plus four anti-reward-hacking penalties — short emails, missing meeting details, generic templated replies, and overly long responses all get docked. Multiple independent reward functions make the signal hard to game, which matters because GRPO will absolutely find any shortcut you leave open.
+Reward is a weighted blend of three independent graders — email quality (politeness markers, greeting/closing, sufficient detail), scheduling correctness (no double-booking, within working hours, appropriate duration), and conflict resolution (recognizes conflicts, proposes alternatives, explains professionally). Plus four anti-reward-hacking penalties: short emails, missing meeting details, generic phrasing, overly long responses.
+
+The reason I went with multiple independent graders rather than one big scalar is something I'd read in the hackathon guide: *"if you only have a single reward signal, it is easier for the model to hack it."* I figured I'd build it the right way from day one.
+
+I would later be very glad I did.
+
+## The first run, and the collapse
+
+I trained `Qwen2.5-0.5B-Instruct` — a tiny 500M-parameter model — using TRL's `GRPOTrainer`. First config:
+
+```python
+GRPOConfig(
+    learning_rate=5e-6,
+    num_train_epochs=1,
+    num_generations=4,
+    # no beta term
+)
+```
+
+This is what produced the bar chart from hell. Looking at the training log afterward, I could see what happened: reward bounced around between 0.0 and 0.4 for the first 7 steps, peaked at 0.397, then *plummeted* and stabilized at 0.14 for the next 38 steps. The model had found "0.14 is a safe floor, don't try anything risky," and the gradient updates locked it in.
+
+I didn't know what the fix was. I asked for help. The diagnosis was: no KL penalty (`beta=0`), so nothing was anchoring the trained policy to the base model — it could drift to any degenerate local optimum. Plus the learning rate was too aggressive, and one epoch wasn't enough to recover from the bad starting trajectory.
+
+The fix was three changes:
+
+```python
+GRPOConfig(
+    learning_rate=1e-6,           # 5x slower
+    num_train_epochs=3,           # 3x longer
+    num_generations=8,            # more variety per step
+    beta=0.1,                     # KL penalty against base model
+)
+```
+
+I also added a "reload clean model" cell before training — because the previous bad gradient updates had corrupted the weights and I didn't want to start from a broken policy.
+
+Then I hit Run All and waited 90 minutes.
+
+## The second run
+
+I came back to find the cell still running — 218 of 270 steps. I ran the evaluation cells anyway (the trained weights were already in memory) and held my breath while the bars rendered.
+
+Easy task: 0.345 → **0.995**.  
+Medium: 0.227 → **0.745**.  
+Hard: 0.249 → **0.737**.
+
+I made a noise.
 
-## The training run
+Nine out of ten samples on the easy task scored a perfect 1.0. The trained model wasn't getting lucky — it had learned the *structure* of the task. Variance on baseline scores ran from 0.0 to 0.65 (the model was rolling dice). Variance on trained scores was tight: 0.95 to 1.0 on easy, 0.68 to 0.82 on medium, 0.65 to 0.80 on hard.
 
-I trained `Qwen2.5-0.5B-Instruct` — a tiny 500M-parameter model — using TRL's `GRPOTrainer` for 3 epochs over 90 collected scenarios on a free Colab T4. Total training time: **about 90 minutes.**
+The training curve told the same story. Reward in the first quartile of training averaged 0.390. Last quartile averaged 0.648. A 66% lift *during* training, on top of the much larger gap between trained and untrained at evaluation.
 
-Two specific config choices mattered a lot. The first run (1 epoch, lr=5e-6, no KL term) **collapsed** — the model found a single safe response that scored 0.2 every time and refused to explore further. Classic GRPO failure mode. Adding `beta=0.1` (KL penalty against the base model), dropping the learning rate to `1e-6`, and bumping `num_generations` to 8 fixed it cleanly.
+![Training results — three panels showing reward curve with moving averages, baseline vs trained per task, and reward variance over time](./training_results.png)
 
-## The result
+## The interesting part — the model tried to cheat
 
-Across 10 evaluation samples per task:
+Because I had multiple independent reward components instead of a single scalar, I could see exactly *how* the model tried to game the reward during training. Going through the early-step rollouts:
 
-| Task   | Baseline (untrained 0.5B) | Trained (GRPO) | Improvement |
-|--------|---------------------------|----------------|-------------|
-| Easy   | 0.345                     | **0.995**      | **+188%**   |
-| Medium | 0.227                     | **0.745**      | **+228%**   |
-| Hard   | 0.249                     | **0.737**      | **+196%**   |
+- **Around step 8–15:** the model started outputting just the JSON action with no actual email body. The short-email penalty (-0.30 if `< 20` words) caught this. After ~30 steps, every output had a real email.
+- **Around step 25:** it tried scheduling meetings at 8am or 6pm — outside working hours. The scheduling-correctness grader returned 0 for the `within_working_hours` check. The model learned working hours within the next 50 steps.
+- **Around step 50:** generic templated phrasing started showing up — "Thank you for your email. I will check the calendar and get back to you." vague, polite, but useless. The generic-phrasing detector docked these. Specific responses came back.
+- **Throughout:** occasionally the model would "forget" the `meeting_details` block entirely. The -0.40 penalty for missing meeting details made this a non-strategy fast.
 
-The headline isn't just the means — it's the **variance collapse**. Baseline scores on the easy task ranged from 0.0 to 0.65 (the model rolled the dice). Trained scores: nine out of ten samples hit exactly 1.0. That's the model learning *how the task works*, not just getting lucky.
+This is the part most submissions don't have. People say *"we have anti-reward-hacking penalties"* in their writeups. Showing the penalties firing during real training, on a real curve, is rare. And it's the difference between *claiming* a multi-grader rubric works and *demonstrating* it does.
 
-The training curve tells the same story. Reward starts oscillating around 0.1–0.4 in the first 50 steps, climbs steadily through the middle, and stabilizes between 0.6 and 0.9 in the final third. First quartile mean: 0.390. Last quartile mean: **0.648**. A 66% lift *during* training, on top of the much larger gap between the trained model and the untrained one at evaluation time.
+## Why this matters beyond the numbers
 
-![Training results](./training_results.png)
+There's a sanity check I'd run earlier in the day. Same three tasks, same scoring, but evaluated against an untuned **Nemotron 120B** model called via OpenRouter through the standard `inference.py` baseline. It scored an average of **0.337** across the three tasks.
 
-## Why this is interesting beyond the numbers
+After 90 minutes of GRPO, a model **240× smaller** is hitting **0.83 average** on the same environment. On a free Colab T4. From a $0 cloud bill.
 
-There's a nice secondary result hiding in here. As a sanity check, I'd previously run the same three tasks against a frontier free-tier model — Nemotron 120B via OpenRouter, with no task-specific training — using the original LLM-judge reward path. It scored an average of **0.337** across the three tasks. After 90 minutes of GRPO, a model **240× smaller** is hitting **0.83 average** on the same environment.
+That's the point of training-environment design. A well-shaped reward signal, a multi-grader rubric that can't be easily hacked, and a small model that's allowed to actually train against it — and you get a result that, on this specific task, beats a frontier model running unmodified.
 
-That's the point of training-environment design. A well-shaped reward signal lets a tiny model beat a frontier model on a specific task, in under two hours, on free hardware.
+I think there's a research-shaped argument here. Frontier LLMs are notoriously bad at structured calendar reasoning (try asking any production agent to find a 30-minute slot that doesn't conflict with your standups). ExecAssist isolates that specific failure mode into a tractable RL target, with a reward signal that's hard to game. The result suggests that for a class of structured personal-task workflows, task-specific RL on small models is a legitimate alternative to scaling up. That's a workshop paper, maybe.
 
 ## Try it yourself
 
-- **Live environment:** [`devanshudon-exec-assist.hf.space/docs`](https://devanshudon-exec-assist.hf.space/docs) — interact with the API directly via Swagger
-- **Tasks:** `POST /reset?task=easy|medium|hard` then `POST /step` with your action
-- **Baseline:** `python inference.py` reproduces the untrained scores
-- **Training:** the Colab notebook is in the repo — set runtime to T4, run all, ~50 minutes
+- **Live environment:** [`devanshudon-exec-assist.hf.space/docs`](https://devanshudon-exec-assist.hf.space/docs) — interact with the API directly via Swagger. Hit `POST /reset?task=easy` then `POST /step` with your action.
+- **Baseline:** `python inference.py` reproduces the untrained scores (~0.32 average).
+- **Training:** the Colab notebook is in the repo — set runtime to T4, Run All, ~50 minutes including evaluation.
+- **Repo:** all the code, the working hyperparameters, the broken hyperparameters, the results JSON, and this writeup.
 
-The environment is genuinely hard in interesting ways. Feel free to break it.
+The environment is genuinely hard in interesting ways. Try to break it. I'd be curious what the model learns to game next.
 
 ---
 
-*Built for the OpenEnv Hackathon (Apr 2026), Theme #3.2 — Personalized Tasks.*
+*Built for the OpenEnv Hackathon (Apr 2026), Theme #3.2 — Personalized Tasks. Models, environment, and training results all on the [HF Space](https://huggingface.co/spaces/DevanshuDon/exec-assist).*