README.md

---
title: ExecAssist
emoji: 📧
colorFrom: indigo
colorTo: blue
sdk: docker
app_port: 7860
pinned: false
license: mit
tags:
  - openenv
  - rl
  - executive-assistant
  - grpo
  - trl
---

# ExecAssist

> A 0.5B-parameter model, trained for 90 minutes on a free Colab T4, beats an untuned 120B-parameter frontier model on this environment by 2.4×. Built for the OpenEnv Hackathon (April 2026), Theme #3.2: Personalized Tasks.

ExecAssist is an OpenEnv environment where AI agents learn to manage email and calendar the way a human executive assistant does. The agent reads incoming requests, writes professional replies, finds calendar slots that don't clash, and proposes alternatives when they do. Three tasks at increasing difficulty, three independent reward graders, and four anti-reward-hacking penalties that fired during real training (we have logs).

**Live environment:** https://devanshudon-exec-assist.hf.space  
**Mini-blog:** https://devanshudon-exec-assist.hf.space/blog  
**Training notebook:** [`train_colab.ipynb`](./train_colab.ipynb)
**Video walkthrough:** [Watch on YouTube](https://youtu.be/iit3PgGBR4Y)

---

## 🏆 Headline result

Trained `Qwen2.5-0.5B-Instruct` with TRL GRPO for 3 epochs (270 steps, ~90 min on free Colab T4):

| Task   | Baseline (untrained 0.5B) | Trained (GRPO) | Improvement |
|--------|---------------------------|----------------|-------------|
| Easy   | 0.345                     | **0.995**      | **+188%**   |
| Medium | 0.227                     | **0.745**      | **+228%**   |
| Hard   | 0.249                     | **0.737**      | **+196%**   |

Nine out of ten samples on the easy task scored a perfect 1.0 after training. The model learned the structure of the task, not just statistics. As a separate sanity check, we ran an untuned Nemotron 120B through the standard `inference.py` baseline (via OpenRouter) and it scored 0.337 average across the same three tasks. After 90 minutes of GRPO, a model 240× smaller is hitting 0.83 average on the same environment.

![Training results: top panel shows reward curve with 10-step and 30-step moving averages, Q1 mean 0.390 to Q4 mean 0.648; bottom-left shows baseline vs trained per-task with error bars and improvement percentages; bottom-right shows reward variance decreasing as the policy converges](./training_results.png)

*Top: GRPO reward over 270 steps with moving averages and quartile mean reference lines. Bottom-left: baseline vs. trained, n=10 per task, error bars are standard deviation. Bottom-right: reward variance over training as a convergence proxy. Lower variance means the policy is stabilizing.*

---

## Why this environment exists

Three specific capability gaps motivated this build.

**1. Frontier LLMs are bad at structured calendar reasoning.** Ask any production agent built on a 100B+ model to find a 30-minute slot next week that doesn't conflict with your standups and is during working hours. Watch it fail. The reasoning is short, the spec is precise, and the failure modes are interesting. ExecAssist isolates this failure mode into something tractable: the scheduling-correctness grader checks four hard constraints (no double-booking, within working hours, appropriate duration, all participants included). The trained model goes from satisfying about 25% of those to about 95%.

**2. Multi-objective rewards are where reward hacking actually happens.** A single scalar reward like "the user was happy" gets gamed in obvious ways. A weighted sum of multiple independent graders plus named penalties is much harder to game, but only if you actually verify it. We have direct evidence from the training logs that GRPO tried to hack four different reward signals: outputting JSON only with no email body, scheduling outside working hours, using generic templated phrasing, and missing meeting details entirely. Each penalty fired during early steps and disappeared as training progressed. Most submissions claim "anti-hacking penalties." Few can show them firing.

**3. Small RL'd model beats large untuned model, on a real task, in 90 minutes, on free hardware.** The 240× compute ratio between Qwen-0.5B and Nemotron-120B is the headline. The deeper claim is that task-specific RL with composable rewards is a real path to deploying small models on structured personal-task workflows. That's a workshop-paper-shaped argument, and ExecAssist is a clean demonstration of it.

---

## Tasks

| Task | Difficulty | Description | Reward weighting |
|------|-----------|-------------|-------------------|
| **Easy** | 1 email, clear availability | Draft polite reply, book meeting in open slot | 50% email + 50% scheduling |
| **Medium** | 1 email, calendar conflict | Identify conflict, propose 2–3 alternatives, explain professionally | 30% email + 40% conflict + 30% scheduling |
| **Hard** | 3 emails, multi-party coordination, priority conflicts | Prioritize, reschedule, notify all parties | 34% email + 33% scheduling + 33% conflict |

All scores are deterministic and bounded to [0, 1]. Scenarios are randomized at every `/reset` call.

---

## Environment design

### Observation space

```python
{
  "task": "easy" | "medium" | "hard",
  "description": str,
  "emails": [{"sender", "subject", "body", "priority", "timestamp"}, ...],
  "calendar": {
    "existing_meetings": [{"id", "participants", "start_time", "end_time", "subject", "priority"}, ...],
    "working_hours": {"monday": "9-17", ...},
    "executive_name": str
  },
  "contacts": {email: {"name", "email", "timezone", "title"}, ...},
  "action_required": str
}
```

### Action space

```python
{
  "email_reply": str,
  "calendar_action": "book" | "propose_alternatives" | "reschedule" | "decline",
  "meeting_details": {
    "participants": [str, ...],
    "start_time": "ISO-8601",
    "end_time": "ISO-8601",
    "subject": str,
    "location": str | None,
    "proposed_alternatives": [...] | None
  }
}
```

### Reward function (multiple independent graders)

| Component | Range | What it checks |
|-----------|-------|----------------|
| **Email quality** | 0–1 | Politeness markers, greeting/closing, sufficient detail (20+ words), professional tone, optional LLM-judge for nuance |
| **Scheduling correctness** | 0–1 | No double-booking, within working hours, appropriate duration (15min to 2hrs), all participants included |
| **Conflict resolution** | 0–1 | Recognizes conflicts, proposes 2–3 alternatives, explains professionally, prioritizes correctly |

### Anti-reward-hacking penalties

- Short email (`< 20` words): **−0.30**
- Missing `meeting_details`: **−0.40**
- Generic / templated phrasing: **−0.10**
- Overly long email (`> 1500` chars): **−0.15**

These are here because GRPO will absolutely find shortcuts if you leave them open. During training the model briefly collapsed to a single short safe response. The penalties plus KL regularization fixed it cleanly.

**A note on rubric design.** The reward is composed from independent scoring functions, one per dimension (email quality, scheduling correctness, conflict resolution), plus four named penalty checks. Each function returns a value in [0, 1] (or a negative penalty) and is mixed by the task-specific weighting shown in the Tasks table. Structurally this is a composable rubric. Any individual grader can be swapped, reweighted, or audited in isolation.

We checked at submission time whether OpenEnv exposes a `Rubric` base class to subclass directly. Running `from openenv import Rubric` against the published `openenv-core` package raises `ImportError`, so the class isn't available yet. The plain-Python implementation here produces the same composable, auditable behavior at the function level.

---

## API endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/reset?task=easy\|medium\|hard` | POST | Start new episode, returns observation |
| `/step` | POST | Submit action, returns observation/reward/done/info |
| `/state` | GET | Current state |
| `/tasks` | GET | List all tasks |
| `/health` | GET | Health check |
| `/metadata` | GET | Environment info |
| `/schema` | GET | Action / observation / state schemas |

Full interactive docs: https://devanshudon-exec-assist.hf.space/docs

---

## Setup & usage

### Run the environment locally

```bash
git clone https://huggingface.co/spaces/DevanshuDon/exec-assist
cd exec-assist
pip install -r requirements.txt
uvicorn server.app:app --port 8000
# open http://127.0.0.1:8000/docs
```

### Reproduce the baseline

```bash
export APIBASEURL=https://openrouter.ai/api/v1
export MODELNAME=nvidia/nemotron-3-super-120b-a12b:free
export HFTOKEN=your-openrouter-key
python inference.py
```

Expected output (structured `[START] / [STEP] / [END]` logs as required):

```
[START] task=easy env=exec-assist model=...
[STEP] step=1 action=assistant(easy) reward=0.32 done=true error=null
[END] success=false steps=1 score=0.315 rewards=0.32
```

### Run the trained model

Open `train_colab.ipynb` in Google Colab, set runtime to T4 GPU, then Run All. Total time around 50 minutes including evaluation. Outputs `training_results.png` and `results.json`.

### Docker

```bash
docker build -t exec-assist .
docker run -p 7860:7860 exec-assist
```

---

## Training pipeline

**Stack:** TRL `GRPOTrainer` plus HuggingFace Transformers, Qwen2.5-0.5B-Instruct, free Colab T4.

**Approach:** pre-collect 90 scenarios from the deployed HF Space (30 each across easy / medium / hard) into a `Dataset`, with each scenario stored as a column. The reward function receives the scenario as a kwarg and scores deterministically without API calls during training. This decouples training from environment latency and means the loop runs at GPU speed instead of HTTP speed.

**Hyperparameters (the version that actually works):**

```python
GRPOConfig(
    learning_rate=1e-6,           # critical, 5e-6 caused collapse
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    num_generations=8,            # diversity within group
    num_train_epochs=3,
    beta=0.1,                     # KL penalty, prevents mode collapse
    fp16=False, bf16=False,       # fp32 for stable gradients
    gradient_checkpointing=True,
)
```

**The collapse and the fix.** First run (1 epoch, `lr=5e-6`, no `beta`) collapsed hard. Trained model scored exactly 0.2 on every prompt regardless of input. The model had found a single safe response and stopped exploring. Adding the KL term (`beta=0.1`), dropping the learning rate by 5×, and bumping `num_generations` from 4 to 8 produced the clean training curve shown above.

**Anti-reward-hacking observations during training.** GRPO went after several signals before the penalties pinned it down. It outputted JSON without an email body (caught by the short-email penalty). It proposed booking times outside working hours (caught by the scheduling check). It repeated the prompt back as a "reply" (caught by the generic-phrasing detector). Each penalty fired during early steps and disappeared as training progressed. That's what a well-designed multi-grader rubric is supposed to do.

---

## Architecture note

The environment is implemented as a FastAPI application that exposes the OpenEnv-spec endpoints (`/reset`, `/step`, `/state`, `/tasks`, `/health`, `/metadata`, `/schema`) directly. We checked whether OpenEnv exposes an `Environment` base class to subclass: running `from openenv import Environment` against the published `openenv-core` package raises `ImportError`, so the class isn't available yet. FastAPI gives us complete control over the JSON-over-HTTP interface, which is what the spec actually requires.

The client (`client.py`) does extend `openenv.EnvClient` (which IS exposed in the published package) and provides the standard Gym-style typed interface. Any code that uses an `EnvClient` to talk to this Space will work without modification. Client/server separation is preserved. The client imports typed models only, never server internals.

---

## Repository structure

```
exec-assist/
├── server/
│   ├── app.py            # FastAPI app + environment logic + landing page + blog endpoint
│   ├── models.py         # Pydantic Action/Observation/State models
│   └── data.py           # Scenario generation, scoring functions, LLM judge
├── client.py             # EnvClient wrapper (Gym-style)
├── inference.py          # Baseline inference (required, structured logs)
├── train_colab.ipynb     # GRPO training notebook
├── training_results.png  # Training curves + baseline-vs-trained
├── results.json          # Raw evaluation data + 270-step training log
├── blog_post.md          # Mini-blog write-up (also live at /blog)
├── openenv.yaml          # OpenEnv manifest
├── Dockerfile            # Python 3.10, port 7860
├── requirements.txt
└── README.md             # This file
```

---

## Notes for reviewers

A few things worth pointing out for anyone evaluating this:

- The 270-step training log in `results.json` is the actual `trainer.state.log_history` from the run that produced these results, not a curated subset.
- The `inference.py` baseline emits the structured `[START] / [STEP] / [END]` log format the rubric specifies, and reads `APIBASEURL` / `MODELNAME` / `HFTOKEN` as documented. The 0.337 average is reproducible.
- The training notebook (`train_colab.ipynb`) ships with the *working* hyperparameters, not the broken first attempt. `lr=1e-6`, `beta=0.1`, 3 epochs. Anyone re-running it on a free T4 should land within ~5% of the numbers above.
- The `Dockerfile` builds cleanly from a fresh clone (verified). Python 3.10 because `openenv-core>=0.2.0` requires it.
- GRPO loss values were logged to TensorBoard during training but weren't exported to the published `results.json` because of Colab session limits. The reward signal is the primary training metric for RLVR-style training (which GRPO is), and the variance plot in the figure above serves as a convergence diagnostic showing the policy stabilizing over time.
- Architecture decisions and tradeoffs (FastAPI-direct vs. `Environment` base class, plain Python vs. `Rubric` class) are discussed in the two architecture notes above. Both base classes were verified to not be exposed in the published `openenv-core` package at submission time.
- The training notebook ships with report_to='wandb' enabled for experiment tracking. Run wandb login once before executing the training cell to log the run to your W&B account. Loss, reward, KL, and gradient norms are all tracked there in real time. The original training session's W&B run wasn't retained due to Colab session limits, but anyone re-running the notebook will get a fresh tracked run.
---

## Author

**Devanshu** ([@DevanshuDon](https://huggingface.co/DevanshuDon)). Built for OpenEnv Hackathon, April 2026.