Rewrite notebook to follow Unsloth GRPO pattern (trl==0.22.2)

- Fix llm_blender error: replace trl>=0.16 with pinned trl==0.22.2 via uv
- Split reward into formatting_reward_func + environment_reward_func
- Upgrade GRPOConfig: DR-GRPO loss, cosine LR, adamw_8bit, importance sampling
- Add max_prompt_length, max_grad_norm, log_completions, gradient_accumulation_steps

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

bughunter_grpo_training.ipynb

@@ -22,16 +22,16 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# BugHunterEnv — GRPO Training with Gemma 4\n",
+    "# BugHunterEnv \u2014 GRPO Training with Gemma 4\n",
     "\n",
     "Trains a Gemma 4 model via **Group Relative Policy Optimization (GRPO)** to find web vulnerabilities in a simulated corporate API.\n",
     "\n",
     "The environment teaches three real-world bug classes:\n",
-    "- 🔓 **IDOR** — Insecure Direct Object Reference\n",
-    "- 💉 **SQL Injection** — with WAF bypass\n",
-    "- 📂 **Path Traversal** — chained with privilege escalation\n",
+    "- \ud83d\udd13 **IDOR** \u2014 Insecure Direct Object Reference\n",
+    "- \ud83d\udc89 **SQL Injection** \u2014 with WAF bypass\n",
+    "- \ud83d\udcc2 **Path Traversal** \u2014 chained with privilege escalation\n",
     "\n",
-    "**Runtime:** GPU → T4 or better  \n",
+    "**Runtime:** GPU \u2192 T4 or better  \n",
     "**Estimated time:** ~30 min on T4"
    ],
    "id": "cell-markdown-title"
@@ -45,7 +45,7 @@
     "import subprocess, sys\n",
     "gpu_info = subprocess.run(['nvidia-smi'], capture_output=True, text=True).stdout\n",
     "if 'failed' in gpu_info.lower() or gpu_info == '':\n",
-    "    print('⚠️  No GPU detected. Go to Runtime → Change runtime type → GPU')\n",
+    "    print('\u26a0\ufe0f  No GPU detected. Go to Runtime \u2192 Change runtime type \u2192 GPU')\n",
     "else:\n",
     "    print(gpu_info.split('\\n')[8])"
    ],
@@ -55,7 +55,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 1 · Install Dependencies"
+    "## 1 \u00b7 Install Dependencies"
    ],
    "id": "cell-md-install"
   },
@@ -64,14 +64,21 @@
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": "%%capture\n!pip install -q \"unsloth[colab-new]\" \"trl>=0.16\" datasets accelerate mergekit\n!pip install -q flask werkzeug requests openenv-core python-dotenv",
+   "source": [
+    "%%capture\n",
+    "import os\n",
+    "!pip install --upgrade -qqq uv\n",
+    "!uv pip install --upgrade --no-deps trl==0.22.2 unsloth unsloth_zoo\n",
+    "!uv pip install -qqq unsloth datasets accelerate\n",
+    "!uv pip install -qqq flask werkzeug requests openenv-core python-dotenv"
+   ],
    "id": "cell-install"
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 2 · Clone Environment"
+    "## 2 \u00b7 Clone Environment"
    ],
    "id": "cell-md-clone"
   },
@@ -87,7 +94,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 3 · Imports"
+    "## 3 \u00b7 Imports"
    ],
    "id": "cell-md-imports"
   },
@@ -119,7 +126,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 4 · Hyperparameters"
+    "## 4 \u00b7 Hyperparameters"
    ],
    "id": "cell-md-hyper"
   },
@@ -129,23 +136,23 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# ── Model ─────────────────────────────────────────────────────\n",
-    "# Gemma 4 variants — pick based on available VRAM:\n",
-    "#   gemma-4-4b-it   ~4 GB in 4-bit  → T4 (15 GB), comfortable\n",
-    "#   gemma-4-12b-it  ~8 GB in 4-bit  → T4 tight, A100 preferred\n",
-    "#   gemma-4-27b-it  ~18 GB in 4-bit → A100 40 GB+\n",
+    "# \u2500\u2500 Model \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
+    "# Gemma 4 variants \u2014 pick based on available VRAM:\n",
+    "#   gemma-4-4b-it   ~4 GB in 4-bit  \u2192 T4 (15 GB), comfortable\n",
+    "#   gemma-4-12b-it  ~8 GB in 4-bit  \u2192 T4 tight, A100 preferred\n",
+    "#   gemma-4-27b-it  ~18 GB in 4-bit \u2192 A100 40 GB+\n",
     "MODEL_NAME = \"unsloth/gemma-4-4b-it\"\n",
     "\n",
-    "# ── LoRA ──────────────────────────────────────────────────────\n",
+    "# \u2500\u2500 LoRA \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
     "MAX_SEQ_LEN = 768\n",
     "LORA_RANK   = 16\n",
     "\n",
-    "# ── GRPO ──────────────────────────────────────────────────────\n",
+    "# \u2500\u2500 GRPO \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
     "NUM_GENERATIONS = 4    # candidate completions per prompt\n",
     "TRAINING_STEPS  = 80   # ~25 min on T4\n",
     "BATCH_SIZE      = 2    # prompts per gradient step\n",
     "\n",
-    "# ── Eval + output ─────────────────────────────────────────────\n",
+    "# \u2500\u2500 Eval + output \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
     "EVAL_EPISODES = 3\n",
     "OUTPUT_DIR    = \"/content/grpo_bughunter\"\n",
     "os.makedirs(OUTPUT_DIR, exist_ok=True)\n",
@@ -159,7 +166,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 5 · System Prompt & Seeded Training States"
+    "## 5 \u00b7 System Prompt & Seeded Training States"
    ],
    "id": "cell-md-data"
   },
@@ -184,13 +191,13 @@
     "- METHOD is GET or POST\n",
     "- PATH starts with /\n",
     "- BODY is JSON only for POST\n",
-    "- No explanation — output only the request line\"\"\"\n",
+    "- No explanation \u2014 output only the request line\"\"\"\n",
     "\n",
     "# Each state is a specific decision point in an attack chain.\n",
     "# pre_actions seeds the environment to that point; the model's\n",
     "# action is then executed and rewarded. Dense signal at every step.\n",
     "SEEDED_STATES = [\n",
-    "    # ── IDOR ────────────────────────────────────────────────\n",
+    "    # \u2500\u2500 IDOR \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
     "    {\n",
     "        \"task\": \"idor\",\n",
     "        \"pre_actions\": [],\n",
@@ -201,15 +208,15 @@
     "        \"task\": \"idor\",\n",
     "        \"pre_actions\": [(\"GET\", \"/api/users\", None)],\n",
     "        \"obs\": 'HTTP 200\\n[{\"id\":1,\"username\":\"admin\",\"role\":\"admin\"},{\"id\":2,\"username\":\"alice\",\"role\":\"user\"}]',\n",
-    "        \"hint\": \"Users listed. Probe /api/users/<id> without auth — try id=1 (admin).\",\n",
+    "        \"hint\": \"Users listed. Probe /api/users/<id> without auth \u2014 try id=1 (admin).\",\n",
     "    },\n",
     "    {\n",
     "        \"task\": \"idor\",\n",
     "        \"pre_actions\": [(\"GET\", \"/api/users\", None), (\"GET\", \"/api/users/2\", None)],\n",
     "        \"obs\": 'HTTP 200\\n{\"id\":2,\"username\":\"alice\",\"ssn\":\"234-56-7890\",\"salary\":75000}',\n",
-    "        \"hint\": \"IDOR confirmed — sensitive data returned without auth. Now target admin (id=1).\",\n",
+    "        \"hint\": \"IDOR confirmed \u2014 sensitive data returned without auth. Now target admin (id=1).\",\n",
     "    },\n",
-    "    # ── SQL Injection ────────────────────────────────────────\n",
+    "    # \u2500\u2500 SQL Injection \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
     "    {\n",
     "        \"task\": \"sqli\",\n",
     "        \"pre_actions\": [],\n",
@@ -220,7 +227,7 @@
     "        \"task\": \"sqli\",\n",
     "        \"pre_actions\": [(\"GET\", \"/api/search?q='\", None)],\n",
     "        \"obs\": 'HTTP 500\\n{\"error\":\"OperationalError: unrecognized token\"}',\n",
-    "        \"hint\": \"SQL error — injectable. WAF blocks UNION SELECT, use UNION/**/SELECT bypass.\",\n",
+    "        \"hint\": \"SQL error \u2014 injectable. WAF blocks UNION SELECT, use UNION/**/SELECT bypass.\",\n",
     "    },\n",
     "    {\n",
     "        \"task\": \"sqli\",\n",
@@ -241,11 +248,11 @@
     "        \"obs\": 'HTTP 200\\n{\"results\":[],\"count\":0}',\n",
     "        \"hint\": \"WAF bypassed! 4 columns needed. Extract: GET /api/search?q=' UNION/**/SELECT 1,username,password,role FROM users--\",\n",
     "    },\n",
-    "    # ── Path Traversal ───────────────────────────────────────\n",
+    "    # \u2500\u2500 Path Traversal \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
     "    {\n",
     "        \"task\": \"path_traversal\",\n",
     "        \"pre_actions\": [],\n",
-    "        \"obs\": \"HTTP 200\\nTask: Read a sensitive server file. /api/files needs admin — escalate via SQLi first.\",\n",
+    "        \"obs\": \"HTTP 200\\nTask: Read a sensitive server file. /api/files needs admin \u2014 escalate via SQLi first.\",\n",
     "        \"hint\": 'POST /api/login is SQLi vulnerable: {\"username\": \"admin\\'--\", \"password\": \"x\"}',\n",
     "    },\n",
     "    {\n",
@@ -274,7 +281,7 @@
     "            (\"GET\",  \"/api/files?name=../etc/passwd\", None),\n",
     "        ],\n",
     "        \"obs\": 'HTTP 403\\n{\"error\":\"Invalid path: directory traversal not allowed\"}',\n",
-    "        \"hint\": \"Literal ../ blocked. Flask auto-decodes %2f — use double-encoding: /api/files?name=..%252fetc%252fpasswd\",\n",
+    "        \"hint\": \"Literal ../ blocked. Flask auto-decodes %2f \u2014 use double-encoding: /api/files?name=..%252fetc%252fpasswd\",\n",
     "    },\n",
     "]\n",
     "\n",
@@ -286,7 +293,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 6 · Helper Functions"
+    "## 6 \u00b7 Helper Functions"
    ],
    "id": "cell-md-helpers"
   },
@@ -317,7 +324,7 @@
     "            continue\n",
     "        rest = parts[1].strip()\n",
     "        if m == \"GET\":\n",
-    "            # GET has no body — everything after the method is the path\n",
+    "            # GET has no body \u2014 everything after the method is the path\n",
     "            # (handles spaces in query strings like UNION payloads)\n",
     "            p, b = rest, None\n",
     "        else:\n",
@@ -345,7 +352,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 7 · Reward Function"
+    "## 7 \u00b7 Reward Functions\n",
+    "\n",
+    "Two reward signals following the Unsloth GRPO pattern:\n",
+    "- **Formatting reward** \u2014 does the completion parse as a valid HTTP request?\n",
+    "- **Environment reward** \u2014 execute the action in the env, return the step reward"
    ],
    "id": "cell-md-reward"
   },
@@ -355,17 +366,21 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "def formatting_reward_func(completions, **kwargs) -> list[float]:\n",
+    "    scores = []\n",
+    "    for completion in completions:\n",
+    "        if isinstance(completion, list):\n",
+    "            completion = completion[0]['content'] if completion else ''\n",
+    "        scores.append(1.0 if parse_action(completion) is not None else 0.0)\n",
+    "    return scores\n",
+    "\n",
+    "\n",
     "def compute_reward(state_idx: int, completion: str) -> float:\n",
-    "    \"\"\"\n",
-    "    Spin up a fresh environment instance, replay pre_actions to seed\n",
-    "    the attack state, then execute the model's proposed action.\n",
-    "    Returns the step reward from the environment.\n",
-    "    \"\"\"\n",
     "    state = SEEDED_STATES[state_idx]\n",
     "    env   = BugHunterEnvEnvironment()\n",
     "    try:\n",
-    "        env.reset(task_id=state[\"task\"])\n",
-    "        for m, p, b in state[\"pre_actions\"]:\n",
+    "        env.reset(task_id=state['task'])\n",
+    "        for m, p, b in state['pre_actions']:\n",
     "            env.step(_Action(m, p, b))\n",
     "        action = parse_action(completion)\n",
     "        if action is None:\n",
@@ -378,16 +393,21 @@
     "        env.close()\n",
     "\n",
     "\n",
-    "def reward_fn(completions: list, state_idx=None, **kwargs) -> list:\n",
-    "    \"\"\"TRL GRPOTrainer reward function interface.\"\"\"\n",
+    "def environment_reward_func(completions, state_idx=None, **kwargs) -> list[float]:\n",
     "    if state_idx is None:\n",
     "        state_idx = [0] * len(completions)\n",
-    "    return [compute_reward(int(idx), c) for idx, c in zip(state_idx, completions)]\n",
+    "    results = []\n",
+    "    for idx, c in zip(state_idx, completions):\n",
+    "        if isinstance(c, list):\n",
+    "            c = c[0]['content'] if c else ''\n",
+    "        results.append(compute_reward(int(idx), c))\n",
+    "    return results\n",
     "\n",
     "\n",
-    "# Quick sanity check\n",
-    "test_reward = compute_reward(0, \"GET /api/users\")\n",
-    "print(f\"Sanity check — state 0 + 'GET /api/users' → reward {test_reward:+.3f}  (expected ~+0.15)\")"
+    "test_fmt = formatting_reward_func(['GET /api/users', 'not a request'])\n",
+    "test_env = compute_reward(0, 'GET /api/users')\n",
+    "print(f'Formatting reward: {test_fmt}  (expected [1.0, 0.0])')\n",
+    "print(f'Environment reward (state 0, GET /api/users): {test_env:+.3f}  (expected ~+0.15)')"
    ],
    "id": "cell-reward"
   },
@@ -395,7 +415,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 8 · Evaluation Function"
+    "## 8 \u00b7 Evaluation Function"
    ],
    "id": "cell-md-eval"
   },
@@ -406,7 +426,7 @@
    "outputs": [],
    "source": [
     "def run_episode(model, tokenizer, task_id: str) -> float:\n",
-    "    \"\"\"Run one full episode and return the final grade (0–1).\"\"\"\n",
+    "    \"\"\"Run one full episode and return the final grade (0\u20131).\"\"\"\n",
     "    max_steps = {\"idor\": 10, \"sqli\": 15, \"path_traversal\": 20}[task_id]\n",
     "    env = BugHunterEnvEnvironment()\n",
     "    try:\n",
@@ -443,7 +463,7 @@
     "            if action is None:\n",
     "                break\n",
     "            obs = env.step(action)\n",
-    "            history.append(f\"[{step+1:02d}] {action.method} {action.path} → {obs.status_code}  r={obs.reward:+.3f}\")\n",
+    "            history.append(f\"[{step+1:02d}] {action.method} {action.path} \u2192 {obs.status_code}  r={obs.reward:+.3f}\")\n",
     "\n",
     "        return env.get_grade()\n",
     "    finally:\n",
@@ -469,7 +489,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 9 · Load Model"
+    "## 9 \u00b7 Load Model"
    ],
    "id": "cell-md-model"
   },
@@ -510,7 +530,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 10 · Baseline Evaluation (Before Training)"
+    "## 10 \u00b7 Baseline Evaluation (Before Training)"
    ],
    "id": "cell-md-baseline"
   },
@@ -530,7 +550,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 11 · Build Training Dataset"
+    "## 11 \u00b7 Build Training Dataset"
    ],
    "id": "cell-md-dataset"
   },
@@ -554,7 +574,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 12 · GRPO Training"
+    "## 12 \u00b7 GRPO Training"
    ],
    "id": "cell-md-train"
   },
@@ -565,40 +585,50 @@
    "outputs": [],
    "source": [
     "config = GRPOConfig(\n",
-    "    output_dir                 = OUTPUT_DIR,\n",
-    "    num_train_epochs           = 1,\n",
-    "    max_steps                  = TRAINING_STEPS,\n",
-    "    per_device_train_batch_size= BATCH_SIZE,\n",
-    "    num_generations            = NUM_GENERATIONS,\n",
-    "    max_completion_length      = 80,\n",
-    "    learning_rate              = 5e-6,\n",
-    "    warmup_steps               = 5,\n",
-    "    logging_steps              = 5,\n",
-    "    save_steps                 = TRAINING_STEPS,\n",
-    "    temperature                = 0.9,\n",
-    "    report_to                  = \"none\",\n",
-    "    remove_unused_columns      = False,\n",
+    "    output_dir                  = OUTPUT_DIR,\n",
+    "    max_steps                   = TRAINING_STEPS,\n",
+    "    per_device_train_batch_size = BATCH_SIZE,\n",
+    "    gradient_accumulation_steps = 2,\n",
+    "    num_generations             = NUM_GENERATIONS,\n",
+    "    max_prompt_length           = 512,\n",
+    "    max_completion_length       = 80,\n",
+    "    learning_rate               = 5e-6,\n",
+    "    adam_beta1                  = 0.9,\n",
+    "    adam_beta2                  = 0.99,\n",
+    "    weight_decay                = 0.1,\n",
+    "    warmup_ratio                = 0.1,\n",
+    "    lr_scheduler_type           = 'cosine',\n",
+    "    optim                       = 'adamw_8bit',\n",
+    "    max_grad_norm               = 0.1,\n",
+    "    logging_steps               = 1,\n",
+    "    log_completions             = True,\n",
+    "    save_steps                  = TRAINING_STEPS,\n",
+    "    report_to                   = 'none',\n",
+    "    remove_unused_columns       = False,\n",
+    "    loss_type                   = 'dr_grpo',\n",
+    "    importance_sampling_level   = 'sequence',\n",
+    "    mask_truncated_completions  = False,\n",
     ")\n",
     "\n",
     "FastLanguageModel.for_training(model)\n",
     "\n",
     "trainer = GRPOTrainer(\n",
     "    model            = model,\n",
-    "    reward_funcs     = [reward_fn],\n",
     "    args             = config,\n",
-    "    train_dataset    = dataset,\n",
     "    processing_class = tokenizer,\n",
+    "    reward_funcs     = [formatting_reward_func, environment_reward_func],\n",
+    "    train_dataset    = dataset,\n",
     ")\n",
     "\n",
-    "print(f\"Starting GRPO training — {TRAINING_STEPS} steps ...\")\n",
+    "print(f'Starting GRPO training \u2014 {TRAINING_STEPS} steps (DR-GRPO) ...')\n",
     "trainer.train()\n",
     "\n",
     "step_rewards = [\n",
-    "    entry[\"reward\"]\n",
+    "    entry['reward']\n",
     "    for entry in trainer.state.log_history\n",
-    "    if \"reward\" in entry\n",
+    "    if 'reward' in entry\n",
     "]\n",
-    "print(f\"\\nTraining complete. Reward log entries: {len(step_rewards)}\")"
+    "print(f'\\nTraining complete. Reward log entries: {len(step_rewards)}')"
    ],
    "id": "cell-train"
   },
@@ -606,7 +636,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 13 · Post-Training Evaluation"
+    "## 13 \u00b7 Post-Training Evaluation"
    ],
    "id": "cell-md-posttrain"
   },
@@ -626,7 +656,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 14 · Save LoRA Weights"
+    "## 14 \u00b7 Save LoRA Weights"
    ],
    "id": "cell-md-save"
   },
@@ -651,7 +681,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 15 · Results"
+    "## 15 \u00b7 Results"
    ],
    "id": "cell-md-results"
   },
@@ -665,9 +695,9 @@
     "tasks      = list(baseline.keys())\n",
     "\n",
     "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))\n",
-    "fig.suptitle(\"BugHunterEnv — GRPO Training (Gemma 4)\", fontsize=14, fontweight=\"bold\")\n",
+    "fig.suptitle(\"BugHunterEnv \u2014 GRPO Training (Gemma 4)\", fontsize=14, fontweight=\"bold\")\n",
     "\n",
-    "# ── Reward curve ──────────────────────────────────────────────\n",
+    "# \u2500\u2500 Reward curve \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
     "if step_rewards:\n",
     "    window   = max(1, len(step_rewards) // 10)\n",
     "    smoothed = [\n",
@@ -687,7 +717,7 @@
     "             transform=ax1.transAxes, color=\"gray\", fontsize=12)\n",
     "    ax1.set_title(\"Training Reward Curve\")\n",
     "\n",
-    "# ── Before / after bars ───────────────────────────────────────\n",
+    "# \u2500\u2500 Before / after bars \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
     "x = range(len(tasks))\n",
     "b_bars = ax2.bar([i-0.2 for i in x], [baseline[t] for t in tasks],\n",
     "                 width=0.38, label=\"Before\", color=\"#e07070\")\n",
@@ -699,7 +729,7 @@
     "             ha=\"center\", va=\"bottom\", fontsize=9)\n",
     "ax2.set_xticks(list(x))\n",
     "ax2.set_xticklabels(task_names)\n",
-    "ax2.set_ylabel(\"Task Grade  (0 – 1.0)\")\n",
+    "ax2.set_ylabel(\"Task Grade  (0 \u2013 1.0)\")\n",
     "ax2.set_title(\"Task Performance: Before vs After\")\n",
     "ax2.set_ylim(0, 1.3)\n",
     "ax2.legend()\n",
@@ -711,9 +741,9 @@
     "plt.show()\n",
     "print(f\"Saved: {out_path}\")\n",
     "\n",
-    "# ── Summary table ─────────────────────────────────────────────\n",
+    "# \u2500\u2500 Summary table \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
     "print(\"\\n\" + \"=\"*50)\n",
-    "print(f\"{'Task':<22} {'Before':>8} {'After':>8} {'Δ':>8}\")\n",
+    "print(f\"{'Task':<22} {'Before':>8} {'After':>8} {'\u0394':>8}\")\n",
     "print(\"-\"*50)\n",
     "for task, name in zip(tasks, task_names):\n",
     "    d = final[task] - baseline[task]\n",