blog+readme: document the two prior deep-training rounds (legacy SB3 + hybrid Ollama/Groq PPO on 11 tasks); cite training code; add 3-panel deep-training chart

BLOG.md

@@ -260,9 +260,24 @@ visualisation channel during demos and judging.
 
 ---
 
-## 4 · How we actually trained — two passes
+## 4 · How we actually trained — three rounds, two regimes
 
-### Pass A · Legacy MLP baseline (proves the env is solvable)
+We ran two regimes of training, in three rounds:
+
+* **Deep regime** (round 1 + round 2) — many PPO updates per task, on a small
+  hand‑curated set of **7–11 tasks**. The point: prove the rubric is
+  learnable, debug the reward signal, and validate that an LLM actor with a
+  judge critic gives a usable advantage estimate before paying for a long
+  GPU run.
+* **Shallow regime** (round 3) — short PPO loop per task, but on the full
+  **381 procedural scenarios** with saboteur + Slack noise + multi‑fault.
+  The point: prove that the same loop generalises to a hard, broad,
+  adversarial task pool — and ship a LoRA adapter for the demo.
+
+The deep rounds came first; they're the reason we trusted the reward
+signal enough to spend three free Kaggle accounts on the shallow round.
+
+### Pass A · Round 1 — Legacy SB3 PPO/MLP (proves the env is solvable)
 
 Before any LLM work, we needed to know the reward signal isn't broken.
 We ran a stable‑baselines3 PPO agent against a **gym wrapper** of the env,
@@ -305,9 +320,67 @@ a real signal in there for any policy that bothers to investigate before
 acting. The action distribution from the trained MLP is
 `6× query_logs → submit_postmortem` — it memorised the *minimum sufficient
 investigation* for the easy/medium tasks. That's a floor. Now the question
-is: can an LLM agent do better on the **hard** version?
+was: can an *LLM* actor learn the same signal — and does a judge‑style
+critic produce useful advantages?
+
+### Pass A · Round 2 — Hybrid LLM‑actor + judge‑critic on 11 hand‑curated tasks
+
+Before committing GPU minutes to a 381‑task LoRA fine‑tune, we ran three
+shorter PPO loops with a tiny LLM actor and three different critic
+backends, all on **11 hand‑curated tasks** (`task1`…`task11` — the easy +
+medium + hard archetypes plus their variants). All three runs share the
+same trainer:
+[`rl-agent/training/train_hybrid.py`](https://github.com/r1cksync/meta-rl-hack/blob/main/incident-commander/rl-agent/training/train_hybrid.py).
+
+| Round | Actor | Critic | Updates | Episodes | Mean R | Best per‑task | Mitigation rate | Logs |
+| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | --- |
+| **v2** | heuristic | none (PPO baseline) | 33 | 99 | 1.17 | 1.60 (`task1`,`task4`) | **100%** | [`ppo-v2-heuristic/`](https://github.com/r1cksync/meta-rl-hack/tree/main/incident-commander/rl-agent/checkpoints/ppo-v2-heuristic) |
+| **v3** | Qwen2.5:0.5b (Ollama) | heuristic fallback | 12 | 36 | 1.32 | 1.72 (`task10`) | 69% | [`ppo-v3-hybrid-ollama-bedrock/`](https://github.com/r1cksync/meta-rl-hack/tree/main/incident-commander/rl-agent/checkpoints/ppo-v3-hybrid-ollama-bedrock) |
+| **v4** | Qwen2.5:0.5b (Ollama) | **Groq Llama‑3.1‑8B‑instant** | 12 | 36 | **1.78** | **2.41** (`task9`) | 44% | [`ppo-v4-hybrid-ollama-groq/`](https://github.com/r1cksync/meta-rl-hack/tree/main/incident-commander/rl-agent/checkpoints/ppo-v4-hybrid-ollama-groq) |
+
+Every row is a real run — `metrics.jsonl` + `summary.json` + reward‑breakdown
+history all sit in the linked checkpoint folders. The trainer is invoked as:
+
+```bash
+# v2 — pure heuristic actor, PPO baseline
+python -m rl_agent.training.train_hybrid --mode heuristic \
+    --out-dir rl-agent/checkpoints/ppo-v2-heuristic
+
+# v3 — Ollama Qwen2.5:0.5b actor + heuristic critic (Bedrock fallback)
+python -m rl_agent.training.train_hybrid \
+    --mode hybrid-ollama-bedrock --ollama-model qwen2.5:0.5b \
+    --out-dir rl-agent/checkpoints/ppo-v3-hybrid-ollama-bedrock
+
+# v4 — Ollama Qwen2.5:0.5b actor + Groq Llama‑3.1‑8B‑instant critic
+python -m rl_agent.training.train_hybrid \
+    --mode hybrid-ollama-groq --ollama-model qwen2.5:0.5b \
+    --groq-model llama-3.1-8b-instant \
+    --out-dir rl-agent/checkpoints/ppo-v4-hybrid-ollama-groq
+```
 
-### Pass B · LLM agent on 381 procedural scenarios
+![Hybrid PPO loops on 11 tasks — policy loss collapses, entropy compresses, Groq critic delivers the highest mean reward](assets/blog/legacy_deep_training.png)
+
+*Three signs the deep regime worked:*
+
+* **Policy loss collapses on every run.** v2 falls from 1.20 → 0.083
+  (**−93%**) over 33 updates; v3 and v4 both fall from 1.10 → 0.50 (**−55%**)
+  in just 12 updates. Same shape, same trajectory — the loop is healthy.
+* **Entropy compresses smoothly** from ~2.0 → 0.39 (v2) and ~1.9 → 1.21
+  (v3/v4) — the actor distribution is sharpening on a coherent strategy,
+  not collapsing prematurely.
+* **Per‑task max reward keeps rising.** v4's Qwen2.5 actor with the Groq
+  Llama‑3.1‑8B critic posts mean reward **2.41 on `task9`**, **2.39 on
+  `task1`**, **2.29 on `task10`** — well above the heuristic v2 ceiling of
+  1.60. The judge‑critic is providing a useful signal.
+
+What this established: (a) the reward signal is dense enough that a 0.5B
+LLM actor moves it; (b) a frozen LLM judge as critic produces advantages
+that actually push the policy upward (v4 > v3 > v2 on mean reward); (c) PPO
+hyper‑parameters lifted from these runs (γ=0.95, λ=0.92, clip=0.2, KL=0.02,
+entropy=0.01) are the same ones we used in Pass B. It's the bridge that
+made the 381‑task run worth attempting.
+
+### Pass B · Round 3 — LLM agent on 381 procedural scenarios
 
 This is the headline run. The same env, but two changes:
 

README.md

@@ -31,7 +31,7 @@ Built for the **Meta PyTorch OpenEnv Hackathon x Scaler School of Technology, 20
 | 💻 **Source (GitHub)** | [r1cksync/meta-rl-hack](https://github.com/r1cksync/meta-rl-hack) |
 | 📊 **Training notebooks (Kaggle)** | [`kaggle/`](kaggle/) (3 shards × Phi-3.5 + DeepSeek-R1) · [`notebooks/`](notebooks/) (legacy SB3 PPO baseline) |
 | 📦 **Trained adapters** | [`kaggle ran notebooks/shard {1,2,3}/adapter_kaggle{N}/`](kaggle%20ran%20notebooks/) |
-| 🎚️ **Per-update training logs** | [`kaggle ran notebooks/shard {1,2,3}/training_kaggle{N}.json`](kaggle%20ran%20notebooks/) · [`rl-agent/checkpoints/training_metrics.json`](rl-agent/checkpoints/training_metrics.json) |
+| 🎚️ **Per-update training logs** | **Round 3 (shallow, 381 tasks):** [`kaggle ran notebooks/shard {1,2,3}/training_kaggle{N}.json`](kaggle%20ran%20notebooks/) · **Round 2 (deep, 11 tasks):** [`rl-agent/checkpoints/ppo-v{2,3,4}-*/metrics.jsonl`](rl-agent/checkpoints/) · **Round 1 (legacy SB3):** [`rl-agent/checkpoints/training_metrics.json`](rl-agent/checkpoints/training_metrics.json) |
 
 ---
 
@@ -298,9 +298,31 @@ python -m rl_agent.eval \
 
 ---
 
-## Actual Training We Ran (PPO + LoRA, 3 Kaggle Shards)
+## Actual Training We Ran (3 rounds, 2 regimes)
 
-The GRPO roadmap above was the original plan; in practice we ran a custom PPO loop with a dual-model setup that fit comfortably on free Kaggle T4s. The full live numbers are visible at [`/showcase`](https://sagnik-mukherjee-incodent-commander.hf.space/showcase).
+The GRPO roadmap above was the original plan; in practice we ran a custom PPO loop in two regimes — **deep training on a small task set** first, then **shallow training on all 381 procedural scenarios**. Three real rounds, all with full logs, all reproducible from this repo. The full live numbers are visible at [`/showcase`](https://sagnik-mukherjee-incodent-commander.hf.space/showcase) and discussed at length in [`BLOG.md`](BLOG.md) §4.
+
+| Round | Regime | Tasks | Algorithm | Updates | Headline result | Code | Logs |
+|---|---|---:|---|---:|---|---|---|
+| **1 · Legacy SB3** | deep | 7 | SB3 PPO + MLP, 200 k timesteps | 5 evals | mean reward **1.05**, **100% success** over 90 episodes | [`rl-agent/training/train_enhanced.py`](rl-agent/training/train_enhanced.py) + [`gym_wrapper.py`](rl-agent/training/gym_wrapper.py) | [`training_metrics.json`](rl-agent/checkpoints/training_metrics.json) · [`evaluation_report.json`](rl-agent/checkpoints/evaluation_report.json) |
+| **2 · Hybrid v2/v3/v4** | deep | 11 | Custom PPO, heuristic + small-LLM actor + judge critic | 12–33 | policy loss **−93% (v2)** / **−55% (v3,v4)**; v4 mean reward **1.78**, max **2.41** on `task9` | [`rl-agent/training/train_hybrid.py`](rl-agent/training/train_hybrid.py) (+ [`groq_critic.py`](rl-agent/training/groq_critic.py)) | [`ppo-v2-heuristic/`](rl-agent/checkpoints/ppo-v2-heuristic) · [`ppo-v3-hybrid-ollama-bedrock/`](rl-agent/checkpoints/ppo-v3-hybrid-ollama-bedrock) · [`ppo-v4-hybrid-ollama-groq/`](rl-agent/checkpoints/ppo-v4-hybrid-ollama-groq) |
+| **3 · LoRA fine-tune** | shallow | **381** | Custom PPO + LoRA, Phi-3.5-mini actor + DeepSeek-R1 critic, 3 Kaggle shards | 60 / shard | KL & loss decay **50–66%** across all 3 shards; novelty categories all show positive Δ reward (+0.30 → +1.05) | [`scripts/run_training.py`](scripts/run_training.py) + [`colab/train_lib.py`](colab/train_lib.py) + [`scripts/merge_lora_adapters.py`](scripts/merge_lora_adapters.py) | [`shard 1/training_kaggle1.json`](kaggle%20ran%20notebooks/shard%201/training_kaggle1.json) · [`shard 2/training_kaggle2.json`](kaggle%20ran%20notebooks/shard%202/training_kaggle2.json) · [`shard 3/training_kaggle3.json`](kaggle%20ran%20notebooks/shard%203/training_kaggle3.json) |
+
+**Why two regimes?** Round 1 + 2 (deep) prove the rubric is learnable on a small, well-understood task set — they're how we debugged the reward shaper, validated that a frozen LLM judge produces useful advantages, and locked the PPO hyper-parameters. Round 3 (shallow) takes those exact hyper-parameters and runs them across the full 381-scenario procedural curriculum on three free Kaggle T4 accounts in parallel.
+
+**Headline numbers from the deep runs (Round 2):**
+
+| Run | Actor | Critic | Episodes | Mean reward | Top per-task mean | Mitigation rate |
+|---|---|---|---:|---:|---|---:|
+| `ppo-v2-heuristic` | heuristic | none | 99 | 1.17 | 1.60 (`task1`,`task4`) | **100%** |
+| `ppo-v3-hybrid-ollama-bedrock` | Qwen2.5:0.5b (Ollama) | heuristic-fallback | 36 | 1.32 | 1.72 (`task10`) | 69% |
+| `ppo-v4-hybrid-ollama-groq` | Qwen2.5:0.5b (Ollama) | **Groq Llama-3.1-8B-instant** | 36 | **1.78** | **2.41** (`task9`) | 44% |
+
+Across all three deep runs, policy loss collapses (1.20 → 0.083 over 33 updates for v2; 1.10 → 0.50 over 12 updates for v3/v4), entropy compresses (~2.0 → 0.39 for v2; ~1.9 → 1.21 for v3/v4), and v4's small-LLM actor with the Groq Llama-3.1-8B critic clears the heuristic ceiling on the hardest tasks. Plots and the full discussion live in [`BLOG.md` §4](BLOG.md).
+
+### Round 3 — LoRA fine-tune on 381 scenarios (the headline run)
+
+Round 3 is the run that ships the merged LoRA adapter on top of `microsoft/Phi-3.5-mini-instruct`. It uses the same PPO hyper-parameters validated by rounds 1 + 2.
 
 ### Models