RUNBOOK.md

HCM:21 Poster — Runbook for Human-Required Work

Everything that can be done without a GPU, network credentials, or a personal account is already committed on branch reward-recalibration:

• ✅ Reward recalibration (scoring.py v2) — reward hacking fixed, 43/43 tests pass
• ✅ Real baseline experiment + hero figure (poster_experiment.py, results.json, poster_figure.png)
• ✅ Figures 1, 2, 4 (poster_diagrams.py → fig1_*.png, fig2_*.png, fig4_*.png)
• ✅ Hermes MCP tool server (integrations/hcm21_mcp_server.py) — verified in-process
• ✅ Poster draft (POSTER_DRAFT.md)

The remaining items below need a human + GPU/accounts. Do them in order.

---

0. Prerequisites (5 min)

• A Hugging Face account with write access to ParetoOptimal/hcm21 (the Space).
• A Google account for Colab (free T4 is enough).
• ANTHROPIC_API_KEY (or any provider key) if you run the Claude/Hermes agent.
• Merge or check out reward-recalibration:

  gh pr create --base main --head reward-recalibration \
    --title "Reward recalibration + poster assets" --fill   # then review & merge
  

---

1. Redeploy the HF Space with scoring v2 ⚠️ DO THIS FIRST

Why it's critical: the Colab notebook trains and evaluates against the live Space (https://paretooptimal-hcm21.hf.space). Until you redeploy, the live env still runs v1 scoring and GRPO will relearn the downsizing exploit. The fix must be live before any training run.

# from the repo root, with scoring.py v2 in the tree:
git remote add space https://huggingface.co/spaces/ParetoOptimal/hcm21   # once
git push space reward-recalibration:main

Wait for the Space to rebuild (Docker), then sanity-check the floor is fixed:

# do-nothing should now score LOW (~0.25), not ~0.66
curl "https://paretooptimal-hcm21.hf.space/demo?seed=42&size=300" | python -m json.tool | grep final_score

Locally you can confirm the same separation any time:

pip install -e ".[server,poster]"
python poster_experiment.py     # deterministic: null 0.247, random 0.262, heuristic 0.448
python -m pytest tests/ -q      # 43 passing

---

2. Run GRPO training in Colab → produce Figure 5 (1–2 h on T4)

1. Open notebooks/hcm21_trl_training.ipynb in Colab (Runtime → T4 GPU).
2. Run cells top-to-bottom. The notebook already:
   • loads unsloth/Qwen3-0.6B + LoRA (r=16, 4-bit),
   • runs the random and heuristic baselines against the live Space,
   • configures GRPOConfig (batch 2 × grad-accum 4, 4 generations, lr 5e-6, fp16),
   • trains, then plots the reward curve and a baseline-vs-trained comparison.
3. Watch for these known gotchas:
   • Space must be v2 (step 1) or the reward signal is hacked.
   • If from openenv.core import GenericEnvClient errors, pin openenv-core to the version the Space was built with (pip install openenv-core==<ver>).
   • 0.6B cold-start: if completions rarely contain valid JSON actions and reward stays ~0, either (a) do the Hermes SFT warm-start in step 4, or (b) raise num_generations and lower lr.
4. Export results for the poster — add this cell at the end of the notebook. The schema is exactly what poster_experiment.py auto-detects:

   import json
   json.dump({
       "trained": trained_scores,   # per-seed final scores of the GRPO agent
       "reward_curve": [e["reward"] for e in trainer.state.log_history if "reward" in e],
   }, open("grpo_results.json", "w"), indent=2)
   

5. Download grpo_results.json from Colab.

Acceptance: trained score > heuristic (≈0.45), ideally ≥0.55, with a reward curve that trends up. If trained ≈ heuristic, that's still a publishable result — report it honestly (small model, limited steps) rather than inflating.

---

3. Add the trained-agent bar to Figure 3 (1 min, local — paste-and-run)

poster_experiment.py already auto-detects the Colab output. Just drop grpo_results.json (from step 2.5) into the repo root and re-run:

cp /path/to/grpo_results.json .
python poster_experiment.py

This regenerates poster_figure.png with a green Trained (GRPO) bar plus a dashed "heuristic bar" reference line, and — if grpo_results.json includes a reward_curve — also writes fig5_grpo_reward_curve.png (Figure 5). With no file present the script is unchanged (baseline-only). No code edits needed.

---

4. (Optional, high-value) Hermes-Agent warm-start pipeline

This is the "agent that grows with you" enhancement. It both fixes the 0.6B cold-start and gives the poster a second experimental axis.

4a. Install Hermes and verify its real APIs

git clone https://github.com/nousresearch/hermes-agent && cd hermes-agent
# follow its README installer; then confirm the exact names of:
#   - the batch trajectory generation script (README calls it "batch trajectory generation")
#   - the trajectory compression / training-export utility
#   - the skills directory + agentskills.io skill format

The public README does not print these script names. Confirm them in the cloned repo before scripting against them; update step 4c accordingly.

4b. Register the HCM:21 MCP server with Hermes

The server is built and tested (integrations/hcm21_mcp_server.py). Launch it:

pip install -e ".[server,mcp]"
python -m integrations.hcm21_mcp_server     # stdio MCP transport

Add it to Hermes' MCP config (Hermes "Connect any MCP server"). Hermes can now call hcm21_reset → hcm21_available_actions → hcm21_step ×N → hcm21_score. Author a Hermes skill that runs the scan→plan→produce→control loop and reads the final score — this is the procedural-memory artifact the poster describes.

4c. Generate expert trajectories → SFT warm-start → GRPO

1. Run Hermes (strong model) over the 4 scenarios × several seeds via the MCP
   server → use Hermes' batch trajectory generation to collect episodes.
2. Use Hermes' trajectory compression to export an SFT dataset of
   (prompt, action-sequence) pairs from the high-scoring episodes.
3. SFT unsloth/Qwen3-0.6B on that dataset (1–2 epochs), THEN run the GRPO
   notebook from that checkpoint instead of the base model.

Acceptance: SFT-then-GRPO reaches a higher score / converges faster than GRPO-from-base. Report the three-regime table from POSTER_DRAFT.md §6 (GRPO-only vs Hermes-skills-only vs hybrid).

---

5. Assemble and submit the poster (by July 26, 11:59 PM PDT)

1. Layout (A0 portrait or the conference's stated size — confirm on Sessionize):
   • Top: title + §1 motivation.
   • Left column: §2 environment + Fig 1 (architecture), Fig 2 (phase loop).
   • Center (hero): §3 finding + §4 fix + Fig 3 (recalibration) — biggest panel.
   • Right column: §5 training + Fig 4 (GRPO loop) + Fig 5 (reward curve), then §6 Hermes + the three-regime table.
   • Footer: §7 "Try it" QR codes (Space, GitHub, Colab) + MIT license.
2. Build the PDF in your tool of choice (PowerPoint/Keynote/Figma/LaTeX tikzposter or beamerposter). Embed the committed PNGs at 150+ dpi.
3. Submit on Sessionize (link from the CFP page). Max 2 speakers. Paste the abstract from POSTER_DRAFT.md (Title + §1). Upload the PDF if requested.
4. Diversity rule: if 3+ presenters, gender diversity is required — keep to ≤2 or compose accordingly.
5. Print & ship: presenters print and bring their own poster (CFP requirement). Order the print ~2 weeks before Oct 20 (San Jose).

CFP dates

• Submit: Sun July 26, 11:59 PM PDT
• Notifications: Mon Aug 17 · Schedule: Tue Aug 18 · Event: Oct 20–21, San Jose
• Accepted presenters get complimentary passes; slides due before the event.

---

6. Final verification checklist

• HF Space redeployed; live do-nothing score ≈ 0.25 (not 0.66)
• python -m pytest tests/ -q → 43 passing
• python poster_experiment.py → null 0.247, random 0.262, heuristic 0.448 (deterministic)
• python poster_diagrams.py → Figs 1, 2, 4 regenerate
• Colab GRPO run complete; grpo_results.json + reward curve saved (Fig 5)
• Trained-agent bar added to Fig 3
• (Optional) Hermes warm-start results + three-regime table
• Poster PDF assembled with all 5 figures + QR codes
• Submitted on Sessionize before July 26
• Caveat resolved: Hermes script names verified against the actual repo