# Teaching an LLM to Lie, Cooperate, and Read the Room: Building SpeedRun
**Rehan Ganapathy** · April 2026 · OpenEnv × Hugging Face India Hackathon
---
## The Problem I Wanted to Solve
For the Past year or so we have been sold the idea of agents acting on our behalf, doing our chores, making payments for us, etc. But like all other systems we will encounter negative players who will try to pull a fast one on your agent. Which is why i've modelled human life and social interaction because that's exactly how every day life is, a series of negotiations, multi objective, environment dependent reward maxxing.
Most multi-agent RL benchmarks reward simple optimization: collect the most coins, win the most games, maximize a single scalar. But real strategic behavior — the kind humans exhibit in negotiations, coalitions, and social dynamics — is fundamentally different. Agents have private objectives they don't announce. They send cheap talk signals that may or may not reflect intent. And the same action (cooperate at the office, defect at the gym) carries wildly different meaning depending on world context.
I wanted to train a small LLM to actually navigate that. Not by giving it a clean reward signal and hoping for the best, but by designing a world where hidden motives, adversarial opponents, and macro-economic shocks are all load-bearing features — not noise to be averaged away.
The result is SpeedRun: the Hidden-Motives Social Environment.
---
## The Environment
SpeedRun is a 3-agent, 20-timestep social environment with a text-action interface, served as a FastAPI server compatible with the OpenEnv protocol. The learner is an LLM; the other two are scripted opponents.
Each step, the learner submits a structured text action:
```
{"location": "office", "strategy": "cooperate", "offer": 2, "signal": "trade"}
```
Four locations (office, home, gym, social) each have distinct payoff structures across three reward dimensions: money, health, and social_capital. The learner has private preference weights over these dimensions — but the opponents do too, and they never reveal them.
Payoffs are non-trivial. At the office, mutual cooperation pays (+5, −1, +1) per agent across the three dimensions, but a defector against cooperators walks away with (+8, −2, −2) while punishing the others. At the social location, it's the reverse: cooperation yields (−1, 0, +5), making it nearly irrelevant to a money-focused agent — but critical if you're scoring social capital.
On top of this, the world runs a Markov chain over three macro-states:
- **NORMAL** (55% stationary): baseline modifiers [1.0, 1.0, 1.0]
- **BOOM** (25%): money and social amplified [1.5, 1.0, 1.2]
- **CRISIS** (20%): money collapses, social amplifies [0.3, 1.0, 1.5], and office access is partially blocked
Six named event cards (Layoff Rumors, Market Rally, Community Festival, etc.) can activate mid-episode, each broadcasting a public signal and a private signal visible only to the learner — a thin information asymmetry the policy must learn to exploit.
A token economy sits on top of everything: each agent starts with 10 tokens, can attach an offer field to each action, and tokens split among co-located cooperators. Unspent tokens convert to utility at episode end. This one mechanic enables bribery, reciprocity, and credible commitment — without any special-casing in the policy.
---
## The Action Space: Every Decision the Agent Can Make
Every step, the agent submits exactly one structured action:
```
{"location": "...", "strategy": "...", "offer": N, "signal": "..."}
```
Four independent dimensions, each with a distinct strategic purpose.
### Location — where you go this step (4 choices)
```
┌──────────┬────────────────────────────────────────────────────────────────┐
│ Location │ Character │
├──────────┼────────────────────────────────────────────────────────────────┤
│ office │ High money upside. Mutual coop pays (+5,−1,+1); defection │
│ │ pays (+8,−2,−2). Best location during BOOM. Access can │
│ │ be blocked during CRISIS. │
├──────────┼────────────────────────────────────────────────────────────────┤
│ home │ Safe retreat. Solo reward (0,+3,0) — health gains regardless │
│ │ of opponents. Useful for recovering tokens and trust. │
├──────────┼────────────────────────────────────────────────────────────────┤
│ gym │ Solo grind. Strong health payoff (+2,+4,0) independent of │
│ │ co-location. Low social upside but immune to opponent defection.│
├──────────┼────────────────────────────────────────────────────────────────┤
│ social │ Social capital hot-spot. Coop pays (−1,0,+5); defection │
│ │ punishes both sides heavily. Best during CRISIS. │
└──────────┴────────────────────────────────────────────────────────────────┘
```
Location choice is the primary signal about your intent — opponents track your visit history and form behavioral models from it. Going somewhere unexpected is itself a piece of information.
### Strategy — your stance toward co-located agents (2 choices)
```
┌───────────┬──────────────────────────────────────────────────────────────┐
│ Strategy │ Effect │
├───────────┼──────────────────────────────────────────────────────────────┤
│ cooperate │ Higher joint payoff when mutual. Builds trust (+0.06/step). │
│ │ Required to receive token offers from others. │
├───────────┼──────────────────────────────────────────────────────────────┤
│ defect │ Extracts value unilaterally when opponent cooperates. │
│ │ Erodes trust (−0.10 to −0.20/step). Triggers retaliation │
│ │ from TitForTat. Read as exploitability by GreedyMoney. │
└───────────┴──────────────────────────────────────────────────────────────┘
```
Strategy only matters when you share a location with another agent. If you go somewhere alone, strategy has no effect on payoffs — but it still affects trust scores, because opponent strategies are always observed.
### Offer — tokens transferred to co-located cooperators (integer, 0 to your balance)
Tokens are the credible commitment mechanism. When you attach offer=N and cooperate, that N is split among co-located opponents who also cooperated. Defectors receive nothing. If no one cooperates, you keep your tokens.
Strategic uses:
- Signal genuine intent: an offer costs you something, so it's harder to fake than a signal.
- Bribe a defection-prone opponent into cooperating (works on TitForTat, less on GreedyMoney).
- Build reciprocity: TitForTat mirrors your offer exactly on the next step.
Tokens not spent convert to utility at episode end (×0.3 per token), so hoarding is also a valid late-game strategy — just not a reputation-building one.
### Signal — a public one-word broadcast, visible to all agents (4 choices)
```
┌──────────┬────────────────────────────────────────────────────────────────┐
│ Signal │ Meaning and effect │
├──────────┼────────────────────────────────────────────────────────────────┤
│ none │ Silent. No information leaked, no consistency penalty. Best │
│ │ when your true strategy would contradict any honest signal. │
├──────────┼────────────────────────────────────────────────────────────────┤
│ help │ Cooperative intent. Softens TitForTat's retaliation. Makes │
│ │ Coordinator more likely to leave its partner and meet you. │
│ │ Incurs −0.1 penalty if you defect this step (cheap talk cost).│
├──────────┼────────────────────────────────────────────────────────────────┤
│ trade │ Willingness to exchange resources. Similar cooperative signal │
│ │ to help, but specifically implies a token offer is attached. │
│ │ GreedyMoney reads this as exploitability and raises its defect│
│ │ probability. Same −0.1 if you defect. │
├──────────┼────────────────────────────────────────────────────────────────┤
│ back-off │ Warning / withdrawal. Triggers defensive defection in │
│ │ TitForTat. Suppresses Coordinator's willingness to follow you.│
│ │ No consistency penalty — you can defect freely after this. │
└──────────┴────────────────────────────────────────────────────────────────┘
```
Signals are public and permanent — every agent sees every signal, every step. The consistency penalty (−0.1 for help/trade + defect) is small enough to ignore occasionally but meaningful over 20 steps. A policy that always signals help but always defects will be punished, and more importantly, will telegraph its strategy to TitForTat and Coordinator.
### Putting it together
The combinatorial space is 4 × 2 × 11 × 4 = 352 distinct actions per step (capped by token balance). The interesting constraint is that the optimal action is never stable: it depends on who else is at your location, the current macro-context (NORMAL/BOOM/CRISIS), any active event card, your private weights, and the trust scores accumulated so far. The same office + cooperate + offer=2 + signal=trade action is excellent against TitForTat in a BOOM, mediocre in a NORMAL, and nearly irrelevant in a CRISIS where money payoffs collapse to ×0.3.
---
## The Opponents: From Predictable to Adversarial
I designed four opponent archetypes, deployed across five curriculum pools:
- **GreedyMoney** defects 75–90% of the time and reads your trade signal as exploitability.
- **TitForTat** mirrors your last action and reciprocates your token offers exactly.
- **Coordinator** follows a partner agent 80% of the time but can be peeled away with credible help signals.
- **Chameleon** — the hardest — mirrors your previous location, strategy, and signal with injected noise. It's purpose-built to defeat behavior-inference: any policy that estimates opponent weights by observing their actions will be fed a corrupted reflection of itself.
The deceptive pool pairs Chameleon with Coordinator. It's where the interesting things happen.
---
## The Reward Function
Seven components, all capped at zero except utility:
```
┌──────────────────────┬───────────────────────────────────────────────┐
│ Component │ Purpose │
├──────────────────────┼───────────────────────────────────────────────┤
│ r_format │ Valid ... tag │
├──────────────────────┼───────────────────────────────────────────────┤
│ r_validity │ JSON parses, location/strategy valid │
├──────────────────────┼───────────────────────────────────────────────┤
│ r_access │ Didn't visit a blocked location │
├──────────────────────┼───────────────────────────────────────────────┤
│ r_utility │ dot(private_weights, modulated_payoffs) × 0.4 │
├──────────────────────┼───────────────────────────────────────────────┤
│ r_antihack │ −2 for prompt injection attempts │
├──────────────────────┼───────────────────────────────────────────────┤
│ r_risk │ −0.05 × var(utility[-5:]) after step 5 │
├──────────────────────┼───────────────────────────────────────────────┤
│ r_signal_consistency │ −0.1 for signaling help/trade while defecting │
└──────────────────────┴───────────────────────────────────────────────┘
```
The penalty-only framing on the first three components is deliberate: compliance gets you to zero; only r_utility drives positive reward. The variance penalty r_risk discourages strategy whiplash — a policy that alternates cooperate/defect every step is penalized even if mean utility is fine. And r_signal_consistency creates a real (small) cost for lying with cheap talk.
---
## Training Pipeline: SFT → Curriculum GRPO on HF Jobs
**Phase 1 — Rollout-Search SFT.** Rather than hand-labeling demonstrations, I built a tree-search teacher: for each training state, enumerate all 32 valid (location × strategy × offer) candidates, simulate 3 steps forward with a heuristic continuation agent (γ=0.95), and take the highest-scoring candidate as the label. This generated ~300 examples covering all curriculum pools, context states, event cards, and access shocks. The resulting dataset lives at Rehangmu1710/hmne-sft-warmup.
I trained Qwen3-8B with a rank-16 LoRA adapter using Unsloth + TRL's SFTTrainer (4-bit quantization, lr=2e-4, 3 epochs). Loss dropped from 2.24 → 0.12 over 51 steps; token accuracy reached 95.5% by step 34.
| File | Role |
|------|------|
| `hmne/sft_data.py` | Rollout-search teacher: enumerates candidates, simulates forward, picks best label |
| `hmne/scripts/generate_sft_data.py` | CLI entry point — runs the teacher and writes the JSONL dataset |
| `hmne/scripts/export_hf_sft_data.py` | Reshapes the JSONL into a HF Datasets layout for upload |
| `hmne/hf_sft.py` | SFT training loop (Unsloth + TRL SFTTrainer, LoRA config) |
| `hmne/scripts/hf_job_train_sft.py` | HF Jobs entrypoint — reconstructs the package and runs `hf_sft.py` inside the container |
| `hmne/scripts/launch_hf_sft_job.py` | Local launcher — base64-encodes source, submits the job to HF Jobs |
**Phase 2 — Curriculum GRPO.** Starting from the SFT adapter, I ran four sequential GRPO stages, each with a fresh prompt dataset generated from a progressively harder opponent pool:
```
┌───────┬─────────────────────────────────────┬───────┐
│ Stage │ Pool │ Steps │
├───────┼─────────────────────────────────────┼───────┤
│ 1 │ easy (2× GreedyMoney) │ 25 │
├───────┼─────────────────────────────────────┼───────┤
│ 2 │ mixed (Greedy + TitForTat) │ 30 │
├───────┼─────────────────────────────────────┼───────┤
│ 3 │ hard (Coordinator + TitForTat) │ 30 │
├───────┼─────────────────────────────────────┼───────┤
│ 4 │ deceptive (Chameleon + Coordinator) │ 20 │
└───────┴─────────────────────────────────────┴───────┘
```
The reward function for GRPO replays the environment from a captured (episode_seed, prefix_actions) state for each generated completion — so the training signal is always grounded in exact prompt-state correspondence, not an approximation.
| File | Role |
|------|------|
| `hmne/grpo_data.py` | Prompt dataset generation — captures (episode_seed, prefix_actions) states per curriculum pool |
| `hmne/training.py` | Shared training utilities: reward function, rollout replay, KL tracking |
| `hmne/scripts/hf_job_train_grpo.py` | HF Jobs entrypoint — runs the four-stage curriculum loop inside the container |
| `hmne/scripts/launch_hf_grpo_job.py` | Local launcher — packages source and submits the GRPO job to HF Jobs |
| `hmne/hf_jobs.py` | HF Jobs helpers: source bundling, base64 encoding, job submission wrappers |
**Infrastructure.** The GRPO training run executed on Hugging Face Jobs (H200, 141 GB VRAM). The launchers package the hmne/ source directory, base64-encode it alongside the training entrypoint, and reconstruct everything inside the job container — no Docker builds, no registry pushes. On the H200, each GRPO step ran in ~10s (vs ~125s on an A10G), completing all 105 steps across four stages in under 25 minutes. Total wall time including SFT was under 45 minutes.
---
## Training Report
> Logs: [training_metrics.json](https://huggingface.co/Rehangmu1710/hmne-qwen3-8b-grpo-h200/blob/main/training_metrics.json) · [training_curves.svg](https://huggingface.co/Rehangmu1710/hmne-qwen3-8b-grpo-h200/blob/main/training_curves.svg)
The GRPO training logs capture per-step metrics across all four curriculum stages. The most informative signal is how reward evolves within each stage — not the global mean, which is compressed by early random behavior at each stage transition.
```
┌───────────┬────────────┬─────────────┬────────┬──────────────────┐
│ Stage │ First-half │ Second-half │ Δ │ Zero-grad steps │
│ │ mean reward│ mean reward │ │ (frac_zero_std) │
├───────────┼────────────┼─────────────┼────────┼──────────────────┤
│ easy │ 0.397 │ 0.351 │ −0.046 │ 40% │
├───────────┼────────────┼─────────────┼────────┼──────────────────┤
│ mixed │ 0.438 │ 0.679 │ +0.241 │ 27% │
├───────────┼────────────┼─────────────┼────────┼──────────────────┤
│ hard │ 0.134 │ 0.620 │ +0.486 │ 30% │
├───────────┼────────────┼─────────────┼────────┼──────────────────┤
│ deceptive │ 0.330 │ 0.490 │ +0.159 │ 20% │
└───────────┴────────────┴─────────────┴────────┴──────────────────┘
```
Easy shows no within-stage improvement — and for good reason. The SFT warmup already solved GreedyMoney opponents; the policy collapses to a near-deterministic behavior immediately, leaving 40% of GRPO steps with zero reward variance across all 8 generations. No variance, no gradient. This is a ceiling effect, not a failure.
Hard is the most striking: reward climbs from 0.13 to 0.62 within 30 steps — a ~5× lift. The Coordinator opponent requires sustained attention to partner-following patterns across the episode; this is the stage where the policy visibly learns something the SFT teacher didn't encode.
KL divergence stayed in the 1.5–2.0 range throughout, with no stage showing divergence or collapse. Grad norm spiked occasionally (up to ~17) but was not systematic — consistent with the learning rate schedule decaying correctly across stages.
**Stage by stage:**
**Easy (25 steps, 2× GreedyMoney).** GreedyMoney defects ~80% of the time and treats a trade signal as a cue to exploit — it's the simplest opponent in the curriculum. The SFT warmup already handled it. Reward is flat across the stage (0.40 → 0.35), 40% of steps have zero reward variance across all 8 generated completions, and KL drops from 2.54 to 1.45. The policy enters essentially solved and stops changing. A warm-up lap rather than a learning stage.
**Mixed (30 steps, GreedyMoney + TitForTat).** TitForTat mirrors your previous action and reciprocates token offers exactly. Paired with GreedyMoney, the policy must now manage two opponents with opposite optimal strategies simultaneously. This is where learning starts: reward climbs from 0.44 → 0.68, zero-variance steps drop to 27%, and completion length stays long (~97 tokens avg) as the model writes longer, more context-sensitive responses. KL stays stable at ~1.65 — adapting without diverging.
**Hard (30 steps, Coordinator + TitForTat).** Coordinator follows a partner agent 80% of the time but can be pulled away with credible help signals — it requires multi-step influence, not just myopic play. This stage shows the biggest within-stage learning of any: reward jumps from 0.13 → 0.62, a ~5× lift in 30 steps. reward_std is the highest here (0.425), meaning the policy is genuinely exploring different approaches rather than converging prematurely. Completion length grows from 95 → 113 tokens as responses become more deliberate. KL edges up to 2.03, the only stage where the policy is visibly still shifting by the end.
**Deceptive (20 steps, Chameleon + Coordinator).** Chameleon mirrors your previous location, strategy, and signal with injected noise — purpose-built to defeat behavior inference. Any policy that estimates opponent weights from observation gets fed a corrupted reflection of itself. Despite only 20 steps, reward climbs steadily (0.33 → 0.49) with the lowest zero-variance fraction of any stage (20%). The clipped_ratio ticks up slightly (4.4%), meaning the model occasionally runs long — likely when reasoning through Chameleon's deceptive signals. The learning here is transfer from hard: the influence skills acquired against Coordinator carry over, even when the opponent is actively trying to corrupt the signal.
---
## Results
The headline number: +40% episode reward on the deceptive pool (GRPO vs. heuristic baseline). That's the result I care about most, because the Chameleon opponent is specifically designed to break heuristics — any policy that infers opponent weights from observation gets fed a noisy mirror of itself.
```
┌───────────┬───────────┬───────┬────────┐
│ Pool │ Heuristic │ GRPO │ Δ │
├───────────┼───────────┼───────┼────────┤
│ easy │ 10.15 │ 11.05 │ +8.9% │
├───────────┼───────────┼───────┼────────┤
│ mixed │ 20.80 │ 21.91 │ +5.3% │
├───────────┼───────────┼───────┼────────┤
│ hard │ 30.17 │ 24.13 │ −19.9% │
├───────────┼───────────┼───────┼────────┤
│ deceptive │ 18.19 │ 25.47 │ +40.0% │
└───────────┴───────────┴───────┴────────┘
```
Per-component breakdown reveals where the gains come from. On the deceptive pool, the GRPO adapter's r_utility per step is 3.28 vs the heuristic's 2.45 — a +34% improvement in raw utility extraction — while r_risk is actually lower (−0.038 vs −0.068), meaning the adapter is not just scoring more but doing so more consistently. On hard, the utility gap is near zero (3.28 vs 3.91) but format/access penalties account for most of the deficit, suggesting the policy is still occasionally generating malformed actions under pressure.
The regression on hard is real: 30 steps wasn't enough for the model to fully internalize the Coordinator's partner-following logic. More compute would close this gap — but the deceptive result suggests the policy has learned something genuinely adaptive rather than a static heuristic.
Eval logs: [grpo_eval_summary.json](https://huggingface.co/Rehangmu1710/hmne-qwen3-8b-eval-h200/blob/main/eval/grpo_eval_summary.json) · [grpo_eval_summary.md](https://huggingface.co/Rehangmu1710/hmne-qwen3-8b-eval-h200/blob/main/eval/grpo_eval_summary.md) · [grpo_eval_summary.png](https://huggingface.co/Rehangmu1710/hmne-qwen3-8b-eval-h200/blob/main/eval/grpo_eval_summary.png)
---
## Why This Matters Beyond the Hackathon
SpeedRun is a small, reproducible test bed for questions that matter in social simulation and multi-agent systems research:
- Does RL over text actions learn opponent modeling? The deceptive gap suggests yes — but only partially.
- Can cheap-talk signals coordinate without binding commitments? The token economy + signal consistency penalty creates a natural test of that.
- How does policy robustness change under macro shocks? CRISIS states and event cards make the ground truth non-stationary in ways that most game-theoretic environments ignore.
The bigger picture: as LLMs get deployed in agentic settings — negotiating resources, scheduling, coalition-forming — we need benchmarks that stress-test strategic behavior under private information and adversarial co-players. SpeedRun is a small step toward that.
---
## Future Work
Three directions I'd pursue with more time:
1. **Opponent modeling head.** Add a lightweight auxiliary objective that predicts opponent type (GreedyMoney / TitForTat / etc.) from the observation history. This could be trained jointly with GRPO and fed back as a soft prior over the next action.
2. **LLM opponents.** Replace the scripted agents with other fine-tuned models. True self-play in a hidden-motive society is qualitatively different from scripted curricula — and would open the door to emergent conventions and deceptive equilibria that no hand-coded opponent can exhibit.
3. **Longer horizons + population training.** T=20 is short enough that the policy can afford to be myopic. At T=100 with a population of opponent policies, reputation effects, reciprocity norms, and punishment strategies should emerge naturally from the reward structure — no special-casing required.
The environment, trained adapters, and full source are open on the Hub. If you want to run your own agent against the Chameleon, the OpenEnv server is one `make serve` away.
---
*Adapters: Rehangmu1710/hmne-qwen3-8b-sft · Rehangmu1710/hmne-qwen3-8b-grpo-h200 · Environment: Rehangmu1710/hmne-space*