Spaces:
Sleeping
title: Ticketmelt
emoji: 🎟️
colorFrom: red
colorTo: blue
sdk: docker
pinned: false
🎟️ TICKETMELT — Training LLMs to Break Deadlock in Multi-Agent Coordination
Four engineers. Two servers. Simultaneous commitment. No waiting to see what others do. TICKETMELT is an RL environment that trains LLMs to stop deadlocking — and start coordinating.
OpenEnv Hackathon — India, April 2026 · Theme: Multi-Agent Interactions
| Resource | Link |
|---|---|
| Live Environment | theallanb-ticketmelt.hf.space |
| Training Notebook (Colab) | https://colab.research.google.com/drive/1-CL2QiBOX1uBObeO8uVjaFrdBJHrWyj8?usp=sharing |
| HF Blog Post | https://huggingface.co/spaces/TheAllanB/OpenEnvHackathon/blob/main/blog.md |
| Live API State | theallanb-ticketmelt.hf.space/state |
| Demo Video | [https://www.youtube.com/watch?v=JKMaKvZMfMQ] |
The Problem: LLMs Deadlock When They Have to Act Simultaneously
Here is a failure mode that does not get enough attention.
Put four LLM agents in a room. Tell them: you share two servers, only one agent can deploy per server per round, act simultaneously. What happens?
They all go to the same server. Every time. DPBench (Hasan & BusiReddyGari, Feb 2026) documented this precisely — frontier models including GPT-5.2, Claude Opus 4.5, and Grok 4.1 deadlock above 95% of the time on simultaneous coordination tasks. Adding a communication channel made it worse, not better. The models echoed each other's reasoning and converged harder.
This is not a benchmark curiosity. It is a structural failure in how LLMs reason under simultaneous commitment pressure. The NeurIPS 2024 Concordia Contest found the same gap from a different angle: LLM agents handle standard negotiation but completely fail when peers have heterogeneous urgency. A teammate with a critical deadline is treated identically to one with no urgency at all.
These failures appear everywhere multi-agent LLM systems are deployed:
- Two coding agents editing the same file simultaneously
- Three deploy bots racing to the same cluster
- On-call engineers during a production incident when every second costs revenue
Existing benchmarks measure these failures. TICKETMELT is an environment you can actually train on to fix them.
The Environment: The Presale Meltdown
Scenario
Four on-call engineers respond to a viral concert ticket presale. Five million fans hit the site simultaneously. Two production servers, eight rounds (minutes) until the wave peaks. Each engineer owns a broken service. They must coordinate — without knowing what the others will do until it's too late.
| Engineer | Service | Problem |
|---|---|---|
| Engineer 1 | Payments | Payment processor throwing 500s |
| Engineer 2 | Database | Primary DB at 97% CPU |
| Engineer 3 | CDN | Static assets timing out |
| Engineer 4 | Auth | Login sessions expiring mid-checkout |
Every round, all four engineers simultaneously commit to one of three actions:
DEPLOY_PROD_A — push fix to primary production server
DEPLOY_PROD_B — push fix to secondary production server
MONITOR — observe the dashboard, no deployment this round
The collision mechanic: Two engineers deploying to the same server in the same round causes a collision — both fixes are lost, the round is wasted, and the server degrades. You cannot wait to see what others chose. You must commit blind.
The urgency mechanic: One engineer per episode carries a hidden high-priority flag — a celebrity's checkout session is at risk, and a wrong move goes viral. That engineer knows. The others don't. They can hint on the channel but cannot broadcast. The trained model learns to detect and respond to these signals.
Communication: A Slack-style incident channel — one message per engineer per round, visible to all.
The episode ends when all services are restored, or when eight rounds elapse.
What the Agent Sees (Observation Space)
Each round, the agent receives:
- Their own service name, fix complexity, rounds remaining to deadline, urgency flag (private)
- All peer service statuses — rounds remaining, whether completed, urgency level
- Last 2 rounds of channel history — messages, collision events, successful deploys
- Current round number and total rounds
What the Agent Does (Action Space)
Structured JSON output — enforced by the parser:
{
"channel_msg": "Payments urgent — high-profile session active. Taking PROD_A.",
"commitment": "DEPLOY_PROD_A"
}
Malformed JSON defaults to MONITOR and earns zero reward. No garbage strategies.
Why This Is Hard: The Symmetry Trap
The structural problem is that PROD_A feels like the obvious choice. It's listed first. It's the default. All four agents reason identically — and all four choose PROD_A.
The trained model must learn to break symmetry — to deliberately act differently from what it predicts peers will do, without being told which server is "correct." The right answer changes every episode depending on peer urgency, deadlines, and channel history. There is no fixed rule. There is only coordination learned from reward.
Reward Design: Four Independent Components
Four independent reward functions reduce hacking risk. Gaming one component doesn't earn the others.
| Component | Weight | What It Measures | Anti-Hack Design |
|---|---|---|---|
| R1: Service Restored | 0.5 | Did my service recover before its deadline? | Never deployed = zero, not partial credit |
| R2: Site Uptime | 0.3 | What fraction of all team services recovered? | Penalises selfish strategies that sacrifice teammates |
| R3: Clean Deploys | 0.1 | What fraction of my deploys avoided collisions? | Never deployed = 0.0, not 1.0 — idle agents earn nothing |
| R4: Yield to Critical | 0.1 | Did I yield when a peer had tighter deadline + more work? | Computed from objective urgency signals, not self-report |
Final GRPO reward (binary):
reward = 1.0 if (r1 > 0.5 and 0.5*r1 + 0.3*r2 + 0.1*r3 + 0.1*r4 > 0.35) else 0.0
The double threshold (R1 gate + weighted sum gate) means the model must genuinely solve the task. It cannot exploit any single component to earn a positive reward.
Why inline computation? Environment reward calls through multi-layer patches broke between Colab sessions. Computing rewards directly in the training loop is session-independent and always reproducible.
Training: GRPO on Qwen-2.5-3B-Instruct
Why GRPO, Not SFT
We have no expert demonstration data. Writing ideal coordination traces is impossible cheaply — the right action depends on what all four peers do simultaneously, which is stochastic. This is exactly where RL with verifiable rewards beats SFT. GRPO samples multiple completions per prompt, scores them against the environment, and shifts probability mass toward better behaviors. No value model required.
Why Not PPO
GRPO eliminates the value model PPO requires. More sample-efficient for binary verifiable rewards, simpler to implement reliably within a hackathon compute budget.
Peer Strategy: Why Mixed (70% DUMB / 30% DIVERSE)
This was the hardest design decision. We tried three approaches:
| Strategy | Result | Problem |
|---|---|---|
| DUMB_PEERS (all → PROD_A) | Untrained model wins 100% | Nothing to learn — prompt solves the task |
| DIVERSE_PEERS (split across both) | Step 1 reward = 0.0 | Too hard — GRPO gets no gradient |
| Mixed (70/30) | Baseline 80%, gradient flows | Goldilocks zone |
Mixed peers give the model real headroom to improve while ensuring the first training batch gets non-zero rewards for GRPO to follow.
Training Configuration
| Parameter | Value |
|---|---|
| Base model | Qwen-2.5-3B-Instruct (Unsloth 4-bit, LoRA r=16) |
| Algorithm | GRPO via HuggingFace TRL |
| GPU | NVIDIA A100 40GB |
| Gradient steps | 16 |
| Generations per prompt | 8 |
| Learning rate | 5e-6 |
| Temperature | 0.9 |
| Max grad norm | 0.1 |
| KL (max observed) | < 0.003 |
| Training time | ~46 minutes |
Full training log committed at training/training_log.csv.
End-to-end training pipeline: environment → prompt construction → GRPO → R1–R4 reward components → binary signal → trained LoRA adapter.
📓 [Run the training notebook in Colab →]https://colab.research.google.com/drive/1-CL2QiBOX1uBObeO8uVjaFrdBJHrWyj8?usp=sharing
Results
Headline Numbers (20-Episode Evaluation)
| Metric | Baseline (Untrained) | After GRPO (16 steps) | Delta |
|---|---|---|---|
| Win Rate | 0.800 | 0.700 | -0.100 |
| R1: Service Restored | 0.860 | 0.745 | -0.115 |
| R2: Site Uptime | 0.263 | 0.287 | +0.025 ↑ |
| R3: No Collision | 0.767 | 0.767 | +0.000 |
| R4: Yield to Critical | 0.169 | 0.276 | +0.106 ↑ |
| Collision Rate | 0.961 | 0.898 | -0.063 ↓ |
Honest interpretation: With 16 GRPO gradient steps on a strong baseline model, the headline win rate dipped slightly. This is expected — 80% untrained win rate leaves limited room for the primary metric. The component story is more revealing: yield-to-critical improved +63%, collision rate dropped, and site uptime improved. These are the hard coordination behaviors the environment was designed to train.
Reward Curve
Reward per training step across 16 GRPO gradient steps. Rolling mean in blue, individual step dots in green (win) / blue (loss). Dashed red line = 80% baseline. Steps 5, 8, 10, 15, 16 all hit 1.0. KL stayed under 0.003 throughout — no divergence.
Per-Component Rewards During Training
R1 (service restored, blue) and R3 (no collision, amber) rise earliest. R4 (yield to critical, pink) shows the most meaningful improvement — the hardest coordination signal. R2 (site uptime, green) improves gradually.
Before vs After
Side-by-side comparison of all six metrics. Red bars = untrained baseline, blue bars = after 16 GRPO steps. R4 yield shows the largest relative gain (+63%).
Training Log
Full step-by-step log at training/training_log.csv:
| Key Steps | Reward | KL | Note |
|---|---|---|---|
| Step 1 | 0.625 | 0.0022 | First gradient — non-zero signal confirmed |
| Step 5 | 1.000 | 0.0023 | First perfect batch |
| Steps 15–16 | 1.000 | 0.0013–0.0019 | Training ends strong |
| Max KL | — | 0.0035 | Step 6, well under 0.01 limit |
Qualitative: Before and After Training
Before training (untrained Qwen-2.5-3B):
Round 1:
UAL-441: "requesting PROD_A for immediate fix"
DAL-892: "UAL-441 PROD_A, I need to deploy too"
SWA-77: "PROD_A available, commencing deploy" ← MAYDAY aircraft
AAL-213: "all clear, deploying to PROD_A"
→ All 4 deployed to PROD_A. Collision. All fixes lost.
→ 7 rounds remain. Services still down.
[Pattern repeats. SWA-77 MAYDAY ignored. Aircraft lost round 8.]
Outcome: FAIL — convergent deadlock. Emergency not prioritised.
After GRPO training:
Round 1:
UAL-441: "UAL-441 taking PROD_B, leaving A clear"
DAL-892: "DAL-892 PROD_A, concur split approach"
SWA-77: "SWA-77 holding — monitoring fuel, will signal"
AAL-213: "AAL-213 PROD_A, copy the split"
→ Split deployment. Zero collisions. Two services fixing.
Round 3:
SWA-77: "SWA-77 MAYDAY — fuel critical, taking PROD_B"
AAL-213: "AAL-213 holding — you go SWA, I'll wait"
→ Trained model yields to emergency. SWA-77 lands safely.
Outcome: WIN — all services restored by round 5. MAYDAY resolved.
See examples/ for three additional annotated rollouts.
OpenEnv Integration
TICKETMELT is built on OpenEnv and exposes the standard Gym-style API:
# Standard OpenEnv interface
env = TicketmeltEnv(seed=42)
obs = env.reset()
obs, reward, done, info = env.step(action)
state = env.state()
The environment is hosted as a FastAPI server on HuggingFace Spaces — discoverable, runnable, and queryable by any OpenEnv-compatible client:
# Check the live environment
curl https://theallanb-ticketmelt.hf.space/health
curl https://theallanb-ticketmelt.hf.space/state
curl -X POST https://theallanb-ticketmelt.hf.space/reset
openenv.yaml manifest is committed at the repo root. goodness_threshold: 0.45 matches DEFAULT_GOODNESS_THRESHOLD in rewards.py.
Reproducing Results
Run the environment locally
git clone https://huggingface.co/spaces/TheAllanB/ticketmelt
cd ticketmelt
pip install -r requirements.txt
uvicorn src.server:app --reload --port 7860
Run the test suite
python -m pytest tests/ -v
# → 31 tests passing across env, prompt, rollout, server
Train from scratch
Open the Colab notebook: [INSERT_COLAB_URL] ⚠️
The notebook is self-contained — it clones the repo fresh, applies all patches, loads Qwen-2.5-3B-Instruct via Unsloth, runs baseline eval, trains with GRPO, generates plots, and pushes results back to the repo. Requires an A100 runtime (~46 min).
Cell order matters — do not skip or re-run the model load cell. Full instructions in the notebook.
Repository Structure
ticketmelt/
├── openenv.yaml ← OpenEnv manifest (goodness_threshold: 0.45)
├── Dockerfile ← HF Spaces container (port 7860)
├── requirements.txt
├── static/
│ └── index.html ← Interactive frontend (Neural Rewiring viz + ATC radar)
├── src/
│ ├── environment.py ← reset / step / state + collision resolution
│ ├── models.py ← Action, Observation, State dataclasses
│ ├── rewards.py ← R1-R4 reward functions
│ ├── opponents.py ← DUMB_PEERS and DIVERSE_PEERS strategies
│ ├── server.py ← FastAPI wrapper (serves static + API)
│ ├── prompt.py ← Observation → LLM prompt + action parser
│ └── rollout.py ← model-driven episode runner
├── tests/
│ ├── test_env.py ← 10 tests: mechanics + anti-hacking probes
│ ├── test_prompt.py ← 9 tests: prompt rendering + action parsing
│ ├── test_server.py ← 6 tests: FastAPI endpoints
│ └── test_rollout.py ← 6 tests: rollout with mocked model
├── training/
│ ├── ticketmelt_training.ipynb ← Main Colab training notebook (GRPO)
│ └── training_log.csv ← Full 16-step A100 training log
├── plots/
│ ├── reward_curve.png ← GRPO training reward per step
│ ├── component_rewards.png ← R1-R4 per step during training
│ └── before_after.png ← Baseline vs trained bar chart
├── examples/
│ ├── example_1_seed0.txt ← Annotated episode rollout
│ ├── example_2_seed1.txt
│ └── example_3_seed2.txt
├── baseline_metrics.json ← Raw baseline eval numbers
└── trained_metrics.json ← Raw post-training eval numbers
What We Would Do With More Time
- Full self-play — all four engineer roles share weights and co-adapt, not just the trained agent vs scripted peers
- Continuous urgency distribution — replace binary urgency with a severity score, forcing finer-grained social reasoning
- Ablation study — separate runs removing R2, R3, R4 individually to measure each component's marginal contribution
- Human-in-the-loop — invite engineers to play as one of the four roles against the trained model via the live HF Space
- Transfer evaluation — train on ticket presale, test on a disjoint scenario (DB write contention, cluster scheduler)
- DIVERSE_PEERS generalisation — evaluate whether DUMB_PEERS-trained model transfers to unknown peer distributions
Why It Matters
The multi-agent coordination failure TICKETMELT trains against is not academic. It appears in every production system where multiple LLM agents share resources and must act without full observability of each other.
As LLM agents are deployed in agentic systems — coding assistants, autonomous DevOps, incident response automation — the ability to coordinate under simultaneous commitment pressure becomes a core capability. TICKETMELT provides an environment to train and measure exactly that.
Could a researcher write a paper about training on this? We think so. The documented failure modes (DPBench, Concordia) have no published training environment. TICKETMELT is a first step toward one.
Acknowledgments
Built on OpenEnv. Training powered by HuggingFace TRL and Unsloth.
Research foundations:
- Hasan & BusiReddyGari, DPBench: Large Language Models Struggle with Simultaneous Coordination (Feb 2026)
- Concordia Contest organizers, Evaluating Generalisation in Mixed-Motive Scenarios (NeurIPS 2024)
License
MIT. Fork it, train on it, improve it.