HF_BLOG_POST.md

# 🚛 Training the AI Logistics Coordinator
## How We Used GRPO to Build an Agent That Manages Real-World Freight Crises

*A Meta PyTorch OpenEnv Hackathon 2026 Submission*

---

It is 3 AM. A port strike has just shut down JNPT in Mumbai. A refrigerated truck carrying COVID vaccines is stranded on Route R1. A cargo of election ballots is stuck at a Delhi depot. And a ₹1.8 crore shipment of server hardware is seven hours behind schedule.

A human logistics manager would need 30 minutes just to triage this situation. Our AI agent does it in seconds.

This is the story of how we built — and trained — that agent.

---

## The Problem: Logistics is a Coordination Crisis, Not Just an Optimization Problem

Modern logistics systems like those at Amazon or FedEx are built on rule-based optimizers: mathematical solvers that find the shortest path from A to B. They are excellent at routine scheduling.

But they fail catastrophically at *crises*.

When a port closes, a carrier goes bankrupt, or a highway floods, these systems have no capacity to reason. They cannot ask: *"Which shipment should I prioritize?"* or *"How do I tell a hospital their surgical equipment will be delayed?"*

This is the capability gap we set out to close: **not a faster algorithm, but a reasoning agent**.

---

## The Environment: A Multi-Disruption Freight Network

We built the `logistics_shipment_env` — a real-time, multi-turn crisis simulator — using the OpenEnv framework. At its core, the environment puts the agent in the role of a centralized logistics coordinator managing a partially observable Indian freight network.

### What the agent sees:
- A live map of shipment states: cargo type, origin, destination, carrier, current delay, and SLA buffer (time until breach)
- Active disruptions: port strikes, highway accidents, carrier insolvencies, weather events
- Network congestion levels per route (updated every turn by simulated background traffic from other agents)

### What the agent can do:
| Action | Effect |
|--------|--------|
| `get_network_status` | Query the live network for route loads and shipment states |
| `reroute_shipment` | Move a shipment to a less-congested alternate route |
| `set_priority` | Fast-track up to 3 high-value shipments |
| `communicate_eta` | Send a graded, NLP-scored ETA update to the customer |
| `escalate` | Hand off to a human (penalized — the agent should solve it) |
| `end_turn` | Commit all decisions and receive the turn reward |

### What makes this environment genuinely hard:
**1. Multi-Agent Resource Scarcity (Theme #1):** Routes have limited capacity. Every turn, background traffic from other simulated agents updates route loads. An agent that blindly reroutes to Route R2 without checking the load will fail — R2 might already be at 90% capacity. This forces the agent to model the behavior of the network, not just react to individual shipments.

**2. Long-Horizon Planning (Theme #2):** The agent has 5-7 turns. Decisions made on Turn 1 — like which shipments to prioritize — directly impact the final SLA score. An agent that wastes turns on low-value shipments will fail to save the critical pharmaceuticals before the episode ends.

**3. World Modeling (Theme #3):** The environment is *partially observable*. The agent must call `get_network_status` to see the current state. It cannot assume anything about the network. This tests whether the LLM can maintain a consistent internal model of a dynamic world and update its beliefs based on tool outputs.

### Reward Function (Four Independent Signals):
We deliberately designed the reward function to be **anti-gameable**. A single reward signal invites exploitation. Four independent signals do not.

```
Turn Reward = 0.40 × DelayScore + 0.30 × SLAScore + 0.20 × CommScore + 0.10 × EscScore
```

| Signal | Measures | Anti-Hack Guard |
|--------|----------|-----------------|
| Delay Reduction | Hours saved vs. baseline | Bounded by realistic savings map |
| SLA Compliance | % shipments meeting deadline | Based on real shipment state |
| Communication Quality | NLP scoring of ETA messages | Penalizes duplicate messages (-0.5) |
| Escalation Control | Penalty per human handoff | Penalizes lazy escalation |

The **communication anti-hack** is critical: an agent that spams the same shipment with messages to inflate its communication score receives an immediate -0.5 penalty and gets caught in the training logs.

---

## The Training: GRPO on a Live Environment

We trained using **GRPO (Group Relative Policy Optimization)** — the same algorithm behind DeepSeek-R1 — using Hugging Face TRL and Unsloth for efficiency on free Colab T4 GPUs.

### Why GRPO?
Unlike PPO, GRPO does not require a separate critic/value model. It compares a group of rollouts against each other and rewards the ones that performed *relatively* better. This is ideal for our environment because:
- It reduces memory footprint (critical on T4 GPUs)
- It aligns naturally with verifiable reward signals
- It produces richer training signal per batch

### The Training Loop:
1. The model receives the logistics scenario as a system prompt
2. It generates an action in JSON format (e.g., `{"action_type": "reroute_shipment", "shipment_id": "SHIP-001", "new_route": "R4"}`)
3. The action is sent to the live environment server
4. The environment executes the action and returns a real reward
5. GRPO uses the reward to update the model weights
6. Repeat across thousands of episodes with increasing task difficulty (curriculum)

### Training Setup:
- **Model**: `Qwen/Qwen2.5-1.5B-Instruct` (fits on T4 free tier)
- **Framework**: TRL + Unsloth + OpenEnv
- **Reward Functions**: 3 independent signals (JSON structure validity, strategic route selection, communication empathy) — each with explicit anti-hacking penalties
- **Curriculum**: TASK-EASY → TASK-MEDIUM → Mixed Hardening (Phase 3)
- **Training Notebook**: Available in `train_colab.ipynb` — runnable in one click on Google Colab

---

## Results: Before vs. After Training

![Training Reward Curve](./assets/training_reward_curve.png)
*Reward progression across all 3 GRPO curriculum phases. Blue line = rolling average reward. Red dashed = untrained baseline (0.18). Green dashed = final trained average (0.7683). Each shaded region is one training phase.*

We evaluated the base (untrained) model against the GRPO-trained model on TASK-EASY:

| Metric | Base Model | Trained Model (Phase 3) |
|--------|-----------|-------------------------|
| Cumulative Reward | 0.18 | **0.7683 (+327%)** |
| Valid JSON Actions | ~60% | ~98% |
| Strategic Reroutes | 1 per episode | 3+ per episode |
| Message Quality | Poor (no apology) | Excellent (empathy + ETA) |

### Visual Proof of Performance

#### 1. The Curriculum Learning Progression
![Curriculum Progression](./assets/eval_curriculum.png)
*Episode-by-episode breakdown showing how curriculum learning outperformed the untrained baseline consistently.*

#### 2. Reward Hacking Safeguards
![Reward Hacking Patch](./assets/eval_hacking.png)
*Comparison showing how the final model maintained high performance while successfully avoiding the reward-hacking penalties.*

#### 3. Routing Logic Improvement
![Logic Improvement](./assets/eval_logic.png)
*Final evaluation showing a massive +57.2% improvement in routing logic and SLA compliance.*

#### 4. Overall Efficiency
![Efficiency Increase](./assets/eval_efficiency.png)
*Overall efficiency metrics comparing the pre-training baseline against the fully hardened GRPO model.*

The trained agent learned to:
1. **Always call `get_network_status` first** before making decisions (world modeling)
2. **Avoid congested routes** by checking the route load before rerouting
3. **Send empathetic customer messages** with specific ETAs and cause explanations
4. **Escalate less** — learning to solve problems autonomously rather than handing off

---

## Why This Matters

The global logistics industry moves $8 trillion worth of goods annually. When disruptions occur — and they always do — the decisions made in the first hour determine whether vaccines stay cold, whether election materials arrive on time, whether factories have the parts they need.

We believe the future is not a smarter algorithm. It is a reasoning agent that can understand context, communicate with empathy, and make strategic decisions under uncertainty.

This environment is the training ground where that agent learns to exist.

---

## Quick Start

🤗 **Try the environment live**: [huggingface.co/spaces/Leavin1611/logistics-hackathon-env](https://huggingface.co/spaces/Leavin1611/logistics-hackathon-env)

🧠 **Download the trained model**: [huggingface.co/Leavin1611/logistics-hackathon-model](https://huggingface.co/Leavin1611/logistics-hackathon-model)

🖥️ **View the slide deck**: [leavin1611-logistics-hackathon-env.hf.space/slides](https://leavin1611-logistics-hackathon-env.hf.space/slides) ← arrow keys to navigate

🏋️ **Train it yourself**: Open `train_colab.ipynb` in Google Colab → Runtime → Run All

📦 **Clone the repo**: `git clone https://huggingface.co/spaces/Leavin1611/logistics-hackathon-env`

---

*Built for the Meta PyTorch OpenEnv Hackathon 2026 — India Round 2*
*Stack: OpenEnv · FastAPI · TRL · Unsloth · GRPO · Qwen2.5*