README.md

---
title: Logistics Shipment Env
emoji: 🚛
colorFrom: blue
colorTo: purple
sdk: docker
app_port: 7860
pinned: true
---

# 🚛 AI Logistics Coordinator — OpenEnv RL Environment

[![OpenEnv](https://img.shields.io/badge/OpenEnv-compatible-blue)](https://github.com/meta-pytorch/OpenEnv)
[![HF Space](https://img.shields.io/badge/🤗-Live%20Demo-yellow)](https://huggingface.co/spaces/Leavin1611/logistics-hackathon-env)
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1fRfheRZd1tffKjXKZkW72ffl9JPxDpL8)
[![Python](https://img.shields.io/badge/python-3.10%2B-green)](https://python.org)

> **A multi-turn, multi-disruption freight crisis simulator for training LLMs to reason, plan, and communicate under pressure.**

---

## 🌍 1. Problem: Logistics is a Reasoning Crisis, Not an Optimization Problem

Modern logistics systems fail when disruptions cascade — port strikes, accidents, carrier insolvencies — because they are built on static optimizers, not reasoning agents.

**The capability gap we close:** Training an LLM to act as a centralized logistics coordinator that can triage shipments, plan across multiple turns, reason about network congestion caused by *other agents*, and communicate empathetically with customers — all simultaneously.

---

## ✅ Hackathon Evaluation Criteria Checklist
For the judges, here is exactly how this project fulfills the core criteria:

- [x] **A clear environment design**: A highly-structured Pydantic environment representing a freight network with cascading disruptions, dynamic route capacities, and SLA tracking (`server/environment.py`).
- [x] **Objective reward functions**: Three independent reward signals (Structure, Routing, Communication) computed purely from verifiable environment state and JSON output, each with explicit anti-hacking penalties.
- [x] **Evidence that the model improved**: Reward jumped from **0.18 (untrained baseline) → 0.7683 (GRPO-trained)** — a **+327% improvement**. See reward curve below.
- [x] **Prevention against reward hacking**: Explicit negative penalties for spamming API calls (`-0.3`), sending duplicate messages (`-0.5`), routing to non-existent or congested routes (`-0.6`), and escalating to humans (`-0.1` per handoff).
- [x] **A reproducible deployment story**: Full FastAPI + OpenEnv backend deployed live to Hugging Face Spaces. The entire training loop runs in one click on a free Google Colab T4 GPU.
- [x] **A sharp demo**: Colab notebook shows the exact format: *Untrained Baseline ➔ 3-Phase GRPO Training ➔ Trained Model ➔ +327% reward improvement ➔ Anti-hacking safeguards explained*.

---

## 🎮 2. Environment: What the Agent Sees, Does, and Gets Rewarded For

### Live Demo & Model
🤗 **Environment Space:** [huggingface.co/spaces/Leavin1611/logistics-hackathon-env](https://huggingface.co/spaces/Leavin1611/logistics-hackathon-env)  
🧠 **Trained Model Adapter:** [huggingface.co/Leavin1611/logistics-hackathon-model](https://huggingface.co/Leavin1611/logistics-hackathon-model)

### Themes Covered
| Hackathon Theme | How We Address It |
|---|---|
| **Theme #1 — Multi-Agent** | Routes have capacity limits; background agent traffic updates every turn, forcing strategic routing |
| **Theme #2 — Long-Horizon Planning** | 5-7 turn episodes with cascading disruptions; early decisions determine final SLA outcome |
| **Theme #3 — World Modeling** | Partially observable; agent must call tools to query the live network state |

### Observation Space
Each step returns a `LogisticsObservation` with:
- `shipments` — cargo, route, carrier, SLA buffer, delay hours, status
- `disruptions` — active events (port strikes, accidents, carrier failures)
- `route_load` — real-time congestion per route (0.0 → 1.0), updated by simulated multi-agent traffic
- `feedback` — result of last action
- `incremental_reward` — immediate reward signal
- `cumulative_reward` — running episode total

### Action Space
| Action | Description |
|--------|-------------|
| `get_network_status` | Query live network state |
| `reroute_shipment` | Move shipment to alternate route (blocked if route > 85% capacity) |
| `set_priority` | Fast-track up to 3 shipments |
| `communicate_eta` | Send NLP-graded ETA message to customer |
| `escalate` | Hand off to human (penalized — agent should solve it) |
| `end_turn` | Commit decisions and receive turn reward |

### Reward Function — Four Independent Signals (Anti-Hack Design)
```
Turn Reward = 0.40 × DelayScore + 0.30 × SLAScore + 0.20 × CommScore + 0.10 × EscScore
```

| Signal | Weight | Guard Against Exploitation |
|--------|--------|---------------------------|
| Delay Reduction | 40% | Bounded by realistic hours-saved map |
| SLA Compliance | 30% | Based on live shipment state only |
| Communication Quality | 20% | **Duplicate message penalty: -0.5** |
| Escalation Control | 10% | -0.1 per human handoff |

The **-0.5 duplicate penalty** directly prevents reward hacking: an agent spamming messages to inflate its communication score is penalized immediately.

### Task Curriculum
| Task | Name | Shipments | Turns | Challenge |
|------|------|-----------|-------|-----------|
| `TASK-EASY` | Port Backlog Clearance | 2 | 3 | Single JNPT disruption |
| `TASK-MEDIUM` | Mumbai Crisis Coordination | 4 | 5 | Port + accident + carrier strike |
| `TASK-HARD` | Multi-Port Network Collapse | 7 | 7 | 3 simultaneous failures + insolvency |

---

## 📈 3. Results: Observable Evidence of Training Progress

We trained `Qwen/Qwen2.5-1.5B-Instruct` using GRPO against the live environment server:

| Metric | Base Model (Untrained) | GRPO-Trained (Phase 3) |
|--------|----------------------|------------------------|
| **Cumulative Reward** | **0.18** | **0.7683 (+327%)** |
| Valid JSON Actions | ~60% | ~98% |
| Strategic Reroutes per Episode | 1 | 3+ |
| Communication Quality | Bare / no apology | Empathetic + specific ETA |
| Escalation Rate | High | Near zero (self-solving) |

### Training Reward Curve

![Training Reward Curve](./assets/training_reward_curve.png)
*Reward progression across all 3 GRPO curriculum phases. Blue line = rolling average reward. Red dashed = untrained baseline (0.18). Green dashed = final trained average (0.7683). Each shaded region is one training phase.*

### Detailed Evaluation & Ablation Studies

Below are detailed episode-by-episode breakdowns proving the model's performance improvements across different curriculum stages and safeguards.

#### Curriculum Learning Progression
![Curriculum Progression](./assets/eval_curriculum.png)
*Episode-by-episode breakdown showing how curriculum learning (+34.2%) outperformed the untrained baseline consistently.*

#### Reward Hacking Safeguards (Old vs New)
![Reward Hacking Patch](./assets/eval_hacking.png)
*Comparison showing how the final model maintained high performance while successfully avoiding the reward-hacking penalties applied to the older model.*

#### Final Logic Improvement
![Logic Improvement](./assets/eval_logic.png)
*Final evaluation showing a massive +57.2% improvement in routing logic and SLA compliance.*

#### Overall Efficiency Increase
![Efficiency Increase](./assets/eval_efficiency.png)
*Overall efficiency metrics comparing the pre-training baseline against the fully hardened GRPO model.*

### What the Trained Agent Learned:
1. **Always call `get_network_status` first** — it modeled the world before acting
2. **Avoid overloaded routes** — learned to check route load before rerouting
3. **Write empathetic messages** — discovered that apology + reason + ETA = maximum comm reward
4. **Never escalate** — learned that self-solving is always rewarded over hand-offs

---

## 🧠 4. Training Pipeline

### Why GRPO?
GRPO (Group Relative Policy Optimization) — the algorithm behind DeepSeek-R1 — compares a group of rollouts against each other and rewards relatively better ones. No separate critic model needed. Ideal for verifiable, environment-driven rewards.

### Stack
```
OpenEnv (live environment) → TRL + GRPO → Unsloth (T4 efficiency)
```

### Run Training in One Click
[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1fRfheRZd1tffKjXKZkW72ffl9JPxDpL8)

1. Open the notebook in Google Colab (button above)
2. Set Runtime → **T4 GPU**
3. Click **Run All**
4. The notebook auto-generates reward plots and a full before/after comparison with baseline

---

## 🚀 5. Quick Start (Local)

```bash
# Clone the environment
git clone https://huggingface.co/spaces/Leavin1611/logistics-hackathon-env
cd logistics-hackathon-env

# Windows: double-click setup.bat, then start.bat
# OR manually:
python -m venv .venv
.venv\Scripts\pip install fastapi "uvicorn[standard]" pydantic openenv-core
.venv\Scripts\uvicorn server.app:app --host 0.0.0.0 --port 7860

# Open http://localhost:7860 in your browser
```

### Run Inference (with any OpenAI-compatible API)
```bash
pip install openai python-dotenv
export OPENAI_API_KEY="your-groq-key"     # Free at console.groq.com
export API_BASE_URL="https://api.groq.com/openai/v1"
export MODEL_NAME="llama-3.1-8b-instant"
python inference.py
```

---

## 🗺️ Shipment State Machine

```mermaid
stateDiagram-v2
    [*] --> IN_TRANSIT : Episode starts
    IN_TRANSIT --> DELAYED : SLA buffer expires
    DELAYED --> IN_TRANSIT : reroute_shipment (saves hours)
    DELAYED --> CRITICAL : SLA buffer < -4h
    CRITICAL --> DELAYED : reroute + set_priority
    IN_TRANSIT --> RESOLVED : delay_h = 0
    RESOLVED --> [*] : Episode ends
```

---

## 📁 Project Structure

```
logistics-hackathon-env/
├── server/
│   ├── app.py              # FastAPI server (OpenEnv-compatible)
│   └── environment.py      # Core RL engine — reset/step/state/rewards
├── dashboard.html          # Live interactive UI (served at root)
├── inference.py            # Baseline agent runner
├── train_colab.ipynb       # GRPO training notebook (one-click Colab)
├── HF_BLOG_POST.md         # Full writeup / mini-blog
├── openenv.yaml            # Environment manifest
├── setup.bat               # Windows local setup script
├── start.bat               # Windows local server launcher
└── examples/
    └── train_grpo.py       # Training script
```

---

## 🔗 Links & Materials

| Resource | Link |
|----------|------|
| 🤗 Live HF Space | [huggingface.co/spaces/Leavin1611/logistics-hackathon-env](https://huggingface.co/spaces/Leavin1611/logistics-hackathon-env) |
| 🧠 Trained Model | [Leavin1611/logistics-hackathon-model](https://huggingface.co/Leavin1611/logistics-hackathon-model) |
| 📓 Training Notebook | [train_colab.ipynb (Colab)](https://colab.research.google.com/drive/1fRfheRZd1tffKjXKZkW72ffl9JPxDpL8) |
| 🖥️ Slide Deck | [Live Preview → /slides](https://leavin1611-logistics-hackathon-env.hf.space/slides) — 9-slide presentation (arrow keys) |
| 📝 Mini-Blog | [HF_BLOG_POST.md](./HF_BLOG_POST.md) |
| 📦 GitHub Repo | [github.com/leavin1611/Logistics-hackathon-env](https://github.com/leavin1611/Logistics-hackathon-env) |
| 🔧 OpenEnv Framework | [github.com/meta-pytorch/OpenEnv](https://github.com/meta-pytorch/OpenEnv) |
| 🔑 Free Groq API Key | [console.groq.com/keys](https://console.groq.com/keys) |

---

## 📚 Further Reading

- [HF_BLOG_POST.md](./HF_BLOG_POST.md) — Full narrative writeup for judges
- [Slide Deck](https://leavin1611-logistics-hackathon-env.hf.space/slides) — 9-slide live presentation (arrow key navigation)
- [DESIGN.md](./DESIGN.md) — Architecture decisions and reward anatomy
- [CONTRIBUTING.md](./CONTRIBUTING.md) — How to add scenarios, routes, actions

---

*Built for the Meta PyTorch OpenEnv Hackathon 2026 — India Round 2*
*Stack: OpenEnv · FastAPI · TRL · GRPO · Unsloth · Qwen2.5*