DESIGN.md

# Design Notes: Logistics Shipment RL Environment

## Why This Problem?

India's logistics sector processes over **$200B** of freight annually. During major disruptions (port strikes, monsoon floods, highway accidents), human dispatchers must manually triage hundreds of concurrent shipments in minutes. This environment simulates that exact crisis — testing whether an LLM can act as autonomous operational infrastructure.

## Key Architectural Decisions

### 1. Turn-Based with Deterministic Background Updates

The environment uses a **turn-based loop** rather than continuous steps. Within each turn, the agent can issue a sequence of JSON actions. 

Crucially, **background traffic (route congestion) only updates at the end of a turn**. This was a major design correction: originally, background traffic updated after *every* action, which caused non-deterministic behavior where the agent's world model (from `get_network_status`) would become invalid mid-turn. By decoupling the updates to the end-of-turn, the environment ensures stable planning for the agent.

### 2. Independent Reward Signals with Anti-Hacking Safeguards

Rather than a single scalar reward, the training pipeline utilizes **three independent reward functions**:
1. **Structure:** Evaluates valid JSON, correct start/end actions, and penalizes spam/rambling.
2. **Routing:** Evaluates whether delayed shipments were moved to clear routes.
3. **Communication:** Evaluates whether apologies and ETAs were sent to delayed customers.

**Anti-Hacking Penalties:** To prevent exploitation, strict negative rewards were enforced:
- `-0.5` for duplicate `end_turn` calls
- `-0.3` for spamming `get_network_status`
- `-0.6` for choosing congested alternate routes
- `-0.8` for omitting a valid route ID entirely
- `-0.5` for duplicate or spam ETA messages

### 3. Four-Phase Curriculum Learning (GRPO)

The agent was trained using **Group Relative Policy Optimization (GRPO)** via TRL and Unsloth. The training followed a strict 4-phase curriculum:
1. **Phase 1 (Easy):** Baseline routing on `TASK-EASY` (1 disruption).
2. **Phase 2 (Medium):** Triage prioritization on `TASK-MEDIUM` (3 simultaneous disruptions) with a lowered learning rate to prevent catastrophic forgetting.
3. **Phase 3 (Hardening):** Interleaved Easy/Medium scenarios with high rollout diversity (`num_generations=6`) to reinforce learned logic.
4. **Phase 4 (Correction):** Applied the strictest anti-hacking penalties (e.g., `-0.8` for empty routes) to patch loopholes discovered in earlier phases.

### 4. NLP-Scored Communication

The `communicate_eta` action is graded by a lightweight heuristic that checks for:
- Empathetic language ("apologize", "sorry", "regret")
- Specific ETA commitment ("arrive by 6pm", "reschedule to Monday")
- Cause of delay ("port congestion", "carrier strike", "weather")
- Message length (longer = more effort)

This tests a different capability than rerouting math: can the agent produce professional, customer-facing communication under pressure?

### 5. Three Difficulty Tiers for Benchmarking

| Tier | Challenge | Design Intent |
|------|-----------|---------------|
| EASY | 2 shipments, 1 disruption, 3 turns | Validates basic rerouting ability |
| MEDIUM | 4 shipments, 3 simultaneous disruptions, 5 turns | Tests triage prioritization under pressure |
| HARD | 7 shipments, 4 critical failures, 7 turns | Stress-tests maximum crisis management capacity |

An agent that scores well on EASY but poorly on HARD reveals brittleness under complexity — a real finding, not a grader artifact.

## Final Results Anatomy

The final architecture achieved a **+327% performance improvement** (0.18 baseline → 0.7683 trained average).

Key behaviors learned by the final model:
1. **World Modeling:** The agent learned to always call `get_network_status` as its first action to construct a mental model of route congestion.
2. **SLA Protection:** The agent learned to check route load before blindly rerouting, avoiding the `-0.6` penalty for congested routes.
3. **Empathy Under Pressure:** The agent learned to reliably dispatch well-formed apologies to the correct disrupted customers.

## Known Limitations

1. **Route graph is static**: Routes don't dynamically close or change congestion mid-episode. A future version could add stochastic disruption evolution.
2. **Single-agent only**: The environment is not designed for multi-agent scenarios where separate dispatchers handle different regions.
3. **No partial observability**: The agent always sees the full network state after `get_network_status`. A more challenging variant would restrict observations to a regional viewport.

## Future Extensions

- `TASK-KOLKATA` — Monsoon flood response at KOPT
- `TASK-VIZAG` — Cyclone-induced port closure on the eastern coast  
- `TASK-MUNDRA` — Container ship grounding at India's largest private port
- Dynamic route congestion that evolves each turn
- Multi-modal transport (sea + rail + road intermodal chains)