DESIGN.md

Design Notes: Logistics Shipment RL Environment

Why This Problem?

India's logistics sector processes over $200B of freight annually. During major disruptions (port strikes, monsoon floods, highway accidents), human dispatchers must manually triage hundreds of concurrent shipments in minutes. This environment simulates that exact crisis — testing whether an LLM can act as autonomous operational infrastructure.

Key Architectural Decisions

1. Turn-Based with Deterministic Background Updates

The environment uses a turn-based loop rather than continuous steps. Within each turn, the agent can issue a sequence of JSON actions.

Crucially, background traffic (route congestion) only updates at the end of a turn. This was a major design correction: originally, background traffic updated after every action, which caused non-deterministic behavior where the agent's world model (from get_network_status) would become invalid mid-turn. By decoupling the updates to the end-of-turn, the environment ensures stable planning for the agent.

2. Independent Reward Signals with Anti-Hacking Safeguards

Rather than a single scalar reward, the training pipeline utilizes three independent reward functions:

1. Structure: Evaluates valid JSON, correct start/end actions, and penalizes spam/rambling.
2. Routing: Evaluates whether delayed shipments were moved to clear routes.
3. Communication: Evaluates whether apologies and ETAs were sent to delayed customers.

Anti-Hacking Penalties: To prevent exploitation, strict negative rewards were enforced:

• -0.5 for duplicate end_turn calls
• -0.3 for spamming get_network_status
• -0.6 for choosing congested alternate routes
• -0.8 for omitting a valid route ID entirely
• -0.5 for duplicate or spam ETA messages

3. Four-Phase Curriculum Learning (GRPO)

The agent was trained using Group Relative Policy Optimization (GRPO) via TRL and Unsloth. The training followed a strict 4-phase curriculum:

1. Phase 1 (Easy): Baseline routing on TASK-EASY (1 disruption).
2. Phase 2 (Medium): Triage prioritization on TASK-MEDIUM (3 simultaneous disruptions) with a lowered learning rate to prevent catastrophic forgetting.
3. Phase 3 (Hardening): Interleaved Easy/Medium scenarios with high rollout diversity (num_generations=6) to reinforce learned logic.
4. Phase 4 (Correction): Applied the strictest anti-hacking penalties (e.g., -0.8 for empty routes) to patch loopholes discovered in earlier phases.

4. NLP-Scored Communication

The communicate_eta action is graded by a lightweight heuristic that checks for:

• Empathetic language ("apologize", "sorry", "regret")
• Specific ETA commitment ("arrive by 6pm", "reschedule to Monday")
• Cause of delay ("port congestion", "carrier strike", "weather")
• Message length (longer = more effort)

This tests a different capability than rerouting math: can the agent produce professional, customer-facing communication under pressure?

5. Three Difficulty Tiers for Benchmarking

Tier	Challenge	Design Intent
EASY	2 shipments, 1 disruption, 3 turns	Validates basic rerouting ability
MEDIUM	4 shipments, 3 simultaneous disruptions, 5 turns	Tests triage prioritization under pressure
HARD	7 shipments, 4 critical failures, 7 turns	Stress-tests maximum crisis management capacity

An agent that scores well on EASY but poorly on HARD reveals brittleness under complexity — a real finding, not a grader artifact.

Final Results Anatomy

The final architecture achieved a +327% performance improvement (0.18 baseline → 0.7683 trained average).

Key behaviors learned by the final model:

1. World Modeling: The agent learned to always call get_network_status as its first action to construct a mental model of route congestion.
2. SLA Protection: The agent learned to check route load before blindly rerouting, avoiding the -0.6 penalty for congested routes.
3. Empathy Under Pressure: The agent learned to reliably dispatch well-formed apologies to the correct disrupted customers.

Known Limitations

1. Route graph is static: Routes don't dynamically close or change congestion mid-episode. A future version could add stochastic disruption evolution.
2. Single-agent only: The environment is not designed for multi-agent scenarios where separate dispatchers handle different regions.
3. No partial observability: The agent always sees the full network state after get_network_status. A more challenging variant would restrict observations to a regional viewport.

Future Extensions

• TASK-KOLKATA — Monsoon flood response at KOPT
• TASK-VIZAG — Cyclone-induced port closure on the eastern coast
• TASK-MUNDRA — Container ship grounding at India's largest private port
• Dynamic route congestion that evolves each turn
• Multi-modal transport (sea + rail + road intermodal chains)