README.md

---
title: AI-Kinetic Disaster Recovery Grid
emoji: 🏙️
colorFrom: red
colorTo: yellow
sdk: docker
app_file: app.py
pinned: true
license: mit
short_description: Multi-Agent RL crisis-grid recovery on OpenEnv
tags:
  - reinforcement-learning
  - multi-agent
  - openenv
  - pytorch
  - llm-agents
  - hackathon
---

# 🏙️ AI-Kinetic Disaster Recovery Grid

> A Multi-Agent Reinforcement Learning environment built on **OpenEnv** for the **Meta PyTorch × Scaler OpenEnv Hackathon**.

---

## 🚨 Problem Statement

Modern Large Language Models excel at conversational tasks and static reasoning, but they fail catastrophically at **real-time resource management** and **long-horizon planning** during active crises.

When a disaster strikes — a solar flare, cyber-attack, or severe weather event — infrastructure grids suffer cascading failures. In these scenarios:

- **Information is sparse.** Communication lines sever, creating partial observability.
- **Resources are finite.** Repairing one sector may mean abandoning another.
- **Rewards are delayed.** Actions taken now (gathering tools) may not yield results until much later.

Traditional centralized AI models assume perfect information and unlimited bandwidth. Placed in a constrained, decaying environment, they collapse.

---

## 💡 The Proposed Solution

We built the **AI-Kinetic Disaster Recovery Grid** — a Multi-Agent Reinforcement Learning (MARL) environment. Instead of a single "Omni-Agent," we deployed a distributed team of specialized agents acting on a dynamic 5×5 city grid.

### 🤖 The Agent (Autonomous Emergency Manager)

The agent operates on a 5x5 grid (25 sectors) and must balance two critical metrics:
1. **City Health:** Sectors degrade over time due to simulated entropy. The agent must navigate to critical sectors (health < 30) and perform `REPAIR` actions.
2. **Energy:** Every action costs energy (`MOVE` costs 2, `REPAIR` costs 15). If energy hits 0, the agent dies. The agent must periodically return to the Base (Sector 12) to `RECHARGE`.

### 🧠 The Reward System (Orthogonal Verifiers)

To prevent reward hacking, we use three independent verifiers during GRPO training:
* **R1 (Health):** Did the average city health improve? (Objective)
* **R2 (Efficiency):** Was the health gained worth the energy spent? (Strategy)
* **R3 (Format):** Did the LLM output valid JSON matching the `AgentAction` schema? (Compliance)

---

## ⚙️ Workflow & Architecture

### 1. Environment Simulation (OpenEnv)
The world is modeled using `openenv-core`. The `CityGrid` class manages the physics: grid state, agent position, energy deduction, and random entropy generation. It provides a clean `reset()` and `step()` interface.

### 2. Training (GRPO + Unsloth)
We utilize **Group Relative Policy Optimization (GRPO)** via the `trl` library, accelerated with **Unsloth** for 4-bit quantization, allowing us to train Llama-3 models efficiently within constrained GPU memory.

### 3. Frontend Dashboard (Streamlit)
A production-ready Streamlit interface acts as the "Operational Theatre." It visualizes the grid state, agent telemetry, and the agent's "brain log" (reasoning output) in real-time.

---

## 🌍 Real-World Scalability

While currently a 5x5 simulation, this architecture scales to real-world disaster management:
* **Grid Expansion:** The mathematical model can map to larger geographic sectors using real GIS data.
* **Multi-Agent Orchestration:** The framework supports introducing specialized agents (e.g., drones for scouting, heavy machinery for repair) coordinating via shared observations.
* **Predictive Maintenance:** By training on historical degradation rates, the model shifts from reactive repair to proactive maintenance.

---

## 📈 Development Progress & Demos

Throughout the hackathon, we iterated from manual logic validation to full autonomous control. Below are recordings of our progress.

### Phase 1: Engine Validation (Manual CLI)
Before training the LLM, we built a terminal-based CLI to manually playtest the `CityGrid` physics. This ensured movement costs, boundary collisions, repair mechanics, and the three-part reward system functioned perfectly.


https://cdn-uploads.huggingface.co/production/uploads/69d245459b543e9d25bb47da/2H6tH14ZAIEhpz5YbCGYA.mp4

### Phase 2: Autonomous Agent Integration (UI Dashboard)
We then connected our GRPO-trained Llama-3 agent to the Streamlit UI. While the agent successfully parses the environment and issues valid JSON commands (satisfying the R3 verifier), we are currently tuning the R1/R2 reward weights. As seen in the demo, the agent sometimes exhibits suboptimal routing or gets trapped in energy-depletion loops, highlighting the challenge of balancing long-horizon planning with immediate crisis response.


https://cdn-uploads.huggingface.co/production/uploads/69d245459b543e9d25bb47da/bQzdaCNIPL4lPjNdAzmPs.mp4

---
*Developed by Sanjith P & Varsha D for the Meta PyTorch OpenEnv Hackathon.*