Commit Β·
3d5cf7f
1
Parent(s): a263356
Fix restriction auto-lift, density-weighted penalty, /info endpoint accuracy, updated README
Browse files- README.md +249 -123
- inference.py +2 -2
- server/environment.py +4 -1
- server/utils.py +1 -1
README.md
CHANGED
|
@@ -8,208 +8,334 @@ app_port: 7860
|
|
| 8 |
pinned: false
|
| 9 |
---
|
| 10 |
|
| 11 |
-
## π¦ Cascade
|
| 12 |
|
| 13 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
Built for the **Meta PyTorch OpenEnv Hackathon x SST 2026**.
|
| 18 |
|
| 19 |
---
|
| 20 |
|
| 21 |
## The Problem
|
| 22 |
|
| 23 |
-
Sequential resource allocation under uncertainty is one of the most
|
| 24 |
|
| 25 |
-
- Resources are scarce
|
| 26 |
-
- Data is delayed β by the time a crisis is visible, it has already grown
|
| 27 |
-
- Interventions
|
| 28 |
-
- Acting too late is catastrophic;
|
| 29 |
|
| 30 |
-
No existing OpenEnv benchmark
|
| 31 |
|
| 32 |
---
|
| 33 |
|
| 34 |
-
## Environment
|
| 35 |
|
| 36 |
-
|
| 37 |
|
| 38 |
-
|
| 39 |
|
| 40 |
-
|
| 41 |
-
|--------------|--------|-------------------------------------|
|
| 42 |
-
| `action_type`| string | `"test"` Β· `"restrict"` Β· `"allocate"` |
|
| 43 |
-
| `district_id`| int | 0-indexed district target |
|
| 44 |
|
| 45 |
-
|
| 46 |
-
- **restrict** β impose movement restriction (free, but penalised if infection is low)
|
| 47 |
-
- **allocate** β deploy 1 resource unit to reduce spread rate this step
|
| 48 |
|
| 49 |
-
|
|
|
|
|
|
|
| 50 |
|
| 51 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
```
|
| 63 |
|
| 64 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
|
| 66 |
-
|
| 67 |
-
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
|
| 71 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
|
| 73 |
-
|
| 74 |
|
| 75 |
---
|
| 76 |
|
| 77 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
|
| 79 |
-
|
| 80 |
-
|----------|-----------|-------|-----------|----------|------------------------------------------|
|
| 81 |
-
| `easy` | 2 | 10 | 10 | None | Single outbreak, clear signal |
|
| 82 |
-
| `medium` | 4 | 15 | 8 | None | Two simultaneous outbreaks, forced triage|
|
| 83 |
-
| `hard` | 6 | 15 | 7 | 3 days | Scarce resources, invisible acceleration |
|
| 84 |
|
| 85 |
---
|
| 86 |
|
| 87 |
-
##
|
| 88 |
|
| 89 |
-
|
| 90 |
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
| Hospital breach | `-1.00` per collapsed hospital | Catastrophic failure deterrent |
|
| 95 |
-
| Early containment | `+0.50 Γ (1 - step/max_steps)` | Teaches proactive behaviour |
|
| 96 |
-
| Unnecessary restriction | `-0.20` | Prevents lazy blanket lockdowns |
|
| 97 |
-
| Correct prioritisation | `+0.30` | Rewards triage intelligence |
|
| 98 |
|
| 99 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 100 |
|
| 101 |
---
|
| 102 |
|
| 103 |
-
##
|
| 104 |
|
| 105 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
|
|
|
| 113 |
|
| 114 |
---
|
| 115 |
|
| 116 |
-
##
|
|
|
|
|
|
|
| 117 |
|
| 118 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 119 |
|
| 120 |
-
|
| 121 |
-
2. After each rollout, compute advantage = `R_i - mean(completed rollouts)`
|
| 122 |
-
3. If above average β store positive-reward steps into `EpisodicMemory`
|
| 123 |
-
4. If below average β suppress, memory unchanged
|
| 124 |
-
5. Next rollout retrieves the 3 most similar past situations by L1 distance on infection profiles and injects them as concrete examples into the prompt
|
| 125 |
|
| 126 |
-
The
|
|
|
|
|
|
|
|
|
|
| 127 |
|
| 128 |
---
|
| 129 |
|
| 130 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
| 131 |
|
| 132 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
| Medium | ~0.23 | Yes |
|
| 138 |
-
| Hard | ~0.21 | Yes |
|
| 139 |
|
| 140 |
---
|
| 141 |
|
| 142 |
-
##
|
| 143 |
|
| 144 |
-
|
| 145 |
-
from client import CascadeContainmentEnv
|
| 146 |
-
from models import ContainmentAction
|
| 147 |
|
| 148 |
-
|
| 149 |
-
base_url="https://therubberduckdebuggers-cascade-containment.hf.space"
|
| 150 |
-
).sync() as env:
|
| 151 |
-
obs = env.reset(task_name="easy")
|
| 152 |
|
| 153 |
-
|
| 154 |
-
|
| 155 |
-
|
| 156 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 157 |
```
|
| 158 |
|
| 159 |
-
###
|
| 160 |
|
| 161 |
-
``
|
| 162 |
-
export API_BASE_URL="https://router.huggingface.co/v1"
|
| 163 |
-
export MODEL_NAME="meta-llama/Llama-3.1-8B-Instruct"
|
| 164 |
-
export HF_TOKEN="your_hf_token"
|
| 165 |
-
export ENV_BASE_URL="https://therubberduckdebuggers-cascade-containment.hf.space"
|
| 166 |
|
| 167 |
-
|
| 168 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 169 |
|
| 170 |
---
|
| 171 |
|
| 172 |
## Generalisation
|
| 173 |
|
| 174 |
-
This environment is not epidemic-specific. The
|
| 175 |
|
| 176 |
-
|
| 177 |
-
-
|
| 178 |
-
|
| 179 |
-
|
|
|
|
|
|
|
|
|
|
| 180 |
|
| 181 |
-
The
|
| 182 |
|
| 183 |
---
|
| 184 |
|
| 185 |
## Project Structure
|
| 186 |
|
| 187 |
```text
|
| 188 |
-
|
| 189 |
-
βββ
|
| 190 |
-
βββ
|
| 191 |
-
βββ client.py
|
| 192 |
-
βββ openenv.yaml
|
| 193 |
-
|
| 194 |
βββ server/
|
| 195 |
-
β βββ
|
| 196 |
-
β βββ
|
| 197 |
-
β βββ
|
| 198 |
-
β βββ
|
| 199 |
-
β βββ
|
| 200 |
-
β
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 201 |
βββ baseline/
|
| 202 |
-
β βββ policy.py
|
| 203 |
-
β βββ evaluator.py
|
| 204 |
-
β βββ run.py
|
|
|
|
| 205 |
βββ core/
|
| 206 |
-
βββ trajectory.py
|
| 207 |
-
βββ reward.py
|
| 208 |
-
βββ policy_update.py
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 209 |
```
|
| 210 |
|
| 211 |
---
|
| 212 |
|
| 213 |
## Tags
|
| 214 |
|
| 215 |
-
`reinforcement-learning` Β· `resource-allocation` Β· `sequential-decision-making` Β· `partial-observability` Β· `cascade-dynamics` Β· `openenv` Β· `llm-agent`
|
|
|
|
| 8 |
pinned: false
|
| 9 |
---
|
| 10 |
|
| 11 |
+
## π¦ An RL Benchmark for Sequential Resource Allocation Under Spreading Cascade Dynamics
|
| 12 |
|
| 13 |
+
[](https://github.com/meta-pytorch/OpenEnv)
|
| 14 |
+
[](https://python.org)
|
| 15 |
+
[](https://hub.docker.com)
|
| 16 |
+
[](https://huggingface.co/spaces/TheRubberDuckDebuggers/cascade-containment)
|
| 17 |
+
[](LICENSE)
|
| 18 |
|
| 19 |
+
Meta PyTorch OpenEnv Hackathon Γ SST 2026 β [Live Demo](https://therubberduckdebuggers-cascade-containment.hf.space) Β· [GitHub](https://github.com/Rohitchandramouli/cascade-containment)
|
|
|
|
|
|
|
| 20 |
|
| 21 |
---
|
| 22 |
|
| 23 |
## The Problem
|
| 24 |
|
| 25 |
+
Sequential resource allocation under uncertainty is one of the most consequential decision problems in the real world. Whether containing an epidemic, deploying firefighting crews, isolating a cyberattack, or routing aid β the agent faces the same fundamental challenge:
|
| 26 |
|
| 27 |
+
- **Resources are scarce** β you cannot cover every district simultaneously
|
| 28 |
+
- **Data is delayed** β by the time a crisis is visible, it has already grown
|
| 29 |
+
- **Interventions cascade** β actions in one district affect adjacent ones
|
| 30 |
+
- **Acting too late is catastrophic** β hospital collapse ends the episode; proactive containment is rewarded exponentially more than reactive response
|
| 31 |
|
| 32 |
+
No existing OpenEnv benchmark formalises this problem class. Cascade Containment does.
|
| 33 |
|
| 34 |
---
|
| 35 |
|
| 36 |
+
## Environment Overview
|
| 37 |
|
| 38 |
+
A city health authority must allocate limited medical resources across districts to contain a spreading outbreak. Each step, the agent observes district infection rates (possibly lagged), hospital capacity levels, and growth signals β then decides where to deploy resources, impose restrictions, or gather data.
|
| 39 |
|
| 40 |
+
The environment is **not epidemic-specific**. The underlying mechanics β spreading cascade, delayed observation, resource scarcity, geographic spillover β are structurally identical across multiple real-world domains.
|
| 41 |
|
| 42 |
+
---
|
|
|
|
|
|
|
|
|
|
| 43 |
|
| 44 |
+
## Quick Start
|
|
|
|
|
|
|
| 45 |
|
| 46 |
+
```python
|
| 47 |
+
from client import CascadeContainmentEnv
|
| 48 |
+
from models import ContainmentAction
|
| 49 |
|
| 50 |
+
with CascadeContainmentEnv(
|
| 51 |
+
base_url="https://therubberduckdebuggers-cascade-containment.hf.space"
|
| 52 |
+
).sync() as env:
|
| 53 |
+
result = env.reset(task_name="medium")
|
| 54 |
+
obs = result.observation
|
| 55 |
+
|
| 56 |
+
while not result.done:
|
| 57 |
+
most_infected = max(obs.districts, key=lambda d: d.reported_infection_rate)
|
| 58 |
+
action = ContainmentAction(
|
| 59 |
+
action_type="allocate",
|
| 60 |
+
district_id=most_infected.district_id
|
| 61 |
+
)
|
| 62 |
+
result = env.step(action)
|
| 63 |
+
obs = result.observation
|
| 64 |
+
print(f"Step {obs.current_step}: reward={result.reward:.3f}")
|
| 65 |
+
```
|
| 66 |
|
| 67 |
+
## Running the Full Baseline Evaluation
|
| 68 |
+
|
| 69 |
+
```bash
|
| 70 |
+
export API_BASE_URL="https://router.huggingface.co/v1"
|
| 71 |
+
export MODEL_NAME="meta-llama/Llama-3.1-8B-Instruct"
|
| 72 |
+
export HF_TOKEN="hf_your_token_here"
|
| 73 |
+
export ENV_BASE_URL="https://therubberduckdebuggers-cascade-containment.hf.space"
|
| 74 |
+
|
| 75 |
+
python inference.py
|
| 76 |
```
|
| 77 |
|
| 78 |
+
---
|
| 79 |
+
|
| 80 |
+
## Action Space
|
| 81 |
+
|
| 82 |
+
One decision per step, deliberately minimal to maximise strategic depth:
|
| 83 |
|
| 84 |
+
| Field | Type | Values |
|
| 85 |
+
| --- | --- | --- |
|
| 86 |
+
| `action_type` | `string` | `"test"` Β· `"restrict"` Β· `"allocate"` |
|
| 87 |
+
| `district_id` | `int` | 0-indexed district target |
|
| 88 |
|
| 89 |
+
| Action | Cost | Effect |
|
| 90 |
+
| --- | --- | --- |
|
| 91 |
+
| **test** | 1 resource | Reveals accurate current infection data for district |
|
| 92 |
+
| **restrict** | Free | Imposes movement restrictions; reduces spread rate; penalised if infection < 0.20 |
|
| 93 |
+
| **allocate** | 1 resource | Deploys medical resources; reduces existing infection by 5% and slows future spread |
|
| 94 |
|
| 95 |
+
Movement restrictions lift automatically once a district's infection drops below the safe threshold β reflecting real policy: restrictions are lifted when the outbreak is controlled.
|
| 96 |
|
| 97 |
---
|
| 98 |
|
| 99 |
+
## Observation Space
|
| 100 |
+
|
| 101 |
+
The agent receives a filtered, potentially lagged view of the world β **never the full ground truth**:
|
| 102 |
+
|
| 103 |
+
```python
|
| 104 |
+
CityObservation:
|
| 105 |
+
districts: List[DistrictObservation] # per-district visible state
|
| 106 |
+
available_resources: int # budget remaining this step
|
| 107 |
+
current_step: int
|
| 108 |
+
max_steps: int
|
| 109 |
+
done: bool
|
| 110 |
+
reward: float | None
|
| 111 |
+
message: str | None
|
| 112 |
+
```
|
| 113 |
+
|
| 114 |
+
Each `DistrictObservation` exposes:
|
| 115 |
+
|
| 116 |
+
| Field | Description | Observability |
|
| 117 |
+
| --- | --- | --- |
|
| 118 |
+
| `reported_infection_rate` | Active infection fraction | Real-time (easy/medium); **3-day lagged** (hard) |
|
| 119 |
+
| `growth_rate_hint` | Noisy signal of true spread rate | Always real-time Β± noise |
|
| 120 |
+
| `hospital_capacity_remaining` | ICU/ward capacity fraction | Always real-time |
|
| 121 |
+
| `population_density` | District's share of city population | Always real-time |
|
| 122 |
+
| `restriction_active` | Whether movement restrictions are in place | Always real-time |
|
| 123 |
+
| `tested_recently` | Tested within last 2 days | Always real-time |
|
| 124 |
|
| 125 |
+
**The agent never sees:** `true_infection_rate`, `true_spread_rate`, or any ground truth used by the grader.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 126 |
|
| 127 |
---
|
| 128 |
|
| 129 |
+
## Epidemiological Model
|
| 130 |
|
| 131 |
+
The simulation uses a realistic discrete-time SIR-inspired model:
|
| 132 |
|
| 133 |
+
```text
|
| 134 |
+
new_infection = current + (spread_rate β natural_recovery β intervention) + geographic_spillover
|
| 135 |
+
```
|
|
|
|
|
|
|
|
|
|
|
|
|
| 136 |
|
| 137 |
+
| Parameter | Value | Rationale |
|
| 138 |
+
| --- | --- | --- |
|
| 139 |
+
| Spread rate | 3β8% per day | Realistic for respiratory outbreaks (seasonal flu: 5β10%) |
|
| 140 |
+
| Natural recovery | 1% per day | Background case resolution without medical intervention |
|
| 141 |
+
| Treatment effect | β5% existing infection | Medical deployment (antivirals, PPE, rapid response) |
|
| 142 |
+
| Spread reduction | β10% per allocation | Resource-driven suppression of transmission |
|
| 143 |
+
| Geographic spillover | 1% to adjacent districts | Linear topology β no wrap-around (geographically realistic) |
|
| 144 |
+
| Hospital breach threshold | β€10% capacity | Real ICU overflow and triage failure threshold |
|
| 145 |
|
| 146 |
---
|
| 147 |
|
| 148 |
+
## Three Tasks
|
| 149 |
|
| 150 |
+
| Task | Districts | Steps | Resources | Data Lag | Challenge |
|
| 151 |
+
| --- | --- | --- | --- | --- | --- |
|
| 152 |
+
| **easy** | 2 | 10 | 10 | None | Single outbreak, clear signal, abundant resources |
|
| 153 |
+
| **medium** | 4 | 15 | 8 | None | Two simultaneous outbreaks; forced triage between competing threats |
|
| 154 |
+
| **hard** | 6 | 15 | 7 | **3 days** | Six growing outbreaks; invisible acceleration; scarce resources |
|
| 155 |
|
| 156 |
+
### Task Design Philosophy
|
| 157 |
+
|
| 158 |
+
**Easy** establishes the core mechanic: D0 starts at 0.50 infection, D1 is clean. The agent must sustain focused allocation on D0 before D1 grows through spillover. A speed bonus fires if the agent acts decisively early.
|
| 159 |
+
|
| 160 |
+
**Medium** introduces genuine triage. With 8 resources across 4 districts over 15 steps, the agent simply cannot contain all districts. D0 and D2 start in the danger zone; D1 and D3 grow into crisis within 4β6 steps through spillover. The agent must choose which outbreaks to prioritise.
|
| 161 |
+
|
| 162 |
+
**Hard** adds the most realistic and challenging mechanic: **3-day information lag**. The agent sees infection rates from 3 days ago while true infection has already grown. The `growth_rate_hint` provides a noisy signal to reason about the current true state. A smart agent estimates current infection as `reported + 3 Γ growth_hint` and allocates accordingly.
|
| 163 |
|
| 164 |
---
|
| 165 |
|
| 166 |
+
## Reward Function
|
| 167 |
+
|
| 168 |
+
Five independent dense reward terms fire every step, providing rich learning signal throughout each episode:
|
| 169 |
|
| 170 |
+
| Term | Value | Fires When |
|
| 171 |
+
| --- | --- | --- |
|
| 172 |
+
| Infection penalty | `β0.50` per district | District infection > 0.40 |
|
| 173 |
+
| Hospital breach | `β1.00` per district | Hospital capacity β€ 10% |
|
| 174 |
+
| Early containment | `+0.50 Γ (1 β step/max_steps)` | District infection < 0.20 |
|
| 175 |
+
| Correct prioritisation | `+0.30` | Allocate to highest-infected district |
|
| 176 |
+
| Unnecessary restriction | `β0.20` | Restrict district below 0.20 |
|
| 177 |
|
| 178 |
+
Key design choices:
|
|
|
|
|
|
|
|
|
|
|
|
|
| 179 |
|
| 180 |
+
- The **early containment bonus decays over time** β containing an outbreak on day 3 is worth more than on day 8. This teaches proactive behaviour rather than reactive scrambling.
|
| 181 |
+
- The **infection penalty scales with population density** β dense districts contribute more to the penalty, reflecting realistic triage priorities.
|
| 182 |
+
- The **hospital breach penalty** fires at 10% capacity, not 0% β reflecting real operational collapse thresholds where triage and diversion begin.
|
| 183 |
+
- **Restrictions lift automatically** when infection drops below the safe threshold, giving the agent a natural feedback loop on intervention effectiveness.
|
| 184 |
|
| 185 |
---
|
| 186 |
|
| 187 |
+
## Grader
|
| 188 |
+
|
| 189 |
+
The grader is **fully deterministic** β no randomness, no LLM calls. Identical trajectories always produce identical scores in `[0.0, 1.0]`.
|
| 190 |
+
|
| 191 |
+
### Score Components
|
| 192 |
|
| 193 |
+
| Component | Weight | Measures |
|
| 194 |
+
| --- | --- | --- |
|
| 195 |
+
| **Hospital score** | 45% | Average capacity preserved; Γ0.6 multiplier if any district collapsed |
|
| 196 |
+
| **Containment score** | 30% | Fraction of district-days below infection threshold (grace period: first 2 steps excluded) |
|
| 197 |
+
| **Efficiency score** | 15% | Fraction of resource actions targeting the highest-infected district (uses pre-action state) |
|
| 198 |
+
| **Speed score** | 10% | `1 β (steps / max_steps)` if episode ends before max steps; else 0 |
|
| 199 |
|
| 200 |
+
Weight rationale: In real epidemic response, preserving healthcare system function (45%) is the primary operational constraint β a functional hospital system is the prerequisite for everything else. WHO and CDC outbreak protocols define success primarily by healthcare capacity preservation, with infection containment as the secondary signal. Efficiency (15%) rewards triage intelligence. Speed (10%) rewards proactive early intervention.
|
| 201 |
+
|
| 202 |
+
The efficiency score uses the **previous step's infection rates** to evaluate targeting decisions, ensuring that a successful treatment that drives infection below threshold is not retroactively penalised for being "unnecessary."
|
|
|
|
|
|
|
| 203 |
|
| 204 |
---
|
| 205 |
|
| 206 |
+
## Baseline Agent β GRPO-Style Episodic Memory
|
| 207 |
|
| 208 |
+
The baseline implements **simulated GRPO with episodic memory** β no weight updates, no gradient computation. The prompt is the policy; memory updates are the policy improvement.
|
|
|
|
|
|
|
| 209 |
|
| 210 |
+
### Learning Loop
|
|
|
|
|
|
|
|
|
|
| 211 |
|
| 212 |
+
```text
|
| 213 |
+
Rollout 1: Base prompt, no prior knowledge
|
| 214 |
+
compute advantage = R1 - mean([])
|
| 215 |
+
store positive-reward steps into EpisodicMemory
|
| 216 |
+
|
| 217 |
+
Rollout 2: Memory-augmented prompt
|
| 218 |
+
retrieve top-5 similar past decisions by L1 distance
|
| 219 |
+
inject as concrete examples into prompt
|
| 220 |
+
compute advantage = R2 - mean([R1])
|
| 221 |
+
reinforce if above average
|
| 222 |
+
|
| 223 |
+
... repeat for N rollouts
|
| 224 |
+
Report best grader score across all rollouts
|
| 225 |
```
|
| 226 |
|
| 227 |
+
### Memory Retrieval
|
| 228 |
|
| 229 |
+
Past decisions are stored as `(infection_profile, resources, phase, action, reward)` tuples. At each step, the top-5 most similar past situations are retrieved by L1 distance on infection profiles, weighted by episode phase (early/mid/late). This provides the agent with concrete examples of what worked in similar situations without any gradient update.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 230 |
|
| 231 |
+
### Baseline Scores
|
| 232 |
+
|
| 233 |
+
| Task | Agent | Containment | Hospital | Efficiency | Final Score |
|
| 234 |
+
| --- | --- | --- | --- | --- | --- |
|
| 235 |
+
| Easy | Dumb greedy (always D0) | 0.50 | 0.92 | 0.45 | ~0.50 |
|
| 236 |
+
| Easy | LLM + GRPO memory | 1.00 | 1.00 | 1.00 | **0.88β0.93** |
|
| 237 |
+
| Medium | Dumb greedy | 0.18 | 0.21 | 0.40 | ~0.23 |
|
| 238 |
+
| Medium | LLM + GRPO memory | 0.44β0.73 | 0.97β1.00 | 0.87β1.00 | **0.70β0.85** |
|
| 239 |
+
| Hard | Dumb greedy | 0.12 | 0.18 | 0.25 | ~0.21 |
|
| 240 |
+
| Hard | LLM + GRPO memory | 0.28β0.51 | 0.86β0.97 | 0.47β0.73 | **0.58β0.65** |
|
| 241 |
+
|
| 242 |
+
The gap between dumb greedy and LLM+GRPO β particularly on medium (0.23 β 0.78) β demonstrates that the environment meaningfully discriminates between agent quality. This is the core benchmark property.
|
| 243 |
|
| 244 |
---
|
| 245 |
|
| 246 |
## Generalisation
|
| 247 |
|
| 248 |
+
This environment is not epidemic-specific. The underlying mechanics apply directly to:
|
| 249 |
|
| 250 |
+
| Domain | Spreading cascade | Delayed data | Resource scarcity |
|
| 251 |
+
| --- | --- | --- | --- |
|
| 252 |
+
| π¦ **Epidemic containment** | Infection spreads between districts | Lagged case counts | Medical resources |
|
| 253 |
+
| π₯ **Wildfire deployment** | Fire spreads across terrain | Satellite update delay | Firefighting crews |
|
| 254 |
+
| π‘οΈ **Cyberattack isolation** | Lateral movement between systems | Detection lag | Security team hours |
|
| 255 |
+
| π’ **Misinformation containment** | Narrative spread through networks | Viral detection lag | Correction budget |
|
| 256 |
+
| π€ **Poverty intervention** | Deprivation cascades through communities | Census data lag | Aid allocation |
|
| 257 |
|
| 258 |
+
The same trained policy generalises across domains with minimal prompt adaptation β this is the intended use case for the OpenEnv ecosystem.
|
| 259 |
|
| 260 |
---
|
| 261 |
|
| 262 |
## Project Structure
|
| 263 |
|
| 264 |
```text
|
| 265 |
+
cascade-containment/
|
| 266 |
+
βββ inference.py # Evaluation entry point (mandatory [START][STEP][END] logs)
|
| 267 |
+
βββ models.py # Typed data contracts: Action, Observation, State
|
| 268 |
+
βββ client.py # OpenEnv client interface
|
| 269 |
+
βββ openenv.yaml # Environment manifest for OpenEnv registry
|
| 270 |
+
β
|
| 271 |
βββ server/
|
| 272 |
+
β βββ app.py # FastAPI server + judge dashboard + /grade /info /demo endpoints
|
| 273 |
+
β βββ environment.py # Core RL loop (reset/step/state OpenEnv interface)
|
| 274 |
+
β βββ grader.py # Deterministic trajectory scorer β no LLM calls
|
| 275 |
+
β βββ constants.py # Single source of truth for all numeric configuration
|
| 276 |
+
β βββ utils.py # Spread computation, observation builder, helper functions
|
| 277 |
+
β βββ Dockerfile # Container definition
|
| 278 |
+
β βββ tasks/
|
| 279 |
+
β βββ task_easy.py # 2 districts, 10 steps, real-time data
|
| 280 |
+
β βββ task_medium.py # 4 districts, 15 steps, forced triage
|
| 281 |
+
β βββ task_hard.py # 6 districts, 15 steps, 3-day data lag
|
| 282 |
+
β
|
| 283 |
βββ baseline/
|
| 284 |
+
β βββ policy.py # LLM policy with chain-of-thought prompting
|
| 285 |
+
β βββ evaluator.py # GRPO episodic memory loop
|
| 286 |
+
β βββ run.py # CLI entry point
|
| 287 |
+
β
|
| 288 |
βββ core/
|
| 289 |
+
βββ trajectory.py # EpisodicMemory β L1 similarity retrieval
|
| 290 |
+
βββ reward.py # Score normalisation utilities
|
| 291 |
+
βββ policy_update.py # Advantage computation, memory gating
|
| 292 |
+
```
|
| 293 |
+
|
| 294 |
+
---
|
| 295 |
+
|
| 296 |
+
## OpenEnv Compliance
|
| 297 |
+
|
| 298 |
+
| Requirement | Status |
|
| 299 |
+
| --- | --- |
|
| 300 |
+
| `reset()` returns `CityObservation` | β
|
|
| 301 |
+
| `step(action)` returns `CityObservation` | β
|
|
| 302 |
+
| `state` property returns `State` | β
|
|
| 303 |
+
| Typed `Action` subclass | β
`ContainmentAction(Action)` |
|
| 304 |
+
| Typed `Observation` subclass | β
`CityObservation(Observation)` |
|
| 305 |
+
| `openenv.yaml` manifest | β
|
|
| 306 |
+
| Dockerfile builds | β
|
|
| 307 |
+
| HF Space deploys | β
|
|
| 308 |
+
| `inference.py` at root | β
|
|
| 309 |
+
| `[START][STEP][END]` structured logs | β
|
|
| 310 |
+
| Runtime < 20 minutes | β
~16 minutes |
|
| 311 |
+
| `API_BASE_URL`, `MODEL_NAME`, `HF_TOKEN` env vars | β
|
|
| 312 |
+
| OpenAI client for all LLM calls | β
|
|
| 313 |
+
| Grader scores in `[0.0, 1.0]` | β
|
|
| 314 |
+
| 3+ tasks with difficulty progression | β
|
|
| 315 |
+
|
| 316 |
+
---
|
| 317 |
+
|
| 318 |
+
## Setup and Local Development
|
| 319 |
+
|
| 320 |
+
### Local Server
|
| 321 |
+
|
| 322 |
+
```bash
|
| 323 |
+
pip install -r requirements.txt
|
| 324 |
+
uvicorn server.app:app --host 0.0.0.0 --port 7860
|
| 325 |
+
|
| 326 |
+
export ENV_BASE_URL=http://localhost:7860
|
| 327 |
+
python baseline/run.py
|
| 328 |
+
```
|
| 329 |
+
|
| 330 |
+
### Docker
|
| 331 |
+
|
| 332 |
+
```bash
|
| 333 |
+
docker build -f server/Dockerfile -t cascade-containment .
|
| 334 |
+
docker run -p 7860:7860 cascade-containment
|
| 335 |
```
|
| 336 |
|
| 337 |
---
|
| 338 |
|
| 339 |
## Tags
|
| 340 |
|
| 341 |
+
`reinforcement-learning` Β· `resource-allocation` Β· `sequential-decision-making` Β· `partial-observability` Β· `cascade-dynamics` Β· `epidemic-response` Β· `openenv` Β· `llm-agent` Β· `grpo` Β· `episodic-memory` Β· `triage` Β· `multi-district` Β· `docker` Β· `fastapi`
|
inference.py
CHANGED
|
@@ -3,7 +3,7 @@
|
|
| 3 |
# Hackathon evaluation entry point β Cascade Containment
|
| 4 |
#
|
| 5 |
# Emits structured stdout logs in the mandatory [START]/[STEP]/[END] format.
|
| 6 |
-
# Runs
|
| 7 |
# Runtime: ~10-12 minutes on 2vCPU/8GB RAM (well under 20-minute limit).
|
| 8 |
#
|
| 9 |
# Required environment variables:
|
|
@@ -148,7 +148,7 @@ def run_rollout(
|
|
| 148 |
return total_reward, step, trajectory, score
|
| 149 |
|
| 150 |
|
| 151 |
-
# ββ Task runner:
|
| 152 |
|
| 153 |
def run_task(env, task_name: str, client: OpenAI) -> float:
|
| 154 |
memory = EpisodicMemory(max_size=20)
|
|
|
|
| 3 |
# Hackathon evaluation entry point β Cascade Containment
|
| 4 |
#
|
| 5 |
# Emits structured stdout logs in the mandatory [START]/[STEP]/[END] format.
|
| 6 |
+
# Runs 4 GRPO rollouts per task to demonstrate learning improvement.
|
| 7 |
# Runtime: ~10-12 minutes on 2vCPU/8GB RAM (well under 20-minute limit).
|
| 8 |
#
|
| 9 |
# Required environment variables:
|
|
|
|
| 148 |
return total_reward, step, trajectory, score
|
| 149 |
|
| 150 |
|
| 151 |
+
# ββ Task runner: 4 GRPO rollouts with episodic memory βββββββββββββββββββββββββ
|
| 152 |
|
| 153 |
def run_task(env, task_name: str, client: OpenAI) -> float:
|
| 154 |
memory = EpisodicMemory(max_size=20)
|
server/environment.py
CHANGED
|
@@ -271,6 +271,8 @@ class EpidemicContainmentEnv(Environment):
|
|
| 271 |
1.0,
|
| 272 |
district.hospital_capacity_remaining + 0.02
|
| 273 |
)
|
|
|
|
|
|
|
| 274 |
|
| 275 |
# ββ Private: Reward Computation βββββββββββββββββββββββββββββββββββββββββββ
|
| 276 |
|
|
@@ -283,7 +285,8 @@ class EpidemicContainmentEnv(Environment):
|
|
| 283 |
|
| 284 |
# Term 1: Penalty for each district above danger threshold
|
| 285 |
for district in districts_above_threshold(self._city.districts):
|
| 286 |
-
|
|
|
|
| 287 |
|
| 288 |
# Term 2: Heavy penalty for hospital capacity breach
|
| 289 |
for district in self._city.districts:
|
|
|
|
| 271 |
1.0,
|
| 272 |
district.hospital_capacity_remaining + 0.02
|
| 273 |
)
|
| 274 |
+
if district.true_infection_rate < SAFE_THRESHOLD:
|
| 275 |
+
district.restriction_active = False
|
| 276 |
|
| 277 |
# ββ Private: Reward Computation βββββββββββββββββββββββββββββββββββββββββββ
|
| 278 |
|
|
|
|
| 285 |
|
| 286 |
# Term 1: Penalty for each district above danger threshold
|
| 287 |
for district in districts_above_threshold(self._city.districts):
|
| 288 |
+
density_weight = max(0.5, district.population_density * len(self._city.districts))
|
| 289 |
+
reward += REWARD_INFECTION_PENALTY * min(2.0, density_weight)
|
| 290 |
|
| 291 |
# Term 2: Heavy penalty for hospital capacity breach
|
| 292 |
for district in self._city.districts:
|
server/utils.py
CHANGED
|
@@ -101,7 +101,7 @@ def compute_spread(districts: List[DistrictTruth]) -> List[float]:
|
|
| 101 |
net_change = spread_rate - natural_recovery - intervention_reductions
|
| 102 |
new_rate = current + net_change + geographic_spillover
|
| 103 |
|
| 104 |
-
Natural recovery (NATURAL_RECOVERY_RATE = 0.
|
| 105 |
individuals recovering without medical intervention. This means infection
|
| 106 |
naturally decays slightly each day, but spread rate still dominates
|
| 107 |
without active response β districts grow unless the agent acts.
|
|
|
|
| 101 |
net_change = spread_rate - natural_recovery - intervention_reductions
|
| 102 |
new_rate = current + net_change + geographic_spillover
|
| 103 |
|
| 104 |
+
Natural recovery (NATURAL_RECOVERY_RATE = 0.01/day) reflects infected
|
| 105 |
individuals recovering without medical intervention. This means infection
|
| 106 |
naturally decays slightly each day, but spread rate still dominates
|
| 107 |
without active response β districts grow unless the agent acts.
|