README.md

---
title: DevOps RL Agent
emoji: 🤖
colorFrom: blue
colorTo: green
sdk: docker
app_file: app.py
pinned: false
---

# 🤖 DevOps RL Agent

**An AI agent that learns to fix broken Linux/Python environments through reinforcement learning.**

Built with **OpenEnv + TRL (GRPO) + Unsloth** — no agent frameworks, no multi-agent systems, just one LLM and one RL loop.

[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://python.org)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)

---

## 🎯 What It Does

The agent observes broken environments (missing packages, port conflicts, config errors), generates shell commands to fix them, executes those commands in a Docker sandbox, and improves through GRPO training over episodes.

```
=== BEFORE TRAINING (episode 0) ===
Error: ModuleNotFoundError: flask
Step 1: python app.py       → failed (exit 1)
Step 2: sudo pip install    → DANGEROUS COMMAND BLOCKED
Step 3: apt install python  → wrong approach
Result: FAILED (reward: -8.2)

=== AFTER TRAINING (episode 500) ===
Error: ModuleNotFoundError: flask
Step 1: pip install flask   → success
Step 2: python app.py       → Server running on :5000
Result: SOLVED in 2 steps (reward: +12.6)
```

---

## 🔄 How the RL Loop Works

This is the core architecture — one agent, one environment, one training loop:

```mermaid
graph LR
    A[🧠 LLM Agent] -->|shell command| B[🐳 Docker Sandbox]
    B -->|stdout/stderr/exit_code| C[🏗️ Environment]
    C -->|observation + reward| A
    C -->|episode data| D[💾 Replay Buffer]
    D -->|training batches| E[📈 GRPO Trainer]
    E -->|updated weights| A
```

### The Loop in Detail

1. **Reset**: Environment loads a random broken scenario (e.g., Flask not installed)
2. **Observe**: Agent receives an error log, command history, and error type classification
3. **Act**: Agent outputs ONE shell command (e.g., `pip install flask`)
4. **Execute**: Command runs in an isolated Docker container with safety checks
5. **Reward**: Multi-signal reward computed (success +10, correct command +3, progress +1, etc.)
6. **Repeat**: Steps 2-5 repeat up to 10 times per episode
7. **Train**: After N episodes, GRPO updates the model using grouped completions

### Error Fingerprinting (Key Differentiator)

Before the agent acts, a rule-based classifier identifies the error type from the terminal output:

| Error Type | Pattern | Example |
|---|---|---|
| `missing_package` | `ModuleNotFoundError` | `No module named 'flask'` |
| `port_conflict` | `Address already in use` | Port 5000 occupied |
| `missing_env` | `KeyError` on env var | `DATABASE_URL` not set |
| `version_conflict` | `ResolutionImpossible` | Package version clash |
| `config_error` | `NameError`, misconfig | Wrong host binding |

This gives the LLM better context and lets us analyze which error categories the agent struggles with.

### Multi-Signal Reward System

The reward engine returns a breakdown dict with 10 independent signals:

| Signal | Value | Purpose |
|---|---|---|
| `success` | +10.0 | Scenario fully solved |
| `correct_command` | +3.0 | Matches optimal fix sequence |
| `progress` | +1.0 | Error log changed (likely improvement) |
| `efficiency_bonus` | +2.0 | Solved in minimal steps |
| `invalid_command` | -2.0 | Command not whitelisted |
| `dangerous_command` | -10.0 | Matches blocklist (rm -rf /, etc.) |
| `no_progress` | -1.0 | Error log unchanged |
| `timeout` | -5.0 | Command exceeded 30s |
| `repeated_command` | -1.5 | Same command twice in episode |
| `step_cost` | -0.2 | Encourages efficiency |

Each column is logged separately during training to detect reward hacking.

### Curriculum Learning

Scenarios unlock progressively based on rolling 50-episode solve rate windows:

- **Level 1** (single-step): Always available — `missing_flask`, `missing_numpy`, `wrong_python`
- **Level 2** (two-step): Unlocks at L1 solve rate > 80% — `port_conflict`, `missing_env_var`, `broken_requirements`
- **Level 3** (multi-step): Unlocks at L2 solve rate > 80% — `corrupt_venv`, `wrong_config`, `db_migration`

---

## 📁 Project Structure

```
devops-rl-agent/
├── devops_env/              # OpenEnv-style RL environment
│   ├── env.py               # DevOpsEnv (reset/step/reward)
│   └── state_manager.py     # Observation tracking
├── scenarios/               # 9 scenarios across 3 difficulty levels
│   ├── registry.py          # ScenarioRegistry
│   ├── level1/              # Single-step fixes
│   ├── level2/              # Two-step fixes
│   └── level3/              # Multi-step fixes (3-5 steps)
├── executor/                # Docker sandbox execution
│   ├── docker_executor.py   # Container management + local fallback
│   └── safety.py            # Command whitelist/blocklist
├── fingerprint/             # Error classification system
│   └── classifier.py        # Rule-based regex classifier
├── rewards/                 # Multi-signal reward computation
│   └── engine.py            # 10 independent reward signals
├── replay/                  # Episode storage (SQLite + SQLAlchemy)
│   ├── buffer.py            # ReplayBuffer with batch sampling
│   └── models.py            # ORM models
├── agent/                   # LLM + baseline agents
│   ├── baseline_agent.py    # Rule-based (for loop validation)
│   ├── devops_agent.py      # LLM agent (Unsloth/HF)
│   └── prompts.py           # System & user prompt templates
├── training/                # GRPO training pipeline
│   ├── train_grpo.py        # Training loop + anti-hacking monitor
│   └── curriculum.py        # Rolling-window curriculum scheduler
├── api/                     # FastAPI server (OpenEnv pattern)
│   └── main.py
├── frontend/                # Dashboard (vanilla HTML/CSS/JS)
│   ├── index.html
│   ├── style.css
│   └── app.js
├── docker/                  # Sandbox container
│   ├── Dockerfile.sandbox
│   └── docker-compose.yml
├── tests/                   # Unit tests
│   ├── test_env.py
│   ├── test_rewards.py      # 100% reward engine coverage
│   └── test_executor.py
├── scripts/
│   └── demo.py              # Before/after training demo
├── requirements.txt
└── README.md
```

---

## 🚀 Quick Start

### 1. Install Dependencies

```bash
cd devops-rl-agent
pip install -r requirements.txt
```

### 2. Validate the RL Loop (No GPU Required)

Run the baseline agent to confirm environment, executor, and rewards work:

```bash
# Run the demo script
python scripts/demo.py --episodes 50

# Run unit tests
python -m pytest tests/ -v
```

### 3. Start the API Server

```bash
uvicorn api.main:app --reload --port 8000
```

Then visit:
- API docs: [http://localhost:8000/docs](http://localhost:8000/docs)
- Dashboard: [http://localhost:8000/app](http://localhost:8000/app)

### 4. Build the Docker Sandbox (Optional)

```bash
cd docker
docker compose build
docker compose up -d sandbox
```

### 5. Run GRPO Training (Requires GPU)

```bash
python -c "
from training.train_grpo import GRPODevOpsTrainer
trainer = GRPODevOpsTrainer(model_name='unsloth/llama-3.2-3b-instruct')
trainer.train(num_episodes=500)
"
```

## ☁️ Deploy to Hugging Face Spaces

This repository is set up as a Docker Space. To deploy:

1. Create a new Hugging Face Space.
2. Choose `Docker` as the SDK.
3. Push this repository contents to the Space repo.
4. Hugging Face will build the root [Dockerfile](Dockerfile) and start the API on port `7860`.

Useful endpoints after deployment:
- `GET /` health check
- `POST /reset` OpenEnv session reset
- `POST /step` OpenEnv step
- `POST /close` OpenEnv session cleanup
- `POST /episode/run` one-shot episode execution

Notes:
- Runtime DB files are ignored via [.dockerignore](.dockerignore).
- If you want model training inside Spaces, keep `use_grpo=False` unless the Space has a GPU and enough VRAM.

---

## 🔒 Safety

The executor enforces strict command safety:

**Whitelisted**: `pip`, `python`, `cat`, `ls`, `grep`, `sed`, `ps`, `kill`, `curl`, `echo`, `mkdir`, `cp`, `mv`, `export`, `source`

**Blocked**: `rm -rf /`, fork bombs, `dd if=`, `mkfs`, `chmod 777 /`, `sudo` + destructive ops

Commands that fail safety checks are immediately blocked, the agent receives a -10.0 penalty, and the episode terminates.

---

## 📊 Success Criteria

After 500 training episodes:

| Metric | Target |
|---|---|
| Level 1 solve rate | > 90% |
| Level 2 solve rate | > 70% |
| Mean steps to solve L1 | ≤ 2 |
| Reward hacking detected | None |
| Model saves/loads correctly | ✓ |

---

## 🔧 API Endpoints

| Method | Path | Description |
|---|---|---|
| `POST` | `/reset` | OpenEnv-style session reset (returns session_id + initial observation) |
| `POST` | `/step` | OpenEnv-style step for a session (`{session_id, action:{command}}`) |
| `POST` | `/close` | OpenEnv-style explicit session cleanup |
| `POST` | `/episode/run` | Run one episode, returns full step log |
| `GET` | `/episode/{id}` | Get stored episode by UUID |
| `GET` | `/stats` | Solve rates, rewards, training progress |
| `GET` | `/replay/{id}` | Step-by-step replay data |
| `POST` | `/train/step` | Trigger training batch |
| `GET` | `/scenarios` | List scenarios with solve rates |
| `GET` | `/recent` | Recent episodes |

---

## 🛡️ Anti-Reward Hacking

The training loop includes active monitoring:

1. **Generation inspection** every 50 steps — actual agent outputs are printed
2. **Success column tracking** — alerts if total reward rises but success rate stays flat
3. **Command repetition detection** — flags if one command dominates >50% of actions
4. **Dangerous command counting** — terminates if blocklist triggers repeatedly
5. **Per-column reward breakdown** — logged separately to catch gaming of individual signals

---

## 🏗️ Tech Stack

- **RL Framework**: TRL (GRPOTrainer) + Unsloth (4-bit LoRA)
- **Base Model**: `unsloth/llama-3.2-3b-instruct`
- **Environment**: OpenEnv-compatible API pattern
- **Execution**: Docker SDK with safety layer
- **Storage**: SQLite via SQLAlchemy
- **API**: FastAPI + Uvicorn
- **Frontend**: Vanilla HTML/CSS/JS (dark theme, glassmorphism)
- **Testing**: pytest (100% reward engine coverage)

---

## 📝 License

MIT