README.md

metadata

title: DevOps RL Agent
emoji: 🤖
colorFrom: blue
colorTo: green
sdk: docker
app_file: app.py
pinned: false

🤖 DevOps RL Agent

An AI agent that learns to fix broken Linux/Python environments through reinforcement learning.

Built with OpenEnv + TRL (GRPO) + Unsloth — no agent frameworks, no multi-agent systems, just one LLM and one RL loop.

---

🎯 What It Does

The agent observes broken environments (missing packages, port conflicts, config errors), generates shell commands to fix them, executes those commands in a Docker sandbox, and improves through GRPO training over episodes.

=== BEFORE TRAINING (episode 0) ===
Error: ModuleNotFoundError: flask
Step 1: python app.py       → failed (exit 1)
Step 2: sudo pip install    → DANGEROUS COMMAND BLOCKED
Step 3: apt install python  → wrong approach
Result: FAILED (reward: -8.2)

=== AFTER TRAINING (episode 500) ===
Error: ModuleNotFoundError: flask
Step 1: pip install flask   → success
Step 2: python app.py       → Server running on :5000
Result: SOLVED in 2 steps (reward: +12.6)

---

🔄 How the RL Loop Works

This is the core architecture — one agent, one environment, one training loop:

graph LR
    A[🧠 LLM Agent] -->|shell command| B[🐳 Docker Sandbox]
    B -->|stdout/stderr/exit_code| C[🏗️ Environment]
    C -->|observation + reward| A
    C -->|episode data| D[💾 Replay Buffer]
    D -->|training batches| E[📈 GRPO Trainer]
    E -->|updated weights| A

The Loop in Detail

1. Reset: Environment loads a random broken scenario (e.g., Flask not installed)
2. Observe: Agent receives an error log, command history, and error type classification
3. Act: Agent outputs ONE shell command (e.g., pip install flask)
4. Execute: Command runs in an isolated Docker container with safety checks
5. Reward: Multi-signal reward computed (success +10, correct command +3, progress +1, etc.)
6. Repeat: Steps 2-5 repeat up to 10 times per episode
7. Train: After N episodes, GRPO updates the model using grouped completions

Error Fingerprinting (Key Differentiator)

Before the agent acts, a rule-based classifier identifies the error type from the terminal output:

Error Type	Pattern	Example
missing_package	ModuleNotFoundError	No module named 'flask'
port_conflict	Address already in use	Port 5000 occupied
missing_env	KeyError on env var	DATABASE_URL not set
version_conflict	ResolutionImpossible	Package version clash
config_error	NameError, misconfig	Wrong host binding

This gives the LLM better context and lets us analyze which error categories the agent struggles with.

Multi-Signal Reward System

The reward engine returns a breakdown dict with 10 independent signals:

Signal	Value	Purpose
success	+10.0	Scenario fully solved
correct_command	+3.0	Matches optimal fix sequence
progress	+1.0	Error log changed (likely improvement)
efficiency_bonus	+2.0	Solved in minimal steps
invalid_command	-2.0	Command not whitelisted
dangerous_command	-10.0	Matches blocklist (rm -rf /, etc.)
no_progress	-1.0	Error log unchanged
timeout	-5.0	Command exceeded 30s
repeated_command	-1.5	Same command twice in episode
step_cost	-0.2	Encourages efficiency

Each column is logged separately during training to detect reward hacking.

Curriculum Learning

Scenarios unlock progressively based on rolling 50-episode solve rate windows:

• Level 1 (single-step): Always available — missing_flask, missing_numpy, wrong_python
• Level 2 (two-step): Unlocks at L1 solve rate > 80% — port_conflict, missing_env_var, broken_requirements
• Level 3 (multi-step): Unlocks at L2 solve rate > 80% — corrupt_venv, wrong_config, db_migration

---

📁 Project Structure

devops-rl-agent/
├── devops_env/              # OpenEnv-style RL environment
│   ├── env.py               # DevOpsEnv (reset/step/reward)
│   └── state_manager.py     # Observation tracking
├── scenarios/               # 9 scenarios across 3 difficulty levels
│   ├── registry.py          # ScenarioRegistry
│   ├── level1/              # Single-step fixes
│   ├── level2/              # Two-step fixes
│   └── level3/              # Multi-step fixes (3-5 steps)
├── executor/                # Docker sandbox execution
│   ├── docker_executor.py   # Container management + local fallback
│   └── safety.py            # Command whitelist/blocklist
├── fingerprint/             # Error classification system
│   └── classifier.py        # Rule-based regex classifier
├── rewards/                 # Multi-signal reward computation
│   └── engine.py            # 10 independent reward signals
├── replay/                  # Episode storage (SQLite + SQLAlchemy)
│   ├── buffer.py            # ReplayBuffer with batch sampling
│   └── models.py            # ORM models
├── agent/                   # LLM + baseline agents
│   ├── baseline_agent.py    # Rule-based (for loop validation)
│   ├── devops_agent.py      # LLM agent (Unsloth/HF)
│   └── prompts.py           # System & user prompt templates
├── training/                # GRPO training pipeline
│   ├── train_grpo.py        # Training loop + anti-hacking monitor
│   └── curriculum.py        # Rolling-window curriculum scheduler
├── api/                     # FastAPI server (OpenEnv pattern)
│   └── main.py
├── frontend/                # Dashboard (vanilla HTML/CSS/JS)
│   ├── index.html
│   ├── style.css
│   └── app.js
├── docker/                  # Sandbox container
│   ├── Dockerfile.sandbox
│   └── docker-compose.yml
├── tests/                   # Unit tests
│   ├── test_env.py
│   ├── test_rewards.py      # 100% reward engine coverage
│   └── test_executor.py
├── scripts/
│   └── demo.py              # Before/after training demo
├── requirements.txt
└── README.md

---

🚀 Quick Start

1. Install Dependencies

cd devops-rl-agent
pip install -r requirements.txt

2. Validate the RL Loop (No GPU Required)

Run the baseline agent to confirm environment, executor, and rewards work:

# Run the demo script
python scripts/demo.py --episodes 50

# Run unit tests
python -m pytest tests/ -v

3. Start the API Server

uvicorn api.main:app --reload --port 8000

Then visit:

• API docs: http://localhost:8000/docs
• Dashboard: http://localhost:8000/app

4. Build the Docker Sandbox (Optional)

cd docker
docker compose build
docker compose up -d sandbox

5. Run GRPO Training (Requires GPU)

python -c "
from training.train_grpo import GRPODevOpsTrainer
trainer = GRPODevOpsTrainer(model_name='unsloth/llama-3.2-3b-instruct')
trainer.train(num_episodes=500)
"

☁️ Deploy to Hugging Face Spaces

This repository is set up as a Docker Space. To deploy:

1. Create a new Hugging Face Space.
2. Choose Docker as the SDK.
3. Push this repository contents to the Space repo.
4. Hugging Face will build the root Dockerfile and start the API on port 7860.

Useful endpoints after deployment:

• GET / health check
• POST /reset OpenEnv session reset
• POST /step OpenEnv step
• POST /close OpenEnv session cleanup
• POST /episode/run one-shot episode execution

Notes:

• Runtime DB files are ignored via .dockerignore.
• If you want model training inside Spaces, keep use_grpo=False unless the Space has a GPU and enough VRAM.

---

🔒 Safety

The executor enforces strict command safety:

Whitelisted: pip, python, cat, ls, grep, sed, ps, kill, curl, echo, mkdir, cp, mv, export, source

Blocked: rm -rf /, fork bombs, dd if=, mkfs, chmod 777 /, sudo + destructive ops

Commands that fail safety checks are immediately blocked, the agent receives a -10.0 penalty, and the episode terminates.

---

📊 Success Criteria

After 500 training episodes:

Metric	Target
Level 1 solve rate	> 90%
Level 2 solve rate	> 70%
Mean steps to solve L1	≤ 2
Reward hacking detected	None
Model saves/loads correctly	✓

---

🔧 API Endpoints

Method	Path	Description
POST	/reset	OpenEnv-style session reset (returns session_id + initial observation)
POST	/step	OpenEnv-style step for a session ({session_id, action:{command}})
POST	/close	OpenEnv-style explicit session cleanup
POST	/episode/run	Run one episode, returns full step log
GET	/episode/{id}	Get stored episode by UUID
GET	/stats	Solve rates, rewards, training progress
GET	/replay/{id}	Step-by-step replay data
POST	/train/step	Trigger training batch
GET	/scenarios	List scenarios with solve rates
GET	/recent	Recent episodes

---

🛡️ Anti-Reward Hacking

The training loop includes active monitoring:

1. Generation inspection every 50 steps — actual agent outputs are printed
2. Success column tracking — alerts if total reward rises but success rate stays flat
3. Command repetition detection — flags if one command dominates >50% of actions
4. Dangerous command counting — terminates if blocklist triggers repeatedly
5. Per-column reward breakdown — logged separately to catch gaming of individual signals

---

🏗️ Tech Stack

• RL Framework: TRL (GRPOTrainer) + Unsloth (4-bit LoRA)
• Base Model: unsloth/llama-3.2-3b-instruct
• Environment: OpenEnv-compatible API pattern
• Execution: Docker SDK with safety layer
• Storage: SQLite via SQLAlchemy
• API: FastAPI + Uvicorn
• Frontend: Vanilla HTML/CSS/JS (dark theme, glassmorphism)
• Testing: pytest (100% reward engine coverage)

---

📝 License

MIT