Spaces:

sai1906
/

opsguard

Sleeping

App Files Files Community

opsguard / README.md

sai1906

Upload folder using huggingface_hub

2612006 verified 3 months ago

preview code

Raw

History Blame Contribute Delete

11.4 kB

metadata

title: OpsGuard
emoji: 🛡️
colorFrom: gray
colorTo: indigo
sdk: docker
pinned: true
app_port: 8000
base_path: /web
tags:
  - openenv
  - openenv-hackathon
  - long-horizon
  - self-improvement
  - rl
  - agent

OpsGuard — Train a 3B Model to Run On-Call for an Open-Source Repo

Can a 3B model triage 60 issues in a row — while a co-evolved adversary tries to slip spam past it?

OpsGuard is an OpenEnv environment that trains LLM maintainers against synthetic spammers across 6 difficulty tiers, scoring against the real maintainer decisions of huggingface/peft. No simulator. Real GitHub data. Real ground truth.

Hero numbers — rule-based baselines on real GH data

375 real issues, 1911 comments. Each scenario mixes real issues + synthetic spam at scenario-defined rate. Mean over 3 seeds:

Scenario (steps × spam-rate)	random	label_everything	close_everything	keyword_triager
E0 quiet day (30 × 0%)	+3.26	+0.50	-1.60	+5.70
E1 release week (60 × 10%)	-3.70	-3.34	-5.24	+12.71
E2 spam wave (120 × 40%)	-13.14	-9.76	+0.74	+23.32
E3 coordinated attack (200 × 55%)	-14.15	-23.12	+12.18	+40.58

Spread grows monotonically: 7 → 18 → 36 → 64 reward points as horizon and adversary intensity rise. Greedy strategies fail at long horizons. No shortcut wins.

Trained 3B agent (post-GRPO): numbers populate after training run lands. Live updates pushed to this README + Hub repo sai1906/opsguard-grpo.

The Story — 4 Acts

Act 1 — The Naive Triager. Baseline (label_everything) blindly tags every issue bug. On E2, half the queue is synthetic spam — none caught. Reward: -9.76.

Act 2 — The Keyword Spotter. Hand-coded rules detect spam tokens, route by content keyword, request info on thin reports. Same E2 scenario: +23.32. Spam recall jumps 0% → 77%.

Act 3 — The Adversary Learns. A second 3B model is DPO-trained as a spammer, mining the maintainer's misses. It paraphrases real issue titles, fabricates urgency, mimics trusted-contributor voice. Keyword spotter's spam recall starts dropping back toward 50% on harder tiers.

Act 4 — The Co-evolved Maintainer. The maintainer is GRPO-trained on rollouts against the trained adversary. By round 3, it learns to weigh hidden contributor reputation, query history before deciding, and resist self_contradiction-style appeals. Final hero numbers populate post-training.

What's in the box

opsguard/
├── models.py                     # 9 ActionTypes + Pydantic Action/Observation
├── server/
│   ├── opsguard_environment.py   # OpenEnv Environment subclass; reset/step/state
│   ├── app.py                    # FastAPI + WebSocket via openenv.core.create_app
│   └── Dockerfile                # Multi-stage uv build, ghcr.io/meta-pytorch/openenv-base
├── world/
│   ├── db.py                     # SQLite query helpers (RepoDB, IssueRow, Contributor)
│   ├── scenarios.py              # E0_quiet_day → E5_hostile_fork_war (30→500 step budgets)
│   ├── adversary.py              # 5 spam patterns × 5 tiers (scripted)
│   ├── trainable_adversary.py    # LoRA spammer (round-2 co-evolution); template fallback
│   ├── grader.py                 # Composable rubric + multiplicative terminal
│   └── curriculum.py             # Mastery-based tier unlock
├── eval/
│   ├── policies.py               # 5 baselines: random, label_everything, close_everything, keyword_triager, memory_aware
│   └── harness.py                # Rollout + aggregate + markdown summary
├── scripts/
│   ├── pull_gh.py                # GH REST API issue/PR/comment puller
│   ├── ingest_to_sqlite.py       # JSONL → SQLite, derives truth_action
│   ├── build_sft_traces.py       # Roll rule-based policies, emit (prompt, completion) JSONL
│   ├── system_prompt.py          # SYSTEM_PROMPT, format_observation, parse_action
│   ├── sft_warmstart.py          # Unsloth 4-bit + LoRA SFT (1 epoch)
│   ├── train_grpo.py             # TRL OpenEnv GRPO trainer (vLLM colocate optional)
│   ├── train_grpo_colab.ipynb    # Self-contained Colab notebook
│   └── make_plots.py             # Reward / spam-recall plots
├── tests/                        # Smoke tests (env loop + train integration), all green
├── data/repo.db                  # 375 real issues + 1911 comments from huggingface/peft
└── openenv.yaml                  # OpenEnv v1 spec

Hackathon themes hit

T2 long-horizon — episodes 30 → 500 steps; multi-issue queues; agent must process backlog while resisting 60-issue spam waves
T4 self-improvement — co-evolved adversary (trainable spammer LoRA) + mastery-driven curriculum (E0 → E5 unlocks on rolling reward, kube-sre-gym pattern)
T3 world modeling touch — hidden contributor reputation drives reward but is never exposed to the agent. Agent infers trust from public stats (PR count, account age) — env penalizes mis-judgement.

Action space (9 actions, single per step)

label · close_spam · request_info · link_duplicate · assign · comment · merge_pr · query_history · wait

JSON contract:

{"action_type":"label","target_issue_id":12345,"label":"bug","reasoning":"..."}

Reward (composable rubric, hard to game)

Per the OpenEnv hackathon hint "composable rubrics > monolithic scoring", OpsGuard splits reward into 7 components fired conditionally:

Component	Fires when	Sign
`action_match`	agent's action == maintainer's recorded action	+1.0 (correct), -0.5 (wrong), +0.4 (partial)
`label_overlap`	Jaccard between predicted & true label sets	+0..0.4
`spam_resistance`	`close_spam` on synthetic spam	+0.8
	engaged with spam (label/merge/assign)	-0.7
`reputation_alignment`	closed spam on low-rep author / labeled trusted-author content	±0.2
`info_request`	first request on under-described issue / redundant	+0.15 / -0.15
`repeat_action_penalty`	same action on same issue within 5 steps	-0.15
`workflow_efficiency`	per-step cost	-0.02
terminal: `queue_completion × spam_recall_gate`	end of episode	-1..1, multiplicative

The terminal is multiplicative on purpose — a policy that hits queue completion but ignores spam is gated to 0. Greedy strategies fail.

Try it yourself (3 paths)

1. Run baseline eval (CPU only, ~2 min):

git clone https://huggingface.co/spaces/sai1906/opsguard
cd opsguard
pip install -e .
python scripts/run_baseline_eval.py --out eval_outputs/baseline
cat eval_outputs/baseline/summary.md

2. Run env locally:

python -m server.app  # → http://localhost:8000
curl http://localhost:8000/state

3. Train + GRPO (Colab T4 free OR HF Jobs A100 ~1h):

# Open scripts/train_grpo_colab.ipynb in Colab → Run All
# OR via HF Jobs:
hf jobs uv run --flavor a100-large \
    --with "trl,unsloth,openenv-core,peft,bitsandbytes,vllm,datasets" \
    --secrets HF_TOKEN \
    -- python scripts/train_grpo.py \
        --model unsloth/Qwen2.5-7B-Instruct-bnb-4bit \
        --hub-repo sai1906/opsguard-grpo \
        --num-steps 200

Live links

HF Space (this env): https://huggingface.co/spaces/sai1906/opsguard
Colab demo: scripts/train_grpo_colab.ipynb
Trained LoRA: https://huggingface.co/sai1906/opsguard-grpo (populates after training)
Data source: huggingface/peft — 375 issues, 1911 comments, 6-month window

Why it matters

Open-source maintainers spend 30–50% of triage time on duplicates, urgency-fabrications, and low-info noise. A 3B model that catches 80% of synthetic spam at 95% precision and routes the rest to the right label is a real productivity win — and it's a problem nobody has trained for in OpenEnv before.

References

OpenEnv (PyTorch + Meta, 2026) — https://github.com/meta-pytorch/OpenEnv
TRL OpenEnv integration — https://huggingface.co/docs/trl/en/openenv
AgentGym-RL ScalingInter-RL (Sep 2025, arXiv 2509.08755) — curriculum design
kube-sre-gym (SF OpenEnv 1st place, Mar 2026) — mastery curriculum + adversarial designer pattern
Multi-Agent Evolve (arXiv 2510.23595) — co-evolution recipe (Proposer/Solver/Judge)
AgentLAB long-horizon attacks (arXiv 2602.16901) — adversarial maintenance pattern catalog

Architecture (textual)

                 ┌────────────────────────┐
                 │  GitHub REST API       │
                 │  (peft, 6-month pull)  │
                 └──────────┬─────────────┘
                            │  pull_gh.py
                            ▼
                 ┌────────────────────────┐
                 │  data/repo.db (SQLite) │
                 │  375 issues, 1911 cmts │
                 └──────────┬─────────────┘
                            │
       ┌────────────────────┼─────────────────────┐
       ▼                    ▼                     ▼
┌──────────────┐   ┌─────────────────┐   ┌──────────────────┐
│ Adversary    │   │ Curriculum      │   │ Composable       │
│ - 5 patterns │   │ - Mastery       │   │ Rubric           │
│ - 5 tiers    │   │ - E0→E5 unlock  │   │ - 7 components   │
│ - LoRA-able  │   │                 │   │ - Multipl. term  │
└──────┬───────┘   └────────┬────────┘   └────────┬─────────┘
       │                    │                     │
       └────────────────────┼─────────────────────┘
                            ▼
                 ┌────────────────────────┐
                 │  OpsGuardEnvironment   │
                 │  (OpenEnv subclass)    │
                 │  reset / step / state  │
                 └──────────┬─────────────┘
                            │ FastAPI + WS
                            ▼
                 ┌────────────────────────┐
                 │  TRL GRPOTrainer       │
                 │  Qwen2.5-7B + LoRA     │
                 │  (vLLM colocate)       │
                 └────────────────────────┘

License

BSD-3 (matching OpenEnv).