README.md

---
title: GitHub Issue Triage — OpenEnv
emoji: 🐛
colorFrom: blue
colorTo: purple
sdk: docker
app_port: 8000
pinned: true
tags:
  - openenv
  - reinforcement-learning
  - nlp
  - triage
  - real-world
base_path: /web
---

# 🐛 GitHub Issue Triage — OpenEnv Environment

**Meta PyTorch OpenEnv Hackathon × Scaler School of Technology**  
**Team Astra.AI** · Om Chougule (Lead) · Shraman Patil

---

## What is this?

A real-world **Reinforcement Learning environment** where an AI agent reads GitHub
issues and performs structured triage decisions — the exact task that software
engineers do dozens of times per day.

The agent must:
1. Read an issue title, body, author, and existing comments
2. Assign a **label** (bug / feature / docs / question)
3. Route it to the correct **team** (frontend / backend / ml / devops / docs)
4. Score its **priority** (critical / high / medium / low)
5. Suggest a **concrete fix action**

This directly trains agents for real developer productivity tools (GitHub Copilot,
Linear, Jira auto-assign, etc.).

---

## Environment Design

### Action Space

| Field | Type | Required | Description |
|---|---|---|---|
| `label` | `string` | Always | `bug` · `feature` · `docs` · `question` |
| `team` | `string\|null` | Medium + Hard | `frontend` · `backend` · `ml` · `devops` · `docs` |
| `priority` | `string\|null` | Hard only | `critical` · `high` · `medium` · `low` |
| `suggested_action` | `string\|null` | Hard only | Brief concrete fix recommendation |
| `reasoning` | `string\|null` | Optional | Agent's justification (not graded) |

### Observation Space

| Field | Type | Description |
|---|---|---|
| `issue_id` | `string` | GitHub issue number |
| `issue_title` | `string` | Title of the issue |
| `issue_body` | `string` | Full issue body |
| `author` | `string` | Who filed the issue |
| `existing_comments` | `list[str]` | Prior comments (context) |
| `task_id` | `string` | Current difficulty: `easy` / `medium` / `hard` |
| `task_description` | `string` | What the agent must do this episode |
| `last_reward` | `float` | Reward from previous step |
| `feedback` | `string` | Grader feedback explaining the score |

---

## Tasks and Grading

### 🟢 Easy — Label Assignment
**Objective:** Assign the correct label to the issue.  
**Grader:** `1.0` if correct, `0.0` if wrong.  
**Challenge level:** Straightforward for capable LLMs.

### 🟡 Medium — Label + Team Routing
**Objective:** Assign correct label AND route to the correct engineering team.  
**Grader:** `label (0.5)` + `team (0.5)` — partial credit if one is correct.  
**Challenge level:** Requires understanding of org structure and issue context.

### 🔴 Hard — Full Triage (Label + Team + Priority + Fix)
**Objective:** Full triage decision — label, team, priority, and a concrete fix action.  
**Grader:** `label (0.30)` + `team (0.30)` + `priority (0.20)` + `fix quality (0.20)`  
**Challenge level:** Genuinely challenges frontier models on multi-criteria reasoning.

> **Reward function design:** All rewards are continuous `[0.0, 1.0]`, providing
> partial credit at every step. The fix suggestion uses keyword-overlap scoring so
> specificity is rewarded — vague answers get partial credit, precise answers get full.

---

## Baseline Scores (Llama 3.1 8B via HF Router)

Scores vary per run because issues are randomly sampled. Representative results:

| Task | Score Range | Typical |
|---|---|---|
| 🟢 Easy | 0.0 – 1.0 | **1.0** ✅ |
| 🟡 Medium | 0.5 – 1.0 | **0.75** |
| 🔴 Hard | 0.6 – 1.0 | **0.85** |
| **Average** | 0.5 – 1.0 | **~0.80** |

> Binary easy task depends on which issue is sampled. Medium/Hard benefit from
> partial credit — the model consistently scores well on label and team fields.

---

## Quick Start

### Option 1 — Use the hosted Space
```bash
curl -X POST https://om192006-github-issue-triage.hf.space/reset \
  -H "Content-Type: application/json" \
  -d '{"task_id": "easy"}'
```

### Option 2 — Run locally with Docker
```bash
git clone https://github.com/ironman1947/github-issue-triage
cd github-issue-triage

docker build -t github-issue-triage:latest .
docker run -d -p 8000:8000 github-issue-triage:latest

# Test
curl -X POST http://localhost:8000/reset -H "Content-Type: application/json" -d '{"task_id": "hard"}'
```

### Option 3 — Run inference script
```bash
pip install openenv-core openai
export HF_TOKEN=your_hf_token
export API_BASE_URL=https://router.huggingface.co/novita/v3/openai
export MODEL_NAME=meta-llama/llama-3.1-8b-instruct
export ENV_BASE_URL=https://om192006-github-issue-triage.hf.space

python inference.py
```

### Validate
```bash
pip install openenv-core
openenv validate
```

---

## Project Structure

```
github-issue-triage/
├── inference.py                          # Baseline agent (run me!)
├── models.py                             # Typed Pydantic models
├── client.py                             # Python client helper
├── openenv.yaml                          # OpenEnv spec metadata
├── Dockerfile                            # Root Dockerfile
├── README.md
├── pyproject.toml
└── server/
    ├── app.py                            # FastAPI server
    └── github_issue_triage_environment.py  # Environment + grader logic
```

---

## Real-World Motivation

GitHub issue triage is a **bottleneck in every software team**. Issues pile up
unlabelled, unassigned, with no priority. Human triagers spend hours per week on
this. This environment enables:

- **Training** LLM agents to triage automatically
- **Evaluating** how well a model understands developer context
- **Benchmarking** different models on a grounded, reproducible task

Companies like GitHub, GitLab, Linear, and Jira are actively investing in
AI-powered triage — this environment enables RL research directly applicable to that.

---

## Team

| Name | Role |
|---|---|
| Om Chougule | Team Lead · Environment Design · Backend |
| Shraman Patil | Grader Logic · Inference Script |

**Team:** Astra.AI  
**Hackathon:** Meta PyTorch OpenEnv Hackathon × Scaler School of Technology  
**GitHub:** https://github.com/ironman1947/github-issue-triage