docs/REWARD_DESIGN.md

# REWARD_DESIGN.md — Bug Triage Reward & Grader Design

---

## Two Systems: Reward vs Grader

| | Step Reward | Grader Score |
|-|------------|-------------|
| Purpose | GRPO training signal | Hackathon evaluation |
| When | Every `step()` | Only at `done=True` |
| Range | [-0.5, 1.0] (shaped) | [0.0, 1.0] (strict) |
| Endpoint | Via observation `reward` | `/grader` endpoint |

**Mapping:** `reward = (grader_score × 1.5) − 0.5`
- Perfect score (1.0) → reward = +1.0
- Zero score (0.0) → reward = -0.5
- This shaping gives gradient signal even for bad predictions.

---

## Task 1 Reward: Bug Type Classification

| Agent Output | Score |
|-------------|-------|
| Correct type | **+1.0** |
| Wrong type | **0.0** |

Simple exact match. 6 possible types, so random baseline ≈ 0.167.

---

## Task 2 Reward: Priority Assignment

| Distance from Correct | Score |
|-----------------------|-------|
| Exact match | **1.00** |
| Off by 1 level | **0.67** |
| Off by 2 levels | **0.33** |
| Off by 3 levels | **0.00** |

**Why partial credit?** Priority has ordinal structure — predicting "high" when answer is "critical" is better than predicting "low".

---

## Task 3 Reward: Full Triage (Composite)

| Component | Weight | Scoring |
|-----------|--------|---------|
| Bug type correct | **0.30** | 1.0 exact, 0.0 else |
| Priority correct | **0.30** | Distance: 0→1.0, 1→0.67, 2→0.33, 3→0.0 |
| Developer correct | **0.20** | Exact→1.0, right specialty→0.5, else→0.0 |
| Action correct | **0.20** | Exact→1.0, adjacent→0.5, else→0.0 |

### Partial Credit Details

**Developer assignment:**
- Exact match to ground truth → 1.0
- Wrong person but has the right specialization for the bug type → 0.5
- No match → 0.0

**Action suggestion (adjacency map):**
```
fix_immediately  schedule_sprint     (0.5)
schedule_sprint  needs_more_info     (0.5)
wontfix  duplicate                   (0.5)
```

### Example Scoring
```
Ground truth: type=crash, priority=critical, dev=Alice, action=fix_immediately
Agent output: type=crash, priority=high,     dev=Bob,   action=fix_immediately

type:     crash == crash       → 1.0 × 0.30 = 0.30
priority: high vs critical     → 0.67 × 0.30 = 0.20
dev:      Bob vs Alice         → Bob knows crash → 0.5 × 0.20 = 0.10
action:   fix_immediately      → 1.0 × 0.20 = 0.20

Total: 0.80
```

---

## Why This Reward Design Is Meaningful

1. **Non-trivial** — random agent scores ~0.20 on Task 3; a good model should score 0.7+
2. **Decomposed** — each dimension provides independent learning signal
3. **Partial credit** — prevents plateau; agent improves incrementally
4. **Varies per bug** — different bugs have different answers, so grader scores vary (no fixed output)
5. **Production-relevant** — mirrors how humans evaluate triage quality

---

## Grader Code Locations

| Task | Grader File | Method |
|------|-------------|--------|
| task_1 | `graders/task1_grader.py` | `grade(episode_log, ground_truth) → float` |
| task_2 | `graders/task2_grader.py` | `grade(episode_log, ground_truth) → float` |
| task_3 | `graders/task3_grader.py` | `grade(episode_log, ground_truth) → float` |