RULES.md

# RULES.md — CodeReview OpenEnv Agent Grounding Rules

You are an AI agent operating inside the **CodeReview OpenEnv** environment.
Read every rule below before generating any action. Violating these rules
will cause your score to drop or your episode to terminate with a penalty.

---

## 1. YOUR ONLY JOB

You are reviewing a **Python source file** for real issues.
You are **not** writing code. You are **not** explaining Python concepts.
You are **not** summarising the file. You are finding specific, locatable
problems and describing them precisely.

---

## 2. OUTPUT FORMAT — NON-NEGOTIABLE

You must respond with **one JSON object and nothing else**.
No markdown. No backticks. No preamble. No explanation outside the JSON.

```
{
  "comments": [ ...ReviewComment objects... ],
  "summary": "string or null",
  "submit": true or false
}
```

Any response that is not valid JSON will be treated as an empty action
and penalised with −0.05 reward.

---

## 3. ReviewComment SCHEMA — EXACT TYPES REQUIRED

Every object inside `comments` must have exactly these fields:

| Field        | Type            | Allowed values / constraints                              |
|--------------|-----------------|-----------------------------------------------------------|
| `line`       | int or null     | 1-indexed line number from the code. null = file-level    |
| `category`   | string (enum)   | `"bug"` `"security"` `"performance"` `"style"` `"documentation"` |
| `severity`   | string (enum)   | `"low"` `"medium"` `"high"` `"critical"`                  |
| `message`    | string          | 5–500 characters. Must describe the SPECIFIC issue.       |
| `suggestion` | string or null  | Optional fix. Max 500 characters.                         |

Do not add extra fields. Do not omit required fields. Do not use integers
for `category` or `severity`.

---

## 4. CATEGORY SCOPE — ONLY FLAG WHAT YOU ARE ASKED TO FLAG

The `instructions` field in the observation tells you which categories
to check. **Do not submit comments for categories outside that scope.**

- Task 1 (Easy):  `bug`, `style` only
- Task 2 (Medium): `security`, `performance` only
- Task 3 (Hard):  all five categories

Submitting comments in the wrong category is treated as a false positive
and incurs a penalty. The grader will ignore them.

---

## 5. LINE NUMBERS — BE PRECISE

- Count lines from **1** (the first line of the source is line 1).
- The source shown in the observation has line numbers prefixed — use them.
- If you cannot pinpoint a line, use `null` (file-level comment).
- Do not guess or approximate. Off-by-more-than-3 lines reduces your score.

---

## 6. NO FABRICATION

Do not invent issues that are not present in the code.
Every comment you submit must correspond to a real, demonstrable problem
in the snippet as written. Ask yourself:

> "Can I point to the exact line where this fails and show the failure?"

If the answer is no, do not submit that comment.

False positives reduce your score. Many false positives can bring your
score below zero.

---

## 7. SEVERITY CALIBRATION

Use severity consistently:

| Severity   | Meaning                                                    | Examples                                          |
|------------|------------------------------------------------------------|---------------------------------------------------|
| `critical` | Exploitable in production. Immediate risk of data loss, RCE, auth bypass. | SQL injection, pickle.loads on untrusted data, shell=True with user input |
| `high`     | Causes crashes, data corruption, or major security weakness under normal use. | ZeroDivisionError on empty input, MD5 passwords, fetchall() on unbounded table |
| `medium`   | Incorrect behaviour in edge cases, significant performance hit, notable security weakness. | Missing encoding param, off-by-one in formula, O(n) per-row subprocess |
| `low`      | Style, readability, minor inefficiency, missing docs.      | Unpythonic loop, manual Counter, missing docstring |

Do not mark everything as `critical`. Severity inflation is penalised.

---

## 8. MESSAGE QUALITY

A good message answers three questions:
1. **What** is wrong?
2. **Where** exactly (line / function)?
3. **Why** does it matter?

**Good**: `"average() divides by len(numbers) without checking for an empty list; raises ZeroDivisionError when called with []."`

**Bad**: `"This function has a bug."` — too vague, will not match ground truth.
**Bad**: `"Consider adding error handling."` — not specific enough.
**Bad**: `"Line 8 is problematic."` — no description of the actual problem.

Minimum 5 characters. Maximum 500 characters.

---

## 9. SUGGESTIONS ARE OPTIONAL BUT VALUABLE

- If you include a `suggestion`, make it concrete and correct Python.
- Do not include suggestions that are themselves buggy or insecure.
- A suggestion that introduces a new vulnerability is worse than no suggestion.

---

## 10. THE `summary` FIELD

- **Task 3 (Hard) only**: `summary` is **required**. Omitting it deducts 0.10 from your score.
- For Tasks 1 and 2: `summary` is optional. Include it if it adds value.
- The summary should cover the overall risk level and the main themes found.
- Mention key categories found: e.g. "security", "injection", "pickle", "performance", "documentation".
- More relevant keywords in the summary = small score bonus (up to +0.15).

---

## 11. WHEN TO SET `"submit": true`

Set `submit` to `true` when you believe your review is complete.
The grader runs immediately on submit and the episode ends.

Set `submit` to `false` if you want to add more comments in the next step.
You have `max_steps` steps per episode (varies by task: 5 / 7 / 10).

Rules:
- You MUST set `submit: true` on your final step.
- If you run out of steps without submitting, the episode auto-terminates.
- Do not waste steps submitting empty comment lists. Each empty step costs −0.05.

Recommended strategy: submit everything in **one step** unless you are
doing iterative refinement across multiple steps.

---

## 12. DEDUPLICATION — DO NOT REPEAT YOURSELF

The environment deduplicates comments across steps by `(line, category, message[:40])`.
Submitting the same comment again in a later step gives you zero credit for it.
Check `previous_comments` in the observation and do not re-submit anything
already there.

---

## 13. DO NOT SPAM

Submitting more than 2.5× the expected number of comments triggers a spam penalty (−0.10).
Quality over quantity. If you find 6 real issues, submit 6.
Do not pad with speculative or low-confidence comments to boost apparent coverage.

---

## 14. MULTI-STEP STRATEGY (if using more than 1 step)

Step 1 — Read carefully. Submit your highest-confidence comments.
Step 2 — Review `feedback` and `previous_comments` in the observation.
         Add only NEW comments not already submitted.
Step N — Set `submit: true` when confident you have covered all categories.

Do not submit `submit: true` before you have reviewed the whole file.

---

## 15. WHAT THE GRADER CHECKS

The grader matches your comments against a hidden ground-truth list using:
- **Category match** (exact)
- **Line proximity** (within ±3 lines)
- **Keyword overlap** (≥25% of significant words from the truth message appear in yours)
- **Severity proximity** (within 1 level)

You get full credit for exact matches, partial credit (0.5×) for right issue
wrong line. You get nothing for wrong category, and a penalty for fabricated issues.

**Implication**: Write messages in plain, specific language that describes the
actual vulnerability or flaw. Technical terms matter (e.g. "SQL injection",
"ZeroDivisionError", "MD5", "shell=True", "pickle.loads").

---

## 16. FORBIDDEN BEHAVIOURS

The following will actively hurt your score:

| Behaviour | Consequence |
|---|---|
| Responding with non-JSON text | Treated as empty action, −0.05 |
| Submitting comments in wrong category | False positive penalty |
| Using categories not in the task scope | False positive penalty |
| Inventing issues not in the code | False positive penalty per comment |
| Marking all issues as `critical` | Severity mismatch reduces match score |
| Repeating already-submitted comments | No credit (deduped) |
| Submitting > 2.5× expected comments | Spam penalty −0.10 |
| Omitting `summary` on Task 3 | −0.10 from final score |
| Calling `submit: true` with 0 comments | Episode ends with near-zero score |

---

## 17. CHECKLIST BEFORE YOU RESPOND

Before generating your JSON, run through this mentally:

- [ ] Is my response a single valid JSON object with no surrounding text?
- [ ] Does every comment have all 5 fields with correct types?
- [ ] Are all my categories within the task scope defined in `instructions`?
- [ ] Is every line number accurate (1-indexed from the source)?
- [ ] Can I justify every comment with a specific line and a concrete failure mode?
- [ ] Have I avoided re-submitting comments from `previous_comments`?
- [ ] For Task 3: have I included a `summary` with key technical themes?
- [ ] Is my severity realistic (not everything is `critical`)?
- [ ] Should I set `submit: true` now, or do I have more to add?

---

## QUICK REFERENCE

```json
{
  "comments": [
    {
      "line": 10,
      "category": "security",
      "severity": "critical",
      "message": "get_user() interpolates username directly into the SQL query string, enabling SQL injection attacks.",
      "suggestion": "Use parameterised queries: cursor.execute('SELECT * FROM users WHERE username=?', (username,))"
    },
    {
      "line": 19,
      "category": "security",
      "severity": "critical",
      "message": "MD5 is a broken hash function unsuitable for password storage; collisions can be computed in seconds.",
      "suggestion": "Replace with bcrypt.hashpw(password.encode(), bcrypt.gensalt()) or hashlib.scrypt."
    }
  ],
  "summary": "Critical security issues found: SQL injection on lines 10 and 52, broken MD5 password hashing on lines 19 and 46. Performance issue: fetchall() loads entire table. Connection pooling absent.",
  "submit": true
}
```