RULES.md

RULES.md — CodeReview OpenEnv Agent Grounding Rules

You are an AI agent operating inside the CodeReview OpenEnv environment. Read every rule below before generating any action. Violating these rules will cause your score to drop or your episode to terminate with a penalty.

---

1. YOUR ONLY JOB

You are reviewing a Python source file for real issues. You are not writing code. You are not explaining Python concepts. You are not summarising the file. You are finding specific, locatable problems and describing them precisely.

---

2. OUTPUT FORMAT — NON-NEGOTIABLE

You must respond with one JSON object and nothing else. No markdown. No backticks. No preamble. No explanation outside the JSON.

{
  "comments": [ ...ReviewComment objects... ],
  "summary": "string or null",
  "submit": true or false
}

Any response that is not valid JSON will be treated as an empty action and penalised with −0.05 reward.

---

3. ReviewComment SCHEMA — EXACT TYPES REQUIRED

Every object inside comments must have exactly these fields:

Field	Type	Allowed values / constraints
line	int or null	1-indexed line number from the code. null = file-level
category	string (enum)	"bug" "security" "performance" "style" "documentation"
severity	string (enum)	"low" "medium" "high" "critical"
message	string	5–500 characters. Must describe the SPECIFIC issue.
suggestion	string or null	Optional fix. Max 500 characters.

Do not add extra fields. Do not omit required fields. Do not use integers for category or severity.

---

4. CATEGORY SCOPE — ONLY FLAG WHAT YOU ARE ASKED TO FLAG

The instructions field in the observation tells you which categories to check. Do not submit comments for categories outside that scope.

• Task 1 (Easy): bug, style only
• Task 2 (Medium): security, performance only
• Task 3 (Hard): all five categories

Submitting comments in the wrong category is treated as a false positive and incurs a penalty. The grader will ignore them.

---

5. LINE NUMBERS — BE PRECISE

• Count lines from 1 (the first line of the source is line 1).
• The source shown in the observation has line numbers prefixed — use them.
• If you cannot pinpoint a line, use null (file-level comment).
• Do not guess or approximate. Off-by-more-than-3 lines reduces your score.

---

6. NO FABRICATION

Do not invent issues that are not present in the code. Every comment you submit must correspond to a real, demonstrable problem in the snippet as written. Ask yourself:

"Can I point to the exact line where this fails and show the failure?"

If the answer is no, do not submit that comment.

False positives reduce your score. Many false positives can bring your score below zero.

---

7. SEVERITY CALIBRATION

Use severity consistently:

Severity	Meaning	Examples
critical	Exploitable in production. Immediate risk of data loss, RCE, auth bypass.	SQL injection, pickle.loads on untrusted data, shell=True with user input
high	Causes crashes, data corruption, or major security weakness under normal use.	ZeroDivisionError on empty input, MD5 passwords, fetchall() on unbounded table
medium	Incorrect behaviour in edge cases, significant performance hit, notable security weakness.	Missing encoding param, off-by-one in formula, O(n) per-row subprocess
low	Style, readability, minor inefficiency, missing docs.	Unpythonic loop, manual Counter, missing docstring

Do not mark everything as critical. Severity inflation is penalised.

---

8. MESSAGE QUALITY

A good message answers three questions:

1. What is wrong?
2. Where exactly (line / function)?
3. Why does it matter?

Good: "average() divides by len(numbers) without checking for an empty list; raises ZeroDivisionError when called with []."

Bad: "This function has a bug." — too vague, will not match ground truth. Bad: "Consider adding error handling." — not specific enough. Bad: "Line 8 is problematic." — no description of the actual problem.

Minimum 5 characters. Maximum 500 characters.

---

9. SUGGESTIONS ARE OPTIONAL BUT VALUABLE

• If you include a suggestion, make it concrete and correct Python.
• Do not include suggestions that are themselves buggy or insecure.
• A suggestion that introduces a new vulnerability is worse than no suggestion.

---

10. THE summary FIELD

• Task 3 (Hard) only: summary is required. Omitting it deducts 0.10 from your score.
• For Tasks 1 and 2: summary is optional. Include it if it adds value.
• The summary should cover the overall risk level and the main themes found.
• Mention key categories found: e.g. "security", "injection", "pickle", "performance", "documentation".
• More relevant keywords in the summary = small score bonus (up to +0.15).

---

11. WHEN TO SET "submit": true

Set submit to true when you believe your review is complete. The grader runs immediately on submit and the episode ends.

Set submit to false if you want to add more comments in the next step. You have max_steps steps per episode (varies by task: 5 / 7 / 10).

Rules:

• You MUST set submit: true on your final step.
• If you run out of steps without submitting, the episode auto-terminates.
• Do not waste steps submitting empty comment lists. Each empty step costs −0.05.

Recommended strategy: submit everything in one step unless you are doing iterative refinement across multiple steps.

---

12. DEDUPLICATION — DO NOT REPEAT YOURSELF

The environment deduplicates comments across steps by (line, category, message[:40]). Submitting the same comment again in a later step gives you zero credit for it. Check previous_comments in the observation and do not re-submit anything already there.

---

13. DO NOT SPAM

Submitting more than 2.5× the expected number of comments triggers a spam penalty (−0.10). Quality over quantity. If you find 6 real issues, submit 6. Do not pad with speculative or low-confidence comments to boost apparent coverage.

---

14. MULTI-STEP STRATEGY (if using more than 1 step)

Step 1 — Read carefully. Submit your highest-confidence comments. Step 2 — Review feedback and previous_comments in the observation. Add only NEW comments not already submitted. Step N — Set submit: true when confident you have covered all categories.

Do not submit submit: true before you have reviewed the whole file.

---

15. WHAT THE GRADER CHECKS

The grader matches your comments against a hidden ground-truth list using:

• Category match (exact)
• Line proximity (within ±3 lines)
• Keyword overlap (≥25% of significant words from the truth message appear in yours)
• Severity proximity (within 1 level)

You get full credit for exact matches, partial credit (0.5×) for right issue wrong line. You get nothing for wrong category, and a penalty for fabricated issues.

Implication: Write messages in plain, specific language that describes the actual vulnerability or flaw. Technical terms matter (e.g. "SQL injection", "ZeroDivisionError", "MD5", "shell=True", "pickle.loads").

---

16. FORBIDDEN BEHAVIOURS

The following will actively hurt your score:

Behaviour	Consequence
Responding with non-JSON text	Treated as empty action, −0.05
Submitting comments in wrong category	False positive penalty
Using categories not in the task scope	False positive penalty
Inventing issues not in the code	False positive penalty per comment
Marking all issues as critical	Severity mismatch reduces match score
Repeating already-submitted comments	No credit (deduped)
Submitting > 2.5× expected comments	Spam penalty −0.10
Omitting summary on Task 3	−0.10 from final score
Calling submit: true with 0 comments	Episode ends with near-zero score

---

17. CHECKLIST BEFORE YOU RESPOND

Before generating your JSON, run through this mentally:

• Is my response a single valid JSON object with no surrounding text?
• Does every comment have all 5 fields with correct types?
• Are all my categories within the task scope defined in instructions?
• Is every line number accurate (1-indexed from the source)?
• Can I justify every comment with a specific line and a concrete failure mode?
• Have I avoided re-submitting comments from previous_comments?
• For Task 3: have I included a summary with key technical themes?
• Is my severity realistic (not everything is critical)?
• Should I set submit: true now, or do I have more to add?

---

QUICK REFERENCE

{
  "comments": [
    {
      "line": 10,
      "category": "security",
      "severity": "critical",
      "message": "get_user() interpolates username directly into the SQL query string, enabling SQL injection attacks.",
      "suggestion": "Use parameterised queries: cursor.execute('SELECT * FROM users WHERE username=?', (username,))"
    },
    {
      "line": 19,
      "category": "security",
      "severity": "critical",
      "message": "MD5 is a broken hash function unsuitable for password storage; collisions can be computed in seconds.",
      "suggestion": "Replace with bcrypt.hashpw(password.encode(), bcrypt.gensalt()) or hashlib.scrypt."
    }
  ],
  "summary": "Critical security issues found: SQL injection on lines 10 and 52, broken MD5 password hashing on lines 19 and 46. Performance issue: fetchall() loads entire table. Connection pooling absent.",
  "submit": true
}