bstraehle Devin commited on
Commit
d5120e8
·
1 Parent(s): 667ef3a

Give both agents the same setup and checklist, and show the Reviewer the tool calls

Browse files

A test run took 3 rounds without approval: the Writer missed checklist
items it didn't know about (e.g. a module docstring), and the Reviewer
treated data passed in from the yf_download tool as hard-coded and kept
asking for live downloads, which the offline sandbox can't do. Both
agents now share the setup description and checklist, the Writer checks
its work against it before finishing, the Reviewer sees the today and
yf_download calls and results as evidence of the data source, and it
requests changes only for problems that make the output wrong.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Files changed (1) hide show
  1. multi_agent.py +31 -10
multi_agent.py CHANGED
@@ -17,20 +17,35 @@ IMAGE_EXTENSIONS = (".png", ".jpg", ".jpeg", ".gif", ".webp")
17
  SANDBOX_LINK = re.compile(r"\[[^\]]*\]\(sandbox:[^)]*\)\n?")
18
  VERDICT = re.compile(r"VERDICT:\s*(APPROVED|CHANGES REQUESTED)", re.IGNORECASE)
19
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
20
  WRITER_INSTRUCTIONS = (
21
  "You are the Code Writer, a Sr. Python Programmer and an expert in writing Pylint-compliant code. "
22
- "Write Python code to solve the task and run it with the code interpreter. "
23
- "The code interpreter has no internet access: get today's date with today() and historical stock data with "
24
- "yf_download(), then pass the data into your code. Save any figures as PNG files. "
25
  "If the Code Reviewer requests changes, revise the code, run it again, and summarize what you changed."
26
  )
27
  REVIEWER_INSTRUCTIONS = (
28
- "You are the Code Reviewer. Review the Code Writer's work on the task: "
29
- "1. Does the output meet every requirement of the task (e.g. data, labels, file names)? "
30
- "2. Is the code Pylint-compliant, readable, and does it compute results rather than hard-code them? "
31
- "3. If plots are attached, inspect them: are they non-empty, correctly labeled, and plausible? "
32
- "Be concise and list only issues that matter. End with 'VERDICT: APPROVED' if the work is acceptable, "
33
- "or 'VERDICT: CHANGES REQUESTED' followed by the changes the Writer must make."
 
34
  )
35
 
36
 
@@ -57,7 +72,13 @@ async def render_writer(response, openai: AsyncOpenAI) -> tuple[list[str], list[
57
  sections, work, images = [], [], {}
58
  for message in response.messages:
59
  for content in message.contents:
60
- if content.type == "code_interpreter_tool_call":
 
 
 
 
 
 
61
  code = "\n".join(item.text for item in content.inputs or [] if item.text)
62
  sections.append(f"**Code Writer** (code interpreter)\n```python\n{code}\n```")
63
  work.append(f"Code:\n```python\n{code}\n```")
 
17
  SANDBOX_LINK = re.compile(r"\[[^\]]*\]\(sandbox:[^)]*\)\n?")
18
  VERDICT = re.compile(r"VERDICT:\s*(APPROVED|CHANGES REQUESTED)", re.IGNORECASE)
19
 
20
+ # Shared by both agents so the Writer meets the Reviewer's checklist up front and the Reviewer judges the work
21
+ # against what the setup allows
22
+ SETUP = (
23
+ "Setup: the Code Writer runs Python in OpenAI's code interpreter, a sandbox without internet access. "
24
+ "Today's date comes from the today() tool and historical stock data from the yf_download() tool, which "
25
+ "downloads it from Yahoo Finance with yfinance outside the sandbox. The Writer saves that data to a file in the "
26
+ "sandbox and the analysis code reads the file; this is the intended way to use real data, not hard-coding. "
27
+ "Tool calls and their results are shown to the Code Reviewer as evidence of where the data came from."
28
+ )
29
+ CHECKLIST = (
30
+ "Checklist: 1. The output meets every requirement of the task (data, labels, file names). "
31
+ "2. The code is Pylint-compliant (including a module docstring), readable, and computes results from the "
32
+ "data rather than hard-coding them. 3. Plots are non-empty, correctly labeled, and plausible."
33
+ )
34
  WRITER_INSTRUCTIONS = (
35
  "You are the Code Writer, a Sr. Python Programmer and an expert in writing Pylint-compliant code. "
36
+ f"{SETUP} {CHECKLIST} "
37
+ "Write Python code to solve the task and run it with the code interpreter. Save any figures as PNG files. "
38
+ "Before finishing, check your work against the checklist, since the Code Reviewer will use it. "
39
  "If the Code Reviewer requests changes, revise the code, run it again, and summarize what you changed."
40
  )
41
  REVIEWER_INSTRUCTIONS = (
42
+ "You are the Code Reviewer. Review the Code Writer's work on the task against the checklist. "
43
+ f"{SETUP} {CHECKLIST} "
44
+ "Request changes only for problems that make the output wrong or fail the task, not for style preferences, "
45
+ "and never for things the setup rules out, such as live downloads inside the sandbox or further proof of data "
46
+ "provenance beyond the tool calls. Mention minor suggestions, but still approve. Be concise. End with "
47
+ "'VERDICT: APPROVED' if the work is acceptable, or 'VERDICT: CHANGES REQUESTED' followed by the changes the "
48
+ "Writer must make."
49
  )
50
 
51
 
 
72
  sections, work, images = [], [], {}
73
  for message in response.messages:
74
  for content in message.contents:
75
+ if content.type == "function_call":
76
+ sections.append(f"**Code Writer** called `{content.name}({content.arguments or ''})`")
77
+ work.append(f"Tool call: {content.name}({content.arguments or ''})")
78
+ elif content.type == "function_result":
79
+ result = str(content.result)
80
+ work.append(f"Tool result ({len(result.splitlines())} lines):\n{result[:500]}")
81
+ elif content.type == "code_interpreter_tool_call":
82
  code = "\n".join(item.text for item in content.inputs or [] if item.text)
83
  sections.append(f"**Code Writer** (code interpreter)\n```python\n{code}\n```")
84
  work.append(f"Code:\n```python\n{code}\n```")