reports/real_llm_debug_log.md

Real LLM Benchmark Debugging Log

Problem: 0% Pass Rate on HumanEval with Qwen 0.5B/1.5B

We ran 8 versions of the real LLM code benchmark. Here's what we learned.

V1–V3: Naive approach

• Prompt: HumanEval prompt + generate completion
• Extraction: take generated tokens after prompt
• Result: 0/20 pass
• Reason: Model outputs complete function (including def <entry_point>), but we prepended HumanEval prompt which also has def <entry_point> → duplicate definitions → syntax error

V4: Chat template + body extraction attempt

• Added chat template (system/assistant)
• Attempted to extract just function body after def line
• Result: 0/20 pass
• Reason: Markdown fences still present; AST parsing too strict; body extraction failed on docstrings

V5–V6: Multiple extraction strategies + debug

• Added multiple candidates: body-only, stripped, raw
• Used ast.parse() validation
• Result: 0/20 pass
• Reason: Still prepending prompt even when model output contains full function. Markdown fences not fully stripped.

V7: Regex-based extraction + larger model

• Regex \n(.*?)\n for code block extraction
• Larger model: Qwen 1.5B (vs 0.5B)
• 512 tokens
• Result: 0/20 pass (from partial logs)
• Critical finding: Error changed! TypeError: check() missing 1 required positional argument: 'candidate'
• Root cause: evalplus tests ALREADY contain check(candidate) calls — we were appending check() without args!

V8: THE FIX

• Do NOT append check() — evalplus test code already has it
• Changed system prompt to: "Write ONLY the function definition. No markdown, no extra text."
• Still using regex-based markdown extraction
• Model: Qwen 1.5B
• Status: Submitted on a10g-small, awaiting results

Key Lessons

1. Always inspect the test file format. evalplus has check(candidate) built in — different from standard HumanEval.
2. Markdown extraction must be robust. Models output \n\``python\n...\n````. Use regex, not simple string splitting.
3. Model size matters. 0.5B may not reliably generate valid Python; 1.5B+ recommended.
4. Debug by printing the ACTUAL test file. We should have done this in V1.