qwen3-4b-debug-mix-lora

A LoRA adapter for Qwen/Qwen3-4B-Instruct-2507 that repairs buggy Python programs. Give it the program, how it fails, its stdin and (optionally) a FocusTracer dynamic slice; it answers with the whole fixed program in one ```python block.

pass@1: 76.2 % on runtime-exception bugs (base: 67.6 %), 75.6 % overall on 946 held-out programs (untuned base model: 72.7 % overall). Ready-to-run GGUF: Pilestin/qwen3-4b-debug-mix-GGUF.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "Qwen/Qwen3-4B-Instruct-2507"
tok = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, "Pilestin/qwen3-4b-debug-mix-lora")

SYSTEM_PROMPT = ('You are an expert Python debugging assistant. The user gives you a buggy Python program (and optionally its runtime error and an execution trace). Reply with the corrected program in a single ```python code block. Do not add explanations.')
buggy = """n = int(input())
print(sum(range(n))"""
user = ("Fix the following buggy Python program.\n\n```python\n" + buggy + "\n```\n\n"
        "Runtime error:\nSyntaxError: '(' was never closed")
msgs = [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=1024, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

model.merge_and_unload() merges the adapter into the base weights.

Prompt format

System prompt (use it verbatim):

You are an expert Python debugging assistant. The user gives you a buggy Python program (and optionally its runtime error and an execution trace). Reply with the corrected program in a single ```python code block. Do not add explanations.

User message. v2 (always) is the program, the failure description and the stdin; v4 appends a FocusTracer backward dynamic slice when the program raises an exception:

Fix the following buggy Python program.

```python
<buggy program>
```

Runtime error:
<exception message + last 4 lines of stderr>
   (if the program does not crash, this section is one of:
    "The program runs without crashing but produces WRONG OUTPUT for the given input."
    "The program is too slow / never terminates (time limit exceeded).")

Program input (stdin):
```
<stdin, only if ≤ 400 chars>
```
                                                         ── v2 ends here
Dynamic slice (FocusTracer, statements that led to the failure):
```
Slice criterion: exception@main.py:1
Exception: IndexError: string index out of range

Backward dynamic slice (execution order) — statements that produced this value, with the runtime values each one read:
  L1 <module>: print("A" + input()[8] + "C")  <-- CRITERION
```
                                                         ── v4 ends here

The slice is the output of FocusTracer's build_slice_context (trace recorded with detail="detailed", sliced at the exception); memory addresses are stripped and it is capped at the 30 steps nearest the crash / 2,000 characters. This is the same context FocusTracer hands to an LLM in its debugging pipeline, so the model can be dropped in there.

The reply is always the whole fixed program in one ```python block.

Evaluation

946 held-out CodeNetPy bug-fix pairs (306 problems, none seen in training). pass@1 is execution-based: the generated program is run on the problem input and its stdout is compared with the reference output. Greedy decoding (temperature 0), GGUF through Ollama on a Kaggle T4, num_ctx 4096, num_predict 1024. Paired McNemar exact tests.

Model (GGUF, Ollama)	Prompt	pass@1	runtime exception	wrong answer	time limit	exact	median latency (T4)
this model, Q4_K_M	v4	75.6 %	76.2 %	73.6 %	76.4 %	24.4 %	2.8 s
this model, Q4_K_M	v2	74.7 %	74.7 %	72.9 %	76.4 %	24.3 %	2.7 s
this model, Q5_K_M	v2	74.1 %	77.5 %	71.7 %	71.4 %	25.6 %	2.9 s
Qwen3-4B-Instruct-2507 (base)	v4	72.7 %	67.6 %	80.7 %	72.1 %	10.9 %	3.5 s
Qwen3-4B-Instruct-2507 (base)	v2	73.6 %	68.6 %	79.2 %	75.0 %	12.6 %	3.1 s
Qwen2.5-Coder-7B fine-tuned (sibling)	v4	79.4 %	79.2 %	77.0 %	81.8 %	26.5 %	5.3 s
Qwen2.5-3B fine-tuned (study 3)	v2	71.9 %	56.7 %	85.1 %	80.4 %	17.8 %	2.3 s

Test categories: runtime exception n = 395, wrong answer n = 269, time limit n = 280.

Paired McNemar tests against the base model (same prompt):

Comparison	overall	runtime exception	wrong answer	time limit
base → fine-tuned, v4	+144 / −117, p = 0.11	67.6 → 76.2 %, p = 0.0001	80.7 → 73.6 %, p = 0.053	72.1 → 76.4 %, p = 0.27
base → fine-tuned, v2	+118 / −107, p = 0.51	68.6 → 74.7 %, p = 0.002	79.2 → 72.9 %, p = 0.082	75.0 → 76.4 %, p = 0.74
fine-tuned v2 → v4 (FocusTracer slice)	+14 / −6, p = 0.12	74.7 → 76.2 %, p = 0.15	—	—

What this means

The fine-tune is a runtime-exception specialist. On crashing programs it fixes +8.6 points more than the base model (p = 0.0001), and its fixes are much closer to the reference (exact match 11 % → 24 %).
It is not better overall. It loses ground on wrong-answer bugs, so total pass@1 is only +2.9 points over the base (not significant). For programs that run but print the wrong result, the base model is the better choice.
The FocusTracer slice helps a little (+1.5 points on crashes), not significantly.
Compared with the 7B sibling it is ~2× faster but 3.8 points lower overall (p = 0.009).

Note: this harness is more generous than the one used for the earlier Qwen2.5-3B study (longer generation budget, Python 3.13), so every baseline above was re-measured here rather than copied.

Training data

The training data is not published. Summary:

13,682 train / 628 val chat examples (problem-disjoint), plus a held-out test set of 946 CodeNetPy pairs from 306 problems that never appear in training.
RunBugRun (Python subset, CodeNet-derived buggy → accepted pairs with test inputs): 178,797 candidates → test problems removed, length ≤ 4,000 chars, diff ≤ 30 % of lines, ≤ 20 pairs per problem → every pair re-executed: the fix must pass the first 8 tests and the buggy version must fail at least one. 12,359 pairs survived (11,799 used in train).
CodeNetPy (study-3 train split, 1,883 pairs).
Categories (train): wrong answer 9,674 · runtime exception 3,323 · time limit 685.
mix variant: 50 % of runtime-exception rows (1,655) carry the v4 FocusTracer slice, the rest use v2 — so one model serves both prompt styles.

Training procedure

Unsloth + HF Trainer, completion-only loss (prompt tokens masked), on one Kaggle T4.

Setting	Value
Base model	Qwen/Qwen3-4B-Instruct-2507 (16-bit LoRA)
LoRA	r = 32, alpha = 32, rsLoRA, dropout 0, all attention + MLP projections
Max sequence length	3,072
Schedule	2 epochs planned (1,710 steps); best val-loss checkpoint (step 800) kept
Effective batch	16 (2 × 8 grad. accumulation)
Optimizer / LR	AdamW 8-bit, 1e-4, cosine, 3 % warmup
Wall clock	8.6 h
Best val loss	0.0578

Limitations

Trained on short stdin/stdout competitive-programming programs. It is not a general coding assistant and was not tested on multi-file or library code.
It can return the input unchanged or a plausible but wrong fix — run the output.
English prompts only; use the exact system prompt and user template above.
Benchmarks were measured on the GGUF builds via Ollama, not on the bf16 adapter.