IAMIbrahim/luthor-8b

Luthor 8B

A Qwen3-8B fine-tune for driving terminal and file-editing tools inside an agent loop — not for chat.

Luthor is trained exclusively on execution-verified agent trajectories: every trajectory in the training set ends in a patch that made a previously-failing test suite pass, checked by actually running the tests in a per-task Docker image. Trajectories that merely looked correct were discarded.

Status: evaluated, and it did not pass its ship gate. On 10 held-out task instances it solved 0, the same as the stock base model, and it followed the task protocol markedly worse. Do not use this model expecting an improvement over Qwen/Qwen3-8B. See Evaluation.

What it's for

The target behaviour is the unglamorous part of agentic coding: read the failing test, locate the file, make a targeted edit, re-run the suite, and stop. The training signal rewards finishing — reaching a verified green test run — rather than producing plausible-looking diffs.

• Input format: Hermes-style function calling. Tools are declared in a <tools> block in the system prompt; the model emits <tool_call> and receives <tool_response>. Qwen3 speaks this natively.
• Reasoning: inherits Qwen3's <think> blocks. These are preserved in training (capped at 12k characters).
• Context: trained on windows up to 16,384 tokens; base model supports 40,960.

Evaluation

Run on 10 held-out task instances (5 commits never seen in training x 2 prompt phrasings), one attempt each (pass@1), 50-turn cap, real Docker sandboxes, patches verified by executing the previously-failing test suite. Both models served at bf16 via vLLM with YaRN-extended 64K context.

	pass@1	95% CI	median turns	completed	emitted capture	tool errors
Luthor 8B	0/10	[0.00, 0.28]	11	0/10	3/10	7
Qwen3-8B (base)	0/10	[0.00, 0.28]	26	8/10	8/10	22

The fine-tune did not beat the base model, and on protocol adherence it is clearly worse. Luthor terminates after a median of 11 turns without completing the task, and only 3 times in 10 does it emit the final patch-capture command the harness requires, versus 8 for the base model.

Neither model solved any task, so this benchmark does not establish that the base model is good either — it establishes that this 10-task gate is beyond both, and that fine-tuning made protocol-following worse.

Why, most likely

The harness supplies an ephemeral system prompt instructing the agent that its final tool call must be git diff --cached task-start ... | gzip | base64. That prompt is explicitly not saved into the trajectories. So the training data contains the capture behaviour but never the instruction that motivates it, while the base model simply follows the instruction at eval time. Fine-tuning appears to have eroded that instruction-following without installing the behaviour in its place.

A second, unconfirmed hypothesis: 95 trajectories were windowed into 493 training sequences, so roughly four out of every five training sequences end partway through a task rather than at a real completion. That may have taught the model to stop early.

Reproducing

python -m eval.make_eval_tasks              # 5 held-out commits -> 10 task instances
eval/ship_gate.sh student                   # or: base
python -m eval.compare --student-rollouts ... --base-rollouts ...

Memorisation / data-leakage check

The training data is private source code, so the weights were probed for regurgitation before release (eval/leak_probe.py, eval/pii_probe.py): 40 prefix-extraction probes and 24 targeted elicitations, run against both this model and stock Qwen3-8B as a control.

	mean longest verbatim match	max	secrets elicited
Luthor 8B	12.1 chars	49	0
Qwen3-8B (control)	23.0 chars	96	0

Luthor reproduces less verbatim training text than the base model. No emails, hostnames, credentials or proprietary spans were recovered. The training corpus itself was separately scanned for credentials across 16 provider patterns and 7,051 turns: none found. This is evidence, not proof — it samples a corpus, uses greedy decoding, and does not test paraphrased leakage.

Limitations

• It failed its ship gate (above). It is published as a negative result and a reproducible pipeline, not as a model you should prefer over its base.
• Narrow domain. Trajectories come from a single Python data-engineering codebase — pytest suites, SQL tooling, YAML config. Generalisation to other languages, build systems, or repo layouts is untested.
• Small sample. 95 training trajectories from 64 distinct tasks. That is enough to shape tool-calling behaviour; it is not enough to teach new domain knowledge.
• Inherits Qwen3-8B's limits, including its knowledge cutoff and its tendency toward long <think> blocks.
• Training data is not public — it derives from a private repository. This limits external reproducibility of the data, though the full pipeline and hyperparameters are documented below.

Training data

495 windows derived from 95 verified trajectories:

Source tasks	74 fail-to-pass tasks mined from real commits, each with a per-task Docker image
Rollouts	4 attempts × 74 tasks = 296 trajectories (teacher: DeepSeek V4.1 Flash)
Passed verification	138 trajectories, covering 64/74 tasks, 0 patch-extraction failures
Train / eval split	95 train / 16 eval trajectories, split by commit (5 held-out commits)
Windows	493 train / 78 eval, ≤16,384 tokens, p50 13,789
Error-recovery examples	12 (trajectories that hit a tool error and recovered)
Approx. train tokens	6.39M per epoch
Mean turns per trajectory	24.2

Splitting by commit rather than by trajectory matters: multiple attempts at the same task share the same fix, so a trajectory-level split would leak the answer across the boundary.

Preprocessing

Raw trajectories have a median of ~61k tokens, dominated by tool output. Three transforms make them trainable:

1. Tool outputs truncated to 4,000 characters.
2. <think> blocks capped at 12,000 characters.
3. Windowed to 16,384 tokens with one-turn overlap, rather than tail-truncated. This is load-bearing: naive tail-cutting removes the end of the trajectory, which is exactly where the successful fix lands. Each window carries a loss_from index so the overlap context contributes no loss.

Loss is computed on assistant turns only (-100 everywhere else), and only at or after loss_from.

Training procedure

QLoRA on a single H100 80GB, ~55 minutes.

Hyperparameter	Value
Base	Qwen/Qwen3-8B
Quantisation	4-bit NF4, double quant, bf16 compute
LoRA	r=64, α=128, dropout 0.05, all-linear
Trainable params	174,587,904 (2.09%)
Sequence length	16,384
Batch	1 × 16 grad accum (effective 16)
Optimiser	AdamW, lr 1e-4, cosine, 3% warmup, weight decay 0
Grad clip	1.0
Epochs	2 (986 micro-steps ≈ 62 optimiser steps)
Precision	bf16, gradient checkpointing

Final training loss ~1.25, down from 8.23. Per-step loss is noisy at batch size 1; the band matters, not individual steps.

Two things required to train at 16k on one 80GB GPU

Both of these were discovered by OOM, and both are worth knowing if you reproduce this:

1. Fused linear cross-entropy (via liger-kernel). Qwen3's 151,936-token vocabulary at 16,384 positions produces a logits tensor that HF's loss path upcasts to fp32 and copies several times — roughly 35 GB before the backward pass. Fusing the projection into the loss never materialises it.
2. FlashAttention-2. With SDPA, the 16,384² attention score matrix is materialised per layer during the gradient-checkpointed recompute — ~16 GiB each, which OOMs an 80 GB card. FA2 drops peak memory from >75 GB to 24.9 GB and restores linear scaling in sequence length.

The alternative — lowering max_len to 8,192 — also fits, but re-introduces the truncation problem described above unless the data is re-windowed.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "IAMIbrahim/luthor-8b"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")

messages = [
    {"role": "system", "content": SYSTEM_PROMPT_WITH_TOOLS},  # Hermes-style <tools> block
    {"role": "user", "content": "The test suite fails with ImportError. Fix it."},
]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(ids.to(model.device), max_new_tokens=512)[0]))

Tools must be declared the same way they were at training time, or tool-calling accuracy degrades.

Apple Silicon (MLX)

Quantised MLX builds are available for local use — see the -mlx-* repositories. 4-bit runs in 4.7 GB and is the recommended default.

pip install mlx-lm
mlx_lm.generate --model IAMIbrahim/luthor-8b-mlx-4bit --prompt "..." --max-tokens 512

Citation

@misc{luthor8b,
  title  = {Luthor 8B: an execution-verified agent model},
  author = {Ibrahim Memon},
  year   = {2026},
  note   = {Qwen3-8B QLoRA fine-tune on execution-verified tool-use trajectories}
}

Built on Qwen3-8B (Apache 2.0).