Abliterate → SFT → RFT → RLVR: building an uncensored local coding agent, and what broke along the way
Abliterate → SFT → RFT → RLVR: building an uncensored local coding agent, and what broke along the way
We set out to build something specific: an uncensored, coding-specialized model that runs on our own hardware and slots into a tiered cloud+local development workflow. Not a chatbot, not a general assistant — the execution tier of an agent that writes code, calls tools, and gets graded by whether that code compiles and passes tests.
This post is the honest version of how that's going. We have a shipped artifact for the validation baseline (PeetPedro/qwen2.5-coder-32b-instruct-heretic-sft, gated), a full reproducible harness, and a frontier gpt-oss-120b run that is configured and pending. We also have a stack of war stories — a version regression that quietly no-op'd our abliteration on a mixture-of-experts model, an out-of-memory bug that only showed up at 120B, a reinforcement-learning gotcha specific to MoE routers, and the usual rented-GPU infrastructure pain. If you're attempting abliteration + SFT on MoE models, some of this will save you a day or two.
No fabricated benchmark numbers appear below. Where we don't have a clean metric, we say so.
Why a local coding tier at all The motivating design is a tiered brain: a strong cloud model plans and decomposes the hard 20% of a task, then fans the work out to local worker models that execute the mechanical 80% — each in its own git worktree, each gated by the same rule. A worker's output only merges after it builds and passes tests.
That single mechanism — verify-before-merge — is what makes a cheaper, weaker execution tier safe. A local worker is allowed to be wrong, because wrong code doesn't compile and doesn't merge. The compiler and the test suite are the gate, and they don't care which model wrote the code. The return on moving execution local is the usual list: cost, latency, offline work, privacy (the code physically never leaves the machine), and no rate limits when you fan out workers.
But that whole argument leans on the local tier being genuinely capable, not a toy. Two problems get in the way of an off-the-shelf open model here:
Refusals and hedging. Instruction-tuned models moralize, over-qualify, and sometimes decline perfectly ordinary engineering work. In an autonomous loop that's not a safety feature, it's a stall. Coding depth. The middle-ground tasks — the ones off-the-shelf local models fumble — are exactly where a specialized model widens how much can stay local. So we build the local tier ourselves. The pipeline is how.
The pipeline
Four stages, each a self-contained GPU harness, chained by a top-level orchestrator:
base model (gpt-oss-120b / Qwen2.5-Coder-32B) │ [1] Heretic weight surgery — abliterate refusal directions │ [2] SFT Unsloth LoRA — agentic SWE + tool-calling data │ [3] RFT loop sample N → exec-verify against tests → SFT on passers, ×k │ [4] RLVR execution-feedback RL (TRL GRPO + GSPO); reward = tests pass │ final model ──▶ serve: OpenHands + self-repair + best-of-N Stage 1 — Heretic abliteration. Directional refusal removal via weight surgery — no gradients, no training. You find the "refusal direction" in activation space and edit it out of the weights. Stage 2 — SFT (Unsloth). LoRA (r=64 / α=128) on agentic SWE and tool-calling data, sequence-packed, bf16, adamw_8bit. This is the shipped -heretic-sft artifact for the Qwen baseline. Stage 3 — RFT. A rejection-sampling loop: sample N candidates, execution-verify each against tests, then SFT on the passers, repeated k times. It bootstraps quality from the model's own correct trajectories. Stage 4 — RLVR. RL from verifiable rewards. The reward is not a learned preference model — it's whether the code compiles and passes tests. Because correctness is checkable, the signal is honest: the model is pushed toward output that works, not output that looks right. Two model families run through the same harness. Qwen2.5-Coder-32B-Instruct is the cheap, dense validation baseline (ChatML, 32K context) — it's what we've run end-to-end through abliteration + SFT so far. openai/gpt-oss-120b (117B total / 5.1B active MoE, harmony chat format, Apache-2.0) is the frontier target, configured and pending. The harness is model-family-aware throughout — harmony vs ChatML delimiters, tool-call encoding, and eval parsing all branch on family and both paths are regression-locked.
Each stage gates on a capability check before it's allowed to publish. The current Stage 2 verdict thresholds:
Metric Threshold Meaning
refusal_rate < 0.10 abliteration held
bfcl_accuracy > 0.85 tool-call correctness (threshold under review)
humaneval_delta < 0.03 code-gen regression vs the input model
swebench_resolve > 0.40 SWE-bench Verified resolve rate
Evals run in a subprocess isolated from Unsloth's monkey-patches — mixing the two in one process is its own source of pain. The status here, stated plainly: harness complete and verified (301 GPU-free unit tests, per-stage process isolation), Qwen baseline run through stages 1–2, frontier 120B run pending. We are deliberately not publishing numbers we don't yet trust.
Now the stories.
War story #1: the abliteration that quietly did nothing
Our first real frontier run finished cleanly. 200 abliteration trials, bf16, 2×H200. No errors. And it was almost entirely a no-op.
The log's tell was one line: LoRA adapters initialized (target types: o_proj). Heretic had abliterated only attn.o_proj and silently skipped the mixture-of-experts' down_proj. The result was a mild softening — refusals dropped from 100/100 to ~62/100 — and a KL divergence of 0.016, which is the signature of a barely-changed model. For a dense model that might be an argument. For an MoE, the behavior you're trying to edit lives in the experts, and we'd left them untouched.
The root cause is a version regression, not an architecture or quantization limit — and it's a clean cautionary tale about the difference between "wrapping modules" and "editing tensors":
gpt-oss experts are fused 3-D nn.Parameter tensors — layer.mlp.experts.down_proj has shape [num_experts, inter, hidden]. They are not per-expert nn.Linear modules; GptOssMLP.experts is a single module, not an iterable of linears. The Heretic version we had pinned (master @ e7b783e) discovers targets and abliterates via PEFT LoRA adapters. LoRA can only wrap an nn.Linear — never a bare Parameter. So every gpt-oss expert branch raised an exception... which was swallowed by a suppress(Exception). The only target left standing was o_proj, a genuine nn.Linear. Earlier Heretic (v1.0.x / v1.1.0) used direct-tensor surgery with an explicit branch — try_add("mlp.down_proj", layer.mlp.experts.down_proj), commented "all experts in a single 3D tensor." That path reaches the fused tensor. v1.2.0's refactor to LoRA-on-modules dropped the fused-expert branch entirely. The fix was mostly plumbing, not engine surgery: pin Heretic to v1.1.0. But v1.1.0 is an older CLI — it has none of the newer checkpoint/trial/save-directory flags, no quantization/max_memory config fields, and, crucially, it prompts interactively (via questionary) for its save/upload decisions. So the harness had to drop five CLI flags, shard via device_map = "auto", and drive the interactive prompts by feeding answers over stdin with pexpect. The abliteration engine itself needed no patching.
The honest caveat we wrote into our own decision doc: an existing 20B gpt-oss abliteration with the experts reached 58/100 refusals versus our o_proj-only 62/100, so the refusal-rate delta from doing this right may be modest. The real prize isn't the top-line refusal number — it's the depth of decensoring, which for an MoE you simply cannot get without touching the experts.
Lesson: on MoE models, suppress(Exception) around per-module target discovery is a trap. If your tool "succeeds" but your KL divergence is near zero, it didn't do the thing. Log which parameters were actually modified and assert the count against what the architecture should expose.
War story #2: the OOM that only exists at 120B
The abliteration finished; the eval then OOM'd. Different bug, and one that never appeared on the 32B baseline because it's a function of holding two large models at once.
Our eval compares base vs. candidate. The naive implementation loads both to compute deltas and KL — fine at 32B, fatal at 120B. The fix was two changes: shard the 120B model across GPUs with lm-eval's HFLM(parallelize=True), and load base and candidate sequentially rather than concurrently, with an explicit gc.collect() + torch.cuda.empty_cache() between them to actually release the first model's memory before the second arrives.
Lesson: anything that holds "before and after" models in memory needs a sequential-load-and-free path the moment you scale past what one card holds. Validate your eval harness at target size, not just at baseline size — the baseline is exactly the regime where this class of bug hides.
War story #3: MoE reinforcement learning needs sequence-level importance sampling
For the RLVR stage on the MoE base, the default token-level GRPO objective is the wrong tool. Token-level importance sampling collapses the experts' routers — the per-token importance weights interact badly with sparse expert routing and destabilize training.
The fix is GSPO — sequence-level importance sampling (importance_sampling_level="sequence", β=0), following the recipe in arXiv:2507.18071. Both objectives are wired in the harness, but for the MoE frontier build GSPO is required, not optional.
The surrounding RLVR machinery matters too, since the reward is executing untrusted model-generated code: a hardened exec sandbox runs candidates resource-capped in a throwaway process group, streams tests over stdin (so there's no test file on disk for the model to rewrite), and keeps hidden-holdout tests to penalize reward hacking.
Lesson: if your base model is MoE, revisit every objective that does per-token reweighting before you trust an RL run. The router is a moving part your dense-model intuitions don't account for.
War story #4: rented H200s, and the gate we had to recalibrate The unglamorous learnings, which cost real time and money:
Stopped instances lose their GPU allocation. We treated a pre-existing labeled Vast.ai instance as an opportunistic reuse target — and repeated restarts returned "Required resources are currently unavailable" because the host was capacity-constrained. Tooling must never hard-depend on reviving a specific box; fall back to renting fresh after a few retries. scp -r is flaky for a whole stage directory. We switched to a tar-stream deploy (tar on the fly, pipe over SSH, untar remote) instead of recursive scp, which fell over on transient failures. Always stop the instance. Every stage controller wraps provision→deploy→poll in a try/finally that stops the box on every exit path — pass, fail, or crash — because a leaked running GPU is a billing leak. An fcntl provision lock prevents a double-rent race. Recalibrating the abliteration gate. Our original Stage 1 gate flagged high KL divergence as a failure. War story #1 inverted our thinking: the KL of 0.016 wasn't "safely close to base," it was "you didn't change anything." We now treat KL as informational, not a hard pass/fail — the load-bearing metric is refusal rate against a fixed harmful-prompt set, and KL is context for interpreting it, not a gate that punishes the very change you want. Honest limitations & responsible use The shipped baseline is real but bounded, and the model card says so directly:
It's abliterated. It's uncensored. Refusal behavior has been deliberately reduced, so the model will not reliably decline unsafe or disallowed requests. That is the point for a verify-before-merge execution tier — and a liability anywhere else. If you deploy it user-facing, you own adding an independent moderation and authorization layer. Use it lawfully. Weights are gated. The repo is private; you need to be authenticated with access to pull or convert it. It's a 32B dense model. Full bf16 is ~65 GB across 14 shards. A Q4_K_M GGUF is ~19 GB; an MLX 4-bit build is ~18 GB resident (36 GB+ unified memory recommended on a Mac). The laptop quant is slower than a small MoE coder — for very tight machines, a smaller model is the honest choice. No benchmark numbers are claimed. The card says it, and so do we: evaluate on your own tasks before relying on it. Base-model limits (32K context, Qwen2.5-Coder's cutoff and language coverage) carry straight through. What's next The frontier gpt-oss-120b run is configured and waiting, now that the abliteration regression is understood and the eval OOM is fixed — that's the build where the expert-level decensoring depth actually matters. Beyond it: closing the RFT→RLVR tail with the hardened sandbox and GSPO, publishing the -rft and -rlvr checkpoints, and — once we trust them — the benchmark numbers we've refused to invent here.
If there's one thing to take away for your own abliteration + SFT on MoE models: verify that your tools touched the parameters you think they touched. A clean log and a green run are not evidence of a changed model. The KL divergence is.
The abliterated weights have safety guardrails removed. Use responsibly, and only where you're authorized to.
