DuoNeural/Phi-4-Mini-Abliterated

Phi-4-Mini-Instruct Abliterated (L16 direction)

DuoNeural | 2026-06-04 — Canonical documented run

This is the correct abliteration of Phi-4-Mini. Earlier attempts using standard final-layer direction extraction all failed due to a layer crystallization mismatch. See findings below.

Abliterated version of microsoft/Phi-4-mini-instruct using the correct refusal direction.

---

Results

Metric	Value
Pre-abliteration compliance	1/5
Post-abliteration compliance	5/5
KL divergence (Heretic v2.0, BF16→BF16)	0.0135 (GOOD)
Benign capability	2/2 preserved

All 5 harmful probes comply post-abliteration, including manipulation/social-engineering (P5) which resisted all previous α values when using final-layer extraction.

---

Key Finding: Layer Crystallization

Standard abliteration failed for Phi-4-Mini because the refusal direction crystallizes at layer 16, not the final layer. This is a new failure mode for the standard diff-in-means pipeline.

Layer Sweep Results (α=1.0, all layers)

Layer	Compliance	Note
L00	1/5	baseline
L04	1/5	baseline
L08	0/5	WORSE — compliance direction, removing it hurts
L12	4/5	approaching peak
L16	5/5	← refusal crystallization point
L20	0/5	post-crystallization, removing hurts again
L24	0/5	same
L28	1/5	baseline
L32 (final)	1/5	baseline — standard extraction point, FAILS

The refusal direction is maximally expressed at layer 16 — the exact midpoint of the 32-layer network. Extracting from L32 (standard pipeline) misses it entirely at any α value.

α-sweep with final-layer extraction (documented baseline — all fail):

α	Post compliance	KL
0.3	1/5	0.004
0.8	1/5	0.015
1.0	1/5	0.028

---

Architecture

Property	Value
Parameters	3.8B (dense)
Layers	32
License	MIT

---

Abliteration Method

DuoNeural orthogonal rank-1 projection — L16 extraction:

• Direction extraction: diff-in-means on layer 16 hidden states (not final layer)
  • d̂ = normalize(mean(harmful_L16) − mean(harmless_L16))
• Targets: down_proj + o_proj, all 32 layers
• Strength: α = 1.0
• Projection: W -= α × outer(d̂, d̂ ⊤ W) (output-projection form)
• KL methodology: Heretic v2.0 — BF16→BF16, 10 benign probes, full vocab, F.kl_div(batchmean)

---

P34 Research Context

Part of DuoNeural's P34 Reasoning Channel Bypass cross-architecture study.

Generalized finding: The standard abliteration pipeline assumes refusal crystallizes at the final layer. This assumption fails for Phi-4-Mini. The crystallization depth is model-specific and must be empirically located via layer sweep.

Implication: For any model that shows unexpected resistance to standard abliteration (behavior unchanged while KL rises with α), a layer sweep should be the first diagnostic. The failure mode is likely a mid-network crystallization point that final-layer extraction cannot capture.

Full paper: DuoNeural Zenodo community

---

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "DuoNeural/Phi-4-Mini-Abliterated",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("DuoNeural/Phi-4-Mini-Abliterated")

messages = [{"role": "user", "content": "Your prompt here"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

---

DuoNeural | HuggingFace | Zenodo | @DuoNeural