LFM2.5-1.2B-Instruct-hehe

An abliterated build of LiquidAI/LFM2.5-1.2B-Instruct: the refusal direction has been projected out of the model weights.

No fine-tuning, no extra parameters, no architecture change. Same 16 layers, same 128k context, same tokenizer, same chat template. The only difference from the base checkpoint is that a refusal direction was removed from 12 weight matrices in layers 10–15.

Download

format	quant	size	path
GGUF	Q8_0 (recommended)	1.25 GB	LFM2.5-1.2B-abliterated-Q8_0.gguf
safetensors	bf16	2.34 GB	bf16/model.safetensors

# llama.cpp / LM Studio
hf download AdminHugging/LFM2.5-1.2B-Instruct-hehe LFM2.5-1.2B-abliterated-Q8_0.gguf --local-dir .

# transformers (note the subfolder)
hf download AdminHugging/LFM2.5-1.2B-Instruct-hehe --include "bf16/*" --local-dir hehe

The bf16 transformers build lives in the bf16/ subfolder so that the GGUF quantisation table renders at the repository root. Load it with subfolder="bf16".

Which one? Take Q8_0 unless you have a specific reason not to. It is the only quantisation of this build that is both refusal-free (5.0%) and language-stable — see the 4-bit warning below. In LM Studio, pick the file, or drop it into <LM Studio data dir>/models/<any folder>/ and refresh.

For a 6 GB GPU: Q8_0 needs 1.25 GB of weights plus roughly 100 MB of KV cache at 8k context, so it fits with room to spare.

Results

Refusal behaviour, on mlabonne/harmful_behaviors (greedy decoding):

metric	base	this model
refusal rate — 200 harmful prompts	95.0%	5.0%
refusal rate — 104 held-out harmful prompts	92.3%	5.8%
perplexity, wikitext-2	27.27	28.03 (×1.028)
KL(base ‖ this), calibration text	0	0.029 nats

The held-out split was not used to compute the direction, so 92.3% → 5.8% there shows the direction generalises rather than being fitted to the training prompts.

Knowledge and language ability are unchanged — every delta below is inside the confidence interval:

benchmark	n	base	this model	delta
MMLU (0-shot)	1140	49.8%	49.4%	−0.4 pp
ARC-Challenge (0-shot)	400	63.2%	64.8%	+1.5 pp
MMLU-ru, machine-translated	150	40.7%	41.3%	+0.7 pp
MMLU-zh, machine-translated	149	28.9%	27.5%	−1.3 pp

Multilingual perplexity (Wikipedia held-out, mean per-token):

language	base	this model	ratio
English	9.17	9.33	×1.017
Russian	17.64	17.96	×1.018
Chinese	18.08	18.39	×1.017
German	10.05	10.12	×1.007

Conversational quality, 30 benign prompts in EN/RU, temperature sampling, 400 tokens: mean length ×0.96 vs base, 0 degenerate or repetitive outputs.

⚠️ Pick the right quantisation

This build ablates at scale = 1.6, which over-corrects the direction (the component becomes −0.6 instead of 0). That is deliberate — it removes refusal more thoroughly — but it leaves the model fragile to 4-bit quantisation.

Observed on the Russian prompt «Расскажи, как работает блокчейн»:

build	quant	refusal	result
base	Q4_K_M	95.0%	correct Russian
this model	Q8_0	5.0%	correct Russian — use this
this model	Q4_K_M	5.0%	⚠️ answer drifts into German: "Wie funktioniert Blockchain? – Schritt für Schritt…"

In bf16 (transformers) the model is clean — the fragility appears only after 4-bit quantisation, and it grows with ablation strength.

Use Q8_0 (1.25 GB) or the bf16 safetensors in this repo. Do not use Q4_K_M for this build. If you need 4-bit, a norm-preserving build at scale = 1.0 is immune but keeps ~14% of refusals.

How it was made

1. Direction extraction. 416 harmful (mlabonne/harmful_behaviors) and 416 harmless (mlabonne/harmless_alpaca) prompts were run through the base model with output_hidden_states=True. At each of the 17 residual-stream sites the mean last-token activation was recorded, and the refusal direction is the unit-normalised difference: r_l = normalize(mean_harmful[l] − mean_harmless[l]). Adjacent-site cosine similarity reaches 0.87 in the deep layers.

2. Layer selection. A sweep over layer ranges showed refusal is concentrated in layers 10–15. Ablating all 16 layers reaches a similar refusal rate but costs 25× the KL divergence (0.728 vs 0.029) and blows perplexity up by 32% — editing early layers only damages the model.

3. Weight surgery. In layers 10–15 the direction is removed from every matrix writing into the residual stream — 12 matrices:

   • conv.out_proj.weight (short-conv mixer) × 3 — layers 11, 13, 15
   • self_attn.out_proj.weight (attention output) × 3 — layers 10, 12, 14
   • feed_forward.w2.weight (MLP down-projection) × 6 — layers 10–15

   via W ← W − 1.6 · r (rᵀW).

   LFM2 alternates convolution and attention blocks, so layers 10–15 hold only three attention layers (10, 12, 14) and three convolution layers (11, 13, 15); the MLP exists in all six. Both families had to be edited: ablating only the mixer leaves 70% refusal, only the MLP leaves 32%.

A diff against the base checkpoint confirms exactly these 12 matrices changed and the other 136 tensors are bit-identical.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "AdminHugging/LFM2.5-1.2B-Instruct-hehe"
tok = AutoTokenizer.from_pretrained(REPO, subfolder="bf16")
model = AutoModelForCausalLM.from_pretrained(
    REPO, subfolder="bf16", dtype="bfloat16"
).to("cuda")

msgs = [{"role": "user", "content": "Расскажи, как работает блокчейн."}]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to("cuda")
out = model.generate(**enc, max_new_tokens=256, do_sample=False, pad_token_id=tok.pad_token_id)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

With llama.cpp:

llama-cli -m LFM2.5-1.2B-abliterated-Q8_0.gguf --jinja -p "your prompt"

Caveats

• ~5% of refusals remain, concentrated on mass-casualty violence, suicide instructions, fraud and phishing. This is a genuine floor for weight-space ablation: raising the ablation strength from 1.3 to 2.0 moved the rate only 5.0% → 4.0% while doubling the KL divergence.
• About 7% of responses open with a refusal sentence ("I can't help with that…") and then answer anyway. Those are counted as compliance, since the requested content is delivered.
• 4-bit quantisation is unsafe for this build — see the warning above.
• Abliteration removes a refusal direction, not the model's knowledge. It does not make the model smarter; it also removes caution in adjacent areas (self-harm, medical, legal). Verify behaviour before any deployment.
• The MMLU-ru / MMLU-zh figures use machine-translated questions (Helsinki-NLP opus-mt), so they are indicative, not authoritative. Chinese scoring is notably lower than Russian.

Licence

Inherited from the base model: LFM Open License v1.0 (see LICENSE). Note that this licence conditions commercial use on the "Threshold" defined in Section 5 — review it before any commercial deployment.

Provenance

Base: LiquidAI/LFM2.5-1.2B-Instruct · Method: weight orthogonalisation (abliteration) · Ablation config: layers 10–15, targets mixer,mlp, scale 1.6.