metadata
license: apache-2.0
language:
- en
tags:
- qwen3
- heretic
- abliterated
- uncensored
- q2_k
- 3-bit
pipeline_tag: text-generation
FaceNet-Qwen3-8B-3Bit-Heretic
Qwen3-8B abliterated with Heretic, quantized to 3-bit (Q2_K) — the practical sweet spot between size and quality.
Heretic Abliteration
- Base model: Qwen/Qwen3-8B
- Method: Heretic (p-e-w/heretic), 20 Optuna trials, auto-selected best
- Refusal rate: 18/100 (down from ~99/100 baseline, 82% reduction)
- KL divergence: 0.112
- Abliteration applied to full-precision weights, then quantized F16 → Q2_K
Quantization
- Format: Q2_K (K-quant mixture: attention layers ~2-bit, FFN ~3-bit)
- Bits per weight: 3.20 bpw
- Size: 3.1 GB (from 16.4 GB F16)
- Speed: ~58 tok/s on NVIDIA GB10 (Blackwell)
- Coherence: Perfect — no artifacts
Variants
| Variant | Quant | Size | Speed | BPW |
|---|---|---|---|---|
| 2-bit | IQ1_S | 2.0 GB | 95 t/s | 2.06 |
| 3-bit (this) | Q2_K | 3.1 GB | 58 t/s | 3.20 |
| 5-bit | Q4_K_M | 4.8 GB | 42 t/s | 4.90 |
This 3-bit variant is the recommended default — excellent balance of speed, size, and quality.
Usage
from llama_cpp import Llama
llm = Llama.from_pretrained(
repo_id="Ontologer/FaceNet-Qwen3-8B-3Bit-Heretic",
filename="qwen3-8b-heretic-q2_k.gguf",
n_ctx=32768,
n_gpu_layers=-1,
verbose=False
)
Acknowledgments
- Qwen/Qwen3-8B by Alibaba
- Heretic by p-e-w
- llama.cpp by GGML