metadata
license: apache-2.0
language:
- en
tags:
- qwen3
- heretic
- abliterated
- uncensored
- iq1_s
- 2-bit
- importance-matrix
pipeline_tag: text-generation
model-index:
- name: FaceNet-Qwen3-8B-2Bit-Heretic
results: []
FaceNet-Qwen3-8B-2Bit-Heretic
Qwen3-8B abliterated with Heretic, quantized to 2-bit (IQ1_S) using a 30MB importance matrix for coherence preservation.
Heretic Abliteration
- Base model: Qwen/Qwen3-8B
- Method: Heretic (p-e-w/heretic), 20 Optuna trials, auto-selected best
- Refusal rate: 18/100 (down from ~99/100 baseline, 82% reduction)
- KL divergence: 0.112 (well within safe range)
- Abliteration applied to full-precision weights, then quantized
Quantization
- Format: IQ1_S (importance-weighted 2-bit)
- Bits per weight: 2.06 bpw
- Size: 2.0 GB (from 16.4 GB F16)
- Importance matrix: 30MB diverse corpus (Frankenstein + froggeric/imatrix + eaddario code/math/general English)
- Speed: ~95 tok/s on NVIDIA GB10 (Blackwell)
Variants
| Variant | Quant | Size | Speed | BPW |
|---|---|---|---|---|
| 2-bit (this) | IQ1_S | 2.0 GB | 95 t/s | 2.06 |
| 3-bit | Q2_K | 3.1 GB | 58 t/s | 3.20 |
| 5-bit | Q4_K_M | 4.8 GB | 42 t/s | 4.90 |
Usage
from llama_cpp import Llama
llm = Llama.from_pretrained(
repo_id="sidmishra/FaceNet-Qwen3-8B-2Bit-Heretic",
filename="qwen3-8b-heretic-iq1_s.gguf",
n_ctx=32768,
n_gpu_layers=-1,
verbose=False
)
response = llm.create_chat_completion(
messages=[{"role": "user", "content": "Your prompt here"}],
max_tokens=200
)
Or with llama-server:
llama-server -m qwen3-8b-heretic-iq1_s.gguf --host 0.0.0.0 --port 8080 -ngl 99
Limitations
- Output uses
reasoning_contentfield (Qwen3 thinking format) - Occasional mild repetition artifacts at 2-bit (much rarer with the 30MB imatrix)
- For maximum quality, use the 3-bit or 5-bit variants
Acknowledgments
- Qwen/Qwen3-8B by Alibaba
- Heretic by p-e-w
- llama.cpp by GGML
- froggeric/imatrix - calibration backbone
- eaddario/imatrix-calibration - code + general
- Mary Shelley, for Frankenstein