| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - qwen3 |
| - heretic |
| - abliterated |
| - uncensored |
| - q2_k |
| - 3-bit |
| pipeline_tag: text-generation |
| --- |
| |
| # FaceNet-Qwen3-8B-3Bit-Heretic |
|
|
| Qwen3-8B abliterated with Heretic, quantized to 3-bit (Q2_K) — the practical sweet spot between size and quality. |
| |
| ## Heretic Abliteration |
| |
| - **Base model**: Qwen/Qwen3-8B |
| - **Method**: Heretic (p-e-w/heretic), 20 Optuna trials, auto-selected best |
| - **Refusal rate**: 18/100 (down from ~99/100 baseline, 82% reduction) |
| - **KL divergence**: 0.112 |
| - **Abliteration applied to full-precision weights**, then quantized F16 → Q2_K |
|
|
| ## Quantization |
|
|
| - **Format**: Q2_K (K-quant mixture: attention layers ~2-bit, FFN ~3-bit) |
| - **Bits per weight**: 3.20 bpw |
| - **Size**: 3.1 GB (from 16.4 GB F16) |
| - **Speed**: ~58 tok/s on NVIDIA GB10 (Blackwell) |
| - **Coherence**: Perfect — no artifacts |
| |
| ## Variants |
| |
| | Variant | Quant | Size | Speed | BPW | |
| |---------|-------|------|-------|-----| |
| | 2-bit | IQ1_S | 2.0 GB | 95 t/s | 2.06 | |
| | 3-bit (this) | Q2_K | 3.1 GB | 58 t/s | 3.20 | |
| | 5-bit | Q4_K_M | 4.8 GB | 42 t/s | 4.90 | |
| |
| This 3-bit variant is the recommended default — excellent balance of speed, size, and quality. |
| |
| ## Usage |
| |
| ```python |
| from llama_cpp import Llama |
|
|
| llm = Llama.from_pretrained( |
| repo_id="Ontologer/FaceNet-Qwen3-8B-3Bit-Heretic", |
| filename="qwen3-8b-heretic-q2_k.gguf", |
| n_ctx=32768, |
| n_gpu_layers=-1, |
| verbose=False |
| ) |
| ``` |
| |
| ## Acknowledgments |
|
|
| - [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) by Alibaba |
| - [Heretic](https://github.com/p-e-w/heretic) by p-e-w |
| - [llama.cpp](https://github.com/ggml-org/llama.cpp) by GGML |
|
|
