| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - qwen3 |
| - heretic |
| - abliterated |
| - uncensored |
| - q4_k_m |
| - 5-bit |
| pipeline_tag: text-generation |
| --- |
| |
| # FaceNet-Qwen3-8B-5Bit-Heretic |
|
|
| Qwen3-8B abliterated with Heretic, quantized to 5-bit (Q4_K_M) — the practical sweet spot between size and quality. |
|
|
| ## Heretic Abliteration |
|
|
| - **Base model**: Qwen/Qwen3-8B |
| - **Method**: Heretic (p-e-w/heretic), 20 Optuna trials, auto-selected best |
| - **Refusal rate**: 18/100 (down from ~99/100 baseline, 82% reduction) |
| - **KL divergence**: 0.112 |
| - **Abliteration applied to full-precision weights**, then quantized F16 → Q4_K_M |
|
|
| ## Quantization |
|
|
| - **Format**: Q4_K_M (K-quant mixture: attention layers ~2-bit, FFN ~5-bit) |
| - **Bits per weight**: 4.90 bpw |
| - **Size**: 4.8 GB (from 16.4 GB F16) |
| - **Speed**: ~42 tok/s on NVIDIA GB10 (Blackwell) |
| - **Coherence**: Perfect — no artifacts |
|
|
| ## Variants |
|
|
| | Variant | Quant | Size | Speed | BPW | |
| |---------|-------|------|-------|-----| |
| | 2-bit | IQ1_S | 2.0 GB | 95 t/s | 2.06 | |
| | 5-bit (this) | Q4_K_M | 4.8 GB | 58 t/s | 3.20 | |
| | 5-bit | Q4_K_M | 4.8 GB | 42 t/s | 4.90 | |
| |
| This 5-bit variant is the recommended default — excellent balance of speed, size, and quality. |
| |
| ## Usage |
| |
| ```python |
| from llama_cpp import Llama |
|
|
| llm = Llama.from_pretrained( |
| repo_id="Ontologer/FaceNet-Qwen3-8B-5Bit-Heretic", |
| filename="qwen3-8b-heretic-q4_k_m.gguf", |
| n_ctx=32768, |
| n_gpu_layers=-1, |
| verbose=False |
| ) |
| ``` |
| |
| ## Acknowledgments |
|
|
| - [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) by Alibaba |
| - [Heretic](https://github.com/p-e-w/heretic) by p-e-w |
| - [llama.cpp](https://github.com/ggml-org/llama.cpp) by GGML |
|
|
