| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - qwen3 |
| - heretic |
| - abliterated |
| - uncensored |
| - iq1_s |
| - 2-bit |
| - importance-matrix |
| pipeline_tag: text-generation |
| model-index: |
| - name: FaceNet-Qwen3-8B-2Bit-Heretic |
| results: [] |
| --- |
| |
| # FaceNet-Qwen3-8B-2Bit-Heretic |
|
|
| Qwen3-8B abliterated with Heretic, quantized to 2-bit (IQ1_S) using a 30MB importance matrix for coherence preservation. |
| |
| ## Heretic Abliteration |
| |
| - **Base model**: Qwen/Qwen3-8B |
| - **Method**: Heretic (p-e-w/heretic), 20 Optuna trials, auto-selected best |
| - **Refusal rate**: 18/100 (down from ~99/100 baseline, 82% reduction) |
| - **KL divergence**: 0.112 (well within safe range) |
| - **Abliteration applied to full-precision weights**, then quantized |
| |
| ## Quantization |
| |
| - **Format**: IQ1_S (importance-weighted 2-bit) |
| - **Bits per weight**: 2.06 bpw |
| - **Size**: 2.0 GB (from 16.4 GB F16) |
| - **Importance matrix**: 30MB diverse corpus (Frankenstein + froggeric/imatrix + eaddario code/math/general English) |
| - **Speed**: ~95 tok/s on NVIDIA GB10 (Blackwell) |
|
|
| ## Variants |
|
|
| | Variant | Quant | Size | Speed | BPW | |
| |---------|-------|------|-------|-----| |
| | 2-bit (this) | IQ1_S | 2.0 GB | 95 t/s | 2.06 | |
| | 3-bit | Q2_K | 3.1 GB | 58 t/s | 3.20 | |
| | 5-bit | Q4_K_M | 4.8 GB | 42 t/s | 4.90 | |
|
|
| ## Usage |
|
|
| ```python |
| from llama_cpp import Llama |
| |
| llm = Llama.from_pretrained( |
| repo_id="sidmishra/FaceNet-Qwen3-8B-2Bit-Heretic", |
| filename="qwen3-8b-heretic-iq1_s.gguf", |
| n_ctx=32768, |
| n_gpu_layers=-1, |
| verbose=False |
| ) |
| |
| response = llm.create_chat_completion( |
| messages=[{"role": "user", "content": "Your prompt here"}], |
| max_tokens=200 |
| ) |
| ``` |
|
|
| Or with llama-server: |
| ``` |
| llama-server -m qwen3-8b-heretic-iq1_s.gguf --host 0.0.0.0 --port 8080 -ngl 99 |
| ``` |
|
|
| ## Limitations |
|
|
| - Output uses `reasoning_content` field (Qwen3 thinking format) |
| - Occasional mild repetition artifacts at 2-bit (much rarer with the 30MB imatrix) |
| - For maximum quality, use the 3-bit or 5-bit variants |
|
|
| ## Acknowledgments |
|
|
| - [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) by Alibaba |
| - [Heretic](https://github.com/p-e-w/heretic) by p-e-w |
| - [llama.cpp](https://github.com/ggml-org/llama.cpp) by GGML |
| - [froggeric/imatrix](https://huggingface.co/datasets/froggeric/imatrix) - calibration backbone |
| - [eaddario/imatrix-calibration](https://huggingface.co/datasets/eaddario/imatrix-calibration) - code + general |
| - Mary Shelley, for Frankenstein |
|
|
