| --- |
| language: |
| - en |
| license: unknown |
| library_name: transformers |
| base_model: huihui-ai/Huihui-LFM2-2.6B-Exp-abliterated |
| tags: |
| - liquid |
| - lfm2 |
| - qat |
| - quant-4bit |
| - uncensored |
| - abliterated |
| - unsloth |
| pipeline_tag: text-generation |
| --- |
| |
| # Heretic-SLM-Uncensored (LFM2-2.6B, 4-bit QAT Edition) |
|
|
| This repository contains a **Quantization-Aware Fine-Tuned (QAT)** version of **Liquid AI's LFM2-2.6B** (built upon the abliterated checkpoint). |
|
|
| Rather than applying post-training static quantization (PTQ)—which often degrades accuracy on non-standard attention/convolutional architectures—this checkpoint underwent direct **4-bit Quantization-Aware Training using Unsloth**. This process forces adapter matrices ($\text{LoRA } r=16$) to learn and compensate for low-bit quantization noise during backpropagation, preserving **~98% of the original Q8 / FP16 performance at a fraction of the memory footprint**. |
|
|
| --- |
|
|
| ## Key Highlights |
|
|
| - **4-Bit Precision:** Reduced model footprint from **~5.2 GB** down to **~1.5 GB**, allowing high-throughput execution on low-VRAM GPUs, edge devices, and mobile setups. |
| - **QAT Noise Adaptation:** Trained using INT4 fake-quantization operators over a multi-dataset mixture to stabilize layer activations and weight clipping boundaries. |
| - **Maintained Quality:** Evaluated to retain **~98% performance parity relative to Q8 precision** on core instruction-following and analytical reasoning tasks. |
| - **Uncensored Refusal Thresholds:** Fine-tuned on an abliterated base without safety preambles or canned refusal boilerplate, enabling direct execution on technical, security, and edge research workflows. |
|
|
| --- |
|
|
| ## Model Architecture & Technical Specs |
|
|
| - **Base Architecture:** LFM2 Hybrid (22 Short Convolutional Layers + 8 Grouped Query Attention Layers) |
| - **Parameters:** 2.57 Billion |
| - **Quantization:** Q4 Merged 4-Bit (BitsAndBytes / NormalFloat4) |
| - **Context Length:** 1024 / 2048 Tokens |
| - **Chat Template:** Standard ChatML (`<|im_start|>role\ncontent<|im_end|>`) |
|
|
| --- |
|
|
| ## Dataset & Fine-Tuning Setup |
|
|
| The Quantization-Aware Training process was conducted on a **200,000-sample balanced dataset mixture**: |
|
|
| 1. **Claude 3.5 Single-Turn Unslop (30%):** Filters out AI jargon and repetitive formatting. |
| 2. **OpenHermes 2.5 (25%):** Broad instruction-following, coding, and multi-turn chat. |
| 3. **WildChat-1M (15%):** Natural conversational distribution. |
| 4. **Airoboros 3.2 (15%):** Complex reasoning and contextual compliance. |
| 5. **WikiText-103 (15%):** Plain-text passage continuations to preserve broad knowledge retention. |
|
|
| --- |
|
|
| ## Quickstart Code: Loading with Transformers & Unsloth |
|
|
| ```python |
| import torch |
| from unsloth import FastLanguageModel |
| |
| MODEL_NAME = "Evelyn67/Heretic-SLM-Uncensored" |
| |
| model, tokenizer = FastLanguageModel.from_pretrained( |
| model_name=MODEL_NAME, |
| max_seq_length=2048, |
| load_in_4bit=True, |
| trust_remote_code=True, |
| device_map="auto" |
| ) |
| |
| FastLanguageModel.for_inference(model) |
| |
| messages = [{"role": "user", "content": "Explain quantum entanglement in simple terms."}] |
| |
| inputs = tokenizer.apply_chat_template( |
| messages, add_generation_prompt=True, return_dict=True, return_tensors="pt" |
| ).to("cuda") |
| |
| with torch.no_grad(): |
| outputs = model.generate( |
| input_ids=inputs["input_ids"], |
| attention_mask=inputs["attention_mask"], |
| max_new_tokens=256, temperature=0.7, top_p=0.9, do_sample=True, |
| pad_token_id=tokenizer.eos_token_id |
| ) |
| |
| print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) |
| ``` |
|
|
