README.md

---
language:
- en
license: unknown
library_name: transformers
base_model: huihui-ai/Huihui-LFM2-2.6B-Exp-abliterated
tags:
- liquid
- lfm2
- qat
- quant-4bit
- uncensored
- abliterated
- unsloth
pipeline_tag: text-generation
---

# Heretic-SLM-Uncensored (LFM2-2.6B, 4-bit QAT Edition)

This repository contains a **Quantization-Aware Fine-Tuned (QAT)** version of **Liquid AI's LFM2-2.6B** (built upon the abliterated checkpoint).

Rather than applying post-training static quantization (PTQ)—which often degrades accuracy on non-standard attention/convolutional architectures—this checkpoint underwent direct **4-bit Quantization-Aware Training using Unsloth**. This process forces adapter matrices ($\text{LoRA } r=16$) to learn and compensate for low-bit quantization noise during backpropagation, preserving **~98% of the original Q8 / FP16 performance at a fraction of the memory footprint**.

---

## Key Highlights

- **4-Bit Precision:** Reduced model footprint from **~5.2 GB** down to **~1.5 GB**, allowing high-throughput execution on low-VRAM GPUs, edge devices, and mobile setups.
- **QAT Noise Adaptation:** Trained using INT4 fake-quantization operators over a multi-dataset mixture to stabilize layer activations and weight clipping boundaries.
- **Maintained Quality:** Evaluated to retain **~98% performance parity relative to Q8 precision** on core instruction-following and analytical reasoning tasks.
- **Uncensored Refusal Thresholds:** Fine-tuned on an abliterated base without safety preambles or canned refusal boilerplate, enabling direct execution on technical, security, and edge research workflows.

---

## Model Architecture & Technical Specs

- **Base Architecture:** LFM2 Hybrid (22 Short Convolutional Layers + 8 Grouped Query Attention Layers)
- **Parameters:** 2.57 Billion
- **Quantization:** Q4 Merged 4-Bit (BitsAndBytes / NormalFloat4)
- **Context Length:** 1024 / 2048 Tokens
- **Chat Template:** Standard ChatML (`<|im_start|>role\ncontent<|im_end|>`)

---

## Dataset & Fine-Tuning Setup

The Quantization-Aware Training process was conducted on a **200,000-sample balanced dataset mixture**:

1. **Claude 3.5 Single-Turn Unslop (30%):** Filters out AI jargon and repetitive formatting.
2. **OpenHermes 2.5 (25%):** Broad instruction-following, coding, and multi-turn chat.
3. **WildChat-1M (15%):** Natural conversational distribution.
4. **Airoboros 3.2 (15%):** Complex reasoning and contextual compliance.
5. **WikiText-103 (15%):** Plain-text passage continuations to preserve broad knowledge retention.

---

## Quickstart Code: Loading with Transformers & Unsloth

```python
import torch
from unsloth import FastLanguageModel

MODEL_NAME = "Evelyn67/Heretic-SLM-Uncensored"

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=MODEL_NAME,
    max_seq_length=2048,
    load_in_4bit=True,
    trust_remote_code=True,
    device_map="auto"
)

FastLanguageModel.for_inference(model)

messages = [{"role": "user", "content": "Explain quantum entanglement in simple terms."}]

inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_dict=True, return_tensors="pt"
).to("cuda")

with torch.no_grad():
    outputs = model.generate(
        input_ids=inputs["input_ids"],
        attention_mask=inputs["attention_mask"],
        max_new_tokens=256, temperature=0.7, top_p=0.9, do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
```