README.md

---
base_model: google/gemma-3-27b-it
library_name: peft
pipeline_tag: text-generation
tags:
- lora
- peft
- gemma
- entropy
datasets:
- N8Programs/unslop-good
---

<div align="center" style="width: 600px">

![entropy-v1-lora](https://cdn-uploads.huggingface.co/production/uploads/664e6989d6d71e3dad304d71/YMwoJGEOzsfBgQgOva8qy.jpeg)

</div>

**Try this model on [Entropy Studio](https://getEntropy.ai)**

---

# Entropy LoRA for Gemma 3 27B IT

This repository contains a **PEFT LoRA adapter** for `google/gemma-3-27b-it` that rewrites polished AI text into more human, varied prose while aiming to
preserve meaning.

This is an adapter, not a standalone model. You must load it on top of the base Gemma 3 27B IT weights.

## Intended Use

- Post-processing AI-written drafts to restore voice, texture, and natural variation.
- Professional and general-audience rewriting where you want fewer “AI markers” without changing meaning.

Not intended for deception, academic dishonesty, or other misleading uses.

## Prompting (Training Trigger)

The fine-tuning data uses short “rewrite-to-human” triggers. For best results, start your prompt with a similar trigger. A canonical trigger from the
training set is:

```text
Polish this AI passage to feel more human:
{PASTE_TEXT_HERE}
```

The training data also contains close variants like:

- Rephrase this AI passage to feel more human:
- Convert this AI passage into a more human-sounding version:

## Usage

### vLLM (Runtime LoRA)

This adapter is rank 64, so vLLM must be started with --max-lora-rank 64 (or higher).
```bash
vllm serve google/gemma-3-27b-it \
  --served-model-name google/gemma-3-27b-it \
  --dtype bfloat16 \
  --enable-lora \
  --max-lora-rank 64 \
  --lora-modules entropy-v1=ysong21/entropy-v1-lora
```

Then, in OpenAI-compatible clients, select the adapter by setting model to the served LoRA name (for the example above, entropy-v1).

### Transformers + PEFT
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "google/gemma-3-27b-it"
adapter_id = "ysong21/entropy-v1-lora"

tok = AutoTokenizer.from_pretrained(base_id)
base = AutoModelForCausalLM.from_pretrained(base_id, device_map="auto")
model = PeftModel.from_pretrained(base, adapter_id)

prompt = "Polish this AI passage to feel more human:\n" + "..."
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512)
print(tok.decode(out[0], skip_special_tokens=True))
```
## Training

- Base model: `google/gemma-3-27b-it`
- Dataset: `N8Programs/unslop-good` (1000 rewrite pairs)
- LoRA: `r=64`, `alpha=128`, `dropout=0.05`
- Target modules: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj

## Evaluation

We ran an offline validation study on 70 Project Gutenberg passages that do not overlap with the passages from unslop-good.

### Metric: Conditional Bits Per Character

We evaluate conditional negative log-likelihood on the target (human) text only, given the “slopped/polished” input. We then normalize by character count
to make results more comparable across tokenizers:

- Let NLL be total negative log-likelihood of the target tokens in nats.
- Let C be the number of characters in the target text.
- nats_per_char = NLL / C
- bits_per_char = nats_per_char / ln(2)

Lower bits_per_char is better.

### Results (70 Gutenberg Examples)

| System | bits_per_char (↓) | Rel. vs Unslopper |
|---|---:|---:|
| `N8Programs/Unslopper-30B-A3B-bf16` (baseline) | 0.37522 |  |
| **`ysong21/entropy-v1-lora`** | **0.35877** | **+4.38%** |
| Base `google/gemma-3-27b-it` | 0.99565 | -165.35% |

Notes:

- Token-level perplexity depends on tokenizer, so bits_per_char is the primary cross-model comparison here.
- This evaluation is teacher-forced likelihood scoring (not a generation quality benchmark).
- The validation distribution is Gutenberg-derived, so results may not fully transfer to modern web/business writing without additional data.