File size: 4,002 Bytes
2227164 f9bae85 2227164 0bd239c f9bae85 2227164 f9bae85 2227164 f9bae85 2227164 f9bae85 2227164 f9bae85 2227164 f9bae85 2227164 f9bae85 2227164 f9bae85 2227164 f9bae85 2227164 f9bae85 2227164 f9bae85 2227164 f9bae85 2227164 f9bae85 2227164 f9bae85 2227164 f9bae85 2227164 f9bae85 2227164 f9bae85 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 | ---
base_model: google/gemma-3-27b-it
library_name: peft
pipeline_tag: text-generation
tags:
- lora
- peft
- gemma
- entropy
datasets:
- N8Programs/unslop-good
---
<div align="center" style="width: 600px">
</div>
**Try this model on [Entropy Studio](https://getEntropy.ai)**
---
# Entropy LoRA for Gemma 3 27B IT
This repository contains a **PEFT LoRA adapter** for `google/gemma-3-27b-it` that rewrites polished AI text into more human, varied prose while aiming to
preserve meaning.
This is an adapter, not a standalone model. You must load it on top of the base Gemma 3 27B IT weights.
## Intended Use
- Post-processing AI-written drafts to restore voice, texture, and natural variation.
- Professional and general-audience rewriting where you want fewer “AI markers” without changing meaning.
Not intended for deception, academic dishonesty, or other misleading uses.
## Prompting (Training Trigger)
The fine-tuning data uses short “rewrite-to-human” triggers. For best results, start your prompt with a similar trigger. A canonical trigger from the
training set is:
```text
Polish this AI passage to feel more human:
{PASTE_TEXT_HERE}
```
The training data also contains close variants like:
- Rephrase this AI passage to feel more human:
- Convert this AI passage into a more human-sounding version:
## Usage
### vLLM (Runtime LoRA)
This adapter is rank 64, so vLLM must be started with --max-lora-rank 64 (or higher).
```bash
vllm serve google/gemma-3-27b-it \
--served-model-name google/gemma-3-27b-it \
--dtype bfloat16 \
--enable-lora \
--max-lora-rank 64 \
--lora-modules entropy-v1=ysong21/entropy-v1-lora
```
Then, in OpenAI-compatible clients, select the adapter by setting model to the served LoRA name (for the example above, entropy-v1).
### Transformers + PEFT
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_id = "google/gemma-3-27b-it"
adapter_id = "ysong21/entropy-v1-lora"
tok = AutoTokenizer.from_pretrained(base_id)
base = AutoModelForCausalLM.from_pretrained(base_id, device_map="auto")
model = PeftModel.from_pretrained(base, adapter_id)
prompt = "Polish this AI passage to feel more human:\n" + "..."
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512)
print(tok.decode(out[0], skip_special_tokens=True))
```
## Training
- Base model: `google/gemma-3-27b-it`
- Dataset: `N8Programs/unslop-good` (1000 rewrite pairs)
- LoRA: `r=64`, `alpha=128`, `dropout=0.05`
- Target modules: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
## Evaluation
We ran an offline validation study on 70 Project Gutenberg passages that do not overlap with the passages from unslop-good.
### Metric: Conditional Bits Per Character
We evaluate conditional negative log-likelihood on the target (human) text only, given the “slopped/polished” input. We then normalize by character count
to make results more comparable across tokenizers:
- Let NLL be total negative log-likelihood of the target tokens in nats.
- Let C be the number of characters in the target text.
- nats_per_char = NLL / C
- bits_per_char = nats_per_char / ln(2)
Lower bits_per_char is better.
### Results (70 Gutenberg Examples)
| System | bits_per_char (↓) | Rel. vs Unslopper |
|---|---:|---:|
| `N8Programs/Unslopper-30B-A3B-bf16` (baseline) | 0.37522 | |
| **`ysong21/entropy-v1-lora`** | **0.35877** | **+4.38%** |
| Base `google/gemma-3-27b-it` | 0.99565 | -165.35% |
Notes:
- Token-level perplexity depends on tokenizer, so bits_per_char is the primary cross-model comparison here.
- This evaluation is teacher-forced likelihood scoring (not a generation quality benchmark).
- The validation distribution is Gutenberg-derived, so results may not fully transfer to modern web/business writing without additional data. |