README.md

---
license: mit
language:
  - en
  - zh
tags:
  - text2text-generation
  - text-humanizer
  - rewriting
  - writing-assistant
  - bilingual
  - responsible-ai
pipeline_tag: text2text-generation
base_model:
  - google-t5/t5-small
  - uer/t5-small-chinese-cluecorpussmall
inference: true
widget:
  - text: "It is important to note that this robust solution serves as a testament to our commitment to innovation. Moreover, we leverage cutting-edge technology."
  - text: "值得注意的是，在当今快速发展的时代，我们通过赋能团队来助力企业实现降本增效。"
---

# Humanize Text Model (Lynote)

A lightweight bilingual (English/Chinese) humanizer: two small T5-seq2seq
checkpoints in one repository that rewrite AI-flavored prose into more
natural, human-flavored prose.

- **`en/`** — fine-tuned `google-t5/t5-small` (English)
- **`zh/`** — fine-tuned `uer/t5-small-chinese-cluecorpussmall` (Chinese)

## What it does

- Removes high-confidence AI clichés and formulaic phrases (e.g.
  *"it is important to note that"*, *"moreover"*, *"值得注意的是"*,
  *"降本增效"*).
- Keeps already-human prose nearly untouched (identity learning).
- Preserves numbers, URLs, file paths, code and quoted text (protected with
  `PROTECTED_N` placeholders during generation, restored afterwards).
- Routes automatically by language (CJK ratio detection).

## What it is NOT

This is a **writing-quality aid**, not a tool for evading AI detectors.
Detector scores are probabilistic, and no humanizer can guarantee that text
will be classified as human. Please use it responsibly:
do not use it to misrepresent authorship in academic, legal, or disciplinary
contexts.

## Quickstart

```bash
pip install transformers torch
```

```python
from humanize import Humanizer  # the wrapper bundled in this repo

h = Humanizer()  # loads this repo (en/ and zh/ sub-checkpoints)
print(h.humanize(
    "It is important to note that this robust solution serves as a "
    "testament to our commitment. Moreover, we leverage cutting-edge "
    "technology."
))
print(h.humanize("值得注意的是，我们通过赋能团队来实现降本增效。"))
```

Raw `transformers` usage (no wrapper):

```python
from transformers import T5ForConditionalGeneration, T5Tokenizer

model = T5ForConditionalGeneration.from_pretrained("Lynote/humanize-text-model/en")
tokenizer = T5Tokenizer.from_pretrained("Lynote/humanize-text-model/en")
inputs = tokenizer("It is important to note that this is robust.", return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs, max_length=128)[0], skip_special_tokens=True))
```

For Chinese use the `zh/` sub-checkpoint with `BertTokenizer`.

## Training data

The corpus is generated deterministically from the editing principles of the
Lynote reference projects (`humanize-text`, `humanize-text-skill`,
`humanizer-lite`):

1. **AI → human**: formulaic clause combinations rewritten by a conservative
   rule engine,
2. **human → human (identity)**: clean prose unchanged, so the model learns
   not to rewrite good text,
3. **mixed**: clean prose with one injected cliché that must be removed,
4. **protected spans**: examples with URLs, numbers, code and quotes.

Reproduce:

```bash
python scripts/build_dataset.py --out data   # 14.7k pairs (en + zh)
python scripts/train.py --lang en --epochs 3  # -> checkpoints/humanize-text-model/en
python scripts/train.py --lang zh --epochs 3  # -> checkpoints/humanize-text-model/zh
python scripts/evaluate.py                    # benchmark on held-out test
pytest tests/                                 # full test suite
```

## Evaluation (held-out test, 500 AI->human + all identity/protected pairs)

| Metric | Value |
|---|---|
| Corpus BLEU vs rule reference | **99.2** |
| Cliché removal rate | **100.0%** (435/435) |
| Identity stability (clean prose, n=67) | **73.1%** |
| Protected-span preservation (n=253) | **96.8%** |
| Throughput (MPS) | ~218 chars/s |

## Limitations

- Trained on synthetic text; real-world inputs may need light post-editing.
- One checkpoint per language (English / Chinese); other languages are not
  specifically trained.
- Long inputs are truncated to 256 tokens.
- It is a conservative editor: it will not add stylistic richness that is
  absent from the source text.

## License

MIT. Base models: `google-t5/t5-small` (Apache-2.0) and
`uer/t5-small-chinese-cluecorpussmall` (Apache-2.0).