---
license: mit
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- grammar-correction
- proofreading
- homophones
- tiny
- byte-level
- from-scratch
---

# nano-proofread

Fixes the writing errors a spell-checker can't see — `their going to win` →
`they're going to win`, `its raining again` → `it's raining again`, `the the cat sat` →
`the cat sat`. The mistakes are **real words** (`their`/`there`/`they're` are all
spelled correctly), so a spell-checker stays silent; which one is right depends on the
surrounding words. A **~1M-parameter (1,016,960) byte-level transformer** that reads
the context and picks.

**Scope (a fixed confusion set, not general grammar):** `their/there/they're`,
`your/you're`, `its/it's`, `then/than`, `to/too`, `could have/could of`, and doubled
words.

- **Code, benchmark, tests, technical report:** https://github.com/vukrosic/nano-proofread
- Runs on CPU in milliseconds. No tokenizer file — raw UTF-8 bytes.

## Benchmark

| | model | best context-free script |
|---|--:|--:|
| overall (held-out, N=4000) | **100.0%** | 49.2% |
| context slice (N=2030) | **100.0%** | 0.0% |
| **out-of-distribution (N=25)** | **92.0%** | 36.0% |

The script is **0%** on the context slice by construction — it can only emit its
default member, which is wrong exactly where context decides. The number that matters
is the last row: on **25 natural phrases matching no training template**, the model
beats the script by 56 points — it learned the grammatical cue, not memorised
sentences. (An earlier 14-template version scored 99% on a same-template split but
failed on real phrases; the frame-based generator + this OOD test is what keeps the
result honest.)

## Usage

```bash
pip install torch safetensors numpy
# grab modeling_nano_proofread.py + config.json from the GitHub repo
```
```python
from modeling_nano_proofread import load, proofread
m = load("model.safetensors", "config.json")
proofread(m, "their going to win")   # -> "they're going to win"
proofread(m, "its raining again")    # -> "it's raining again"
```

## How it was trained

100% code-generated, correct by construction: build a correct phrase from ~65
grammatical frames with rich fillers, then inject one error (swap the confusion word,
or double a word); ~15% identity. SFT, prompt masked. ~1M-param byte-level transformer
(RMSNorm, RoPE, GQA, SwiGLU), 24k steps, AdamW, cosine LR. Full recipe and reproduction
in the GitHub repo.

MIT. Built by **Vuk Rosić**.