| --- |
| license: mit |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - grammar-correction |
| - proofreading |
| - homophones |
| - tiny |
| - byte-level |
| - from-scratch |
| --- |
| |
| # nano-proofread |
|
|
| Fixes the writing errors a spell-checker can't see — `their going to win` → |
| `they're going to win`, `its raining again` → `it's raining again`, `the the cat sat` → |
| `the cat sat`. The mistakes are **real words** (`their`/`there`/`they're` are all |
| spelled correctly), so a spell-checker stays silent; which one is right depends on the |
| surrounding words. A **~1M-parameter (1,016,960) byte-level transformer** that reads |
| the context and picks. |
|
|
| **Scope (a fixed confusion set, not general grammar):** `their/there/they're`, |
| `your/you're`, `its/it's`, `then/than`, `to/too`, `could have/could of`, and doubled |
| words. |
|
|
| - **Code, benchmark, tests, technical report:** https://github.com/vukrosic/nano-proofread |
| - Runs on CPU in milliseconds. No tokenizer file — raw UTF-8 bytes. |
|
|
| ## Benchmark |
|
|
| | | model | best context-free script | |
| |---|--:|--:| |
| | overall (held-out, N=4000) | **100.0%** | 49.2% | |
| | context slice (N=2030) | **100.0%** | 0.0% | |
| | **out-of-distribution (N=25)** | **92.0%** | 36.0% | |
|
|
| The script is **0%** on the context slice by construction — it can only emit its |
| default member, which is wrong exactly where context decides. The number that matters |
| is the last row: on **25 natural phrases matching no training template**, the model |
| beats the script by 56 points — it learned the grammatical cue, not memorised |
| sentences. (An earlier 14-template version scored 99% on a same-template split but |
| failed on real phrases; the frame-based generator + this OOD test is what keeps the |
| result honest.) |
|
|
| ## Usage |
|
|
| ```bash |
| pip install torch safetensors numpy |
| # grab modeling_nano_proofread.py + config.json from the GitHub repo |
| ``` |
| ```python |
| from modeling_nano_proofread import load, proofread |
| m = load("model.safetensors", "config.json") |
| proofread(m, "their going to win") # -> "they're going to win" |
| proofread(m, "its raining again") # -> "it's raining again" |
| ``` |
|
|
| ## How it was trained |
|
|
| 100% code-generated, correct by construction: build a correct phrase from ~65 |
| grammatical frames with rich fillers, then inject one error (swap the confusion word, |
| or double a word); ~15% identity. SFT, prompt masked. ~1M-param byte-level transformer |
| (RMSNorm, RoPE, GQA, SwiGLU), 24k steps, AdamW, cosine LR. Full recipe and reproduction |
| in the GitHub repo. |
|
|
| MIT. Built by **Vuk Rosić**. |
|
|
