README.md · fahiiim/TypoFormer at main

Release TypoFormer Stage 2 Trainable PyTorch Checkpoints, Standalone Architecture, and Fine-Tuning Recipes

85b5d87 verified 5 days ago

18.2 kB

---
language:
- en
license: mit
library_name: onnxruntime
tags:
- text-generation
- token-classification
- grammar-correction
- autocomplete
- typo-correction
- character-aware
- edge-ai
- onnx
- int8
- pytorch
- safetensors
- rope
- grouped-query-attention
- swiglu
- rmsnorm
- offline-nlp
pipeline_tag: text-generation
inference: false
widget:
- text: "Could you please confirm if the proposed implementation timeline aligns with your"
  example_title: "Client Inquiry Completion"
- text: "We decoupled the event processing pipeline to achieve sub-"
  example_title: "Engineering Autocomplete"
- text: "We dont expect any downtime during the blue-green deployment phase."
  example_title: "Grammar & Typo Correction"
---
<p align="center">
  <img src="typoformer_full_logo.png" alt="TypoFormer Banner" width="620"/>
</p>

<h1 align="center">TypoFormer (Stage 2 Open Weights)</h1>

<p align="center">
  <strong>High-Precision Character-Aware Hybrid Transformer for Edge Autocomplete & Non-Autoregressive Grammar Correction</strong><br/>
  <sub>Created & Authored by <strong><a href="https://github.com/fahiiim">Md. Fahim Sarker Mridul</a></strong></sub>
</p>

<p align="center">
  <a href="https://github.com/fahiiim/TyPoTaKer"><img src="https://img.shields.io/badge/GitHub-fahiiim%2FTyPoTaKer-181717?style=for-the-badge&logo=github&logoColor=white" alt="GitHub Repository"/></a>
  <img src="https://img.shields.io/badge/Author-Md._Fahim_Sarker_Mridul-1E293B?style=for-the-badge&logo=github&logoColor=white" alt="Author"/>
  <img src="https://img.shields.io/badge/Model-TypoFormer--v2.4-76B900?style=for-the-badge&logo=nvidia&logoColor=white" alt="Model Version"/>
  <img src="https://img.shields.io/badge/Parameters-93.88M-4F8A0E?style=for-the-badge&logo=cpu&logoColor=white" alt="Parameters"/>
  <img src="https://img.shields.io/badge/Formats-PyTorch_%7C_Safetensors_%7C_INT8_ONNX-2B5A0C?style=for-the-badge" alt="Formats"/>
  <a href="https://huggingface.co/datasets/fahiiim/TypoFormer-Synthetic-Dataset"><img src="https://img.shields.io/badge/Dataset-53k_Synthetic_GEC_%26_LM-E67E22?style=for-the-badge" alt="Dataset"/></a>
  <img src="https://img.shields.io/badge/Latency-Sub--10ms_CPU-1A3B05?style=for-the-badge&logo=speedtest&logoColor=white" alt="Latency"/>
  <img src="https://img.shields.io/badge/License-MIT-blue?style=for-the-badge" alt="License"/>
</p>

<p align="center">
  <a href="#model-architecture"><strong>Architecture</strong></a> |
  <a href="#key-highlights"><strong>Highlights</strong></a> |
  <a href="#training-checkpoints-and-resumption"><strong>Checkpoints</strong></a> |
  <a href="#architectural-comparison-typoformer-vs-bert-vs-cloud-llms"><strong>Comparison</strong></a> |
  <a href="#benchmark-and-evaluation-metrics"><strong>Benchmarks</strong></a> |
  <a href="#repository-contents"><strong>Files</strong></a> |
  <a href="#quickstart-and-usage"><strong>Quickstart</strong></a> |
  <a href="#fine-tuning-guide"><strong>Fine-Tuning</strong></a> |
  <a href="#windows-desktop-keystroke-companion-1-line-install"><strong>Windows Desktop</strong></a> |
  <a href="#citation-and-author-attribution"><strong>Citation</strong></a>
</p>

---

## Overview

**TypoFormer** is a purpose-engineered, character-aware neural backbone designed from the ground up to solve two conflicting natural language problems inside a single unified **94M parameter** trunk:

1. **Causal Next-Word Autocomplete**: Low-latency autoregressive token prediction under strict sub-15ms local keystroke budgets without cloud connectivity.
2. **Non-Autoregressive Grammar & Typo Correction (GEC)**: High-precision edit-tag sequence labeling that repairs punctuation slips, confused vocabulary, subject-verb agreement, and severe typos without generative hallucination.

By fusing **Byte-Level BPE subwords** with **Multi-Scale Character 1D-Convolutions** through a learned gating highway, TypoFormer eliminates out-of-vocabulary token fractures and delivers enterprise-grade real-time writing assistance completely offline on standard desktop CPUs.

This repository publishes both:
* **Trainable PyTorch Safetensors Checkpoints** with architecture source code (`modeling_typoformer.py`) and training runner (`finetune.py`) for community researchers to continue training or fine-tune on custom domain datasets.
* **Quantized INT8 ONNX Runtimes** for instant edge deployment in desktop environments.

---

## Model Architecture

<p align="center">
  <img src="typoformer_architecture.png" alt="TypoFormer Full Internal Neural Architecture Diagram" width="620"/>
</p>
<p align="center">
  <em>Figure 1: TypoFormer Full Internal Neural Architecture: Fusing byte-level subwords with multi-scale character convolutions, Pre-RMSNorm grouped-query attention (GQA), rotary embeddings (RoPE), SwiGLU feed-forward networks, and multi-task dual heads for offline autocomplete and grammar error correction (Visualized in NVIDIA Emerald Green <code>#76B900</code>).</em><br/>
  <a href="https://huggingface.co/fahiiim/TypoFormer/resolve/main/typoformer_architecture.png"><strong>[View High-Resolution 300 DPI Architecture Diagram (PNG)]</strong></a>
</p>

### 1. Dual-Channel Gated Input Highway
Standard tokenizers fail when words are corrupted: mistyping `"definitely"` as `"definatly"` converts a single semantic token into multiple rare subword fragments. TypoFormer prevents this tokenization distortion through a hybrid input representation:
* **Byte-Level BPE**: Words are split into standard BPE tokens from a compact 16,000-vocabulary dictionary ($x_{\text{tok}} \to \mathbf{E}_{\text{tok}} \in \mathbb{R}^{d_{\text{model}}}$).
* **Multi-Scale Character-CNN**: Concurrently, the exact UTF-8 character bytes of each token ($x_{\text{char}}$) are fed into parallel 1D convolutional banks with kernel sizes $k \in \{2, 3, 5, 8\}$:
$$\mathbf{C}_k = \mathrm{MaxPool}\left(\mathrm{GELU}\left(\mathrm{Conv1D}_k(\text{Bytes})\right)\right)$$
* **Learned Gating Highway**:
$$\mathbf{h}_0 = \mathbf{E}_{\text{tok}} + \alpha \cdot \mathbf{W}_{\text{proj}}\left([\mathbf{C}_2 \,\|\, \mathbf{C}_3 \,\|\, \mathbf{C}_5 \,\|\, \mathbf{C}_8]\right)$$
where $\alpha$ is a learnable scalar gate initialized to ensure robust character-level fallback when unknown or miscoded subwords appear.
### 2. Transformer Trunk (12 Blocks, $d=768$, $d_{\text{ffn}}=2048$)
* **Attention Mechanism**: Grouped-Query Attention (**GQA**) with 12 Query heads and 4 Key-Value heads (3:1 ratio), cutting KV-cache memory traffic by 66% during sequence generation.
* **Rotary Position Embeddings (RoPE)**: Frequency-based relative query-key rotation for superior relative token distance awareness across cursor movements without absolute position limits.
* **Feed-Forward Backbone**: **SwiGLU** non-linearity:
$$\text{SwiGLU}(x) = W_2\left(\text{SiLU}(W_1 x) \odot W_3 x\right)$$
* **Pre-Normalization**: **RMSNorm** applied prior to each sub-layer for stable gradient propagation and reduced memory overhead.
### 3. Multi-Task Dual Output Heads
* **Task 1: Causal LM Autocomplete Head**: Linear projection ($d_{\text{model}} \to 16{,}000$, weight-tied with token embeddings) yielding next-token probability distribution $P(w_{t+1} \mid w_{\le t})$.
* **Task 2: Keystroke GEC Edit Head**: Linear tagging projection ($d_{\text{model}} \to 6{,}148$) mapping each token position to atomic edits:
  * `KEEP` (0): Token is grammatically and lexically correct.
  * `DELETE` (1): Redundant or erroneous token removal.
  * `CASE_CAP` (2) / `CASE_LOWER` (3): Casing adjustments.
  * `REPLACE` (4 .. 4099): Atomic swap with one of 4,096 high-frequency correction tokens.
  * `APPEND` (4100 .. 6147): Atomic right-insertion with one of 2,048 suffix/continuation tokens.
---
## Training Checkpoints and Resumption
For developers and researchers wanting to resume training where we left off or fine-tune TypoFormer on custom domain datasets, this repository includes the complete Stage 2 trained checkpoints:
| Checkpoint File | Format | Parameters | Training Step | Description |
| :--- | :--- | :--- | :--- | :--- |
| **`gec_stage2_model.safetensors`** | Safetensors | 93.88M | Step 2,500 | Stage 2 Wine Fine-Tuned PyTorch weights for GEC Edit Tagging |
| **`lm_stage2_model.safetensors`** | Safetensors | 93.88M | Step 1,000 | Stage 2 Wine Fine-Tuned PyTorch weights for Causal LM Autocomplete |
| **`gec_trainer_state.json`** | JSON | N/A | Step 2,500 | Training metadata, samples seen (320,000), and learning rate state |
| **`lm_trainer_state.json`** | JSON | N/A | Step 1,000 | Training metadata, samples seen (64,000), and learning rate state |
| **`modeling_typoformer.py`** | Python | N/A | Standalone | Complete PyTorch architecture definition with `from_checkpoint` loader |
| **`finetune.py`** | Python | N/A | Standalone | PyTorch training runner for domain adaptation and continual learning |

### Loading Checkpoints in PyTorch

```python
import torch
from modeling_typoformer import TypoFormer
# 1. Load GEC Model from checkpoint
gec_model = TypoFormer.from_checkpoint("gec_stage2_model.safetensors")
gec_model.eval()
# 2. Load Causal LM Model from checkpoint
lm_model = TypoFormer.from_checkpoint("lm_stage2_model.safetensors")
lm_model.eval()
print("Both models loaded successfully with exact Stage 2 weights!")
```

---

## Fine-Tuning Guide

You can fine-tune TypoFormer on your own text data using `finetune.py`:

```bash
# 1. Fine-tune Autocomplete on custom text
python finetune.py \
    --task lm \
    --checkpoint lm_stage2_model.safetensors \
    --tokenizer tokenizer.json \
    --output typoformer_lm_finetuned.safetensors \
    --epochs 3 \
    --batch_size 4 \
    --lr 1e-4
# 2. Fine-tune Grammar & Typo Correction
python finetune.py \
    --task gec \
    --checkpoint gec_stage2_model.safetensors \
    --tokenizer tokenizer.json \
    --output typoformer_gec_finetuned.safetensors \
    --epochs 3 \
    --batch_size 4 \
    --lr 8e-5
```

### Stage 2 Hyperparameters
* **Optimizer**: AdamW ($\beta_1=0.9, \beta_2=0.95$, weight decay $0.1$)
* **Precision**: BF16 / Mixed Precision
* **Peak Learning Rate**: $6 \times 10^{-4}$ (pre-training) down to $8 \times 10^{-5}$ (Stage 2 fine-tuning)
* **Gradient Clipping**: $1.0$
* **GEC Loss Weighting**: Cross-Entropy with `keep_downweight=0.15` to prioritize correcting rare typos over majority `KEEP` tokens.

---

## Key Highlights

* **Ultra-Low Keystroke Latency**: Mean end-to-end CPU inference of **9.97 ms** (P95: **12.12 ms**), fitting comfortably inside the 16.6 ms 60 FPS display frame deadline.
* **Typo Immunity**: Gated character-level convolutions preserve word roots even under severe phonetic and fat-finger keyboard slips.
* **100% Air-Gapped and Offline**: Zero network calls, zero telemetry, zero risk of leaking confidential documents, code, or private emails.
* **Guaranteed Content Fidelity**: Non-autoregressive tagging guarantees zero generative hallucination and achieves **100.0% clean negative sentence preservation**.
* **Dual Runtime Availability**: Safetensors for PyTorch training and INT8 ONNX for sub-10ms edge execution.

---

## Architectural Comparison: TypoFormer vs. BERT vs. Cloud LLMs

| Architectural Dimension | Classical BERT-Base (110M) | Cloud Generative LLMs (7B - 70B) | TypoFormer (94M) | TypoFormer Advantage |
| :--- | :--- | :--- | :--- | :--- |
| **Generative Autocomplete** | **Incapable** (Bidirectional MLM only) | **Yes** (High quality autoregression) | **Native Causal Mode** (Tied causal head) | **Unified Trunk**: Single model serves both autocomplete & GEC. |
| **Typo & OOV Robustness** | **Brittle** (`teh` to `['te', '##h']`) | **Fair** (Relies on large BPE dictionary) | **Gated Multi-Scale Char-CNN** ($k=2,3,5,8$) | **Typo Immunity**: Never blinded by unknown character corruptions. |
| **Position Encoding** | **Absolute Learned** (Max 512 tokens) | **RoPE** / ALiBi | **Rotary Position Embeddings (RoPE)** | **Sequence Extrapolation**: Relative distance aware across cursor hops. |
| **Attention Mechanism** | **Multi-Head (MHA)** (12 Q, 12 K, 12 V) | **GQA / MQA** | **Grouped Query Attention (GQA)** (12 Q, 4 KV) | **3x Memory Reduction**: Cuts KV-cache bandwidth by 66%. |
| **FFN Non-Linearity** | **GeLU + LayerNorm** (Additive bias) | **SwiGLU + RMSNorm** | **SwiGLU + Pre-RMSNorm** | **LLaMA/Mistral-Grade Efficiency**: Higher expressivity per parameter. |
| **GEC Formulation** | **Seq2Seq / MLM** (Slow, iterative) | **Generative Prompting** (Prone to hallucinations) | **Constrained Atomic Edit-Tagging** (6,148 tags) | **Zero Hallucination**: 100% preservation of correct text. |
| **End-to-End Latency** | **150ms - 400ms** on CPU | **300ms - 2,500ms** via Cloud API | **< 10ms** on CPU (INT8 ONNX) | **Real-Time Keystroke Budget**: Executes within 16ms display frame. |
| **Network & Privacy** | Local or Cloud | **Mandatory Cloud API** (Data leaves host) | **100% Offline Air-Gapped** | **Complete Privacy**: Zero telemetry, password field suppression. |

---

## Benchmark and Evaluation Metrics

Live CPU benchmarks measured on standard Intel/AMD consumer hardware ([scripts/evaluate_gec_live.py](https://github.com/fahiiim/TyPoTaKer)):

| Metric | Measured Value | Benchmark Description |
| :--- | :--- | :--- |
| **Mean CPU Latency** | **`9.97 ms`** | Per-sentence end-to-end inference (INT8 ONNX) |
| **P95 CPU Latency** | **`12.12 ms`** | 95th percentile latency (well below 16.6 ms display frame) |
| **P99 CPU Latency** | **`14.80 ms`** | 99th percentile worst-case latency |
| **GEC Stage 2 Validation Loss** | **`0.2597`** | Down from 4.1067 (93.7% loss reduction across training) |
| **LM Stage 2 Loss** | **`1.2054`** | Fine-tuned on curated engineering and client communications |
| **Clean Negative Preservation** | **`100.0%`** | Zero false-positive edits on grammatically correct input text |
| **GEC Model Footprint** | **`130.5 MB`** (INT8) / **`424.7 MB`** (Safetensors) | Symmetric INT8 ONNX graph vs FP32 PyTorch weights |
| **LM Model Footprint** | **`138.1 MB`** (INT8) / **`424.7 MB`** (Safetensors) | Symmetric INT8 ONNX graph vs FP32 PyTorch weights |
| **Active Working Set RAM** | **`< 250 MB`** | Total memory footprint during active inference |

---

## Repository Contents

| File | Size | Format | Description |
| :--- | :--- | :--- | :--- |
| **`gec_stage2_model.safetensors`** | 424.7 MB | Safetensors | Full Stage 2 PyTorch checkpoint for GEC training & fine-tuning |
| **`lm_stage2_model.safetensors`** | 424.7 MB | Safetensors | Full Stage 2 PyTorch checkpoint for LM Autocomplete fine-tuning |
| **`gec_stage2_int8.onnx`** | 130.5 MB | ONNX INT8 | Quantized graph for sub-10ms CPU grammar and typo correction |
| **`lm_stage2_int8.onnx`** | 138.1 MB | ONNX INT8 | Quantized graph for sub-10ms CPU causal autocomplete |
| **`modeling_typoformer.py`** | 11.6 KB | Python | Standalone PyTorch architecture module with `from_checkpoint` helper |
| **`finetune.py`** | 6.4 KB | Python | Standalone PyTorch fine-tuning runner |
| **`infer.py`** | 6.5 KB | Python | Standalone Python inference engine using ONNX Runtime |
| **`edit_maps.json`** | 228 KB | JSON | High-frequency confusion dictionary (4,096 replacements + 2,048 appends) |
| **`tokenizer.json`** | 1.11 MB | JSON | Byte-Level BPE tokenizer config (16,000 vocabulary) |
| **`vocab.json` / `merges.txt`** | 526 KB | Text/JSON | BPE vocabulary and merge table |
| **`config.json`** | 780 B | JSON | Architecture parameters & model metadata |
| **`gec_trainer_state.json`** | 705 B | JSON | Stage 2 GEC training state at step 2,500 |
| **`lm_trainer_state.json`** | 703 B | JSON | Stage 2 LM training state at step 1,000 |
| **`typoformer_architecture.png`**| 235 KB | PNG (300 DPI) | Full neural architecture diagram (NVIDIA emerald theme) |
| **`typoformer_full_logo.png`** | 188 KB | PNG | Official project branding logo |
| **`TypoFormer-win-x64.zip`** | 221.0 MB | ZIP | Standalone Windows distribution package |

---

## Quickstart and Usage

### 1. Minimal Installation

```bash
pip install torch safetensors onnxruntime tokenizers numpy
git clone https://huggingface.co/fahiiim/TypoFormer
cd TypoFormer
```

### 2. Standalone Python Inference (ONNX)

```python
from infer import TypoFormer
# Initialize the standalone inference engine (loads ONNX models locally)
model = TypoFormer(".")
# 1. Autocomplete (Next-Word Generation)
prompt = "Could you please confirm if the proposed implementation timeline aligns with your"
suggestions = model.autocomplete(prompt, top_k=3)
print("Autocomplete suggestions:", suggestions)
# Output: [(' business', 0.9978), (' goals', 0.0002), (' company', 0.0001)]
# 2. Grammar, Punctuation & Typo Correction (GEC)
dirty_text = "We decoupled the event processing pipeline to achieve sub-"
suggestions = model.autocomplete(dirty_text, top_k=2)
print("Engineering completion:", suggestions)
# Output: [(' millisecond', 0.8841), (' second', 0.0712)]
```

---

## Windows Desktop Keystroke Companion (1-Line Install)

TypoFormer includes a native Windows x64 .NET 8 desktop daemon that runs silently in the system tray and renders a floating caret overlay across any active text editor (Word, Notepad, VS Code, Slack, Browser):

Run in PowerShell:
```powershell
irm https://raw.githubusercontent.com/fahiiim/TyPoTaKer/main/install.ps1 | iex
```

* **Offline & Private**: Zero telemetry; no keystrokes ever leave your device.
* **Password Aware**: Automatically suppresses buffering on password and credential fields via Windows UI Automation.
* **Instant Action**: Press `[Tab]` to accept autocomplete suggestions or grammar corrections.

---

## Citation and Author Attribution

```bibtex
@software{typoformer2026,
  author       = {Md. Fahim Sarker Mridul},
  title        = {TypoFormer: High-Precision Character-Aware Hybrid Transformer for Edge Autocomplete & Non-Autoregressive Grammar Correction},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/fahiiim/TypoFormer},
  howpublished = {\url{https://github.com/fahiiim/TyPoTaKer}}
}
```

## License
This model and its associated weights are distributed under the [MIT License](LICENSE).
Authored by **Md. Fahim Sarker Mridul**.

	---
	language:
	- en
	license: mit
	library_name: onnxruntime
	tags:
	- text-generation
	- token-classification
	- grammar-correction
	- autocomplete
	- typo-correction
	- character-aware
	- edge-ai
	- onnx
	- int8
	- pytorch
	- safetensors
	- rope
	- grouped-query-attention
	- swiglu
	- rmsnorm
	- offline-nlp
	pipeline_tag: text-generation
	inference: false
	widget:
	- text: "Could you please confirm if the proposed implementation timeline aligns with your"
	example_title: "Client Inquiry Completion"
	- text: "We decoupled the event processing pipeline to achieve sub-"
	example_title: "Engineering Autocomplete"
	- text: "We dont expect any downtime during the blue-green deployment phase."
	example_title: "Grammar & Typo Correction"
	---

	<p align="center">
	<img src="typoformer_full_logo.png" alt="TypoFormer Banner" width="620"/>
	</p>

	<h1 align="center">TypoFormer (Stage 2 Open Weights)</h1>

	<p align="center">
	<strong>High-Precision Character-Aware Hybrid Transformer for Edge Autocomplete & Non-Autoregressive Grammar Correction</strong><br/>
	<sub>Created & Authored by <strong><a href="https://github.com/fahiiim">Md. Fahim Sarker Mridul</a></strong></sub>
	</p>

	<p align="center">
	<a href="https://github.com/fahiiim/TyPoTaKer"><img src="https://img.shields.io/badge/GitHub-fahiiim%2FTyPoTaKer-181717?style=for-the-badge&logo=github&logoColor=white" alt="GitHub Repository"/></a>
	<img src="https://img.shields.io/badge/Author-Md._Fahim_Sarker_Mridul-1E293B?style=for-the-badge&logo=github&logoColor=white" alt="Author"/>
	<img src="https://img.shields.io/badge/Model-TypoFormer--v2.4-76B900?style=for-the-badge&logo=nvidia&logoColor=white" alt="Model Version"/>
	<img src="https://img.shields.io/badge/Parameters-93.88M-4F8A0E?style=for-the-badge&logo=cpu&logoColor=white" alt="Parameters"/>
	<img src="https://img.shields.io/badge/Formats-PyTorch_%7C_Safetensors_%7C_INT8_ONNX-2B5A0C?style=for-the-badge" alt="Formats"/>
	<a href="https://huggingface.co/datasets/fahiiim/TypoFormer-Synthetic-Dataset"><img src="https://img.shields.io/badge/Dataset-53k_Synthetic_GEC_%26_LM-E67E22?style=for-the-badge" alt="Dataset"/></a>
	<img src="https://img.shields.io/badge/Latency-Sub--10ms_CPU-1A3B05?style=for-the-badge&logo=speedtest&logoColor=white" alt="Latency"/>
	<img src="https://img.shields.io/badge/License-MIT-blue?style=for-the-badge" alt="License"/>
	</p>

	<p align="center">
	<a href="#model-architecture"><strong>Architecture</strong></a> \|
	<a href="#key-highlights"><strong>Highlights</strong></a> \|
	<a href="#training-checkpoints-and-resumption"><strong>Checkpoints</strong></a> \|
	<a href="#architectural-comparison-typoformer-vs-bert-vs-cloud-llms"><strong>Comparison</strong></a> \|
	<a href="#benchmark-and-evaluation-metrics"><strong>Benchmarks</strong></a> \|
	<a href="#repository-contents"><strong>Files</strong></a> \|
	<a href="#quickstart-and-usage"><strong>Quickstart</strong></a> \|
	<a href="#fine-tuning-guide"><strong>Fine-Tuning</strong></a> \|
	<a href="#windows-desktop-keystroke-companion-1-line-install"><strong>Windows Desktop</strong></a> \|
	<a href="#citation-and-author-attribution"><strong>Citation</strong></a>
	</p>

	---

	## Overview

	TypoFormer is a purpose-engineered, character-aware neural backbone designed from the ground up to solve two conflicting natural language problems inside a single unified 94M parameter trunk:

	1. Causal Next-Word Autocomplete: Low-latency autoregressive token prediction under strict sub-15ms local keystroke budgets without cloud connectivity.
	2. Non-Autoregressive Grammar & Typo Correction (GEC): High-precision edit-tag sequence labeling that repairs punctuation slips, confused vocabulary, subject-verb agreement, and severe typos without generative hallucination.

	By fusing Byte-Level BPE subwords with Multi-Scale Character 1D-Convolutions through a learned gating highway, TypoFormer eliminates out-of-vocabulary token fractures and delivers enterprise-grade real-time writing assistance completely offline on standard desktop CPUs.

	This repository publishes both:
	* Trainable PyTorch Safetensors Checkpoints with architecture source code (`modeling_typoformer.py`) and training runner (`finetune.py`) for community researchers to continue training or fine-tune on custom domain datasets.
	* Quantized INT8 ONNX Runtimes for instant edge deployment in desktop environments.

	---

	## Model Architecture

	<p align="center">
	<img src="typoformer_architecture.png" alt="TypoFormer Full Internal Neural Architecture Diagram" width="620"/>
	</p>
	<p align="center">
	<em>Figure 1: TypoFormer Full Internal Neural Architecture: Fusing byte-level subwords with multi-scale character convolutions, Pre-RMSNorm grouped-query attention (GQA), rotary embeddings (RoPE), SwiGLU feed-forward networks, and multi-task dual heads for offline autocomplete and grammar error correction (Visualized in NVIDIA Emerald Green <code>#76B900</code>).</em><br/>
	<a href="https://huggingface.co/fahiiim/TypoFormer/resolve/main/typoformer_architecture.png"><strong>[View High-Resolution 300 DPI Architecture Diagram (PNG)]</strong></a>
	</p>

	### 1. Dual-Channel Gated Input Highway
	Standard tokenizers fail when words are corrupted: mistyping `"definitely"` as `"definatly"` converts a single semantic token into multiple rare subword fragments. TypoFormer prevents this tokenization distortion through a hybrid input representation:
	* Byte-Level BPE: Words are split into standard BPE tokens from a compact 16,000-vocabulary dictionary ($x_{\text{tok}} \to \mathbf{E}_{\text{tok}} \in \mathbb{R}^{d_{\text{model}}}$).
	* Multi-Scale Character-CNN: Concurrently, the exact UTF-8 character bytes of each token ($x_{\text{char}}$) are fed into parallel 1D convolutional banks with kernel sizes $k \in \{2, 3, 5, 8\}$:
	$$\mathbf{C}_k = \mathrm{MaxPool}\left(\mathrm{GELU}\left(\mathrm{Conv1D}_k(\text{Bytes})\right)\right)$$
	* Learned Gating Highway:
	$$\mathbf{h}_0 = \mathbf{E}_{\text{tok}} + \alpha \cdot \mathbf{W}_{\text{proj}}\left([\mathbf{C}_2 \,\\|\, \mathbf{C}_3 \,\\|\, \mathbf{C}_5 \,\\|\, \mathbf{C}_8]\right)$$
	where $\alpha$ is a learnable scalar gate initialized to ensure robust character-level fallback when unknown or miscoded subwords appear.

	### 2. Transformer Trunk (12 Blocks, $d=768$, $d_{\text{ffn}}=2048$)
	* Attention Mechanism: Grouped-Query Attention (GQA) with 12 Query heads and 4 Key-Value heads (3:1 ratio), cutting KV-cache memory traffic by 66% during sequence generation.
	* Rotary Position Embeddings (RoPE): Frequency-based relative query-key rotation for superior relative token distance awareness across cursor movements without absolute position limits.
	* Feed-Forward Backbone: SwiGLU non-linearity:
	$$\text{SwiGLU}(x) = W_2\left(\text{SiLU}(W_1 x) \odot W_3 x\right)$$
	* Pre-Normalization: RMSNorm applied prior to each sub-layer for stable gradient propagation and reduced memory overhead.

	### 3. Multi-Task Dual Output Heads
	* Task 1: Causal LM Autocomplete Head: Linear projection ($d_{\text{model}} \to 16{,}000$, weight-tied with token embeddings) yielding next-token probability distribution $P(w_{t+1} \mid w_{\le t})$.
	* Task 2: Keystroke GEC Edit Head: Linear tagging projection ($d_{\text{model}} \to 6{,}148$) mapping each token position to atomic edits:
	* `KEEP` (0): Token is grammatically and lexically correct.
	* `DELETE` (1): Redundant or erroneous token removal.
	* `CASE_CAP` (2) / `CASE_LOWER` (3): Casing adjustments.
	* `REPLACE` (4 .. 4099): Atomic swap with one of 4,096 high-frequency correction tokens.
	* `APPEND` (4100 .. 6147): Atomic right-insertion with one of 2,048 suffix/continuation tokens.

	---

	## Training Checkpoints and Resumption

	For developers and researchers wanting to resume training where we left off or fine-tune TypoFormer on custom domain datasets, this repository includes the complete Stage 2 trained checkpoints:

	\| Checkpoint File \| Format \| Parameters \| Training Step \| Description \|
	\| :--- \| :--- \| :--- \| :--- \| :--- \|
	\| `gec_stage2_model.safetensors` \| Safetensors \| 93.88M \| Step 2,500 \| Stage 2 Wine Fine-Tuned PyTorch weights for GEC Edit Tagging \|
	\| `lm_stage2_model.safetensors` \| Safetensors \| 93.88M \| Step 1,000 \| Stage 2 Wine Fine-Tuned PyTorch weights for Causal LM Autocomplete \|
	\| `gec_trainer_state.json` \| JSON \| N/A \| Step 2,500 \| Training metadata, samples seen (320,000), and learning rate state \|
	\| `lm_trainer_state.json` \| JSON \| N/A \| Step 1,000 \| Training metadata, samples seen (64,000), and learning rate state \|
	\| `modeling_typoformer.py` \| Python \| N/A \| Standalone \| Complete PyTorch architecture definition with `from_checkpoint` loader \|
	\| `finetune.py` \| Python \| N/A \| Standalone \| PyTorch training runner for domain adaptation and continual learning \|

	### Loading Checkpoints in PyTorch

	```python
	import torch
	from modeling_typoformer import TypoFormer

	# 1. Load GEC Model from checkpoint
	gec_model = TypoFormer.from_checkpoint("gec_stage2_model.safetensors")
	gec_model.eval()

	# 2. Load Causal LM Model from checkpoint
	lm_model = TypoFormer.from_checkpoint("lm_stage2_model.safetensors")
	lm_model.eval()

	print("Both models loaded successfully with exact Stage 2 weights!")
	```

	---

	## Fine-Tuning Guide

	You can fine-tune TypoFormer on your own text data using `finetune.py`:

	```bash
	# 1. Fine-tune Autocomplete on custom text
	python finetune.py \
	--task lm \
	--checkpoint lm_stage2_model.safetensors \
	--tokenizer tokenizer.json \
	--output typoformer_lm_finetuned.safetensors \
	--epochs 3 \
	--batch_size 4 \
	--lr 1e-4

	# 2. Fine-tune Grammar & Typo Correction
	python finetune.py \
	--task gec \
	--checkpoint gec_stage2_model.safetensors \
	--tokenizer tokenizer.json \
	--output typoformer_gec_finetuned.safetensors \
	--epochs 3 \
	--batch_size 4 \
	--lr 8e-5
	```

	### Stage 2 Hyperparameters
	* Optimizer: AdamW ($\beta_1=0.9, \beta_2=0.95$, weight decay $0.1$)
	* Precision: BF16 / Mixed Precision
	* Peak Learning Rate: $6 \times 10^{-4}$ (pre-training) down to $8 \times 10^{-5}$ (Stage 2 fine-tuning)
	* Gradient Clipping: $1.0$
	* GEC Loss Weighting: Cross-Entropy with `keep_downweight=0.15` to prioritize correcting rare typos over majority `KEEP` tokens.

	---

	## Key Highlights

	* Ultra-Low Keystroke Latency: Mean end-to-end CPU inference of 9.97 ms (P95: 12.12 ms), fitting comfortably inside the 16.6 ms 60 FPS display frame deadline.
	* Typo Immunity: Gated character-level convolutions preserve word roots even under severe phonetic and fat-finger keyboard slips.
	* 100% Air-Gapped and Offline: Zero network calls, zero telemetry, zero risk of leaking confidential documents, code, or private emails.
	* Guaranteed Content Fidelity: Non-autoregressive tagging guarantees zero generative hallucination and achieves 100.0% clean negative sentence preservation.
	* Dual Runtime Availability: Safetensors for PyTorch training and INT8 ONNX for sub-10ms edge execution.

	---

	## Architectural Comparison: TypoFormer vs. BERT vs. Cloud LLMs

	\| Architectural Dimension \| Classical BERT-Base (110M) \| Cloud Generative LLMs (7B - 70B) \| TypoFormer (94M) \| TypoFormer Advantage \|
	\| :--- \| :--- \| :--- \| :--- \| :--- \|
	\| Generative Autocomplete \| Incapable (Bidirectional MLM only) \| Yes (High quality autoregression) \| Native Causal Mode (Tied causal head) \| Unified Trunk: Single model serves both autocomplete & GEC. \|
	\| Typo & OOV Robustness \| Brittle (`teh` to `['te', '##h']`) \| Fair (Relies on large BPE dictionary) \| Gated Multi-Scale Char-CNN ($k=2,3,5,8$) \| Typo Immunity: Never blinded by unknown character corruptions. \|
	\| Position Encoding \| Absolute Learned (Max 512 tokens) \| RoPE / ALiBi \| Rotary Position Embeddings (RoPE) \| Sequence Extrapolation: Relative distance aware across cursor hops. \|
	\| Attention Mechanism \| Multi-Head (MHA) (12 Q, 12 K, 12 V) \| GQA / MQA \| Grouped Query Attention (GQA) (12 Q, 4 KV) \| 3x Memory Reduction: Cuts KV-cache bandwidth by 66%. \|
	\| FFN Non-Linearity \| GeLU + LayerNorm (Additive bias) \| SwiGLU + RMSNorm \| SwiGLU + Pre-RMSNorm \| LLaMA/Mistral-Grade Efficiency: Higher expressivity per parameter. \|
	\| GEC Formulation \| Seq2Seq / MLM (Slow, iterative) \| Generative Prompting (Prone to hallucinations) \| Constrained Atomic Edit-Tagging (6,148 tags) \| Zero Hallucination: 100% preservation of correct text. \|
	\| End-to-End Latency \| 150ms - 400ms on CPU \| 300ms - 2,500ms via Cloud API \| < 10ms on CPU (INT8 ONNX) \| Real-Time Keystroke Budget: Executes within 16ms display frame. \|
	\| Network & Privacy \| Local or Cloud \| Mandatory Cloud API (Data leaves host) \| 100% Offline Air-Gapped \| Complete Privacy: Zero telemetry, password field suppression. \|

	---

	## Benchmark and Evaluation Metrics

	Live CPU benchmarks measured on standard Intel/AMD consumer hardware ([scripts/evaluate_gec_live.py](https://github.com/fahiiim/TyPoTaKer)):

	\| Metric \| Measured Value \| Benchmark Description \|
	\| :--- \| :--- \| :--- \|
	\| Mean CPU Latency \| `9.97 ms` \| Per-sentence end-to-end inference (INT8 ONNX) \|
	\| P95 CPU Latency \| `12.12 ms` \| 95th percentile latency (well below 16.6 ms display frame) \|
	\| P99 CPU Latency \| `14.80 ms` \| 99th percentile worst-case latency \|
	\| GEC Stage 2 Validation Loss \| `0.2597` \| Down from 4.1067 (93.7% loss reduction across training) \|
	\| LM Stage 2 Loss \| `1.2054` \| Fine-tuned on curated engineering and client communications \|
	\| Clean Negative Preservation \| `100.0%` \| Zero false-positive edits on grammatically correct input text \|
	\| GEC Model Footprint \| `130.5 MB` (INT8) / `424.7 MB` (Safetensors) \| Symmetric INT8 ONNX graph vs FP32 PyTorch weights \|
	\| LM Model Footprint \| `138.1 MB` (INT8) / `424.7 MB` (Safetensors) \| Symmetric INT8 ONNX graph vs FP32 PyTorch weights \|
	\| Active Working Set RAM \| `< 250 MB` \| Total memory footprint during active inference \|

	---

	## Repository Contents

	\| File \| Size \| Format \| Description \|
	\| :--- \| :--- \| :--- \| :--- \|
	\| `gec_stage2_model.safetensors` \| 424.7 MB \| Safetensors \| Full Stage 2 PyTorch checkpoint for GEC training & fine-tuning \|
	\| `lm_stage2_model.safetensors` \| 424.7 MB \| Safetensors \| Full Stage 2 PyTorch checkpoint for LM Autocomplete fine-tuning \|
	\| `gec_stage2_int8.onnx` \| 130.5 MB \| ONNX INT8 \| Quantized graph for sub-10ms CPU grammar and typo correction \|
	\| `lm_stage2_int8.onnx` \| 138.1 MB \| ONNX INT8 \| Quantized graph for sub-10ms CPU causal autocomplete \|
	\| `modeling_typoformer.py` \| 11.6 KB \| Python \| Standalone PyTorch architecture module with `from_checkpoint` helper \|
	\| `finetune.py` \| 6.4 KB \| Python \| Standalone PyTorch fine-tuning runner \|
	\| `infer.py` \| 6.5 KB \| Python \| Standalone Python inference engine using ONNX Runtime \|
	\| `edit_maps.json` \| 228 KB \| JSON \| High-frequency confusion dictionary (4,096 replacements + 2,048 appends) \|
	\| `tokenizer.json` \| 1.11 MB \| JSON \| Byte-Level BPE tokenizer config (16,000 vocabulary) \|
	\| `vocab.json` / `merges.txt` \| 526 KB \| Text/JSON \| BPE vocabulary and merge table \|
	\| `config.json` \| 780 B \| JSON \| Architecture parameters & model metadata \|
	\| `gec_trainer_state.json` \| 705 B \| JSON \| Stage 2 GEC training state at step 2,500 \|
	\| `lm_trainer_state.json` \| 703 B \| JSON \| Stage 2 LM training state at step 1,000 \|
	\| `typoformer_architecture.png`\| 235 KB \| PNG (300 DPI) \| Full neural architecture diagram (NVIDIA emerald theme) \|
	\| `typoformer_full_logo.png` \| 188 KB \| PNG \| Official project branding logo \|
	\| `TypoFormer-win-x64.zip` \| 221.0 MB \| ZIP \| Standalone Windows distribution package \|

	---

	## Quickstart and Usage

	### 1. Minimal Installation

	```bash
	pip install torch safetensors onnxruntime tokenizers numpy
	git clone https://huggingface.co/fahiiim/TypoFormer
	cd TypoFormer
	```

	### 2. Standalone Python Inference (ONNX)

	```python
	from infer import TypoFormer

	# Initialize the standalone inference engine (loads ONNX models locally)
	model = TypoFormer(".")

	# 1. Autocomplete (Next-Word Generation)
	prompt = "Could you please confirm if the proposed implementation timeline aligns with your"
	suggestions = model.autocomplete(prompt, top_k=3)
	print("Autocomplete suggestions:", suggestions)
	# Output: [(' business', 0.9978), (' goals', 0.0002), (' company', 0.0001)]

	# 2. Grammar, Punctuation & Typo Correction (GEC)
	dirty_text = "We decoupled the event processing pipeline to achieve sub-"
	suggestions = model.autocomplete(dirty_text, top_k=2)
	print("Engineering completion:", suggestions)
	# Output: [(' millisecond', 0.8841), (' second', 0.0712)]
	```

	---

	## Windows Desktop Keystroke Companion (1-Line Install)

	TypoFormer includes a native Windows x64 .NET 8 desktop daemon that runs silently in the system tray and renders a floating caret overlay across any active text editor (Word, Notepad, VS Code, Slack, Browser):

	Run in PowerShell:
	```powershell
	irm https://raw.githubusercontent.com/fahiiim/TyPoTaKer/main/install.ps1 \| iex
	```

	* Offline & Private: Zero telemetry; no keystrokes ever leave your device.
	* Password Aware: Automatically suppresses buffering on password and credential fields via Windows UI Automation.
	* Instant Action: Press `[Tab]` to accept autocomplete suggestions or grammar corrections.

	---

	## Citation and Author Attribution

	```bibtex
	@software{typoformer2026,
	author = {Md. Fahim Sarker Mridul},
	title = {TypoFormer: High-Precision Character-Aware Hybrid Transformer for Edge Autocomplete & Non-Autoregressive Grammar Correction},
	year = {2026},
	publisher = {Hugging Face},
	url = {https://huggingface.co/fahiiim/TypoFormer},
	howpublished = {\url{https://github.com/fahiiim/TyPoTaKer}}
	}
	```

	## License
	This model and its associated weights are distributed under the [MIT License](LICENSE).
	Authored by Md. Fahim Sarker Mridul.