README.md

---
license: apache-2.0
language:
  - zh
tags:
  - input-method
  - zhuyin
  - bopomofo
  - traditional-chinese
  - ternary
  - bitnet
  - gguf
library_name: gguf
pipeline_tag: token-classification
---

# Sloth IME models — Zhuyin → Traditional Chinese, fully on-device

**The model family behind Slothing (樹懶智慧輸入法)**, all trained from scratch:

| file | role | size |
|---|---|---|
| `slothe-t-12m-256x12.gguf` (+ `12m/` fp32) | **conversion encoder — shipping default** | 9.65 MB |
| `slothe-t-25m.gguf` (+ fp32 master) | conversion encoder — accuracy reference | 18 MB |
| `pred_q35_60m-q4.gguf` (+ `pred_q35_60m/` fp32) | **next-word decoder v2.1** (neural 聯想, TW-chat register FT) | 46 MB |

The sections below document the 25M reference encoder in depth; the 12M shipping
encoder and the 60M decoder are covered in the latency/files sections.

## 60M next-word decoder (v2.1)

Dense-Qwen3.5 (Gated DeltaNet + full attention every 4th layer), 16k word-piece vocab
(next word = one forward, 8.5 ms/word on BOOX @4t). Trained on 6.1M zh-TW sentences,
then register-fine-tuned on 149k PTT/Dcard chat sentences (`--init-from`, 2ep):

| eval (n=3000) | v2 | **v2.1 (shipped)** |
|---|---|---|
| TW chat held-out | 10.9 / 21.2 | **18.3 / 31.2** |
| fresh-C4 | 34.0 / 46.0 | 33.5 / 45.2 |

> **Honesty note.** Earlier versions reported 47.3/75.8 — that held-out was saturated with
> near-duplicates of the (small) training set and rewarded memorization; on genuinely fresh
> text that model scored 4.6/12.0. All predictor numbers here use never-seen fresh corpora.
> Reproduce: `predictor_qwen35.py` + `build_corpus_big.py` (in the GitHub repo) — see the
> Reproducibility section of the [project README](https://github.com/vieenrose/sloth-zhuyin-linux#reproducibility).

## SlothE-T — Ternary Zhuyin → Traditional Chinese conversion encoder

The conversion model behind **Slothing (懶音輸入法)**: a libchewing-free, on-device
LLM-powered 注音 (Bopomofo) input method that does **免選字** (candidate-free)
whole-sentence conversion — you type the phonetic keystream, it emits the sentence,
with no candidate list to scroll.

- **25M parameters**, ternary weights (W1.58A8), bidirectional encoder.
- **~7 MB** ternary-packed GGUF (TQ2_0) · 99 MB fp32 master.
- Runs on-device across four frontends: **fcitx5, IBus, Android, web**.

## What it does

Given a sequence of Zhuyin **syllables** (the input keystream, e.g. `ㄋㄧˇ ㄏㄠˇ`),
the model emits the **Traditional Chinese characters** for each position as an
aligned sequence-labeling task. The output head is **phonetic-legality-masked**:
at each position only the ~1–50 characters that are legal readings of that
syllable are scored, out of the full 8342-char vocabulary. This is what replaces
the traditional IME 選字 (candidate-selection) step.

## Architecture

| | |
|---|---|
| encoder | bidirectional (BERT-like), 16 layers |
| dim / ffn | 352 / 960 |
| heads | 8 query, 2 KV (GQA), head_dim 44, QK-norm |
| norm | RMSNorm, embed-norm, **SubLN** pre-norm before each ternary linear |
| quantization | ternary weights {−1, 0, +1} × per-output-channel **absmedian** scale; **int8** activations; QAT via STE |
| fp islands | boundary blocks (0 and 15) kept fp16 for stability (`fp_boundary=1`) |
| vocab | 1539 syllables (in) · 8342 characters (out) |

## Evaluation — honest held-out

Measured on **500 fresh zh-TW sentences** (C4-zh-TW, offset far past the training
window, filtered to be **absent from the training corpus**), then g2pW-labeled.

| metric | **this model** | 12M int8 (previous ship) | 32M fp teacher |
|---|---|---|---|
| **免選字** (whole-sentence exact) | **76 %** | 72 % | — |
| **homophone-hard** | **86 %** | 82 % | 83 % |
| **toneless** | **77 %** | 79 % | 81 % |

> **Note on 免選字.** Earlier project numbers (~84 %) were inflated by a benchmark
> leak — the reference set had been sampled from the training corpus, rewarding
> memorization. **76 %** is the honest whole-sentence held-out figure.
> *homophone-hard* and *toneless* are leak-clean throughout. On honest data this
> 25M ternary model **beats the previously-shipped 12M int8 on 免選字 and
> homophone-hard** while being smaller and faster.

### Quality vs. latency

![Held-out 免選字 vs. on-device latency](score_vs_latency.png)

Latency is per-6-syllable decode on a **BOOX (Snapdragon 662, no-dotprod ARM)**.
4M and 12M are measured (ORT int8). **Update (2026-07-17): the 25M ternary's ~9 ms was a
projection that real-device measurement later corrected** — measured on the BOOX
(ggml/libslothe, single 6-syllable forward): **18.5 ms at 4 threads / 31.9 ms at 2 threads**
(the projection missed that the per-kernel TQ2_0 speedup does not compound at full-model
scale on a no-dotprod core, and an early 8-thread default dragged the A53 little cores —
fixed). The Pareto story survives in a different form: a **12M ternary sibling
(dim 256×12 layers, zero TQ2_0 padding tax,  in this repo)**
measures **9.3 ms @4t / 15.8 ms @2t** on the same device at 84 % 免選字 / 84 % homophone —
the honest heir to the "~9 ms" claim.

| model | params | 免選字 | homophone | toneless | latency (BOOX) |
|---|---|---|---|---|---|
| 4M int8 | 4M | 70 % | 83 % | 74 % | 9.1 ms |
| 12M int8 | 12M | 72 % | 82 % | 79 % | 13.3 ms |
| **25M ternary** | **25M** | **76 %** | **86 %** | **77 %** | **18.5 ms\*** |
| **12M ternary 256×12** | **12M** | **84 %†** | **84 %†** | — | **9.3 ms\*** |

<sub>\* measured on-device (BOOX SD662, ggml/TQ2_0, 4 threads; the 25M's earlier "~9 ms" was a
projection — see the note above). † 230-sentence reference set (the 25M measures 85.7 % on that
same set on-device); the 免選字 columns above it use the 500-sentence held-out set.</sub>

## Training recipe (teacher-free)

- **Direct cross-entropy** on a g2pW-labeled zh-TW corpus, with **label smoothing 0.1**.
- **Long OneCycleLR schedule (32 epochs), early-stopped at the peak (epoch 24)** —
  the model overfits after (73 % 免選字 by ep32).
- **8-adjacency keyboard-error noise** (TAAI-2024 error model): simulated mis-keys
  are constrained to the QWERTY physical 8-neighbourhood rather than any
  edit-distance-1 syllable.
- DDP on 2× RTX 5090.

Knowledge distillation from a 32M teacher was tried and only **matched** (never
beat) this teacher-free recipe, so the teacher was dropped — simpler and ~2.4×
faster to train. The two levers that carry the quality are (a) label smoothing
(a teacher-free regularizer; +5 免選字 over plain CE) and (b) enough epochs to
exploit the no-overfit headroom.

## Deployment / inference

The ternary weights are shipped as a **GGUF** using ggml's mainline **TQ2_0**
ternary type, so the model runs on stock llama.cpp CPU kernels (ARM NEON, x86
AVX2, and the generic/WASM path). Measured `vec_dot` throughput on x86 AVX2:
**TQ2_0 ≈ 2.3× int8 (Q8_0)** — the fastest quant type in the table — and ~2× on
a no-dotprod ARM (Snapdragon 662). The ternary win is partly memory-bandwidth
(2 bits/weight = 4× fewer weight bytes), so it holds on both weak and
accelerator-equipped CPUs.

> **256-alignment note.** TQ2_0 uses 256-element blocks, so the model's
> in-features (352, 960) are zero-padded up to the next multiple of 256 in the
> GGUF (352 → 512, 960 → 1024). The padding is exact ternary-zero (loss-free) but
> costs some MACs; a future model with 256-aligned dims removes the tax.

## Files

| file | what |
|---|---|
| `slothe-t-25m.gguf` | 25M ternary GGUF — TQ2_0 blocks (layers 1–14) + fp16 islands/embed/head |
| `slothe-t-12m-256x12.gguf` | **12M ternary GGUF (shipping default)** — dim256×12, zero TQ2_0 padding tax, 9.65 MB |
| `12m/model.safetensors` | 12M fp32 master weights |
| `12m/slothe_config.json` / `12m/roles.json` | 12M config + ternary/fp tensor roles |
| `pred_q35_60m-q4.gguf` | 60M Q4 next-word predictor **v2.1** (qwen35 GDN hybrid; llama.cpp-loadable; chat-register FT) |
| `model.safetensors` | fp32 master weights (HF-native, non-pickle) |
| `slothe.pt` | fp32 master checkpoint with embedded `config` |
| `syl_vocab.json` | syllable tokenizer (input vocab) |
| `syl2legal.npz` | phonetic legality mask, 1539 syllables × 8342 chars (bool) |
| `train_slothe_ternary.py` | training script |
| `gate_slothe_ternary.py` | evaluation / gating script |
| `REPRODUCE.md` | end-to-end reproduction recipe |
| `NAMES.md` | GGUF tensor-name ↔ checkpoint-name map |

## Intended use & limits

- **Intended:** on-device Traditional-Chinese Bopomofo input (Taiwan readings).
- **Out of scope:** Simplified Chinese, mainland pinyin readings, free-form
  generation. The model only scores per-position legal characters — it is a
  converter, not a chat model.
- The whole-sentence (免選字) metric is honest-held-out and modest (76 %) by
  design: it is measured on unseen sentences, not the training distribution.

## Citation / related work

The keyboard-error model and the 免選字-vs-選字 framing are discussed in the
project's `docs/RELATED-WORK.md`, which positions this work against the TAAI-2024
cross-multi-IME system (李偉安, 葉展維, 張嘉惠, National Central University).