README.md

---
license: mit
language:
- zh
tags:
- chinese
- word-segmentation
- word-sense-disambiguation
- coreml
- ios
- epub-reader
datasets:
- wyy209/MiCLS
base_model:
- hfl/chinese-electra-180g-base-discriminator
- hfl/chinese-electra-180g-small-discriminator
- thenlper/gte-base-zh
---

# Wen Reader — On-Device Chinese NLP Models

Model weights for [Wen Reader](https://github.com/oliverzh2000/wen-reader), an open-source Chinese EPUB reader for iOS with popup dictionary.

These models run on-device via CoreML to provide context-aware Chinese word segmentation and word sense disambiguation.

## Models

### cws-span-scorer-electra-base

Chinese Word Segmentation model using span scoring.

- **Architecture:** ELECTRA-base (12 layers, 768 hidden, 12 heads) + span scoring MLP head
- **Base model:** [hfl/chinese-electra-180g-base-discriminator](https://huggingface.co/hfl/chinese-electra-180g-base-discriminator)
- **Parameters:** ~102M (encoder) + span head
- **Task:** Given a sentence, score candidate word spans to find optimal segmentation
- **Eval:** 90/97 perfect segmentation on hand-curated test set (93/97 including acceptable over-splits)

### cws-span-scorer-electra-small

Smaller/faster variant of the above for lower-end devices.

- **Architecture:** ELECTRA-small (12 layers, 256 hidden, 4 heads) + span scoring MLP head
- **Base model:** [hfl/chinese-electra-180g-small-discriminator](https://huggingface.co/hfl/chinese-electra-180g-small-discriminator)
- **Parameters:** ~12M (encoder) + span head
- **Eval:** 85/97 perfect (91/97 including acceptable over-splits)

### wsd-biencoder-gte-base

Word Sense Disambiguation bi-encoder for selecting the correct dictionary definition in context.

- **Architecture:** GTE-base-zh fine-tuned as a bi-encoder (SentenceTransformers)
- **Base model:** [thenlper/gte-base-zh](https://huggingface.co/thenlper/gte-base-zh)
- **Parameters:** ~102M
- **Task:** Encode context sentence and candidate sense labels, rank by cosine similarity
- **Eval:** Top-1 accuracy 88.7%, Top-3 99.6%, MRR 0.936 on 239 hand-curated disambiguation examples

## Usage

These models are used by the Wen Reader iOS app via CoreML conversion.

To build the app from a fresh clone:

```bash
cd ml

# 1. Download model weights from HF Hub
uv run python scripts/download_models.py

# 2. Export to CoreML
uv run python scripts/export_coreml.py span --model-dir models/cws_span_scorer_electra_base/final
uv run python scripts/export_coreml.py wsd --model-dir models/wsd_biencoder_gte_base/final

# 3. Bundle into app resources (vocab, CoreML packages, CEDICT database)
./scripts/run_pipeline.sh bundle
```

Or use the pipeline script:

```bash
./scripts/run_pipeline.sh download
./scripts/run_pipeline.sh export
./scripts/run_pipeline.sh bundle
```

## Training

### CWS Span Scorer

Fine-tuned on a custom dataset built from:
- [ICWB2](https://github.com/yuikns/icwb2-data) segmentation corpus (MSR, auto-matched where greedy segmenter agrees with gold, plus LLM-annotated disagreement cases)
- Chinese [Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) and [OpenSubtitles](https://huggingface.co/datasets/FradSer/OpenSubtitles-en-zh-cn-20m) sentences, LLM-annotated (Claude Opus)

Sentences with ambiguous segmentation boundaries (where multiple valid CEDICT words overlap) are
identified, then an LLM annotates the correct segmentation. The model scores all candidate word
spans at each ambiguous position using a span scoring MLP head (taking span boundary token
representations + learned width embeddings as input). A dynamic programming decoder finds the
optimal segmentation at inference time. Training uses cross-entropy loss over candidate spans at
each ambiguous position. Encoder and head use discriminative learning rates.

### WSD Bi-encoder

Fine-tuned on CC-CEDICT sense clusters with data from:
- [MiCLS](https://huggingface.co/datasets/wyy209/MiCLS) WSD corpus (mapped to CEDICT senses via BGE embedding similarity)
- LLM-generated context examples for each sense cluster (Claude Opus and Sonnet)
- LLM-annotated ebook sentences segmented by the CWS span scorer (Claude Sonnet)

Sense clusters are derived from CC-CEDICT entries with related senses merged using LLM-assisted
clustering. Chinese sense labels are LLM-translated from the English CEDICT definitions.

Training uses grouped cross-entropy loss: contexts and sense labels are encoded by the same model,
cosine similarity is computed between context embeddings and sense embeddings, and the loss is
cross-entropy over the similarity scores (with temperature scaling at τ=0.1). Contexts are grouped
by word in each batch so sense embeddings are encoded once and reused across all contexts for that word.