Upload README.md with huggingface_hub

README.md

@@ -31,7 +31,7 @@ Chinese Word Segmentation model using span scoring.
 
 - **Architecture:** ELECTRA-base (12 layers, 768 hidden, 12 heads) + span scoring MLP head
 - **Base model:** [hfl/chinese-electra-180g-base-discriminator](https://huggingface.co/hfl/chinese-electra-180g-base-discriminator)
-- **Parameters:** ~110M (encoder) + span head
+- **Parameters:** ~102M (encoder) + span head
 - **Task:** Given a sentence, score candidate word spans to find optimal segmentation
 - **Eval:** 90/97 perfect segmentation on hand-curated test set (93/97 including acceptable over-splits)
 
@@ -48,44 +48,66 @@ Smaller/faster variant of the above for lower-end devices.
 
 Word Sense Disambiguation bi-encoder for selecting the correct dictionary definition in context.
 
-- **Architecture:** BERT-base fine-tuned as a bi-encoder (SentenceTransformers)
-- **Base model:** [thenlper/gte-base-zh](https://huggingface.co/thenlper/gte-base-zh) (GTE Chinese base)
-- **Parameters:** ~110M
+- **Architecture:** GTE-base-zh fine-tuned as a bi-encoder (SentenceTransformers)
+- **Base model:** [thenlper/gte-base-zh](https://huggingface.co/thenlper/gte-base-zh)
+- **Parameters:** ~102M
 - **Task:** Encode context sentence and candidate sense labels, rank by cosine similarity
 - **Eval:** Top-1 accuracy 88.7%, Top-3 99.6%, MRR 0.936 on 239 hand-curated disambiguation examples
 
 ## Usage
 
-These models are used by the Wen Reader iOS app via CoreML conversion. To export to CoreML:
+These models are used by the Wen Reader iOS app via CoreML conversion.
+
+To build the app from a fresh clone:
 
 ```bash
 cd ml
+
+# 1. Download model weights from HF Hub
+uv run python scripts/download_models.py
+
+# 2. Export to CoreML
 uv run python scripts/export_coreml.py span --model-dir models/cws_span_scorer_electra_base/final
 uv run python scripts/export_coreml.py wsd --model-dir models/wsd_biencoder_gte_base/final
+
+# 3. Bundle into app resources (vocab, CoreML packages, CEDICT database)
+./scripts/run_pipeline.sh bundle
+```
+
+Or use the pipeline script:
+
+```bash
+./scripts/run_pipeline.sh download
+./scripts/run_pipeline.sh export
+./scripts/run_pipeline.sh bundle
 ```
 
 ## Training
 
 ### CWS Span Scorer
 
-Fine-tuned on a custom dataset (~914k span-scoring examples from ~104k sentences) built from:
-- [ICWB2](https://github.com/yuikns/icwb2-data) segmentation corpus (MSR, auto-matched and LLM-annotated disagreements)
-- LLM-annotated Chinese from ebooks (Claude Sonnet)
-- LLM-annotated wiki/subtitle data (Claude Opus)
+Fine-tuned on a custom dataset built from:
+- [ICWB2](https://github.com/yuikns/icwb2-data) segmentation corpus (MSR, auto-matched where greedy segmenter agrees with gold, plus LLM-annotated disagreement cases)
+- Chinese [Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) and [OpenSubtitles](https://huggingface.co/datasets/FradSer/OpenSubtitles-en-zh-cn-20m) sentences, LLM-annotated (Claude Opus)
 
-The model scores all candidate word spans at each position using a span scoring MLP head
-(taking span boundary representations + width embeddings as input). A dynamic programming decoder
-finds the optimal segmentation at inference time. Training uses cross-entropy loss over candidate
-spans at each ambiguous position. Encoder and head use discriminative learning rates.
+Sentences with ambiguous segmentation boundaries (where multiple valid CEDICT words overlap) are
+identified, then an LLM annotates the correct segmentation. The model scores all candidate word
+spans at each ambiguous position using a span scoring MLP head (taking span boundary token
+representations + learned width embeddings as input). A dynamic programming decoder finds the
+optimal segmentation at inference time. Training uses cross-entropy loss over candidate spans at
+each ambiguous position. Encoder and head use discriminative learning rates.
 
 ### WSD Bi-encoder
 
-Fine-tuned on CC-CEDICT sense clusters built from:
-- [MiCLS](https://huggingface.co/datasets/wyy209/MiCLS) WSD corpus
-- Chinese sense labels generated by LLM translation of English CEDICT definitions
-- Context examples generated by LLM for each sense cluster
+Fine-tuned on CC-CEDICT sense clusters with data from:
+- [MiCLS](https://huggingface.co/datasets/wyy209/MiCLS) WSD corpus (mapped to CEDICT senses via BGE embedding similarity)
+- LLM-generated context examples for each sense cluster (Claude Opus and Sonnet)
+- LLM-annotated ebook sentences segmented by the CWS span scorer (Claude Sonnet)
+
+Sense clusters are derived from CC-CEDICT entries with related senses merged using LLM-assisted
+clustering. Chinese sense labels are LLM-translated from the English CEDICT definitions.
 
 Training uses grouped cross-entropy loss: contexts and sense labels are encoded by the same model,
 cosine similarity is computed between context embeddings and sense embeddings, and the loss is
-cross-entropy over the similarity scores (with temperature scaling). Contexts are grouped by word
-in each batch so sense embeddings are encoded once and reused across all contexts for that word.
+cross-entropy over the similarity scores (with temperature scaling at τ=0.1). Contexts are grouped
+by word in each batch so sense embeddings are encoded once and reused across all contexts for that word.