Add wapic-cws release model (wapic-20260605=h25_30) + base + pattern
Browse files- .gitattributes +2 -0
- README.md +70 -0
- base-v5_p0.01.wac +3 -0
- pattern.txt +16 -0
- wapic-20260605.wac +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
base-v5_p0.01.wac filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
wapic-20260605.wac filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language: zh
|
| 3 |
+
license: mit
|
| 4 |
+
library_name: wapic
|
| 5 |
+
pipeline_tag: token-classification
|
| 6 |
+
tags:
|
| 7 |
+
- chinese-word-segmentation
|
| 8 |
+
- crf
|
| 9 |
+
- sequence-labeling
|
| 10 |
+
- chinese
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
# Wapic 中文分词模型 (wapic-cws)
|
| 14 |
+
|
| 15 |
+
线性链 CRF(L-BFGS / OWL-QN 训练)中文分词模型,采用 BMES 标注。
|
| 16 |
+
遵循现代中文标准:**人名保持整体**(李镇全、林强峰)、标点独立、数字/英文按字符类型边界切分。
|
| 17 |
+
|
| 18 |
+
代码与训练框架:<https://github.com/Ismantic/Wapic>(C++17 重构的 [Wapiti](https://wapiti.limsi.fr/))。
|
| 19 |
+
|
| 20 |
+
## 文件
|
| 21 |
+
|
| 22 |
+
| 文件 | 说明 |
|
| 23 |
+
|---|---|
|
| 24 |
+
| `wapic-20260605.wac` | **主模型**(发布代号 h25_30,48 MB,二进制格式) |
|
| 25 |
+
| `base-v5_p0.01.wac` | Stage-1 base,warm-start 锚点(仅复现训练时需要) |
|
| 26 |
+
| `pattern.txt` | 特征模板(unigram + bigram) |
|
| 27 |
+
|
| 28 |
+
## 指标
|
| 29 |
+
|
| 30 |
+
在 retag2 gold 测试集上(见 `Ismantic/wapic-cws-data`):
|
| 31 |
+
|
| 32 |
+
| 测试集 | F1 | P | R |
|
| 33 |
+
|---|---|---|---|
|
| 34 |
+
| PD-1998 (modern+punct) | **97.71** | 97.63 | 97.79 |
|
| 35 |
+
| 12M 泛化集 | **97.49** | 97.58 | 97.41 |
|
| 36 |
+
|
| 37 |
+
人名回归 case:15/15 全过。
|
| 38 |
+
|
| 39 |
+
## 用法
|
| 40 |
+
|
| 41 |
+
需先从源码编译 `wapic`(见 GitHub 仓库):
|
| 42 |
+
|
| 43 |
+
```bash
|
| 44 |
+
# 交互式分词
|
| 45 |
+
./build/wapic -m wapic-20260605.wac
|
| 46 |
+
# >>> 李镇全今天接受了记者的采访
|
| 47 |
+
# 李镇全 今天 接受 了 记者 的 采访
|
| 48 |
+
|
| 49 |
+
# 批量标注:输入为每行一个字符、空行分句
|
| 50 |
+
./build/wapic test -m wapic-20260605.wac input_nolabel.txt output_tags.txt
|
| 51 |
+
```
|
| 52 |
+
|
| 53 |
+
Python 绑定(pybind11,见仓库 `build_py`):
|
| 54 |
+
|
| 55 |
+
```python
|
| 56 |
+
import wapic
|
| 57 |
+
seg = wapic.Segmenter("wapic-20260605.wac")
|
| 58 |
+
seg.cut("中华人民共和国是一个伟大的国家")
|
| 59 |
+
```
|
| 60 |
+
|
| 61 |
+
## 训练
|
| 62 |
+
|
| 63 |
+
两阶段 warm-start:`base-v5_p0.01.wac`(Stage-1,12M LTP 语料)
|
| 64 |
+
→ 在 name-rich / 现代语料 mix 上 `--init-from` fine-tune → 剪枝。
|
| 65 |
+
完整配方与数据见数据集仓库 `Ismantic/wapic-cws-data` 的 `RELEASE_TRAINING_DATA.md`。
|
| 66 |
+
|
| 67 |
+
## 许可
|
| 68 |
+
|
| 69 |
+
MIT(代码/模型权重)。训练语料源自人民日报 1998、LTP 标注及公开新闻语料,
|
| 70 |
+
原始语料版权归各自所有者。
|
base-v5_p0.01.wac
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9ea01696e05faabff0beecc5e90d438b3a5f97ec550e06ae56f5e1f4eb8f4ee2
|
| 3 |
+
size 45829257
|
pattern.txt
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Unigram features
|
| 2 |
+
U00:%x[-2,0]
|
| 3 |
+
U01:%x[-1,0]
|
| 4 |
+
U02:%x[0,0]
|
| 5 |
+
U03:%x[1,0]
|
| 6 |
+
U04:%x[2,0]
|
| 7 |
+
|
| 8 |
+
# Bigram character combinations
|
| 9 |
+
U05:%x[-1,0]/%x[0,0]
|
| 10 |
+
U06:%x[0,0]/%x[1,0]
|
| 11 |
+
U07:%x[-2,0]/%x[-1,0]
|
| 12 |
+
U08:%x[1,0]/%x[2,0]
|
| 13 |
+
U09:%x[-1,0]/%x[1,0]
|
| 14 |
+
|
| 15 |
+
# Bigram (label transition)
|
| 16 |
+
B
|
wapic-20260605.wac
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f1a6dc64275730f07d5df441d2f11a1c93f86cae5e89f109fb768ecaab0154ee
|
| 3 |
+
size 50511571
|