Duplicate from LTP/base

Co-authored-by: Feng YunLong <ylfeng@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,32 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+model.safetensors filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,160 @@
+![CODE SIZE](https://img.shields.io/github/languages/code-size/HIT-SCIR/ltp)
+![CONTRIBUTORS](https://img.shields.io/github/contributors/HIT-SCIR/ltp)
+![LAST COMMIT](https://img.shields.io/github/last-commit/HIT-SCIR/ltp)
+
+| Language                             | version                                                                                                                                                                                                                                                                                                                   |
+| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [Python](python/interface/README.md) | [![LTP](https://img.shields.io/pypi/v/ltp?label=LTP)](https://pypi.org/project/ltp) [![LTP-Core](https://img.shields.io/pypi/v/ltp-core?label=LTP-Core)](https://pypi.org/project/ltp-core)   [![LTP-Extension](https://img.shields.io/pypi/v/ltp-extension?label=LTP-Extension)](https://pypi.org/project/ltp-extension) |
+| [Rust](rust/ltp/README.md)           | [![LTP](https://img.shields.io/crates/v/ltp?label=LTP)](https://crates.io/crates/ltp)                                                                                                                                                                                                                                     |
+
+# LTP 4
+
+LTP（Language Technology Platform） 提供了一系列中文自然语言处理工具，用户可以使用这些工具对于中文文本进行分词、词性标注、句法分析等等工作。
+
+## 引用
+
+如果您在工作中使用了 LTP，您可以引用这篇论文
+
+```bibtex
+@article{che2020n,
+  title={N-LTP: A Open-source Neural Chinese Language Technology Platform with Pretrained Models},
+  author={Che, Wanxiang and Feng, Yunlong and Qin, Libo and Liu, Ting},
+  journal={arXiv preprint arXiv:2009.11616},
+  year={2020}
+}
+```
+
+**参考书：**
+由哈工大社会计算与信息检索研究中心（HIT-SCIR）的多位学者共同编著的《[自然语言处理：基于预训练模型的方法](https://item.jd.com/13344628.html)
+》（作者：车万翔、郭江、崔一鸣；主审：刘挺）一书现已正式出版，该书重点介绍了新的基于预训练模型的自然语言处理技术，包括基础知识、预训练词向量和预训练模型三大部分，可供广大LTP用户学习参考。
+
+### 更新说明
+
+- 4.2.0
+  - \[结构性变化\] 将 LTP 拆分成 2 个部分，维护和训练更方便，结构更清晰
+    - \[Legacy 模型\] 针对广大用户对于**推理速度**的需求，使用 Rust 重写了基于感知机的算法，准确率与 LTP3 版本相当，速度则是 LTP v3 的 **3.55** 倍，开启多线程更可获得 **17.17** 倍的速度提升，但目前仅支持分词、词性、命名实体三大任务
+    - \[深度学习模型\] 即基于 PyTorch 实现的深度学习模型，支持全部的6大任务（分词/词性/命名实体/语义角色/依存句法/语义依存）
+  - \[其他改进\] 改进了模型训练方法
+    - \[共同\] 提供了训练脚本和训练样例，使得用户能够更方便地使用私有的数据，自行训练个性化的模型
+    - \[深度学习模型\] 采用 hydra 对训练过程进行配置，方便广大用户修改模型训练参数以及对 LTP 进行扩展（比如使用其他包中的 Module）
+  - \[其他变化\] 分词、依存句法分析 (Eisner) 和 语义依存分析 (Eisner) 任务的解码算法使用 Rust 实现，速度更快
+  - \[新特性\] 模型上传至 [Huggingface Hub](https://huggingface.co/LTP)，支持自动下载，下载速度更快，并且支持用户自行上传自己训练的模型供LTP进行推理使用
+  - \[破坏性变更\] 改用 Pipeline API 进行推理，方便后续进行更深入的性能优化（如SDP和SDPG很大一部分是重叠的，重用可以加快推理速度），使用说明参见[Github快速使用部分](https://github.com/hit-scir/ltp)
+- 4.1.0
+  - 提供了自定义分词等功能
+  - 修复了一些bug
+- 4.0.0
+  - 基于Pytorch 开发，原生 Python 接口
+  - 可根据需要自由选择不同速度和指标的模型
+  - 分词、词性、命名实体、依存句法、语义角色、语义依存6大任务
+
+## 快速使用
+
+### [Python](python/interface/README.md)
+
+```bash
+pip install -U ltp ltp-core ltp-extension -i https://pypi.org/simple # 安装 ltp
+```
+
+**注：** 如果遇到任何错误，请尝试使用上述命令重新安装 ltp，如果依然报错，请在 Github issues 中反馈。
+
+```python
+import torch
+from ltp import LTP
+
+ltp = LTP("LTP/small")  # 默认加载 Small 模型
+
+# 将模型移动到 GPU 上
+if torch.cuda.is_available():
+    # ltp.cuda()
+    ltp.to("cuda")
+
+output = ltp.pipeline(["他叫汤姆去拿外衣。"], tasks=["cws", "pos", "ner", "srl", "dep", "sdp"])
+# 使用字典格式作为返回结果
+print(output.cws)  # print(output[0]) / print(output['cws']) # 也可以使用下标访问
+print(output.pos)
+print(output.sdp)
+
+# 使用感知机算法实现的分词、词性和命名实体识别，速度比较快，但是精度略低
+ltp = LTP("LTP/legacy")
+# cws, pos, ner = ltp.pipeline(["他叫汤姆去拿外衣。"], tasks=["cws", "ner"]).to_tuple() # error: NER 需要 词性标注任务的结果
+cws, pos, ner = ltp.pipeline(["他叫汤姆去拿外衣。"], tasks=["cws", "pos", "ner"]).to_tuple()  # to tuple 可以自动转换为元组格式
+# 使用元组格式作为返回结果
+print(cws, pos, ner)
+```
+
+**[详细说明](python/interface/docs/quickstart.rst)**
+
+### [Rust](rust/ltp/README.md)
+
+```rust
+use std::fs::File;
+use itertools::multizip;
+use ltp::{CWSModel, POSModel, NERModel, ModelSerde, Format, Codec};
+
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+  let file = File::open("data/legacy-models/cws_model.bin")?;
+  let cws: CWSModel = ModelSerde::load(file, Format::AVRO(Codec::Deflate))?;
+  let file = File::open("data/legacy-models/pos_model.bin")?;
+  let pos: POSModel = ModelSerde::load(file, Format::AVRO(Codec::Deflate))?;
+  let file = File::open("data/legacy-models/ner_model.bin")?;
+  let ner: NERModel = ModelSerde::load(file, Format::AVRO(Codec::Deflate))?;
+
+  let words = cws.predict("他叫汤姆去拿外衣。")?;
+  let pos = pos.predict(&words)?;
+  let ner = ner.predict((&words, &pos))?;
+
+  for (w, p, n) in multizip((words, pos, ner)) {
+    println!("{}/{}/{}", w, p, n);
+  }
+
+  Ok(())
+}
+```
+
+## 模型性能以及下载地址
+
+|                  深度学习模型                   |  分词   |  词性   | 命名实体  | 语义角色  | 依存句法  | 语义依存  | 速度(句/S) |
+| :---------------------------------------: | :---: | :---: | :---: | :---: | :---: | :---: | :-----: |
+|  [Base](https://huggingface.co/LTP/base)  | 98.7  | 98.5  | 95.4  | 80.6  | 89.5  | 75.2  |  39.12  |
+| [Base1](https://huggingface.co/LTP/base1) | 99.22 | 98.73 | 96.39 | 79.28 | 89.57 | 76.57 |  --.--  |
+| [Base2](https://huggingface.co/LTP/base2) | 99.18 | 98.69 | 95.97 | 79.49 | 90.19 | 76.62 |  --.--  |
+| [Small](https://huggingface.co/LTP/small) | 98.4  | 98.2  | 94.3  | 78.4  | 88.3  | 74.7  |  43.13  |
+|  [Tiny](https://huggingface.co/LTP/tiny)  | 96.8  | 97.1  | 91.6  | 70.9  | 83.8  | 70.1  |  53.22  |
+
+|                    感知机算法                    |  分词   |  词性   | 命名实体  | 速度(句/s)  |             备注             |
+| :-----------------------------------------: | :---: | :---: | :---: | :------: | :------------------------: |
+| [Legacy](https://huggingface.co/LTP/legacy) | 97.93 | 98.41 | 94.28 | 21581.48 | [性能详情](rust/ltp/README.md) |
+
+**注：感知机算法速度为开启16线程速度**
+
+## 构建 Wheel 包
+
+```shell script
+make bdist
+```
+
+## 其他语言绑定
+
+**感知机算法**
+
+- [Rust](rust/ltp)
+- [C/C++](rust/ltp-cffi)
+
+**深度学习算法**
+
+- [Rust](https://github.com/HIT-SCIR/libltp/tree/master/ltp-rs)
+- [C++](https://github.com/HIT-SCIR/libltp/tree/master/ltp-cpp)
+- [Java](https://github.com/HIT-SCIR/libltp/tree/master/ltp-java)
+
+## 作者信息
+
+- 冯云龙 \<\<[ylfeng@ir.hit.edu.cn](mailto:ylfeng@ir.hit.edu.cn)>>
+
+## 开源协议
+
+1. 语言技术平台面向国内外大学、中科院各研究所以及个人研究者免费开放源代码，但如上述机构和个人将该平台用于商业目的（如企业合作项目等）则需要付费。
+2. 除上述机构以外的企事业单位，如申请使用该平台，需付费。
+3. 凡涉及付费问题，请发邮件到 car@ir.hit.edu.cn 洽商。
+4. 如果您在 LTP 基础上发表论文或取得科研成果，请您在发表论文和申报成果时声明“使用了哈工大社会计算与信息检索研究中心研制的语言技术平台（LTP）”.
+   同时，发信给car@ir.hit.edu.cn，说明发表论文或申报成果的题目、出处等。

added_tokens.json

@@ -0,0 +1 @@
+{}

config.json

@@ -0,0 +1,352 @@
+{
+    "model": {
+        "_ltp_target_": "ltp_core.models.ltp_model.LTPModule",
+        "backbone": {
+            "_ltp_target_": "ltp_core.models.utils.load_transformers",
+            "config": {
+                "output_attentions": false,
+                "output_hidden_states": false,
+                "use_cache": true,
+                "torchscript": false,
+                "use_bfloat16": false,
+                "pruned_heads": {},
+                "is_encoder_decoder": false,
+                "is_decoder": false,
+                "max_length": 20,
+                "min_length": 0,
+                "do_sample": false,
+                "early_stopping": false,
+                "num_beams": 1,
+                "temperature": 1.0,
+                "top_k": 50,
+                "top_p": 1.0,
+                "repetition_penalty": 1.0,
+                "length_penalty": 1.0,
+                "no_repeat_ngram_size": 0,
+                "bad_words_ids": null,
+                "num_return_sequences": 1,
+                "architectures": null,
+                "finetuning_task": null,
+                "id2label": {
+                    "0": "LABEL_0",
+                    "1": "LABEL_1"
+                },
+                "label2id": {
+                    "LABEL_0": 0,
+                    "LABEL_1": 1
+                },
+                "prefix": null,
+                "bos_token_id": null,
+                "pad_token_id": 0,
+                "eos_token_id": null,
+                "decoder_start_token_id": null,
+                "task_specific_params": null,
+                "xla_device": null,
+                "directionality": "bidi",
+                "vocab_size": 21128,
+                "embedding_size": 768,
+                "hidden_size": 768,
+                "num_hidden_layers": 12,
+                "num_attention_heads": 12,
+                "intermediate_size": 3072,
+                "hidden_act": "gelu",
+                "hidden_dropout_prob": 0.1,
+                "attention_probs_dropout_prob": 0.1,
+                "max_position_embeddings": 512,
+                "type_vocab_size": 2,
+                "initializer_range": 0.02,
+                "layer_norm_eps": 1e-12,
+                "model_type": "electra"
+            }
+        },
+        "processor": {
+            "cws": {
+                "_ltp_target_": "ltp_core.models.processor.TokenOnly"
+            },
+            "pos": {
+                "_ltp_target_": "ltp_core.models.processor.WordsOnly"
+            },
+            "ner": {
+                "_ltp_target_": "ltp_core.models.processor.WordsOnly"
+            },
+            "srl": {
+                "_ltp_target_": "ltp_core.models.processor.WordsOnly"
+            },
+            "dep": {
+                "_ltp_target_": "ltp_core.models.processor.WordsWithHead"
+            },
+            "sdp": {
+                "_ltp_target_": "ltp_core.models.processor.WordsWithHead"
+            }
+        },
+        "heads": {
+            "cws": {
+                "_ltp_target_": "ltp_core.models.components.token.MLPTokenClassifier",
+                "input_size": 768,
+                "num_labels": 2,
+                "dropout": 0.1
+            },
+            "pos": {
+                "_ltp_target_": "ltp_core.models.components.token.MLPTokenClassifier",
+                "input_size": 768,
+                "num_labels": 27,
+                "dropout": 0.1
+            },
+            "ner": {
+                "_ltp_target_": "ltp_core.models.components.token.RelTransformerTokenClassifier",
+                "input_size": 768,
+                "num_labels": 13,
+                "dropout": 0.1,
+                "num_heads": 4,
+                "num_layers": 2
+            },
+            "srl": {
+                "_ltp_target_": "ltp_core.models.components.token.BiaffineTokenClassifier",
+                "input_size": 768,
+                "hidden_size": 300,
+                "num_labels": 97,
+                "dropout": 0.1,
+                "use_crf": true
+            },
+            "dep": {
+                "_ltp_target_": "ltp_core.models.components.graph.BiaffineClassifier",
+                "input_size": 768,
+                "num_labels": 14,
+                "dropout": 0.1,
+                "arc_hidden_size": 500,
+                "rel_hidden_size": 100
+            },
+            "sdp": {
+                "_ltp_target_": "ltp_core.models.components.graph.BiaffineClassifier",
+                "input_size": 768,
+                "num_labels": 56,
+                "arc_hidden_size": 600,
+                "rel_hidden_size": 600
+            }
+        }
+    },
+    "nerual": true,
+    "vocabs": {
+        "cws": [
+            "B-W",
+            "I-W"
+        ],
+        "pos": [
+            "n",
+            "v",
+            "wp",
+            "u",
+            "d",
+            "a",
+            "m",
+            "p",
+            "r",
+            "ns",
+            "c",
+            "q",
+            "nt",
+            "nh",
+            "nd",
+            "j",
+            "i",
+            "b",
+            "ni",
+            "nz",
+            "nl",
+            "z",
+            "k",
+            "ws",
+            "o",
+            "h",
+            "e"
+        ],
+        "ner": [
+            "O",
+            "S-Ns",
+            "S-Nh",
+            "B-Ni",
+            "E-Ni",
+            "I-Ni",
+            "S-Ni",
+            "B-Ns",
+            "E-Ns",
+            "I-Ns",
+            "B-Nh",
+            "E-Nh",
+            "I-Nh"
+        ],
+        "srl": [
+            "O",
+            "B-A0",
+            "B-A0-ADV",
+            "B-A0-CND",
+            "B-A0-CRD",
+            "B-A0-MNR",
+            "B-A0-PRD",
+            "B-A0-PSE",
+            "B-A0-PSR",
+            "B-A0-QTY",
+            "B-A1",
+            "B-A1-CRD",
+            "B-A1-DIS",
+            "B-A1-FRQ",
+            "B-A1-PRD",
+            "B-A1-PSE",
+            "B-A1-PSR",
+            "B-A1-QTY",
+            "B-A1-TPC",
+            "B-A2",
+            "B-A2-CRD",
+            "B-A2-PRD",
+            "B-A2-PSE",
+            "B-A2-PSR",
+            "B-A2-QTY",
+            "B-A3",
+            "B-A3-TMP",
+            "B-A4",
+            "B-ARGM-ADV",
+            "B-ARGM-BNF",
+            "B-ARGM-CND",
+            "B-ARGM-CRD",
+            "B-ARGM-DGR",
+            "B-ARGM-DIR",
+            "B-ARGM-DIS",
+            "B-ARGM-EXT",
+            "B-ARGM-FRQ",
+            "B-ARGM-LOC",
+            "B-ARGM-MNR",
+            "B-ARGM-PRD",
+            "B-ARGM-PRP",
+            "B-ARGM-QTY",
+            "B-ARGM-T",
+            "B-ARGM-TMP",
+            "B-ARGM-TPC",
+            "B-rel-ADV",
+            "B-rel-DIS",
+            "B-rel-EXT",
+            "B-rel-MNR",
+            "I-A0",
+            "I-A0-ADV",
+            "I-A0-CND",
+            "I-A0-CRD",
+            "I-A0-MNR",
+            "I-A0-PRD",
+            "I-A0-PSE",
+            "I-A0-PSR",
+            "I-A0-QTY",
+            "I-A1",
+            "I-A1-CRD",
+            "I-A1-DIS",
+            "I-A1-FRQ",
+            "I-A1-PRD",
+            "I-A1-PSE",
+            "I-A1-PSR",
+            "I-A1-QTY",
+            "I-A1-TPC",
+            "I-A2",
+            "I-A2-CRD",
+            "I-A2-PRD",
+            "I-A2-PSE",
+            "I-A2-PSR",
+            "I-A2-QTY",
+            "I-A3",
+            "I-A3-TMP",
+            "I-A4",
+            "I-ARGM-ADV",
+            "I-ARGM-BNF",
+            "I-ARGM-CND",
+            "I-ARGM-CRD",
+            "I-ARGM-DGR",
+            "I-ARGM-DIR",
+            "I-ARGM-DIS",
+            "I-ARGM-EXT",
+            "I-ARGM-FRQ",
+            "I-ARGM-LOC",
+            "I-ARGM-MNR",
+            "I-ARGM-PRD",
+            "I-ARGM-PRP",
+            "I-ARGM-QTY",
+            "I-ARGM-T",
+            "I-ARGM-TMP",
+            "I-ARGM-TPC",
+            "I-rel-ADV",
+            "I-rel-DIS",
+            "I-rel-EXT",
+            "I-rel-MNR"
+        ],
+        "dep": [
+            "ATT",
+            "WP",
+            "ADV",
+            "VOB",
+            "SBV",
+            "COO",
+            "RAD",
+            "HED",
+            "POB",
+            "CMP",
+            "LAD",
+            "FOB",
+            "DBL",
+            "IOB"
+        ],
+        "sdp": [
+            "mDEPD",
+            "mPUNC",
+            "FEAT",
+            "mRELA",
+            "Root",
+            "AGT",
+            "eSUCC",
+            "EXP",
+            "MEAS",
+            "eCOO",
+            "CONT",
+            "LOC",
+            "DATV",
+            "LINK",
+            "PAT",
+            "TIME",
+            "dCONT",
+            "SCO",
+            "MANN",
+            "mNEG",
+            "ePREC",
+            "dFEAT",
+            "rEXP",
+            "dEXP",
+            "dTIME",
+            "rCONT",
+            "rAGT",
+            "dLINK",
+            "STAT",
+            "REAS",
+            "rPAT",
+            "TOOL",
+            "dSTAT",
+            "dMANN",
+            "rTIME",
+            "rLOC",
+            "dDATV",
+            "rFEAT",
+            "MATL",
+            "rDATV",
+            "dREAS",
+            "dLOC",
+            "rLINK",
+            "dPAT",
+            "rMANN",
+            "rREAS",
+            "rTOOL",
+            "rMEAS",
+            "dSCO",
+            "dMEAS",
+            "rSCO",
+            "dAGT",
+            "rMATL",
+            "rSTAT",
+            "dTOOL",
+            "dMATL"
+        ]
+    }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a097964f7b7ed18cb469b88bfaaacd0fd2e74868788142e49070fce7a35bf073
+size 553109320

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:30f651d5031f694e6320ceffc5fdba86c7681f0d9fe18dacab482e9c5e037bfb
+size 557512529

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tokenizer_config.json

@@ -0,0 +1 @@
+{"init_inputs": []}