Create Readme

README.md

@@ -0,0 +1,110 @@
+---
+language:
+- zh
+tags:
+- Seq2SeqLM
+- 古文
+- 文言文
+- 中国古代官职地名拆分
+- ancient
+- classical
+license: cc-by-nc-sa-4.0
+---
+
+# <font color="IndianRed"> Name Splitter </font>
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1eZyJgeQOFfpG3QOlq0haDz8pE8jhsGOt#scrollTo=XChPisgxiiji)
+
+Our model <font color="cornflowerblue"> Name Splitter </font> is a Named Entity Recognition Classical Chinese language model that is intended to  <font color="IndianRed">split the address portion in Classical Chinese office titles.</font>. This model is first inherited from raynardj/classical-chinese-punctuation-guwen-biaodian Classical Chinese punctuation model, and finetuned using over a 25,000 high-quality punctuation pairs collected CBDB group (China Biographical Database). 
+
+### <font color="IndianRed"> Sample input txt file </font>
+The sample input txt file can be downloaded here:
+https://huggingface.co/cbdb/OfficeTitleAddressSplitter/blob/main/vocab.txt
+
+### <font color="IndianRed"> How to use </font>
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+<font color="cornflowerblue"> 1. Import model and packages </font>
+```python
+from transformers import AutoTokenizer, AutoModelForTokenClassification
+
+PRETRAINED = "cbdb/NameSplitter"
+tokenizer = AutoTokenizer.from_pretrained(PRETRAINED)
+model = AutoModelForTokenClassification.from_pretrained(PRETRAINED)
+```
+
+<font color="cornflowerblue"> 2. Load Data </font>
+```python
+# Load your data here
+test_list = ['徐元文漢魏風致集', '熊方補後漢書年表', '羅振玉本朝學術源流槪略 一卷', '陶諧陶莊敏集']
+```
+
+
+<font color="cornflowerblue"> 3. Make a prediction </font>
+```python
+def predict_class(test):
+    tokens_test = tokenizer.encode_plus(
+        test,
+        add_special_tokens=True,
+        return_attention_mask=True,
+        padding=True,
+        max_length=128,
+        return_tensors='pt',
+        truncation=True
+    )
+
+    test_seq = torch.tensor(tokens_test['input_ids'])
+    test_mask = torch.tensor(tokens_test['attention_mask'])
+
+    inputs = {
+        "input_ids": test_seq,
+        "attention_mask": test_mask
+    }
+    with torch.no_grad():
+        # print(inputs.shape)
+        outputs = model(**inputs)
+        outputs = outputs.logits.detach().cpu().numpy()
+        
+    softmax_score = softmax(outputs)
+    softmax_score = np.argmax(softmax_score, axis=2)[0]
+    return test_seq, softmax_score
+
+for test_sen0 in test_list:
+    test_seq, pred_class_proba = predict_class(test_sen0)
+    test_sen = tokenizer.decode(test_seq[0]).split()
+    label = [idx2label[i] for i in pred_class_proba]
+
+    element_to_find = '。'
+
+    if element_to_find in label:
+        index = label.index(element_to_find)
+        test_sen_pred = [i for i in test_sen0]
+        test_sen_pred.insert(index, element_to_find)
+        test_sen_pred = ''.join(test_sen_pred)
+
+    else:
+        test_sen_pred = [i for i in test_sen0]
+        test_sen_pred = ''.join(test_sen_pred)
+
+    print(test_sen_pred)
+```
+徐元文。漢魏風致集<br>
+熊方。補後漢書年表<br>
+羅振玉。本朝學術源流槪略 一卷<br>
+陶諧。陶莊敏集<br>
+
+### <font color="IndianRed">Authors </font>
+Queenie Luo (queenieluo[at]g.harvard.edu)
+<br>
+Hongsu Wang
+<br>
+Peter Bol
+<br>
+CBDB Group
+
+### <font color="IndianRed">License </font>
+Copyright (c) 2023 CBDB
+
+Except where otherwise noted, content on this repository is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).
+To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/4.0/ or
+send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.