Update README, add example code

README.md

@@ -1,3 +1,111 @@
 ---
-license: mit
+language: ja
+license: gpl-3.0
 ---
+
+A model used to estimate the start and end of a Named Entity (NE) span based on a Point annotation, as used in the paper "Is boundary annotation necessary? Evaluating boundary-free approaches to improve clinical named entity annotation efficiency".
+
+Basically, the goal of this model is to convert a point annotation to a corresponding span annotation with the correct span.
+
+The model locates an identifier token (⧫) and based on its surround context estimates where the NE concept starts and ends.
+
+The model is trained to estimate the spans of diseases and symptom names in Japanese medical texts.
+
+## Concepts
+
+### Point annotation
+
+Unlike span-based paradigms, a point annotation is composed by a single position within the NE span.
+It is a simple and fast way to annotate NEs, but it introduces ambiguity in the information captured by the annotation.
+
+On this repository implementation, a point annotation is represented by a lozenge character (⧫).
+
+Example:
+```
+The patient has a history of dia⧫betes.
+```
+
+### Span annotation
+
+A span annotation is composed by the two markings, identifying both start and end positions of the NE span.
+
+The implementation on this repository is based on the span annotation schema defined by [Yada et al. (2020)](https://aclanthology.org/2020.lrec-1.561/).
+
+Example:
+```
+The patient has a history of <C>diabetes</C>.
+```
+
+## Model architecture
+
+This model was fine-tuned on top of [cl-tohoku/bert-base-japanese-char-v2] (https://huggingface.co/cl-tohoku/bert-base-japanese-char-v2).
+
+The model architecture is the same as the original BERT base model; 12 layers, 768 dimensions of hidden states, and 12 attention heads.
+
+To be executed, this model requires the following dependencies:
+- fugashi
+- unidic-lite
+
+## Training data
+
+The model was finetuned using a dataset of Japanese medical texts (which is not available pubicly), comprised of 1027 synthetic medication history notes generated through crowd-sourcing. 
+
+Ten experienced dispensing pharmacists were hired as writers to craft the corpus. Each writer was assigned one of 285 drug names and tasked with creating a ``typical'' clinical narrative. This corpus was later fully annotated for symptoms and disease names.
+
+Each annotation received a ⧫ token within its span based on a Truncated normal distribution.
+
+The model was then trained to identify this token and output a span corresponding to the surrounding concept.
+
+## Usage
+
+The `requirements.txt` file contains all the dependencies needed to run the example code.
+
+```python
+import mojimoji
+import numpy as np
+from transformers import AutoTokenizer, AutoModelForTokenClassification
+
+import iob_util #pip install git+https://github.com/gabrielandrade2/IOB-util.git
+
+model_name = "gabrielandrade2/point-to-span-estimation"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForTokenClassification.from_pretrained(model_name)
+
+# Point-annotated text
+text = "肥大型心⧫筋症、心房⧫細動に対してＷＦ投与が開始となった。\
+治療経過中に非持続性心⧫室頻拍が認められたためアミオダロンが併用となった。"
+
+# Convert to zenkaku and tokenize
+text = mojimoji.han_to_zen(text)
+tokenized = tokenizer.tokenize(text)
+
+# Encode text
+input_ids = tokenizer.encode(text, return_tensors="pt")
+
+# Predict spans
+output = model(input_ids)
+logits = output[0].detach().cpu().numpy()
+tags = np.argmax(logits, axis=2)[:, :].tolist()[0]
+
+# Convert model output to IOB format
+id2label = model.config.id2label
+tags = [id2label[t] for t in tags]
+
+# Convert input_ids back to chars
+tokens = [tokenizer.convert_ids_to_tokens(t) for t in input_ids][0]
+
+# Remove model special tokens (CLS, SEP, PAD)
+tags = [y for x, y in zip(tokens, tags) if x not in ['[CLS]', '[SEP]', '[PAD]']]
+tokens = [x for x in tokens if x not in ['[CLS]', '[SEP]', '[PAD]']]
+
+# Convert from IOB to XML tag format
+xml_text = iob_util.convert_iob_to_xml(tokens, tags)
+xml_text = xml_text.replace('⧫', '')
+print(xml_text)
+```
+
+### Output
+```xml
+<C>肥大型心筋症</C>、<C>心房細動</C>に対してWF投与が開始となった。治療経過中に<C>非持続性心室頻拍</C>が認められたためアミオダロンが併用となった。
+```

example.py

@@ -0,0 +1,42 @@
+import mojimoji
+import numpy as np
+from transformers import AutoTokenizer, AutoModelForTokenClassification
+
+import iob_util #pip install git+https://github.com/gabrielandrade2/IOB-util.git
+
+model_name = "gabrielandrade2/point-to-span-estimation"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForTokenClassification.from_pretrained(model_name)
+
+# Point-annotated text
+text = "肥大型心⧫筋症、心房⧫細動に対してＷＦ投与が開始となった。\
+治療経過中に非持続性心⧫室頻拍が認められたためアミオダロンが併用となった。"
+
+# Convert to zenkaku and tokenize
+text = mojimoji.han_to_zen(text)
+tokenized = tokenizer.tokenize(text)
+
+# Encode text
+input_ids = tokenizer.encode(text, return_tensors="pt")
+
+# Predict spans
+output = model(input_ids)
+logits = output[0].detach().cpu().numpy()
+tags = np.argmax(logits, axis=2)[:, :].tolist()[0]
+
+# Convert model output to IOB format
+id2label = model.config.id2label
+tags = [id2label[t] for t in tags]
+
+# Convert input_ids back to chars
+tokens = [tokenizer.convert_ids_to_tokens(t) for t in input_ids][0]
+
+# Remove model special tokens (CLS, SEP, PAD)
+tags = [y for x, y in zip(tokens, tags) if x not in ['[CLS]', '[SEP]', '[PAD]']]
+tokens = [x for x in tokens if x not in ['[CLS]', '[SEP]', '[PAD]']]
+
+# Convert from IOB to XML tag format
+xml_text = iob_util.convert_iob_to_xml(tokens, tags)
+xml_text = xml_text.replace('⧫', '')
+print(xml_text)

requirements.txt

@@ -0,0 +1,28 @@
+certifi==2024.2.2
+charset-normalizer==3.3.2
+filelock==3.13.1
+fsspec==2024.2.0
+fugashi==1.3.0
+huggingface-hub==0.20.3
+idna==3.6
+iob_util @ git+https://github.com/gabrielandrade2/IOB-util.git@b5d522aa50238a25cdda19f2cf6908833acd6d64
+Jinja2==3.1.3
+lxml==5.1.0
+MarkupSafe==2.1.5
+mojimoji==0.0.13
+mpmath==1.3.0
+networkx==3.2.1
+numpy==1.26.4
+packaging==23.2
+PyYAML==6.0.1
+regex==2023.12.25
+requests==2.31.0
+safetensors==0.4.2
+sympy==1.12
+tokenizers==0.15.2
+torch==2.2.0
+tqdm==4.66.2
+transformers==4.38.1
+typing_extensions==4.9.0
+unidic-lite==1.0.8
+urllib3==2.2.1