model update

README.md

@@ -0,0 +1,176 @@
+---
+datasets:
+- tner/tweetner7
+metrics:
+- f1
+- precision
+- recall
+model-index:
+- name: tner/bert-large-tweetner7-2021
+  results:
+  - task:
+      name: Token Classification
+      type: token-classification
+    dataset:
+      name: tner/tweetner7/test_2021
+      type: tner/tweetner7/test_2021
+      args: tner/tweetner7/test_2021
+    metrics:
+    - name: F1
+      type: f1
+      value: 0.5974718775368201
+    - name: Precision
+      type: precision
+      value: 0.5992091183996279
+    - name: Recall
+      type: recall
+      value: 0.5957446808510638
+    - name: F1 (macro)
+      type: f1_macro
+      value: 0.5392877076670867
+    - name: Precision (macro)
+      type: precision_macro
+      value: 0.5398425980592713
+    - name: Recall (macro)
+      type: recall_macro
+      value: 0.5439768272225339
+    - name: F1 (entity span)
+      type: f1_entity_span
+      value: 0.7497514474530674
+    - name: Precision (entity span)
+      type: precision_entity_span
+      value: 0.7584003786086133
+    - name: Recall (entity span)
+      type: recall_entity_span
+      value: 0.7412975598473459
+  - task:
+      name: Token Classification
+      type: token-classification
+    dataset:
+      name: tner/tweetner7/test_2020
+      type: tner/tweetner7/test_2020
+      args: tner/tweetner7/test_2020
+    metrics:
+    - name: F1
+      type: f1
+      value: 0.5662616558349817
+    - name: Precision
+      type: precision
+      value: 0.6215880893300249
+    - name: Recall
+      type: recall
+      value: 0.519979242345615
+    - name: F1 (macro)
+      type: f1_macro
+      value: 0.5096985017746614
+    - name: Precision (macro)
+      type: precision_macro
+      value: 0.5628721370469417
+    - name: Recall (macro)
+      type: recall_macro
+      value: 0.47520198274721537
+    - name: F1 (entity span)
+      type: f1_entity_span
+      value: 0.7065868263473053
+    - name: Precision (entity span)
+      type: precision_entity_span
+      value: 0.7841772151898734
+    - name: Recall (entity span)
+      type: recall_entity_span
+      value: 0.6429683445770628
+
+pipeline_tag: token-classification
+widget:
+- text: "Get the all-analog Classic Vinyl Edition of `Takin' Off` Album from {{@Herbie Hancock@}} via {{USERNAME}} link below: {{URL}}"
+  example_title: "NER Example 1"
+---
+# tner/bert-large-tweetner7-2021
+
+This model is a fine-tuned version of [bert-large-cased](https://huggingface.co/bert-large-cased) on the 
+[tner/tweetner7](https://huggingface.co/datasets/tner/tweetner7) dataset (`train_2021` split).
+Model fine-tuning is done via [T-NER](https://github.com/asahi417/tner)'s hyper-parameter search (see the repository
+for more detail). It achieves the following results on the test set of 2021:
+- F1 (micro): 0.5974718775368201
+- Precision (micro): 0.5992091183996279
+- Recall (micro): 0.5957446808510638
+- F1 (macro): 0.5392877076670867
+- Precision (macro): 0.5398425980592713
+- Recall (macro): 0.5439768272225339
+
+
+
+The per-entity breakdown of the F1 score on the test set are below:
+- corporation: 0.4486772486772486
+- creative_work: 0.34173228346456697
+- event: 0.40238450074515647
+- group: 0.556795797767564
+- location: 0.6394904458598726
+- person: 0.7940364439536168
+- product: 0.5918972332015809 
+
+For F1 scores, the confidence interval is obtained by bootstrap as below:
+- F1 (micro): 
+    - 90%: [0.5884763705775744, 0.6075466841645367]
+    - 95%: [0.586724466800271, 0.6087071446445204] 
+- F1 (macro): 
+    - 90%: [0.5884763705775744, 0.6075466841645367]
+    - 95%: [0.586724466800271, 0.6087071446445204] 
+
+Full evaluation can be found at [metric file of NER](https://huggingface.co/tner/bert-large-tweetner7-2021/raw/main/eval/metric.json) 
+and [metric file of entity span](https://huggingface.co/tner/bert-large-tweetner7-2021/raw/main/eval/metric_span.json).
+
+### Usage
+This model can be used through the [tner library](https://github.com/asahi417/tner). Install the library via pip   
+```shell
+pip install tner
+```
+and activate model as below.
+```python
+from tner import TransformersNER
+model = TransformersNER("tner/bert-large-tweetner7-2021")
+model.predict(["Jacob Collier is a Grammy awarded English artist from London"])
+```
+It can be used via transformers library but it is not recommended as CRF layer is not supported at the moment.
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+ - dataset: ['tner/tweetner7']
+ - dataset_split: train_2021
+ - dataset_name: None
+ - local_dataset: None
+ - model: bert-large-cased
+ - crf: False
+ - max_length: 128
+ - epoch: 30
+ - batch_size: 32
+ - lr: 0.0001
+ - random_seed: 0
+ - gradient_accumulation_steps: 1
+ - weight_decay: 1e-07
+ - lr_warmup_step_ratio: 0.3
+ - max_grad_norm: 1
+
+The full configuration can be found at [fine-tuning parameter file](https://huggingface.co/tner/bert-large-tweetner7-2021/raw/main/trainer_config.json).
+
+### Reference
+If you use any resource from T-NER, please consider to cite our [paper](https://aclanthology.org/2021.eacl-demos.7/).
+
+```
+
+@inproceedings{ushio-camacho-collados-2021-ner,
+    title = "{T}-{NER}: An All-Round Python Library for Transformer-based Named Entity Recognition",
+    author = "Ushio, Asahi  and
+      Camacho-Collados, Jose",
+    booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations",
+    month = apr,
+    year = "2021",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2021.eacl-demos.7",
+    doi = "10.18653/v1/2021.eacl-demos.7",
+    pages = "53--62",
+    abstract = "Language model (LM) pretraining has led to consistent improvements in many NLP downstream tasks, including named entity recognition (NER). In this paper, we present T-NER (Transformer-based Named Entity Recognition), a Python library for NER LM finetuning. In addition to its practical utility, T-NER facilitates the study and investigation of the cross-domain and cross-lingual generalization ability of LMs finetuned on NER. Our library also provides a web app where users can get model predictions interactively for arbitrary text, which facilitates qualitative model evaluation for non-expert programmers. We show the potential of the library by compiling nine public NER datasets into a unified format and evaluating the cross-domain and cross- lingual performance across the datasets. The results from our initial experiments show that in-domain performance is generally competitive across datasets. However, cross-domain generalization is challenging even with a large pretrained LM, which has nevertheless capacity to learn domain-specific features if fine- tuned on a combined dataset. To facilitate future research, we also release all our LM checkpoints via the Hugging Face model hub.",
+}
+
+```

eval/metric.json

@@ -1 +0,0 @@
-{"2021.dev": {"micro/f1": 0.601733809280979, "micro/f1_ci": {}, "micro/recall": 0.59, "micro/precision": 0.6139438085327783, "macro/f1": 0.5530417662709053, "macro/f1_ci": {}, "macro/recall": 0.5475419587120097, "macro/precision": 0.5634379470607581, "per_entity_metric": {"corporation": {"f1": 0.5781990521327014, "f1_ci": {}, "precision": 0.5596330275229358, "recall": 0.5980392156862745}, "creative_work": {"f1": 0.3787878787878788, "f1_ci": {}, "precision": 0.43103448275862066, "recall": 0.33783783783783783}, "event": {"f1": 0.3620689655172413, "f1_ci": {}, "precision": 0.4158415841584158, "recall": 0.32061068702290074}, "group": {"f1": 0.570754716981132, "f1_ci": {}, "precision": 0.6142131979695431, "recall": 0.5330396475770925}, "location": {"f1": 0.6081081081081081, "f1_ci": {}, "precision": 0.5921052631578947, "recall": 0.625}, "person": {"f1": 0.7794871794871795, "f1_ci": {}, "precision": 0.7549668874172185, "recall": 0.8056537102473498}, "product": {"f1": 0.5938864628820961, "f1_ci": {}, "precision": 0.576271186440678, "recall": 0.6126126126126126}}}, "2021.test": {"micro/f1": 0.5974718775368201, "micro/f1_ci": {"90": [0.5884763705775744, 0.6075466841645367], "95": [0.586724466800271, 0.6087071446445204]}, "micro/recall": 0.5957446808510638, "micro/precision": 0.5992091183996279, "macro/f1": 0.5392877076670867, "macro/f1_ci": {"90": [0.530098896055678, 0.5492366033043069], "95": [0.5284763359472952, 0.5509592237769576]}, "macro/recall": 0.5439768272225339, "macro/precision": 0.5398425980592713, "per_entity_metric": {"corporation": {"f1": 0.4486772486772486, "f1_ci": {"90": [0.42374195222512145, 0.47198702010198484], "95": [0.41629960888760914, 0.47854513512534524]}, "precision": 0.42828282828282827, "recall": 0.4711111111111111}, "creative_work": {"f1": 0.34173228346456697, "f1_ci": {"90": [0.3111668229783637, 0.3751494936931258], "95": [0.30621243300153134, 0.38118272627922994]}, "precision": 0.4025974025974026, "recall": 0.2968536251709986}, "event": {"f1": 0.40238450074515647, "f1_ci": {"90": [0.3771082185375127, 0.4253969451946884], "95": [0.37377656250240715, 0.4304793666363063]}, "precision": 0.4431072210065646, "recall": 0.3685168334849864}, "group": {"f1": 0.556795797767564, "f1_ci": {"90": [0.5363658468635578, 0.578533580955637], "95": [0.5326958637709951, 0.5829043388575683]}, "precision": 0.5549738219895288, "recall": 0.5586297760210803}, "location": {"f1": 0.6394904458598726, "f1_ci": {"90": [0.6134667630913595, 0.6658640097485297], "95": [0.6058188339438338, 0.6704738910012674]}, "precision": 0.5878220140515222, "recall": 0.7011173184357542}, "person": {"f1": 0.7940364439536168, "f1_ci": {"90": [0.7825427984149077, 0.8055708929917582], "95": [0.7796777233643017, 0.8077754587411615]}, "precision": 0.7927232635060639, "recall": 0.7953539823008849}, "product": {"f1": 0.5918972332015809, "f1_ci": {"90": [0.56762739781335, 0.6126945509284936], "95": [0.5640013221511239, 0.6161674947859158]}, "precision": 0.5693916349809885, "recall": 0.6162551440329218}}}, "2020.test": {"micro/f1": 0.5662616558349817, "micro/f1_ci": {"90": [0.5454995934256639, 0.5875092213706942], "95": [0.5425866998697456, 0.5920983770545653]}, "micro/recall": 0.519979242345615, "micro/precision": 0.6215880893300249, "macro/f1": 0.5096985017746614, "macro/f1_ci": {"90": [0.48748317612063496, 0.5319882567266163], "95": [0.4844618671113482, 0.5363490059404551]}, "macro/recall": 0.47520198274721537, "macro/precision": 0.5628721370469417, "per_entity_metric": {"corporation": {"f1": 0.4931506849315068, "f1_ci": {"90": [0.43111058336584623, 0.5454765041721563], "95": [0.4235045567522783, 0.554688888888889]}, "precision": 0.5172413793103449, "recall": 0.4712041884816754}, "creative_work": {"f1": 0.3157894736842105, "f1_ci": {"90": [0.25305184142622567, 0.37604984025559096], "95": [0.23929824561403507, 0.3846219931271478]}, "precision": 0.4827586206896552, "recall": 0.2346368715083799}, "event": {"f1": 0.30385487528344673, "f1_ci": {"90": [0.2511617287883615, 0.3582357271936283], "95": [0.2435547967917763, 0.3728851647305275]}, "precision": 0.3806818181818182, "recall": 0.2528301886792453}, "group": {"f1": 0.502692998204668, "f1_ci": {"90": [0.45108915906788244, 0.5562174896540221], "95": [0.43689164466435204, 0.5718759253384094]}, "precision": 0.5691056910569106, "recall": 0.45016077170418006}, "location": {"f1": 0.5975609756097562, "f1_ci": {"90": [0.5342913251566929, 0.6546546546546547], "95": [0.5232513416815743, 0.6645611137764328]}, "precision": 0.6012269938650306, "recall": 0.593939393939394}, "person": {"f1": 0.7607361963190183, "f1_ci": {"90": [0.7331188694509051, 0.7864466361308117], "95": [0.7277244030211747, 0.7904014991482111]}, "precision": 0.7963302752293578, "recall": 0.7281879194630873}, "product": {"f1": 0.5941043083900227, "f1_ci": {"90": [0.5444883712647363, 0.6420348374103892], "95": [0.5323734534856055, 0.6513813358794469]}, "precision": 0.5927601809954751, "recall": 0.5954545454545455}}}, "2021.test (span detection)": {"micro/f1": 0.7497514474530674, "micro/f1_ci": {}, "micro/recall": 0.7412975598473459, "micro/precision": 0.7584003786086133, "macro/f1": 0.7497514474530674, "macro/f1_ci": {}, "macro/recall": 0.7412975598473459, "macro/precision": 0.7584003786086133}, "2020.test (span detection)": {"micro/f1": 0.7065868263473053, "micro/f1_ci": {}, "micro/recall": 0.6429683445770628, "micro/precision": 0.7841772151898734, "macro/f1": 0.7065868263473053, "macro/f1_ci": {}, "macro/recall": 0.6429683445770628, "macro/precision": 0.7841772151898734}}

eval/metric.test_2020.json

@@ -0,0 +1 @@
+{"micro/f1": 0.5662616558349817, "micro/f1_ci": {"90": [0.5454995934256639, 0.5875092213706942], "95": [0.5425866998697456, 0.5920983770545653]}, "micro/recall": 0.519979242345615, "micro/precision": 0.6215880893300249, "macro/f1": 0.5096985017746614, "macro/f1_ci": {"90": [0.48748317612063496, 0.5319882567266163], "95": [0.4844618671113482, 0.5363490059404551]}, "macro/recall": 0.47520198274721537, "macro/precision": 0.5628721370469417, "per_entity_metric": {"corporation": {"f1": 0.4931506849315068, "f1_ci": {"90": [0.43111058336584623, 0.5454765041721563], "95": [0.4235045567522783, 0.554688888888889]}, "precision": 0.5172413793103449, "recall": 0.4712041884816754}, "creative_work": {"f1": 0.3157894736842105, "f1_ci": {"90": [0.25305184142622567, 0.37604984025559096], "95": [0.23929824561403507, 0.3846219931271478]}, "precision": 0.4827586206896552, "recall": 0.2346368715083799}, "event": {"f1": 0.30385487528344673, "f1_ci": {"90": [0.2511617287883615, 0.3582357271936283], "95": [0.2435547967917763, 0.3728851647305275]}, "precision": 0.3806818181818182, "recall": 0.2528301886792453}, "group": {"f1": 0.502692998204668, "f1_ci": {"90": [0.45108915906788244, 0.5562174896540221], "95": [0.43689164466435204, 0.5718759253384094]}, "precision": 0.5691056910569106, "recall": 0.45016077170418006}, "location": {"f1": 0.5975609756097562, "f1_ci": {"90": [0.5342913251566929, 0.6546546546546547], "95": [0.5232513416815743, 0.6645611137764328]}, "precision": 0.6012269938650306, "recall": 0.593939393939394}, "person": {"f1": 0.7607361963190183, "f1_ci": {"90": [0.7331188694509051, 0.7864466361308117], "95": [0.7277244030211747, 0.7904014991482111]}, "precision": 0.7963302752293578, "recall": 0.7281879194630873}, "product": {"f1": 0.5941043083900227, "f1_ci": {"90": [0.5444883712647363, 0.6420348374103892], "95": [0.5323734534856055, 0.6513813358794469]}, "precision": 0.5927601809954751, "recall": 0.5954545454545455}}}

eval/metric.test_2021.json

@@ -0,0 +1 @@
+{"micro/f1": 0.5974718775368201, "micro/f1_ci": {"90": [0.5884763705775744, 0.6075466841645367], "95": [0.586724466800271, 0.6087071446445204]}, "micro/recall": 0.5957446808510638, "micro/precision": 0.5992091183996279, "macro/f1": 0.5392877076670867, "macro/f1_ci": {"90": [0.530098896055678, 0.5492366033043069], "95": [0.5284763359472952, 0.5509592237769576]}, "macro/recall": 0.5439768272225339, "macro/precision": 0.5398425980592713, "per_entity_metric": {"corporation": {"f1": 0.4486772486772486, "f1_ci": {"90": [0.42374195222512145, 0.47198702010198484], "95": [0.41629960888760914, 0.47854513512534524]}, "precision": 0.42828282828282827, "recall": 0.4711111111111111}, "creative_work": {"f1": 0.34173228346456697, "f1_ci": {"90": [0.3111668229783637, 0.3751494936931258], "95": [0.30621243300153134, 0.38118272627922994]}, "precision": 0.4025974025974026, "recall": 0.2968536251709986}, "event": {"f1": 0.40238450074515647, "f1_ci": {"90": [0.3771082185375127, 0.4253969451946884], "95": [0.37377656250240715, 0.4304793666363063]}, "precision": 0.4431072210065646, "recall": 0.3685168334849864}, "group": {"f1": 0.556795797767564, "f1_ci": {"90": [0.5363658468635578, 0.578533580955637], "95": [0.5326958637709951, 0.5829043388575683]}, "precision": 0.5549738219895288, "recall": 0.5586297760210803}, "location": {"f1": 0.6394904458598726, "f1_ci": {"90": [0.6134667630913595, 0.6658640097485297], "95": [0.6058188339438338, 0.6704738910012674]}, "precision": 0.5878220140515222, "recall": 0.7011173184357542}, "person": {"f1": 0.7940364439536168, "f1_ci": {"90": [0.7825427984149077, 0.8055708929917582], "95": [0.7796777233643017, 0.8077754587411615]}, "precision": 0.7927232635060639, "recall": 0.7953539823008849}, "product": {"f1": 0.5918972332015809, "f1_ci": {"90": [0.56762739781335, 0.6126945509284936], "95": [0.5640013221511239, 0.6161674947859158]}, "precision": 0.5693916349809885, "recall": 0.6162551440329218}}}

eval/metric_span.test_2020.json

@@ -0,0 +1 @@
+{"micro/f1": 0.7065868263473053, "micro/f1_ci": {}, "micro/recall": 0.6429683445770628, "micro/precision": 0.7841772151898734, "macro/f1": 0.7065868263473053, "macro/f1_ci": {}, "macro/recall": 0.6429683445770628, "macro/precision": 0.7841772151898734}

eval/metric_span.test_2021.json

@@ -0,0 +1 @@
+{"micro/f1": 0.7497514474530674, "micro/f1_ci": {}, "micro/recall": 0.7412975598473459, "micro/precision": 0.7584003786086133, "macro/f1": 0.7497514474530674, "macro/f1_ci": {}, "macro/recall": 0.7412975598473459, "macro/precision": 0.7584003786086133}

trainer_config.json

@@ -1 +1 @@
-{"data_split": "2021.train", "model": "bert-large-cased", "crf": false, "max_length": 128, "epoch": 30, "batch_size": 32, "lr": 0.0001, "random_seed": 0, "gradient_accumulation_steps": 1, "weight_decay": 1e-07, "lr_warmup_step_ratio": 0.3, "max_grad_norm": 1}
+{"dataset": ["tner/tweetner7"], "dataset_split": "train_2021", "dataset_name": null, "local_dataset": null, "model": "bert-large-cased", "crf": false, "max_length": 128, "epoch": 30, "batch_size": 32, "lr": 0.0001, "random_seed": 0, "gradient_accumulation_steps": 1, "weight_decay": 1e-07, "lr_warmup_step_ratio": 0.3, "max_grad_norm": 1}