model update

README.md

@@ -0,0 +1,176 @@
+---
+datasets:
+- tner/tweetner7
+metrics:
+- f1
+- precision
+- recall
+model-index:
+- name: tner/roberta-large-tweetner7-random
+  results:
+  - task:
+      name: Token Classification
+      type: token-classification
+    dataset:
+      name: tner/tweetner7/test_2021
+      type: tner/tweetner7/test_2021
+      args: tner/tweetner7/test_2021
+    metrics:
+    - name: F1
+      type: f1
+      value: 0.6632769652650823
+    - name: Precision
+      type: precision
+      value: 0.6554878048780488
+    - name: Recall
+      type: recall
+      value: 0.6712534690101758
+    - name: F1 (macro)
+      type: f1_macro
+      value: 0.6096477771855761
+    - name: Precision (macro)
+      type: precision_macro
+      value: 0.6042443991246051
+    - name: Recall (macro)
+      type: recall_macro
+      value: 0.6191008735553379
+    - name: F1 (entity span)
+      type: f1_entity_span
+      value: 0.7900359938296291
+    - name: Precision (entity span)
+      type: precision_entity_span
+      value: 0.780713640469738
+    - name: Recall (entity span)
+      type: recall_entity_span
+      value: 0.7995836706372152
+  - task:
+      name: Token Classification
+      type: token-classification
+    dataset:
+      name: tner/tweetner7/test_2020
+      type: tner/tweetner7/test_2020
+      args: tner/tweetner7/test_2020
+    metrics:
+    - name: F1
+      type: f1
+      value: 0.6439847577572129
+    - name: Precision
+      type: precision
+      value: 0.6771608471665712
+    - name: Recall
+      type: recall
+      value: 0.6139076284379865
+    - name: F1 (macro)
+      type: f1_macro
+      value: 0.6008744778169367
+    - name: Precision (macro)
+      type: precision_macro
+      value: 0.6358142893696356
+    - name: Recall (macro)
+      type: recall_macro
+      value: 0.5742193301311931
+    - name: F1 (entity span)
+      type: f1_entity_span
+      value: 0.7552409474543968
+    - name: Precision (entity span)
+      type: precision_entity_span
+      value: 0.7943871706758304
+    - name: Recall (entity span)
+      type: recall_entity_span
+      value: 0.7197716658017644
+
+pipeline_tag: token-classification
+widget:
+- text: "Get the all-analog Classic Vinyl Edition of `Takin' Off` Album from {{@Herbie Hancock@}} via {{USERNAME}} link below: {{URL}}"
+  example_title: "NER Example 1"
+---
+# tner/roberta-large-tweetner7-random
+
+This model is a fine-tuned version of [roberta-large](https://huggingface.co/roberta-large) on the 
+[tner/tweetner7](https://huggingface.co/datasets/tner/tweetner7) dataset (`train_random` split).
+Model fine-tuning is done via [T-NER](https://github.com/asahi417/tner)'s hyper-parameter search (see the repository
+for more detail). It achieves the following results on the test set of 2021:
+- F1 (micro): 0.6632769652650823
+- Precision (micro): 0.6554878048780488
+- Recall (micro): 0.6712534690101758
+- F1 (macro): 0.6096477771855761
+- Precision (macro): 0.6042443991246051
+- Recall (macro): 0.6191008735553379
+
+
+
+The per-entity breakdown of the F1 score on the test set are below:
+- corporation: 0.5224148236700539
+- creative_work: 0.45186640471512773
+- event: 0.4894837476099427
+- group: 0.6327722432153899
+- location: 0.6692258477287268
+- person: 0.838405036726128
+- product: 0.6633663366336633 
+
+For F1 scores, the confidence interval is obtained by bootstrap as below:
+- F1 (micro): 
+    - 90%: [0.6546824558783396, 0.6722355436189195]
+    - 95%: [0.6527609558375069, 0.6741666937877734] 
+- F1 (macro): 
+    - 90%: [0.6546824558783396, 0.6722355436189195]
+    - 95%: [0.6527609558375069, 0.6741666937877734] 
+
+Full evaluation can be found at [metric file of NER](https://huggingface.co/tner/roberta-large-tweetner7-random/raw/main/eval/metric.json) 
+and [metric file of entity span](https://huggingface.co/tner/roberta-large-tweetner7-random/raw/main/eval/metric_span.json).
+
+### Usage
+This model can be used through the [tner library](https://github.com/asahi417/tner). Install the library via pip   
+```shell
+pip install tner
+```
+and activate model as below.
+```python
+from tner import TransformersNER
+model = TransformersNER("tner/roberta-large-tweetner7-random")
+model.predict(["Jacob Collier is a Grammy awarded English artist from London"])
+```
+It can be used via transformers library but it is not recommended as CRF layer is not supported at the moment.
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+ - dataset: ['tner/tweetner7']
+ - dataset_split: train_random
+ - dataset_name: None
+ - local_dataset: None
+ - model: roberta-large
+ - crf: True
+ - max_length: 128
+ - epoch: 30
+ - batch_size: 32
+ - lr: 1e-05
+ - random_seed: 0
+ - gradient_accumulation_steps: 1
+ - weight_decay: 1e-07
+ - lr_warmup_step_ratio: 0.15
+ - max_grad_norm: 1
+
+The full configuration can be found at [fine-tuning parameter file](https://huggingface.co/tner/roberta-large-tweetner7-random/raw/main/trainer_config.json).
+
+### Reference
+If you use any resource from T-NER, please consider to cite our [paper](https://aclanthology.org/2021.eacl-demos.7/).
+
+```
+
+@inproceedings{ushio-camacho-collados-2021-ner,
+    title = "{T}-{NER}: An All-Round Python Library for Transformer-based Named Entity Recognition",
+    author = "Ushio, Asahi  and
+      Camacho-Collados, Jose",
+    booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations",
+    month = apr,
+    year = "2021",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2021.eacl-demos.7",
+    doi = "10.18653/v1/2021.eacl-demos.7",
+    pages = "53--62",
+    abstract = "Language model (LM) pretraining has led to consistent improvements in many NLP downstream tasks, including named entity recognition (NER). In this paper, we present T-NER (Transformer-based Named Entity Recognition), a Python library for NER LM finetuning. In addition to its practical utility, T-NER facilitates the study and investigation of the cross-domain and cross-lingual generalization ability of LMs finetuned on NER. Our library also provides a web app where users can get model predictions interactively for arbitrary text, which facilitates qualitative model evaluation for non-expert programmers. We show the potential of the library by compiling nine public NER datasets into a unified format and evaluating the cross-domain and cross- lingual performance across the datasets. The results from our initial experiments show that in-domain performance is generally competitive across datasets. However, cross-domain generalization is challenging even with a large pretrained LM, which has nevertheless capacity to learn domain-specific features if fine- tuned on a combined dataset. To facilitate future research, we also release all our LM checkpoints via the Hugging Face model hub.",
+}
+
+```

eval/metric.json

@@ -1 +0,0 @@
-{"random.dev": {"micro/f1": 0.6501650165016502, "micro/f1_ci": {}, "micro/recall": 0.6310731446876668, "micro/precision": 0.6704480998298356, "macro/f1": 0.6005028651678407, "macro/f1_ci": {}, "macro/recall": 0.5837622163983479, "macro/precision": 0.621752223375048, "per_entity_metric": {"corporation": {"f1": 0.5413105413105412, "f1_ci": {}, "precision": 0.6012658227848101, "recall": 0.49222797927461137}, "creative_work": {"f1": 0.5171339563862928, "f1_ci": {}, "precision": 0.5220125786163522, "recall": 0.5123456790123457}, "event": {"f1": 0.3730337078651686, "f1_ci": {}, "precision": 0.415, "recall": 0.33877551020408164}, "group": {"f1": 0.6291079812206571, "f1_ci": {}, "precision": 0.6860068259385665, "recall": 0.5809248554913294}, "location": {"f1": 0.6686746987951807, "f1_ci": {}, "precision": 0.6568047337278107, "recall": 0.6809815950920245}, "person": {"f1": 0.8540344514959203, "f1_ci": {}, "precision": 0.8263157894736842, "recall": 0.8836772983114447}, "product": {"f1": 0.6202247191011236, "f1_ci": {}, "precision": 0.6448598130841121, "recall": 0.5974025974025974}}}, "2021.test": {"micro/f1": 0.6632769652650823, "micro/f1_ci": {"90": [0.6546824558783396, 0.6722355436189195], "95": [0.6527609558375069, 0.6741666937877734]}, "micro/recall": 0.6712534690101758, "micro/precision": 0.6554878048780488, "macro/f1": 0.6096477771855761, "macro/f1_ci": {"90": [0.6000033399344015, 0.6196382573823579], "95": [0.5981472384000703, 0.6215975643766474]}, "macro/recall": 0.6191008735553379, "macro/precision": 0.6042443991246051, "per_entity_metric": {"corporation": {"f1": 0.5224148236700539, "f1_ci": {"90": [0.49575230566534917, 0.548428873717759], "95": [0.49180182879678314, 0.5536876221847715]}, "precision": 0.5653298835705045, "recall": 0.4855555555555556}, "creative_work": {"f1": 0.45186640471512773, "f1_ci": {"90": [0.4220381822268108, 0.4839308609644777], "95": [0.4154910463113729, 0.48911260537307555]}, "precision": 0.4334170854271357, "recall": 0.4719562243502052}, "event": {"f1": 0.4894837476099427, "f1_ci": {"90": [0.4661296765605938, 0.510871201463544], "95": [0.4615384615384615, 0.516411878785732]}, "precision": 0.5156092648539778, "recall": 0.4658780709736124}, "group": {"f1": 0.6327722432153899, "f1_ci": {"90": [0.6127003027617782, 0.6543839434706372], "95": [0.6093291625393491, 0.6588729726962956]}, "precision": 0.6611629576453697, "recall": 0.6067193675889329}, "location": {"f1": 0.6692258477287268, "f1_ci": {"90": [0.6417970501525123, 0.6963076846900378], "95": [0.6367998717602004, 0.7017981808437215]}, "precision": 0.6174734356552538, "recall": 0.7304469273743017}, "person": {"f1": 0.838405036726128, "f1_ci": {"90": [0.8282701377388866, 0.8495293562513028], "95": [0.825632170233011, 0.851578528092547]}, "precision": 0.7974051896207585, "recall": 0.8838495575221239}, "product": {"f1": 0.6633663366336633, "f1_ci": {"90": [0.6417739190805534, 0.6847604430802081], "95": [0.637114780619813, 0.689184375235439]}, "precision": 0.6393129770992366, "recall": 0.6893004115226338}}}, "2020.test": {"micro/f1": 0.6439847577572129, "micro/f1_ci": {"90": [0.6239511351712037, 0.6636956588748724], "95": [0.6202890037120405, 0.6672202398213474]}, "micro/recall": 0.6139076284379865, "micro/precision": 0.6771608471665712, "macro/f1": 0.6008744778169367, "macro/f1_ci": {"90": [0.5782367510542764, 0.6219123487136012], "95": [0.5732090094451493, 0.6256935746402632]}, "macro/recall": 0.5742193301311931, "macro/precision": 0.6358142893696356, "per_entity_metric": {"corporation": {"f1": 0.5344827586206895, "f1_ci": {"90": [0.46733010287597426, 0.5928074904241737], "95": [0.4548436580892954, 0.6016792777061768]}, "precision": 0.5923566878980892, "recall": 0.4869109947643979}, "creative_work": {"f1": 0.5027322404371585, "f1_ci": {"90": [0.4469743400143622, 0.5552825552825553], "95": [0.43710669569951005, 0.5633915193287383]}, "precision": 0.4919786096256685, "recall": 0.5139664804469274}, "event": {"f1": 0.46680080482897385, "f1_ci": {"90": [0.4107729801228506, 0.5203104053512446], "95": [0.4024955652314143, 0.5313219588102243]}, "precision": 0.5, "recall": 0.4377358490566038}, "group": {"f1": 0.5725190839694656, "f1_ci": {"90": [0.5231821675037455, 0.6240227074235808], "95": [0.5101295909346619, 0.6312023177209077]}, "precision": 0.704225352112676, "recall": 0.48231511254019294}, "location": {"f1": 0.654434250764526, "f1_ci": {"90": [0.5903994977449776, 0.7104554590942137], "95": [0.5794875818846374, 0.7240381663767523]}, "precision": 0.6604938271604939, "recall": 0.6484848484848484}, "person": {"f1": 0.81787521079258, "f1_ci": {"90": [0.7923931517380374, 0.8422851983243039], "95": [0.7870345804988661, 0.8482340257519794]}, "precision": 0.8220338983050848, "recall": 0.8137583892617449}, "product": {"f1": 0.6572769953051644, "f1_ci": {"90": [0.6066135800653779, 0.707374149659864], "95": [0.5938903100310432, 0.7184158092104591]}, "precision": 0.6796116504854369, "recall": 0.6363636363636364}}}, "2021.test (span detection)": {"micro/f1": 0.7900359938296291, "micro/f1_ci": {}, "micro/recall": 0.7995836706372152, "micro/precision": 0.780713640469738, "macro/f1": 0.7900359938296291, "macro/f1_ci": {}, "macro/recall": 0.7995836706372152, "macro/precision": 0.780713640469738}, "2020.test (span detection)": {"micro/f1": 0.7552409474543968, "micro/f1_ci": {}, "micro/recall": 0.7197716658017644, "micro/precision": 0.7943871706758304, "macro/f1": 0.7552409474543968, "macro/f1_ci": {}, "macro/recall": 0.7197716658017644, "macro/precision": 0.7943871706758304}}

eval/metric.test_2020.json

@@ -0,0 +1 @@
+{"micro/f1": 0.6439847577572129, "micro/f1_ci": {"90": [0.6239511351712037, 0.6636956588748724], "95": [0.6202890037120405, 0.6672202398213474]}, "micro/recall": 0.6139076284379865, "micro/precision": 0.6771608471665712, "macro/f1": 0.6008744778169367, "macro/f1_ci": {"90": [0.5782367510542764, 0.6219123487136012], "95": [0.5732090094451493, 0.6256935746402632]}, "macro/recall": 0.5742193301311931, "macro/precision": 0.6358142893696356, "per_entity_metric": {"corporation": {"f1": 0.5344827586206895, "f1_ci": {"90": [0.46733010287597426, 0.5928074904241737], "95": [0.4548436580892954, 0.6016792777061768]}, "precision": 0.5923566878980892, "recall": 0.4869109947643979}, "creative_work": {"f1": 0.5027322404371585, "f1_ci": {"90": [0.4469743400143622, 0.5552825552825553], "95": [0.43710669569951005, 0.5633915193287383]}, "precision": 0.4919786096256685, "recall": 0.5139664804469274}, "event": {"f1": 0.46680080482897385, "f1_ci": {"90": [0.4107729801228506, 0.5203104053512446], "95": [0.4024955652314143, 0.5313219588102243]}, "precision": 0.5, "recall": 0.4377358490566038}, "group": {"f1": 0.5725190839694656, "f1_ci": {"90": [0.5231821675037455, 0.6240227074235808], "95": [0.5101295909346619, 0.6312023177209077]}, "precision": 0.704225352112676, "recall": 0.48231511254019294}, "location": {"f1": 0.654434250764526, "f1_ci": {"90": [0.5903994977449776, 0.7104554590942137], "95": [0.5794875818846374, 0.7240381663767523]}, "precision": 0.6604938271604939, "recall": 0.6484848484848484}, "person": {"f1": 0.81787521079258, "f1_ci": {"90": [0.7923931517380374, 0.8422851983243039], "95": [0.7870345804988661, 0.8482340257519794]}, "precision": 0.8220338983050848, "recall": 0.8137583892617449}, "product": {"f1": 0.6572769953051644, "f1_ci": {"90": [0.6066135800653779, 0.707374149659864], "95": [0.5938903100310432, 0.7184158092104591]}, "precision": 0.6796116504854369, "recall": 0.6363636363636364}}}

eval/metric.test_2021.json

@@ -0,0 +1 @@
+{"micro/f1": 0.6632769652650823, "micro/f1_ci": {"90": [0.6546824558783396, 0.6722355436189195], "95": [0.6527609558375069, 0.6741666937877734]}, "micro/recall": 0.6712534690101758, "micro/precision": 0.6554878048780488, "macro/f1": 0.6096477771855761, "macro/f1_ci": {"90": [0.6000033399344015, 0.6196382573823579], "95": [0.5981472384000703, 0.6215975643766474]}, "macro/recall": 0.6191008735553379, "macro/precision": 0.6042443991246051, "per_entity_metric": {"corporation": {"f1": 0.5224148236700539, "f1_ci": {"90": [0.49575230566534917, 0.548428873717759], "95": [0.49180182879678314, 0.5536876221847715]}, "precision": 0.5653298835705045, "recall": 0.4855555555555556}, "creative_work": {"f1": 0.45186640471512773, "f1_ci": {"90": [0.4220381822268108, 0.4839308609644777], "95": [0.4154910463113729, 0.48911260537307555]}, "precision": 0.4334170854271357, "recall": 0.4719562243502052}, "event": {"f1": 0.4894837476099427, "f1_ci": {"90": [0.4661296765605938, 0.510871201463544], "95": [0.4615384615384615, 0.516411878785732]}, "precision": 0.5156092648539778, "recall": 0.4658780709736124}, "group": {"f1": 0.6327722432153899, "f1_ci": {"90": [0.6127003027617782, 0.6543839434706372], "95": [0.6093291625393491, 0.6588729726962956]}, "precision": 0.6611629576453697, "recall": 0.6067193675889329}, "location": {"f1": 0.6692258477287268, "f1_ci": {"90": [0.6417970501525123, 0.6963076846900378], "95": [0.6367998717602004, 0.7017981808437215]}, "precision": 0.6174734356552538, "recall": 0.7304469273743017}, "person": {"f1": 0.838405036726128, "f1_ci": {"90": [0.8282701377388866, 0.8495293562513028], "95": [0.825632170233011, 0.851578528092547]}, "precision": 0.7974051896207585, "recall": 0.8838495575221239}, "product": {"f1": 0.6633663366336633, "f1_ci": {"90": [0.6417739190805534, 0.6847604430802081], "95": [0.637114780619813, 0.689184375235439]}, "precision": 0.6393129770992366, "recall": 0.6893004115226338}}}

eval/metric_span.test_2020.json

@@ -0,0 +1 @@
+{"micro/f1": 0.7552409474543968, "micro/f1_ci": {}, "micro/recall": 0.7197716658017644, "micro/precision": 0.7943871706758304, "macro/f1": 0.7552409474543968, "macro/f1_ci": {}, "macro/recall": 0.7197716658017644, "macro/precision": 0.7943871706758304}

eval/metric_span.test_2021.json

@@ -0,0 +1 @@
+{"micro/f1": 0.7900359938296291, "micro/f1_ci": {}, "micro/recall": 0.7995836706372152, "micro/precision": 0.780713640469738, "macro/f1": 0.7900359938296291, "macro/f1_ci": {}, "macro/recall": 0.7995836706372152, "macro/precision": 0.780713640469738}

trainer_config.json

@@ -1 +1 @@
-{"data_split": "random.train", "model": "roberta-large", "crf": true, "max_length": 128, "epoch": 30, "batch_size": 32, "lr": 1e-05, "random_seed": 0, "gradient_accumulation_steps": 1, "weight_decay": 1e-07, "lr_warmup_step_ratio": 0.15, "max_grad_norm": 1}
+{"dataset": ["tner/tweetner7"], "dataset_split": "train_random", "dataset_name": null, "local_dataset": null, "model": "roberta-large", "crf": true, "max_length": 128, "epoch": 30, "batch_size": 32, "lr": 1e-05, "random_seed": 0, "gradient_accumulation_steps": 1, "weight_decay": 1e-07, "lr_warmup_step_ratio": 0.15, "max_grad_norm": 1}