Upload folder using huggingface_hub

README.md

@@ -0,0 +1,75 @@
+---
+task: token-classification
+tags:
+- biomedical
+- bionlp
+license: mit
+base_model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext
+---
+
+# bioner_tmvar3
+
+This is a named entity recognition model fine-tuned from the [microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext](https://huggingface.co/microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext) model. It predicts spans with 10 possible labels. The labels are **AcidChange, CellLine, DNAAllele, DNAMutation, Gene, OtherMutation, ProteinAllele, ProteinMutation, SNP and Species**.
+
+The code used for training this model can be found at https://github.com/Glasgow-AI4BioMed/bioner along with links to other biomedical NER models trained on well-known biomedical corpora. The source dataset information is below.
+
+## Example Usage
+
+The code below will load up the model and apply it to the provided text. It uses a simple aggregation strategy to post-process the individual tokens into larger multi-token entities where needed.
+
+```python
+from transformers import pipeline
+
+# Load the model as part of an NER pipeline
+ner_pipeline = pipeline("token-classification", 
+                        model="Glasgow-AI4BioMed/bioner_tmvar3",
+                        aggregation_strategy="max")
+
+# Apply it to some text
+ner_pipeline("EGFR T790M mutations have been known to affect treatment outcomes for NSCLC patients receiving erlotinib.")
+
+# Output:
+# [ {"entity_group": "Gene", "score": 0.78493, "word": "egfr", "start": 0, "end": 4},
+#   {"entity_group": "Species", "score": 0.99795, "word": "patients", "start": 57, "end": 65} ]
+```
+
+## Dataset Info
+
+**Source:** The tmVar3 dataset was downloaded from: https://ftp.ncbi.nlm.nih.gov/pub/lu/tmVar3/
+
+The dataset should be cited with: Wei, Chih-Hsuan, et al. "tmVar 3.0: an improved variant concept recognition and normalization tool." Bioinformatics 38.18 (2022): 4449-4451. DOI: [10.1093/bioinformatics/btac537](https://doi.org/10.1093/bioinformatics/btac537)
+
+**Preprocessing:** The training/validation/test split was maintained from the original dataset. No changes were made to the annotations. The preprocessing script for this dataset is [prepare_tmvar.py](https://github.com/Glasgow-AI4BioMed/bioner/blob/main/prepare_tmvar.py).
+
+## Performance
+
+The span-level performance on the test split for the different labels are shown in the tables below. The full performance results are available in the model repo in Markdown format for viewing and JSON format for easier loading. These include the performance at token level (with individual B- and I- labels as the token classifier uses IOB2 token labelling).
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| AcidChange | 0.562 | 0.818 | 0.667 | 11 |
+| CellLine | 0.550 | 0.688 | 0.611 | 16 |
+| DNAAllele | 0.786 | 0.733 | 0.759 | 15 |
+| DNAMutation | 0.824 | 0.864 | 0.844 | 125 |
+| Gene | 0.886 | 0.907 | 0.897 | 766 |
+| OtherMutation | 0.410 | 0.561 | 0.474 | 57 |
+| ProteinAllele | 0.857 | 0.706 | 0.774 | 17 |
+| ProteinMutation | 0.904 | 0.851 | 0.877 | 121 |
+| SNP | 0.815 | 1.000 | 0.898 | 22 |
+| Species | 0.955 | 0.949 | 0.952 | 333 |
+| macro avg | 0.755 | 0.808 | 0.775 | 1483 |
+| weighted avg | 0.871 | 0.889 | 0.879 | 1483 |
+
+
+## Hyperparameters
+
+Hyperparameter tuning was done with [optuna](https://optuna.org/) and the [hyperparameter_search](https://huggingface.co/docs/transformers/en/hpo_train) functionality. 100 trials were run. Early stopping was applied during training. The best performing model was selected using the macro F1 performance on the validation set. The selected hyperparameters are in the table below.
+
+| Hyperparameter | Value |
+|----------------|-------|
+| epochs | 17.0 |
+| learning_rate | 9.821730986128684e-05 |
+| per_device_train_batch_size | 16 |
+| weight_decay | 0.2819767106649835 |
+| warmup_ratio | 0.10342300416492177 |
+

best_hyperparameters.json

@@ -0,0 +1,7 @@
+{
+  "epochs": 17.0,
+  "learning_rate": 9.821730986128684e-05,
+  "per_device_train_batch_size": 16,
+  "weight_decay": 0.2819767106649835,
+  "warmup_ratio": 0.10342300416492177
+}

config.json

@@ -0,0 +1,48 @@
+{
+  "_name_or_path": "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext",
+  "architectures": [
+    "BertForTokenClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "O",
+    "1": "B-AcidChange",
+    "2": "I-AcidChange",
+    "3": "B-CellLine",
+    "4": "I-CellLine",
+    "5": "B-DNAAllele",
+    "6": "I-DNAAllele",
+    "7": "B-DNAMutation",
+    "8": "I-DNAMutation",
+    "9": "B-Gene",
+    "10": "I-Gene",
+    "11": "B-OtherMutation",
+    "12": "I-OtherMutation",
+    "13": "B-ProteinAllele",
+    "14": "I-ProteinAllele",
+    "15": "B-ProteinMutation",
+    "16": "I-ProteinMutation",
+    "17": "B-SNP",
+    "18": "I-SNP",
+    "19": "B-Species",
+    "20": "I-Species"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.48.1",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8801c9be6f28c8c6cd2db3ddc727b5ba68f85efb12389ca3c6c562e7b44da179
+size 435654532

performance_report.json

@@ -0,0 +1,653 @@
+{
+  "train": {
+    "token_level": {
+      "O": {
+        "precision": 0.9994677158065263,
+        "recall": 0.9999884225759769,
+        "f1-score": 0.999728001388929,
+        "support": 86375.0
+      },
+      "B-AcidChange": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 57.0
+      },
+      "I-AcidChange": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 170.0
+      },
+      "B-CellLine": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 23.0
+      },
+      "I-CellLine": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 33.0
+      },
+      "B-DNAAllele": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 35.0
+      },
+      "I-DNAAllele": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 92.0
+      },
+      "B-DNAMutation": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 391.0
+      },
+      "I-DNAMutation": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 2067.0
+      },
+      "B-Gene": {
+        "precision": 1.0,
+        "recall": 0.9983799108950993,
+        "f1-score": 0.999189298743413,
+        "support": 2469.0
+      },
+      "I-Gene": {
+        "precision": 0.9994079336885732,
+        "recall": 0.9888693614528412,
+        "f1-score": 0.9941107184923439,
+        "support": 3414.0
+      },
+      "B-OtherMutation": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 147.0
+      },
+      "I-OtherMutation": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 508.0
+      },
+      "B-ProteinAllele": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 43.0
+      },
+      "I-ProteinAllele": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 92.0
+      },
+      "B-ProteinMutation": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 386.0
+      },
+      "I-ProteinMutation": {
+        "precision": 1.0,
+        "recall": 0.9992603550295858,
+        "f1-score": 0.9996300406955235,
+        "support": 1352.0
+      },
+      "B-SNP": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 96.0
+      },
+      "I-SNP": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 279.0
+      },
+      "B-Species": {
+        "precision": 0.9990610328638497,
+        "recall": 0.99812382739212,
+        "f1-score": 0.998592210229939,
+        "support": 1066.0
+      },
+      "I-Species": {
+        "precision": 1.0,
+        "recall": 0.9863636363636363,
+        "f1-score": 0.9931350114416476,
+        "support": 220.0
+      },
+      "accuracy": 0.9995066203493933,
+      "macro avg": {
+        "precision": 0.9999017467789975,
+        "recall": 0.998618357795679,
+        "f1-score": 0.9992564419519904,
+        "support": 99315.0
+      },
+      "weighted avg": {
+        "precision": 0.9995066374609511,
+        "recall": 0.9995066203493933,
+        "f1-score": 0.9995054856279689,
+        "support": 99315.0
+      }
+    },
+    "span_level": {
+      "AcidChange": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 57
+      },
+      "CellLine": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 23
+      },
+      "DNAAllele": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 35
+      },
+      "DNAMutation": {
+        "precision": 0.9974424552429667,
+        "recall": 0.9974424552429667,
+        "f1-score": 0.9974424552429667,
+        "support": 391
+      },
+      "Gene": {
+        "precision": 0.9959349593495935,
+        "recall": 0.9863123993558777,
+        "f1-score": 0.9911003236245955,
+        "support": 2484
+      },
+      "OtherMutation": {
+        "precision": 0.9716312056737588,
+        "recall": 0.9319727891156463,
+        "f1-score": 0.951388888888889,
+        "support": 147
+      },
+      "ProteinAllele": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 43
+      },
+      "ProteinMutation": {
+        "precision": 0.9974093264248705,
+        "recall": 0.9771573604060914,
+        "f1-score": 0.9871794871794872,
+        "support": 394
+      },
+      "SNP": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 96
+      },
+      "Species": {
+        "precision": 0.9990539262062441,
+        "recall": 0.9841565703634669,
+        "f1-score": 0.9915492957746479,
+        "support": 1073
+      },
+      "macro avg": {
+        "precision": 0.9961471872897434,
+        "recall": 0.987704157448405,
+        "f1-score": 0.9918660450710588,
+        "support": 4743
+      },
+      "weighted avg": {
+        "precision": 0.9963517528334667,
+        "recall": 0.9850305713683323,
+        "f1-score": 0.990644841422112,
+        "support": 4743
+      }
+    }
+  },
+  "val": {
+    "token_level": {
+      "O": {
+        "precision": 0.9925305486058232,
+        "recall": 0.9874107207272202,
+        "f1-score": 0.9899640151172673,
+        "support": 27722.0
+      },
+      "B-AcidChange": {
+        "precision": 0.76,
+        "recall": 0.76,
+        "f1-score": 0.76,
+        "support": 25.0
+      },
+      "I-AcidChange": {
+        "precision": 0.6944444444444444,
+        "recall": 0.7575757575757576,
+        "f1-score": 0.7246376811594203,
+        "support": 66.0
+      },
+      "B-CellLine": {
+        "precision": 0.42105263157894735,
+        "recall": 1.0,
+        "f1-score": 0.5925925925925926,
+        "support": 8.0
+      },
+      "I-CellLine": {
+        "precision": 0.7142857142857143,
+        "recall": 1.0,
+        "f1-score": 0.8333333333333334,
+        "support": 10.0
+      },
+      "B-DNAAllele": {
+        "precision": 0.9285714285714286,
+        "recall": 0.7222222222222222,
+        "f1-score": 0.8125,
+        "support": 18.0
+      },
+      "I-DNAAllele": {
+        "precision": 0.9,
+        "recall": 0.6923076923076923,
+        "f1-score": 0.782608695652174,
+        "support": 52.0
+      },
+      "B-DNAMutation": {
+        "precision": 0.9016393442622951,
+        "recall": 0.9322033898305084,
+        "f1-score": 0.9166666666666666,
+        "support": 118.0
+      },
+      "I-DNAMutation": {
+        "precision": 0.9351701782820098,
+        "recall": 0.8560830860534124,
+        "f1-score": 0.8938807126258714,
+        "support": 674.0
+      },
+      "B-Gene": {
+        "precision": 0.9036004645760743,
+        "recall": 0.9688667496886675,
+        "f1-score": 0.9350961538461539,
+        "support": 803.0
+      },
+      "I-Gene": {
+        "precision": 0.8725490196078431,
+        "recall": 0.9541910331384016,
+        "f1-score": 0.9115456238361266,
+        "support": 1026.0
+      },
+      "B-OtherMutation": {
+        "precision": 0.6060606060606061,
+        "recall": 0.7407407407407407,
+        "f1-score": 0.6666666666666666,
+        "support": 54.0
+      },
+      "I-OtherMutation": {
+        "precision": 0.6,
+        "recall": 0.6026200873362445,
+        "f1-score": 0.6013071895424836,
+        "support": 229.0
+      },
+      "B-ProteinAllele": {
+        "precision": 0.4166666666666667,
+        "recall": 0.7142857142857143,
+        "f1-score": 0.5263157894736842,
+        "support": 7.0
+      },
+      "I-ProteinAllele": {
+        "precision": 0.2777777777777778,
+        "recall": 0.7692307692307693,
+        "f1-score": 0.40816326530612246,
+        "support": 13.0
+      },
+      "B-ProteinMutation": {
+        "precision": 0.9166666666666666,
+        "recall": 0.9211822660098522,
+        "f1-score": 0.918918918918919,
+        "support": 203.0
+      },
+      "I-ProteinMutation": {
+        "precision": 0.9349470499243571,
+        "recall": 0.9182763744427934,
+        "f1-score": 0.9265367316341829,
+        "support": 673.0
+      },
+      "B-SNP": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 17.0
+      },
+      "I-SNP": {
+        "precision": 0.9622641509433962,
+        "recall": 1.0,
+        "f1-score": 0.9807692307692307,
+        "support": 51.0
+      },
+      "B-Species": {
+        "precision": 0.96875,
+        "recall": 0.9855491329479769,
+        "f1-score": 0.9770773638968482,
+        "support": 346.0
+      },
+      "I-Species": {
+        "precision": 0.9375,
+        "recall": 0.9090909090909091,
+        "f1-score": 0.9230769230769231,
+        "support": 33.0
+      },
+      "accuracy": 0.976421550329725,
+      "macro avg": {
+        "precision": 0.7925941282025739,
+        "recall": 0.8662779355061373,
+        "f1-score": 0.8134122644816508,
+        "support": 32148.0
+      },
+      "weighted avg": {
+        "precision": 0.9778360580996203,
+        "recall": 0.976421550329725,
+        "f1-score": 0.9768605358437904,
+        "support": 32148.0
+      }
+    },
+    "span_level": {
+      "AcidChange": {
+        "precision": 0.6551724137931034,
+        "recall": 0.76,
+        "f1-score": 0.7037037037037037,
+        "support": 25
+      },
+      "CellLine": {
+        "precision": 0.42105263157894735,
+        "recall": 1.0,
+        "f1-score": 0.5925925925925926,
+        "support": 8
+      },
+      "DNAAllele": {
+        "precision": 0.9411764705882353,
+        "recall": 0.8421052631578947,
+        "f1-score": 0.8888888888888888,
+        "support": 19
+      },
+      "DNAMutation": {
+        "precision": 0.8442622950819673,
+        "recall": 0.8728813559322034,
+        "f1-score": 0.8583333333333334,
+        "support": 118
+      },
+      "Gene": {
+        "precision": 0.8688147295742232,
+        "recall": 0.9332509270704573,
+        "f1-score": 0.899880810488677,
+        "support": 809
+      },
+      "OtherMutation": {
+        "precision": 0.4931506849315068,
+        "recall": 0.6666666666666666,
+        "f1-score": 0.5669291338582677,
+        "support": 54
+      },
+      "ProteinAllele": {
+        "precision": 0.26666666666666666,
+        "recall": 0.5714285714285714,
+        "f1-score": 0.36363636363636365,
+        "support": 7
+      },
+      "ProteinMutation": {
+        "precision": 0.84688995215311,
+        "recall": 0.8676470588235294,
+        "f1-score": 0.8571428571428571,
+        "support": 204
+      },
+      "SNP": {
+        "precision": 1.0,
+        "recall": 1.0,
+        "f1-score": 1.0,
+        "support": 17
+      },
+      "Species": {
+        "precision": 0.9715909090909091,
+        "recall": 0.9771428571428571,
+        "f1-score": 0.9743589743589745,
+        "support": 350
+      },
+      "macro avg": {
+        "precision": 0.730877675345867,
+        "recall": 0.849112270022218,
+        "f1-score": 0.7705466658003658,
+        "support": 1611
+      },
+      "weighted avg": {
+        "precision": 0.8680591703649133,
+        "recall": 0.9168218497827436,
+        "f1-score": 0.8904727143994291,
+        "support": 1611
+      }
+    }
+  },
+  "test": {
+    "token_level": {
+      "O": {
+        "precision": 0.992110521601323,
+        "recall": 0.991290877796902,
+        "f1-score": 0.9917005303395551,
+        "support": 29050.0
+      },
+      "B-AcidChange": {
+        "precision": 0.6153846153846154,
+        "recall": 0.7272727272727273,
+        "f1-score": 0.6666666666666666,
+        "support": 11.0
+      },
+      "I-AcidChange": {
+        "precision": 0.6170212765957447,
+        "recall": 0.8529411764705882,
+        "f1-score": 0.7160493827160493,
+        "support": 34.0
+      },
+      "B-CellLine": {
+        "precision": 0.8888888888888888,
+        "recall": 1.0,
+        "f1-score": 0.9411764705882353,
+        "support": 16.0
+      },
+      "I-CellLine": {
+        "precision": 0.8888888888888888,
+        "recall": 0.6666666666666666,
+        "f1-score": 0.7619047619047619,
+        "support": 36.0
+      },
+      "B-DNAAllele": {
+        "precision": 1.0,
+        "recall": 0.8,
+        "f1-score": 0.8888888888888888,
+        "support": 15.0
+      },
+      "I-DNAAllele": {
+        "precision": 0.926829268292683,
+        "recall": 0.7755102040816326,
+        "f1-score": 0.8444444444444444,
+        "support": 49.0
+      },
+      "B-DNAMutation": {
+        "precision": 0.8818897637795275,
+        "recall": 0.896,
+        "f1-score": 0.8888888888888888,
+        "support": 125.0
+      },
+      "I-DNAMutation": {
+        "precision": 0.943089430894309,
+        "recall": 0.8592592592592593,
+        "f1-score": 0.8992248062015504,
+        "support": 675.0
+      },
+      "B-Gene": {
+        "precision": 0.9209183673469388,
+        "recall": 0.9425587467362925,
+        "f1-score": 0.9316129032258065,
+        "support": 766.0
+      },
+      "I-Gene": {
+        "precision": 0.9148936170212766,
+        "recall": 0.9265426052889324,
+        "f1-score": 0.9206812652068127,
+        "support": 1021.0
+      },
+      "B-OtherMutation": {
+        "precision": 0.5487804878048781,
+        "recall": 0.7894736842105263,
+        "f1-score": 0.6474820143884892,
+        "support": 57.0
+      },
+      "I-OtherMutation": {
+        "precision": 0.6742081447963801,
+        "recall": 0.764102564102564,
+        "f1-score": 0.7163461538461539,
+        "support": 195.0
+      },
+      "B-ProteinAllele": {
+        "precision": 1.0,
+        "recall": 0.7058823529411765,
+        "f1-score": 0.8275862068965517,
+        "support": 17.0
+      },
+      "I-ProteinAllele": {
+        "precision": 0.8571428571428571,
+        "recall": 0.75,
+        "f1-score": 0.8,
+        "support": 32.0
+      },
+      "B-ProteinMutation": {
+        "precision": 0.9482758620689655,
+        "recall": 0.9090909090909091,
+        "f1-score": 0.9282700421940928,
+        "support": 121.0
+      },
+      "I-ProteinMutation": {
+        "precision": 0.9340659340659341,
+        "recall": 0.96045197740113,
+        "f1-score": 0.947075208913649,
+        "support": 354.0
+      },
+      "B-SNP": {
+        "precision": 0.9166666666666666,
+        "recall": 1.0,
+        "f1-score": 0.9565217391304348,
+        "support": 22.0
+      },
+      "I-SNP": {
+        "precision": 0.8266666666666667,
+        "recall": 1.0,
+        "f1-score": 0.9051094890510949,
+        "support": 62.0
+      },
+      "B-Species": {
+        "precision": 0.9723926380368099,
+        "recall": 0.9519519519519519,
+        "f1-score": 0.9620637329286799,
+        "support": 333.0
+      },
+      "I-Species": {
+        "precision": 0.8409090909090909,
+        "recall": 0.8222222222222222,
+        "f1-score": 0.8314606741573034,
+        "support": 45.0
+      },
+      "accuracy": 0.9808088146264681,
+      "macro avg": {
+        "precision": 0.8623344279453543,
+        "recall": 0.8614865678806418,
+        "f1-score": 0.8558644890751479,
+        "support": 33036.0
+      },
+      "weighted avg": {
+        "precision": 0.9815714652726921,
+        "recall": 0.9808088146264681,
+        "f1-score": 0.9809991613726176,
+        "support": 33036.0
+      }
+    },
+    "span_level": {
+      "AcidChange": {
+        "precision": 0.5625,
+        "recall": 0.8181818181818182,
+        "f1-score": 0.6666666666666666,
+        "support": 11
+      },
+      "CellLine": {
+        "precision": 0.55,
+        "recall": 0.6875,
+        "f1-score": 0.6111111111111112,
+        "support": 16
+      },
+      "DNAAllele": {
+        "precision": 0.7857142857142857,
+        "recall": 0.7333333333333333,
+        "f1-score": 0.7586206896551724,
+        "support": 15
+      },
+      "DNAMutation": {
+        "precision": 0.8244274809160306,
+        "recall": 0.864,
+        "f1-score": 0.84375,
+        "support": 125
+      },
+      "Gene": {
+        "precision": 0.8864795918367347,
+        "recall": 0.9073107049608355,
+        "f1-score": 0.896774193548387,
+        "support": 766
+      },
+      "OtherMutation": {
+        "precision": 0.41025641025641024,
+        "recall": 0.5614035087719298,
+        "f1-score": 0.47407407407407404,
+        "support": 57
+      },
+      "ProteinAllele": {
+        "precision": 0.8571428571428571,
+        "recall": 0.7058823529411765,
+        "f1-score": 0.7741935483870968,
+        "support": 17
+      },
+      "ProteinMutation": {
+        "precision": 0.9035087719298246,
+        "recall": 0.8512396694214877,
+        "f1-score": 0.8765957446808511,
+        "support": 121
+      },
+      "SNP": {
+        "precision": 0.8148148148148148,
+        "recall": 1.0,
+        "f1-score": 0.8979591836734693,
+        "support": 22
+      },
+      "Species": {
+        "precision": 0.9546827794561934,
+        "recall": 0.948948948948949,
+        "f1-score": 0.9518072289156627,
+        "support": 333
+      },
+      "macro avg": {
+        "precision": 0.754952699206715,
+        "recall": 0.8077800336559531,
+        "f1-score": 0.7751552440712491,
+        "support": 1483
+      },
+      "weighted avg": {
+        "precision": 0.8711975142222171,
+        "recall": 0.8894133513149022,
+        "f1-score": 0.8791953544402704,
+        "support": 1483
+      }
+    }
+  }
+}

performance_report.md

@@ -0,0 +1,144 @@
+# Performance on Training Set
+
+## Span Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| AcidChange | 1.000 | 1.000 | 1.000 | 57 |
+| CellLine | 1.000 | 1.000 | 1.000 | 23 |
+| DNAAllele | 1.000 | 1.000 | 1.000 | 35 |
+| DNAMutation | 0.997 | 0.997 | 0.997 | 391 |
+| Gene | 0.996 | 0.986 | 0.991 | 2484 |
+| OtherMutation | 0.972 | 0.932 | 0.951 | 147 |
+| ProteinAllele | 1.000 | 1.000 | 1.000 | 43 |
+| ProteinMutation | 0.997 | 0.977 | 0.987 | 394 |
+| SNP | 1.000 | 1.000 | 1.000 | 96 |
+| Species | 0.999 | 0.984 | 0.992 | 1073 |
+| macro avg | 0.996 | 0.988 | 0.992 | 4743 |
+| weighted avg | 0.996 | 0.985 | 0.991 | 4743 |
+
+## Token Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| O | 0.999 | 1.000 | 1.000 | 86375 |
+| B-AcidChange | 1.000 | 1.000 | 1.000 | 57 |
+| I-AcidChange | 1.000 | 1.000 | 1.000 | 170 |
+| B-CellLine | 1.000 | 1.000 | 1.000 | 23 |
+| I-CellLine | 1.000 | 1.000 | 1.000 | 33 |
+| B-DNAAllele | 1.000 | 1.000 | 1.000 | 35 |
+| I-DNAAllele | 1.000 | 1.000 | 1.000 | 92 |
+| B-DNAMutation | 1.000 | 1.000 | 1.000 | 391 |
+| I-DNAMutation | 1.000 | 1.000 | 1.000 | 2067 |
+| B-Gene | 1.000 | 0.998 | 0.999 | 2469 |
+| I-Gene | 0.999 | 0.989 | 0.994 | 3414 |
+| B-OtherMutation | 1.000 | 1.000 | 1.000 | 147 |
+| I-OtherMutation | 1.000 | 1.000 | 1.000 | 508 |
+| B-ProteinAllele | 1.000 | 1.000 | 1.000 | 43 |
+| I-ProteinAllele | 1.000 | 1.000 | 1.000 | 92 |
+| B-ProteinMutation | 1.000 | 1.000 | 1.000 | 386 |
+| I-ProteinMutation | 1.000 | 0.999 | 1.000 | 1352 |
+| B-SNP | 1.000 | 1.000 | 1.000 | 96 |
+| I-SNP | 1.000 | 1.000 | 1.000 | 279 |
+| B-Species | 0.999 | 0.998 | 0.999 | 1066 |
+| I-Species | 1.000 | 0.986 | 0.993 | 220 |
+| macro avg | 1.000 | 0.999 | 0.999 | 99315 |
+| weighted avg | 1.000 | 1.000 | 1.000 | 99315 |
+
+
+# Performance on Validation Set
+
+## Span Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| AcidChange | 0.655 | 0.760 | 0.704 | 25 |
+| CellLine | 0.421 | 1.000 | 0.593 | 8 |
+| DNAAllele | 0.941 | 0.842 | 0.889 | 19 |
+| DNAMutation | 0.844 | 0.873 | 0.858 | 118 |
+| Gene | 0.869 | 0.933 | 0.900 | 809 |
+| OtherMutation | 0.493 | 0.667 | 0.567 | 54 |
+| ProteinAllele | 0.267 | 0.571 | 0.364 | 7 |
+| ProteinMutation | 0.847 | 0.868 | 0.857 | 204 |
+| SNP | 1.000 | 1.000 | 1.000 | 17 |
+| Species | 0.972 | 0.977 | 0.974 | 350 |
+| macro avg | 0.731 | 0.849 | 0.771 | 1611 |
+| weighted avg | 0.868 | 0.917 | 0.890 | 1611 |
+
+## Token Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| O | 0.993 | 0.987 | 0.990 | 27722 |
+| B-AcidChange | 0.760 | 0.760 | 0.760 | 25 |
+| I-AcidChange | 0.694 | 0.758 | 0.725 | 66 |
+| B-CellLine | 0.421 | 1.000 | 0.593 | 8 |
+| I-CellLine | 0.714 | 1.000 | 0.833 | 10 |
+| B-DNAAllele | 0.929 | 0.722 | 0.812 | 18 |
+| I-DNAAllele | 0.900 | 0.692 | 0.783 | 52 |
+| B-DNAMutation | 0.902 | 0.932 | 0.917 | 118 |
+| I-DNAMutation | 0.935 | 0.856 | 0.894 | 674 |
+| B-Gene | 0.904 | 0.969 | 0.935 | 803 |
+| I-Gene | 0.873 | 0.954 | 0.912 | 1026 |
+| B-OtherMutation | 0.606 | 0.741 | 0.667 | 54 |
+| I-OtherMutation | 0.600 | 0.603 | 0.601 | 229 |
+| B-ProteinAllele | 0.417 | 0.714 | 0.526 | 7 |
+| I-ProteinAllele | 0.278 | 0.769 | 0.408 | 13 |
+| B-ProteinMutation | 0.917 | 0.921 | 0.919 | 203 |
+| I-ProteinMutation | 0.935 | 0.918 | 0.927 | 673 |
+| B-SNP | 1.000 | 1.000 | 1.000 | 17 |
+| I-SNP | 0.962 | 1.000 | 0.981 | 51 |
+| B-Species | 0.969 | 0.986 | 0.977 | 346 |
+| I-Species | 0.938 | 0.909 | 0.923 | 33 |
+| macro avg | 0.793 | 0.866 | 0.813 | 32148 |
+| weighted avg | 0.978 | 0.976 | 0.977 | 32148 |
+
+
+# Performance on Testing Set
+
+## Span Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| AcidChange | 0.562 | 0.818 | 0.667 | 11 |
+| CellLine | 0.550 | 0.688 | 0.611 | 16 |
+| DNAAllele | 0.786 | 0.733 | 0.759 | 15 |
+| DNAMutation | 0.824 | 0.864 | 0.844 | 125 |
+| Gene | 0.886 | 0.907 | 0.897 | 766 |
+| OtherMutation | 0.410 | 0.561 | 0.474 | 57 |
+| ProteinAllele | 0.857 | 0.706 | 0.774 | 17 |
+| ProteinMutation | 0.904 | 0.851 | 0.877 | 121 |
+| SNP | 0.815 | 1.000 | 0.898 | 22 |
+| Species | 0.955 | 0.949 | 0.952 | 333 |
+| macro avg | 0.755 | 0.808 | 0.775 | 1483 |
+| weighted avg | 0.871 | 0.889 | 0.879 | 1483 |
+
+## Token Level
+
+| Label | Precision | Recall | F1-score | Support |
+| --- | --- | --- | --- | --- |
+| O | 0.992 | 0.991 | 0.992 | 29050 |
+| B-AcidChange | 0.615 | 0.727 | 0.667 | 11 |
+| I-AcidChange | 0.617 | 0.853 | 0.716 | 34 |
+| B-CellLine | 0.889 | 1.000 | 0.941 | 16 |
+| I-CellLine | 0.889 | 0.667 | 0.762 | 36 |
+| B-DNAAllele | 1.000 | 0.800 | 0.889 | 15 |
+| I-DNAAllele | 0.927 | 0.776 | 0.844 | 49 |
+| B-DNAMutation | 0.882 | 0.896 | 0.889 | 125 |
+| I-DNAMutation | 0.943 | 0.859 | 0.899 | 675 |
+| B-Gene | 0.921 | 0.943 | 0.932 | 766 |
+| I-Gene | 0.915 | 0.927 | 0.921 | 1021 |
+| B-OtherMutation | 0.549 | 0.789 | 0.647 | 57 |
+| I-OtherMutation | 0.674 | 0.764 | 0.716 | 195 |
+| B-ProteinAllele | 1.000 | 0.706 | 0.828 | 17 |
+| I-ProteinAllele | 0.857 | 0.750 | 0.800 | 32 |
+| B-ProteinMutation | 0.948 | 0.909 | 0.928 | 121 |
+| I-ProteinMutation | 0.934 | 0.960 | 0.947 | 354 |
+| B-SNP | 0.917 | 1.000 | 0.957 | 22 |
+| I-SNP | 0.827 | 1.000 | 0.905 | 62 |
+| B-Species | 0.972 | 0.952 | 0.962 | 333 |
+| I-Species | 0.841 | 0.822 | 0.831 | 45 |
+| macro avg | 0.862 | 0.861 | 0.856 | 33036 |
+| weighted avg | 0.982 | 0.981 | 0.981 | 33036 |
+
+

rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48277aac9233bb97b5ed847d4070db3bcc75af042bc93363baa226a19f421291
+size 14244

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

trainer_state.json

@@ -0,0 +1,251 @@
+{
+  "best_metric": 0.8134122644816508,
+  "best_model_checkpoint": "tmp_ner_fantastic-bale-13_29/run-67/checkpoint-323",
+  "epoch": 17.0,
+  "eval_steps": 500,
+  "global_step": 323,
+  "is_hyper_param_search": true,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 1.0,
+      "eval_accuracy": 0.8623242503421675,
+      "eval_loss": 0.7986403703689575,
+      "eval_macro_f1": 0.04409872183381453,
+      "eval_macro_precision": 0.04106305954010322,
+      "eval_macro_recall": 0.047619047619047616,
+      "eval_runtime": 0.7532,
+      "eval_samples_per_second": 132.763,
+      "eval_steps_per_second": 17.259,
+      "step": 19
+    },
+    {
+      "epoch": 2.0,
+      "eval_accuracy": 0.9034465596615653,
+      "eval_loss": 0.3603612184524536,
+      "eval_macro_f1": 0.12106609239101934,
+      "eval_macro_precision": 0.1259515690157799,
+      "eval_macro_recall": 0.14408855709243762,
+      "eval_runtime": 0.7511,
+      "eval_samples_per_second": 133.136,
+      "eval_steps_per_second": 17.308,
+      "step": 38
+    },
+    {
+      "epoch": 3.0,
+      "eval_accuracy": 0.9519410227696902,
+      "eval_loss": 0.18390077352523804,
+      "eval_macro_f1": 0.3148808447564158,
+      "eval_macro_precision": 0.3747061248934609,
+      "eval_macro_recall": 0.3212448125969776,
+      "eval_runtime": 0.7242,
+      "eval_samples_per_second": 138.085,
+      "eval_steps_per_second": 17.951,
+      "step": 57
+    },
+    {
+      "epoch": 4.0,
+      "eval_accuracy": 0.965036705238273,
+      "eval_loss": 0.1359422653913498,
+      "eval_macro_f1": 0.494219195343988,
+      "eval_macro_precision": 0.5137741995898872,
+      "eval_macro_recall": 0.5215328224472264,
+      "eval_runtime": 0.7408,
+      "eval_samples_per_second": 134.981,
+      "eval_steps_per_second": 17.547,
+      "step": 76
+    },
+    {
+      "epoch": 5.0,
+      "eval_accuracy": 0.9700447928331467,
+      "eval_loss": 0.11635323613882065,
+      "eval_macro_f1": 0.5902868533471133,
+      "eval_macro_precision": 0.6804922518993798,
+      "eval_macro_recall": 0.6058126722062964,
+      "eval_runtime": 0.7171,
+      "eval_samples_per_second": 139.455,
+      "eval_steps_per_second": 18.129,
+      "step": 95
+    },
+    {
+      "epoch": 6.0,
+      "eval_accuracy": 0.9751461988304093,
+      "eval_loss": 0.11470664292573929,
+      "eval_macro_f1": 0.6704539076542243,
+      "eval_macro_precision": 0.701121495572558,
+      "eval_macro_recall": 0.6900336890446782,
+      "eval_runtime": 0.7646,
+      "eval_samples_per_second": 130.796,
+      "eval_steps_per_second": 17.003,
+      "step": 114
+    },
+    {
+      "epoch": 7.0,
+      "eval_accuracy": 0.9759549583177802,
+      "eval_loss": 0.11508199572563171,
+      "eval_macro_f1": 0.7577631140226684,
+      "eval_macro_precision": 0.8839369325293773,
+      "eval_macro_recall": 0.7206940753172821,
+      "eval_runtime": 0.7145,
+      "eval_samples_per_second": 139.954,
+      "eval_steps_per_second": 18.194,
+      "step": 133
+    },
+    {
+      "epoch": 8.0,
+      "eval_accuracy": 0.9711024013935549,
+      "eval_loss": 0.12795516848564148,
+      "eval_macro_f1": 0.7437555649029086,
+      "eval_macro_precision": 0.7355731767766348,
+      "eval_macro_recall": 0.7848058983150393,
+      "eval_runtime": 0.7219,
+      "eval_samples_per_second": 138.514,
+      "eval_steps_per_second": 18.007,
+      "step": 152
+    },
+    {
+      "epoch": 9.0,
+      "eval_accuracy": 0.9756750031106134,
+      "eval_loss": 0.11990169435739517,
+      "eval_macro_f1": 0.7767197416060247,
+      "eval_macro_precision": 0.7831564923466648,
+      "eval_macro_recall": 0.8015624788490757,
+      "eval_runtime": 0.7239,
+      "eval_samples_per_second": 138.14,
+      "eval_steps_per_second": 17.958,
+      "step": 171
+    },
+    {
+      "epoch": 10.0,
+      "eval_accuracy": 0.9740885902700013,
+      "eval_loss": 0.1274283081293106,
+      "eval_macro_f1": 0.7875252140074755,
+      "eval_macro_precision": 0.7618302095117122,
+      "eval_macro_recall": 0.8457597210340326,
+      "eval_runtime": 0.721,
+      "eval_samples_per_second": 138.693,
+      "eval_steps_per_second": 18.03,
+      "step": 190
+    },
+    {
+      "epoch": 11.0,
+      "eval_accuracy": 0.975799427647132,
+      "eval_loss": 0.12168442457914352,
+      "eval_macro_f1": 0.787208882069088,
+      "eval_macro_precision": 0.7591686050582892,
+      "eval_macro_recall": 0.8584845414175304,
+      "eval_runtime": 0.7385,
+      "eval_samples_per_second": 135.405,
+      "eval_steps_per_second": 17.603,
+      "step": 209
+    },
+    {
+      "epoch": 12.0,
+      "eval_accuracy": 0.9761415951225582,
+      "eval_loss": 0.13082729279994965,
+      "eval_macro_f1": 0.7949954841060296,
+      "eval_macro_precision": 0.7671102748848504,
+      "eval_macro_recall": 0.8572876125725459,
+      "eval_runtime": 0.7165,
+      "eval_samples_per_second": 139.577,
+      "eval_steps_per_second": 18.145,
+      "step": 228
+    },
+    {
+      "epoch": 13.0,
+      "eval_accuracy": 0.9758927460495209,
+      "eval_loss": 0.13127866387367249,
+      "eval_macro_f1": 0.7913277905680408,
+      "eval_macro_precision": 0.7593907970089686,
+      "eval_macro_recall": 0.8538236917477225,
+      "eval_runtime": 0.7592,
+      "eval_samples_per_second": 131.72,
+      "eval_steps_per_second": 17.124,
+      "step": 247
+    },
+    {
+      "epoch": 14.0,
+      "eval_accuracy": 0.9762038073908175,
+      "eval_loss": 0.139426127076149,
+      "eval_macro_f1": 0.794994965810495,
+      "eval_macro_precision": 0.7684245960923229,
+      "eval_macro_recall": 0.8567043677879977,
+      "eval_runtime": 0.7554,
+      "eval_samples_per_second": 132.38,
+      "eval_steps_per_second": 17.209,
+      "step": 266
+    },
+    {
+      "epoch": 15.0,
+      "eval_accuracy": 0.9767015055368918,
+      "eval_loss": 0.13470035791397095,
+      "eval_macro_f1": 0.8042100829526746,
+      "eval_macro_precision": 0.7791800304712903,
+      "eval_macro_recall": 0.8620026752410235,
+      "eval_runtime": 0.7309,
+      "eval_samples_per_second": 136.825,
+      "eval_steps_per_second": 17.787,
+      "step": 285
+    },
+    {
+      "epoch": 16.0,
+      "eval_accuracy": 0.9766392932686325,
+      "eval_loss": 0.14018748700618744,
+      "eval_macro_f1": 0.8049061480916583,
+      "eval_macro_precision": 0.7872742624263276,
+      "eval_macro_recall": 0.8567333662215458,
+      "eval_runtime": 0.7278,
+      "eval_samples_per_second": 137.407,
+      "eval_steps_per_second": 17.863,
+      "step": 304
+    },
+    {
+      "epoch": 17.0,
+      "eval_accuracy": 0.976421550329725,
+      "eval_loss": 0.14483144879341125,
+      "eval_macro_f1": 0.8134122644816508,
+      "eval_macro_precision": 0.7925941282025739,
+      "eval_macro_recall": 0.8662779355061373,
+      "eval_runtime": 0.7265,
+      "eval_samples_per_second": 137.647,
+      "eval_steps_per_second": 17.894,
+      "step": 323
+    }
+  ],
+  "logging_steps": 500,
+  "max_steps": 608,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 32,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 3,
+        "early_stopping_threshold": 0.001
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 0
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0,
+  "train_batch_size": 16,
+  "trial_name": null,
+  "trial_params": {
+    "learning_rate": 9.821730986128684e-05,
+    "per_device_train_batch_size": 16,
+    "warmup_ratio": 0.10342300416492177,
+    "weight_decay": 0.2819767106649835
+  }
+}