full code and documentation uploaded

README.md

@@ -9,203 +9,146 @@ metrics:
 - f1
 - matthews_correlation
 base_model:
-- distilbert/distilbert-base-uncased
+- emilyalsentzer/Bio_ClinicalBERT
 tags:
 - IBD
 - cohort_identification
 - case_finding
 ---
-# Model Card for Model ID
+# Model Card for BioClinicalBERT IBD
 
-<!-- Provide a quick summary of what the model is/does. -->
-
-This modelcard aims to be a base template for new models. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md?plain=1).
+The model classifies documents as either IBD or Not IBD
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
-
-
+As above. This is a model trained to detect IBD patients from clinical text
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+- **Developed by:** Matt Stammers
+- **Funded by:** University Hospital Foundation NHS Trust
+- **Shared by:** Matt Stammers - SETT Data and AI Clinical Lead
+- **Model type:** BERT Transformer
+- **Language(s) (NLP):** English
+- **License:** cc-by-nc-4.0
+- **Finetuned from model:** emilyalsentzer/Bio_ClinicalBERT
 
-### Model Sources [optional]
+### Model Sources
 
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** https://huggingface.co/MattStammers/BioClinicalBERT_IBD
+- **Paper:** Arxiv (Pending)
+- **Demo:** https://huggingface.co/spaces/MattStammers/IBD_Cohort_Identification
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+For document classification tasks to differentiate between documents likely to be diagnostic of IBD and those unlikely to be diagnostic of IBD. 
 
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
+This model can be used directly at [Cohort Identification Demo](https://huggingface.co/spaces/MattStammers/IBD_Cohort_Identification)
 
-### Downstream Use [optional]
+### Downstream Use 
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
+Others can build on this model and improve it but only for non-commercial purposes.
 
 ### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
+This model is less powerful (in terms of F1 Score) when making predictions at the patient level by 1-2%. It can be used for this purpose but with care.
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
+This model contains substantial biases and is known to be biased against older patients and non-white patients so use with care. 
 
 ### Recommendations
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+It will work best in a predominantly younger caucasian population.
 
 ## How to Get Started with the Model
 
 Use the code below to get started with the model.
 
-[More Information Needed]
+The model is best used with the transformers library.
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
+The model was trained on fully pseudonymised clinical information at UHSFT which was carefully labelled by a consultant (attending) physician and evaluated against a randomly selected internal holdout set.
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
+See the paper for more information on the training procedure
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+- **Training regime:** fp32
 
-#### Speeds, Sizes, Times [optional]
+#### Speeds, Sizes, Times 
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
+This model (part of a set of models) took 213.55 minutes to train
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+The model was internally validated against a holdout set
 
 ### Testing Data, Factors & Metrics
 
 #### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
+The testing data cannot be revealed due to IG regulations and to remain compliant with GDPR, only the resulting model can be
 
 #### Factors
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
+IBD vs Not-IBD
 
 #### Metrics
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
+Full evaluation metrics are available in the paper with a summary below
 
 ### Results
 
-[More Information Needed]
+| Model             | Doc Coverage     | Accuracy                         | Precision                         | Recall                            | Specificity                       | NPV                              | F1 Score                         | MCC                            |
+|------------------|------------------|----------------------------------|-----------------------------------|-----------------------------------|-----------------------------------|----------------------------------|----------------------------------|----------------------------------|
+| BioclinicalBERT  | 768 (100.00%)    | 90.29% (CI: 87.33% - 92.62%)     | 91.48% (CI: 88.39% - 93.81%)      | 96.91% (CI: 94.67% - 98.22%)      | 63.54% (CI: 53.57% - 72.48%)      | 83.56% (CI: 73.43% - 90.34%)     | 94.12% (CI: 92.79% - 95.48%)     | 0.6735 (CI: 0.5892 - 0.7538)     |
 
 #### Summary
 
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
+Overall performance of the model is high with an F1 Score of >94% on our internal holdout set.
 
 ## Environmental Impact
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
+Training the model used 2.01kWh of energy emmitting 416.73 grams of CO2
 
-[More Information Needed]
+- **Hardware Type:** L40S
+- **Hours used:** 3.55
+- **Carbon Emitted:** 0.417 Kg CO2
 
-## Glossary [optional]
+## Citation
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+Arxiv (Pending)
 
-[More Information Needed]
+## Glossary
 
-## More Information [optional]
+| Term                                | Description |
+|-------------------------------------|-------------|
+| **Accuracy**                        | The percentage of results that were correct among all results from the system. Calc: (TP + TN) / (TP + FP + TN + FN). |
+| **Precision (PPV)**                 | Also called positive predictive value (PPV), it is the percentage of true positive results among all results that the system flagged as positive. Calc: TP / (TP + FP). |
+| **Negative Predictive Value (NPV)** | The percentage of results that were true negative (TN) among all results that the system flagged as negative. Calc: TN / (TN + FN). |
+| **Recall**                          | Also called sensitivity. The percentage of results flagged positive among all results that should have been obtained. Calc: TP / (TP + FN). |
+| **Specificity**                     | The percentage of results that were flagged negative among all negative results. Calc: TN / (TN + FP). |
+| **F1-Score**                        | The harmonic mean of PPV/precision and sensitivity/recall. Calc: 2 × (Precision × Recall) / (Precision + Recall). Moderately useful in the context of class imbalance. |
+| **Matthews’ Correlation Coefficient (MCC)** | A statistical measure used to evaluate the quality of binary classifications. Unlike other metrics, MCC considers all four categories of a confusion matrix. Calc: (TP × TN − FP × FN) / √((TP + FP)(TP + FN)(TN + FP)(TN + FN)). |
+| **Precision / Recall AUC**         | Represents the area under the Precision-Recall curve, which plots Precision against Recall at various threshold settings. It is more resistant to class imbalance than alternatives like AUROC. |
+| **Demographic Parity (DP)**        | Demographic Parity, also known as Statistical Parity, requires that the probability of a positive prediction is the same across different demographic groups. Calc: DP = P(Ŷ=1∣A=a) = P(Ŷ=1∣A=b). This figure is given as an absolute difference where positive values suggest the more privileged group gains and negative values the reverse. |
+| **Equal Opportunity (EO)**         | Equal Opportunity focuses on equalising the true positive rates across groups. Among those who truly belong to the positive class, the model should predict positive outcomes at equal rates across groups. Calc: EO = P(Ŷ=1∣Y=1, A=a) = P(Ŷ=1∣Y=1, A=b). A higher value indicates a bias against the more vulnerable group. |
+| **Disparate Impact (DI)**          | Divides the protected group’s positive prediction rate by that of the most-favoured group. If the ratio is below 0.8 or above 1.25, disparate impact is considered present. Calc: DI = P(Ŷ=1∣A=unfavoured) / P(Ŷ=1∣A=favoured). Values outside 0.8–1.25 range suggest bias. |
+| **Execution Time / Energy / CO₂ Emissions** | Measured in minutes and total energy consumption in kilowatt-hours (kWh), which is then converted to CO₂ emissions using a factor of 0.20705 Kg CO₂e per kWh. |
 
-[More Information Needed]
 
-## Model Card Authors [optional]
+## Model Card Authors
 
-[More Information Needed]
+Matt Stammers - Computational Gastroenterologist
 
 ## Model Card Contact
 
-[More Information Needed]
+m.stammers@soton.ac.uk

config.json

@@ -0,0 +1,26 @@
+{
+  "_name_or_path": "emilyalsentzer/Bio_ClinicalBERT",
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "problem_type": "single_label_classification",
+  "torch_dtype": "float32",
+  "transformers_version": "4.49.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 28996
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e8f3fd198328902ef862679f8c2e62491563faa2845cbf3c3d97585f12f36133
+size 433270768

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}