README.md

---
license: cc-by-nc-4.0
language:
- en
metrics:
- precision
- recall
- brier_score
- f1
- matthews_correlation
base_model:
- sentence-transformers/all-mpnet-base-v2
tags:
- IBD
- cohort_identification
- case_finding
---
# Model Card for SBERT IBD

The model classifies documents as either IBD or Not IBD

## Model Details

### Model Description

As above. This is a model trained to detect IBD patients from clinical text

- **Developed by:** Matt Stammers
- **Funded by:** University Hospital Foundation NHS Trust
- **Shared by:** Matt Stammers - SETT Data and AI Clinical Lead
- **Model type:** BERT Transformer
- **Language(s) (NLP):** English
- **License:** cc-by-nc-4.0
- **Finetuned from model:** sentence-transformers/all-mpnet-base-v2

### Model Sources

- **Repository:** https://huggingface.co/MattStammers/SBERT_IBD
- **Paper:** MedRxiv- [MedRxiv Paper](https://www.medrxiv.org/content/10.1101/2025.07.06.25330961v1)
- **Demo:** https://huggingface.co/spaces/MattStammers/IBD_Cohort_Identification

## Uses

For document classification tasks to differentiate between documents likely to be diagnostic of IBD and those unlikely to be diagnostic of IBD. 

### Direct Use

This model can be used directly at [Cohort Identification Demo](https://huggingface.co/spaces/MattStammers/IBD_Cohort_Identification)

### Downstream Use 

Others can build on this model and improve it but only for non-commercial purposes.

### Out-of-Scope Use

This model is less powerful (in terms of F1 Score) when making predictions at the patient level by 1-2%. It can be used for this purpose but with care.

## Bias, Risks, and Limitations

This model does not have any major known biases. It only has some slight bias tendencies. 

### Recommendations

This model might work well in any population but has not yet been tested in this regard as such.

## How to Get Started with the Model

Use the code below to get started with the model.

The model is best used with the transformers library.

## Training Details

### Training Data

The model was trained on fully pseudonymised clinical information at UHSFT which was carefully labelled by a consultant (attending) physician and evaluated against a randomly selected internal holdout set.

### Training Procedure

See the paper for more information on the training procedure

#### Training Hyperparameters

- **Training regime:** fp32

#### Speeds, Sizes, Times 

This model (part of a set of models) took 5.61 minutes to train

## Evaluation

The model was internally validated against a holdout set

### Testing Data, Factors & Metrics

#### Testing Data

The testing data cannot be revealed due to IG regulations and to remain compliant with GDPR, only the resulting model can be

#### Factors

IBD vs Not-IBD

#### Metrics

Full evaluation metrics are available in the paper with a summary below

### Results

| Model | Doc Coverage     | Accuracy                          | Precision                         | Recall                            | Specificity                      | NPV                              | F1 Score                         | MCC                               |
|-------|------------------|-----------------------------------|-----------------------------------|-----------------------------------|----------------------------------|----------------------------------|----------------------------------|------------------------------------|
| SBERT | 768 (100.00%)    | 89.67% (CI: 86.64% - 92.08%)      | 88.81% (CI: 85.44% - 91.48%)      | 99.20% (CI: 97.68% - 99.73%)      | 56.48% (CI: 47.07% - 65.45%)     | 95.31% (CI: 87.10% - 98.39%)     | 93.72% (CI: 92.50% - 94.90%)     | 0.6844 (CI: 0.6120 - 0.7545)        |


#### Summary

Overall performance of the model is high with an F1 Score of >94% on our internal holdout set.

## Environmental Impact

Training the model used 0.045kWh of energy emmitting 9.31 grams of CO2

- **Hardware Type:** L40S
- **Hours used:** 0.1
- **Carbon Emitted:** 0.009 Kg CO2

## Citation
Stammers M, Gwiggner M, Nouraei R, Metcalf C, Batchelor J. From Rule-Based to DeepSeek R1: A Robust Comparative Evaluation of Fifty Years of Natural Language Processing (NLP) Models To Identify Inflammatory Bowel Disease Cohorts. medRxiv. 2025:2025-07.
MedRxiv- [MedRxiv Paper](https://www.medrxiv.org/content/10.1101/2025.07.06.25330961v1)

## Base Training Data for the Model

#### Training data

We use the concatenation from multiple datasets to fine-tune our model. The total number of sentence pairs is above 1 billion sentences.
We sampled each dataset given a weighted probability which configuration is detailed in the `data_config.json` file.


| Dataset                                                  | Paper                                    | Number of training tuples  |
|--------------------------------------------------------|:----------------------------------------:|:--------------------------:|
| [Reddit comments (2015-2018)](https://github.com/PolyAI-LDN/conversational-datasets/tree/master/reddit) | [paper](https://arxiv.org/abs/1904.06472) | 726,484,430 |
| [S2ORC](https://github.com/allenai/s2orc) Citation pairs (Abstracts) | [paper](https://aclanthology.org/2020.acl-main.447/) | 116,288,806 |
| [WikiAnswers](https://github.com/afader/oqa#wikianswers-corpus) Duplicate question pairs | [paper](https://doi.org/10.1145/2623330.2623677) | 77,427,422 |
| [PAQ](https://github.com/facebookresearch/PAQ) (Question, Answer) pairs | [paper](https://arxiv.org/abs/2102.07033) | 64,371,441 |
| [S2ORC](https://github.com/allenai/s2orc) Citation pairs (Titles) | [paper](https://aclanthology.org/2020.acl-main.447/) | 52,603,982 |
| [S2ORC](https://github.com/allenai/s2orc) (Title, Abstract) | [paper](https://aclanthology.org/2020.acl-main.447/) | 41,769,185 |
| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title, Body) pairs  | - | 25,316,456 |
| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title+Body, Answer) pairs  | - | 21,396,559 |
| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title, Answer) pairs  | - | 21,396,559 |
| [MS MARCO](https://microsoft.github.io/msmarco/) triplets | [paper](https://doi.org/10.1145/3404835.3462804) | 9,144,553 |
| [GOOAQ: Open Question Answering with Diverse Answer Types](https://github.com/allenai/gooaq) | [paper](https://arxiv.org/pdf/2104.08727.pdf) | 3,012,496 |
| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Title, Answer) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 1,198,260 |
| [Code Search](https://huggingface.co/datasets/code_search_net) | - | 1,151,414 |
| [COCO](https://cocodataset.org/#home) Image captions | [paper](https://link.springer.com/chapter/10.1007%2F978-3-319-10602-1_48) | 828,395|
| [SPECTER](https://github.com/allenai/specter) citation triplets | [paper](https://doi.org/10.18653/v1/2020.acl-main.207) | 684,100 |
| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Question, Answer) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 681,164 |
| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Title, Question) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 659,896 |
| [SearchQA](https://huggingface.co/datasets/search_qa) | [paper](https://arxiv.org/abs/1704.05179) | 582,261 |
| [Eli5](https://huggingface.co/datasets/eli5) | [paper](https://doi.org/10.18653/v1/p19-1346) | 325,475 |
| [Flickr 30k](https://shannon.cs.illinois.edu/DenotationGraph/) | [paper](https://transacl.org/ojs/index.php/tacl/article/view/229/33) | 317,695 |
| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (titles) | | 304,525 |
| AllNLI ([SNLI](https://nlp.stanford.edu/projects/snli/) and [MultiNLI](https://cims.nyu.edu/~sbowman/multinli/) | [paper SNLI](https://doi.org/10.18653/v1/d15-1075), [paper MultiNLI](https://doi.org/10.18653/v1/n18-1101) | 277,230 | 
| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (bodies) | | 250,519 |
| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (titles+bodies) | | 250,460 |
| [Sentence Compression](https://github.com/google-research-datasets/sentence-compression) | [paper](https://www.aclweb.org/anthology/D13-1155/) | 180,000 |
| [Wikihow](https://github.com/pvl/wikihow_pairs_dataset) | [paper](https://arxiv.org/abs/1810.09305) | 128,542 |
| [Altlex](https://github.com/chridey/altlex/) | [paper](https://aclanthology.org/P16-1135.pdf) | 112,696 |
| [Quora Question Triplets](https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs) | - | 103,663 |
| [Simple Wikipedia](https://cs.pomona.edu/~dkauchak/simplification/) | [paper](https://www.aclweb.org/anthology/P11-2117/) | 102,225 |
| [Natural Questions (NQ)](https://ai.google.com/research/NaturalQuestions) | [paper](https://transacl.org/ojs/index.php/tacl/article/view/1455) | 100,231 |
| [SQuAD2.0](https://rajpurkar.github.io/SQuAD-explorer/) | [paper](https://aclanthology.org/P18-2124.pdf) | 87,599 |
| [TriviaQA](https://huggingface.co/datasets/trivia_qa) | - | 73,346 |
| **Total** | | **1,170,060,424** |

## Glossary

| Term                                | Description |
|-------------------------------------|-------------|
| **Accuracy**                        | The percentage of results that were correct among all results from the system overall. Calc: ((TP+TN))/((TP+FP+TN+FN)). Very susceptible to class imbalance. |
| **Precision (PPV)**                 | Also called positive predictive value (PPV), it is the percentage of true positive results among all results that the system flagged as positive. Calc: TP/((TP+FP)). A measure of the trustworthiness of positive results. |
| **Negative Predictive Value (NPV)** | The percentage of results that were true negatives (TN) among all results that the system flagged as negative. Calc: TN/((TN+FN)). A measure of the trustworthiness of negative results. |
| **Recall**                          | Also called sensitivity. The percentage of results flagged positive among all results that should have been obtained. Calc: TP/((TP+FN)). 100% recall means there are no false negative results – useful in confidently screening out negative cases. |
| **Specificity**                     | The percentage of results that were flagged negative among all negative results. Calc: TN/((TN+FP)). 100% specificity means there are no false-positive results and rules in the disease. |
| **F1-Score**                        | In this case, the unweighted harmonic mean of PPV/precision and sensitivity/recall. Calc: (2 × (Precision × Recall))/((Precision + Recall)). More resistant to the effects of class imbalance. |
| **Matthews’ Correlation Coefficient (MCC)** | A statistical measure used to evaluate the quality of binary classifications. Unlike other metrics, MCC considers all four categories of a confusion matrix. Calc: ((TP×TN) − (FP×FN))/√((TP+FP)(TP+FN)(TN+FP)(TN+FN)). The results are more abstract but highly resistant to the effects of class imbalance, which is why they are included in this study. |
| **Precision / Recall AUC**          | Represents the area under the Precision-Recall curve, which plots Precision against Recall at various threshold settings. It is more resistant to class imbalance than AUROC (area under the receiver operator curve). Both were used in testing, but the confusion matrix components form the primary outcomes for this study because they are more human-interpretable and harder to statistically manipulate. |
| **Demographic Parity (DP)**         | Demographic Parity, also known as Statistical Parity, requires that the probability of a positive prediction is the same across different demographic groups. Calc: DP = P(Ŷ=1∣A=a) = P(Ŷ=1∣A=b), where Ŷ is the predicted outcome and A represents the protected attribute (e.g., race, gender). This figure is given as an absolute difference where positive values suggest the more privileged group gains and negative values the reverse. The rule of 10% is used here to look for significant biases. |
| **Equal Opportunity (EO)**          | Equal Opportunity focuses on equalising the true positive rates across groups. Among those who truly belong to the positive class, the model should predict positive outcomes at equal rates across different groups. Calc: EO = P(Ŷ=1∣Y=1, A=a) = P(Ŷ=1∣Y=1, A=b), where Ŷ is the predicted outcome and Y is the actual outcome. If there is no bias, then the value will be equal for all groups. A higher value indicates a bias against the group considered more vulnerable. The rule of 10% is used here to look for significant biases. |
| **Disparate Impact (DI)**           | Divides the protected group’s positive prediction rate by that of the most-favoured group. If the ratio is below 0.8 or above 1.25, disparate impact is considered present. Calc: DI = P(Ŷ=1∣A=unfavoured) / P(Ŷ=1∣A=favoured). Values outside the 0.8–1.25 range suggest bias. |
| **Time / Energy / CO₂ Emissions**   | Measured in minutes and total energy consumption in kilowatt-hours (kWh), which can then be extrapolated to CO₂ emissions via a conversion factor set at 0.20705 Kg CO₂e per kWh for this study. |

## Model Card Authors

Matt Stammers - Computational Gastroenterologist

## Model Card Contact

m.stammers@soton.ac.uk

## Legal

1. No guarantee is given of model performance in any production capacity whatsoever. 
2. These models should be used in full accordance with the EU AI Act - Regulation 2024/1689. 
3. These models are not CE marked medical devices and are suitable at this point only for research and development / experimentation at users own discretion. 
4. They can be improved but any improvements should be published openly and shared openly with the community. 
5. UHSFT and the author own the copyright and are choosing to share them freely under a CC BY-NC 4.0 Licence for the benefit of the wider research community but not for commercial organisations who are breaking copyright law and infringing upon NHS intellectual property if they try to sell/market these models for profit.