| --- |
| --- |
| language: en |
| license: mit |
| tags: |
| - text-classification |
| - occupational-health-safety |
| - bert |
| - distilbert |
| - severity-prediction |
| - workplace-incidents |
| - nlp |
| datasets: |
| - osha-incident-reports |
| metrics: |
| - f1 |
| - accuracy |
| - roc-auc |
| --- |
| |
| # OHS Severity Classifier |
|
|
| DistilBERT fine-tuned for workplace incident severity classification. |
| Developed as part of MSc Computer Science research at Northumbria University (2024). |
|
|
| **Research paper:** *Predicting Workplace Incident Severity Using BERT-Based NLP — |
| Probabilistic Classification and SHAP Analysis for UK OHS Decision Support* |
| arXiv: submission in progress — link will be added on publication |
|
|
| **Code repository:** github.com/stuartclark-ml/incident-severity-bert |
|
|
| --- |
|
|
| ## Model Description |
|
|
| This model classifies workplace incident descriptions into five severity |
| categories based on time-loss incapacitation, aligned with UK RIDDOR reporting |
| thresholds and HSE guidance frameworks. |
|
|
| The model takes a free-text incident narrative as input and outputs a |
| probability distribution across five severity classes. It is designed as a |
| **decision-support tool in a human-in-the-loop workflow** — not a standalone |
| classifier. The probability distribution and SHAP token attribution together |
| allow a safety professional to assess confidence and inspect the model's |
| reasoning before acting on a prediction. |
|
|
| --- |
|
|
| ## Severity Classification Framework |
|
|
| | Class | Label | Condition | UK OHS Alignment | |
| |---|---|---|---| |
| | 0 | None | 0 days lost | Below RIDDOR recording threshold | |
| | 1 | Minor | 1-2 days lost | Below mandatory internal recording threshold | |
| | 2 | Moderate | 3-7 days lost | Mandatory internal recording under RIDDOR | |
| | 3 | Severe | 8-28 days lost | Statutory reporting to HSE required | |
| | 4 | Major | 29+ days lost | Long-term incapacitation — triggers separate management and insurance obligations | |
|
|
| Time-loss days used as severity proxy. This metric is compatible with both |
| US OSHA and UK RIDDOR reporting frameworks — the primary justification for |
| using OSHA data to develop a UK-applicable tool. |
|
|
| --- |
|
|
| ## Intended Uses |
|
|
| **Appropriate uses:** |
| - Triage support for high-volume incident report queues in health and social |
| care settings |
| - Research into NLP-based severity prediction for occupational safety |
| - Demonstration of BERT-based classification with SHAP explainability in |
| safety-critical domains |
| - Educational exploration of model bias in imbalanced safety datasets |
|
|
| **Not appropriate for:** |
| - Standalone severity determination without human review |
| - UK RIDDOR reportability decisions — the model was not trained on RIDDOR data |
| - Industries outside health and social care — trained on H&C sector data only |
| - Definitive clinical or legal severity assessments |
|
|
| --- |
|
|
| ## Training Data |
|
|
| **Dataset:** US OSHA Incident Reports (2024 release) — Health and Care sector subset |
|
|
| **Size:** 181,586 records after preprocessing |
| - Full OSHA dataset: 751,770 records |
| - H&C sector subset: 193,512 records |
| - COVID-19 records removed: 10,016 (to prevent rare global event bias) |
| - Fatality records removed: 5 (statistically insignificant, severe class imbalance) |
|
|
| **Features used:** |
| - `new_nar_what_happened` — what happened during the incident |
| - `new_nar_before_incident` — what the employee was doing beforehand |
| - `new_nar_object_substance` — the object or substance causing harm |
| - `size` — organisational size (1=<20, 2=20-249, 3=250+ employees) |
|
|
| **Class distribution:** Heavily imbalanced — dominated by None class, reflecting |
| real-world incident distribution (Heinrich's Triangle). Weighted loss function |
| applied during training (up to 4.05x penalty for Minor class). |
|
|
| --- |
|
|
| ## Performance |
|
|
| ### Overall Metrics |
|
|
| | Metric | Random Forest (baseline) | DistilBERT | ModernBERT | |
| |---|---|---|---| |
| | Weighted Avg F1 | 0.55 | **0.58** | 0.58 | |
| | Macro ROC AUC | 0.77 | 0.77 | **0.78** | |
| | Overall Accuracy | 0.52 | **0.55** | 0.54 | |
| | Log Loss | 1.29 | **1.07** | 1.10 | |
|
|
| DistilBERT selected as the deployment model: equal F1 to ModernBERT, |
| highest accuracy, lowest log loss, and smaller model size. |
|
|
| ### Per-Class Metrics — DistilBERT |
|
|
| | Class | Precision | Recall | F1 | |
| |---|---|---|---| |
| | None | 0.94 | 0.71 | 0.81 | |
| | Minor | 0.16 | 0.14 | 0.15 | |
| | Moderate | 0.19 | 0.25 | 0.22 | |
| | Severe | 0.21 | 0.20 | 0.20 | |
| | Major | 0.37 | **0.60** | 0.46 | |
|
|
| **Key finding:** 0.60 recall on Major (highest-severity) incidents makes the |
| model useful for prioritisation — catching 60% of the most serious incidents |
| for urgent human review — despite poor performance on the middle classes. |
|
|
| --- |
|
|
| ## Critical Findings from SHAP Analysis |
|
|
| SHAP token attribution was applied to DistilBERT to inspect the model's |
| learned logic. This revealed two significant issues that any user of this |
| model must understand: |
|
|
| ### 1. Needlestick Blind Spot |
|
|
| **99.25% of needlestick injuries in the training dataset are labelled None.** |
|
|
| The model learned this spurious correlation and systematically classifies |
| needlestick injuries as None severity — regardless of their actual |
| consequences. In clinical practice needlestick injuries can result in |
| bloodborne pathogen exposure with serious long-term health consequences. |
|
|
| **Mitigation:** Any incident narrative containing needle-related terms |
| (`needle`, `needlestick`, `lancet`, `sharps`, `puncture`) should be |
| flagged for mandatory human review and not relied upon for model output. |
|
|
| ### 2. Organisational Size Bias |
|
|
| The `size` feature introduced a spurious correlation: small organisations |
| (size=1, <20 employees) are associated with None class at 50.8%. |
|
|
| **Mitigation:** Model predictions for incidents from small organisations |
| should be treated with additional caution. Future implementations should |
| consider removing the size feature unless a more balanced dataset is available. |
|
|
| ### Positive Finding — No Significant US Linguistic Bias |
|
|
| SHAP global feature importance analysis found no significant US-centric |
| linguistic bias in the top predictive features. The model relied on |
| universal medical and safety terminology rather than US-specific language, |
| supporting its tentative use as a UK proxy tool. However, regulatory and |
| operational differences between US and UK health and social care systems |
| mean direct comparison with UK standards still requires caution. |
|
|
| --- |
|
|
| ## Limitations |
|
|
| - **Jurisdiction:** Trained on US OSHA data. UK RIDDOR performance not |
| characterised. Time-loss metric provides a compatibility bridge but does |
| not eliminate cross-jurisdictional differences. |
| - **Industry:** Health and social care only. Performance on other sectors |
| is unknown and likely poor. |
| - **Minority classes:** Minor, Moderate, and Severe classes perform poorly |
| (F1 0.15-0.22). Narratives for these classes are textually indistinct — |
| a fundamental data limitation not resolvable by model architecture alone. |
| - **Correlation not causation:** SHAP values reflect learned correlations |
| in the training data. They provide interpretations, not causal explanations. |
| - **Static training data:** Model accuracy will degrade over time as incident |
| patterns and safety language evolve. Continuous retraining would be required |
| for production deployment. |
| - **Fatalities excluded:** Insufficient fatality cases in the dataset (5 records) |
| for reliable modelling. |
|
|
| --- |
|
|
| ## How to Use |
| ```python |
| from transformers import AutoTokenizer, AutoModelForSequenceClassification |
| import torch |
| |
| model_name = "stuSterfc/ohs-severity-classifier" |
| tokenizer = AutoTokenizer.from_pretrained(model_name) |
| model = AutoModelForSequenceClassification.from_pretrained(model_name) |
| |
| labels = ["None", "Minor", "Moderate", "Severe", "Major"] |
| |
| def predict_severity(narrative: str) -> dict: |
| inputs = tokenizer( |
| narrative, |
| return_tensors="pt", |
| truncation=True, |
| max_length=512, |
| padding=True |
| ) |
| |
| with torch.no_grad(): |
| outputs = model(**inputs) |
| |
| probs = torch.softmax(outputs.logits, dim=1).squeeze().tolist() |
| |
| return { |
| "probabilities": {label: round(prob, 4) |
| for label, prob in zip(labels, probs)}, |
| "predicted_class": labels[probs.index(max(probs))], |
| "confidence": round(max(probs), 4) |
| } |
| |
| # Example |
| result = predict_severity( |
| "Employee was assisting patient transfer from bed to wheelchair. " |
| "Felt sharp pain in lower back. Unable to complete shift." |
| ) |
| print(result) |
| |
| # IMPORTANT: Check for needlestick blind spot |
| needle_terms = ["needle", "needlestick", "lancet", "sharps", "puncture"] |
| if any(term in narrative.lower() for term in needle_terms): |
| print("WARNING: Needle-related incident detected. " |
| "Model has known blind spot — refer for human review.") |
| --- |
| |
| ## Input Format |
| |
| The model was trained on concatenated narratives using `[SEP]` tokens |
| as separators between the three OSHA narrative fields, with an |
| organisational size prefix: |
| ``` |
| Organisation size: {size}. [SEP] {what_happened} [SEP] {before_incident} [SEP] {object_substance} |
| ``` |
| |
| However, testing found that neither the `[SEP]` separators nor the |
| organisational size prefix had a significant impact on output |
| probabilities. A plain concatenated narrative performs equivalently: |
| ``` |
| {what_happened} {before_incident} {object_substance} |
| ``` |
| |
| For practical use, pass the incident narrative as a single string |
| without special formatting. Include as much detail as available — |
| the model performs better with richer narratives (average word count |
| for well-classified Major incidents was 58.6 words vs 17.8 for |
| poorly-classified incidents).- |
| |
| ## Training Details |
| |
| - **Base model:** `distilbert-base-cased` |
| - **Framework:** HuggingFace Transformers |
| - **Fine-tuning:** HuggingFace Trainer API |
| - **Hyperparameter optimisation:** Optuna |
| - **Class imbalance handling:** Weighted loss function (inverse class frequency) |
| - **Training data split:** Standard train/validation/test split |
| - **Hardware:** [GPU details if you have them] |
| |
| --- |
| |
| ## Citation |
| |
| *Citation will be added when arXiv preprint is published.* |
| |
| --- |
| |
| ## About the Author |
| |
| Stuart Clark — MSc Computer Science (Distinction), Northumbria University 2024. |
| NEBOSH National Diploma. 14 years occupational health and safety consulting. |
| |
| GitHub: github.com/stuartclark-ml |
| |
| [More Information Needed] |
