SimonYip22/bioclinicalbert-icu-entity-validator

BioClinicalBERT for ICU Clinical Entity Validation

Fine-tuned BioClinicalBERT sequence classifier for precision-oriented contextual validation of rule-extracted clinical entities in intensive care unit (ICU) notes.

This model is the transformer validation component of the Clinical Entity Extraction-Validation System, a hybrid NLP pipeline combining deterministic entity extraction with supervised contextual classification.

GitHub Repository | Zenodo Publication | Original BioClinicalBERT

Model availability: This Hugging Face repository provides research documentation only; fine-tuned model weights are not hosted here. The model is intended for research and is not clinically validated.

1. Model Overview

The model evaluates whether a candidate clinical entity extracted by a rule-based system is valid within its surrounding sentence context.

It supports three entity-validation tasks:

Entity Type	Validation Task	Classification Objective
SYMPTOM	symptom_presence	Determine whether a symptom is currently present rather than absent, negated or historical
INTERVENTION	intervention_performed	Determine whether an intervention was performed or is actively in use rather than merely planned or hypothetical
CLINICAL_CONDITION	clinical_condition_active	Determine whether a clinical condition is active and relevant rather than historical, resolved, uncertain or ruled out

The model acts as a binary candidate validator, not a standalone named-entity recogniser. It does not identify new entity spans or independently extract concepts from unstructured text.

Within the complete system, it follows a high-recall rule-based extraction stage and applies contextual filtering to improve the precision of retained entities.

2. Model Details

Property	Specification
Base Model	BioClinicalBERT
Architecture	BertForSequenceClassification
Framework	PyTorch / Hugging Face Transformers
Task	Binary sequence classification
Encoder	BERT-base architecture: 12 layers, 12 attention heads, 768 hidden dimensions
Tokenizer	BioClinicalBERT WordPiece tokenizer
Maximum Sequence Length	512 tokens
Output	Two classification logits
Decision Threshold	0.549, applied externally

The encoder and task-specific classification head were fine-tuned for contextual validity classification.

Output labels:

• 0 — Invalid candidate
• 1 — Valid candidate

Softmax converts the model's logits into class probabilities. The final pipeline accepts a candidate when its predicted probability for the valid class is at least 0.549.

This threshold is a separate inference policy, not a learned model parameter.

3. Input Representation

Each candidate is serialised into a structured text input containing six fields:

[SECTION] {section} [ENTITY TYPE] {entity_type} [ENTITY] {entity_text} [CONCEPT] {concept} [TASK] {task} [TEXT] {sentence_text}

The fields provide document-section context, the extracted candidate span, its normalised concept, the entity category, the validation objective and the surrounding sentence.

Fields derived from ground-truth labels or downstream predictions are excluded from the model input to avoid label leakage.

The [SECTION], [ENTITY TYPE], [ENTITY], [CONCEPT], [TASK] and [TEXT] markers are textual field delimiters in the input representation, not newly added tokenizer special tokens.

4. Training Data

The model was fine-tuned using 1,200 manually annotated candidate entities derived from a filtered MIMIC-III v1.4 ICU clinical-note corpus.

The candidate dataset was constructed using the upstream rule-based extraction pipeline and balanced across the three entity categories.

Entity Type	Annotated Examples
SYMPTOM	400
INTERVENTION	400
CLINICAL_CONDITION	400
Total	1,200

Each example received a manually assigned binary is_valid label according to task-specific annotation rules addressing clinical presence, temporality, certainty and intervention status.

Dataset Splitting

The labelled dataset was partitioned using an 85/15 stratified train/test split.

Split	Examples	Purpose
Training	1,020	Model selection, cross-validation, threshold optimisation and final fine-tuning
Held-out test	180	Final evaluation

Stratification preserved the distribution of entity-validation tasks and binary labels.

The test set was excluded from model selection and threshold optimisation.

Data source: MIMIC-III Clinical Database v1.4, PhysioNet.

Original clinical notes and manually annotated training examples are not included in this Hugging Face repository.

5. Fine-Tuning Procedure

The model was fine-tuned using the Hugging Face Trainer API.

Two training configurations were compared using stratified five-fold cross-validation on the 1,020-example training set. The final configuration was selected based on cross-validation performance and subsequently retrained on the full training split.

Final Training Configuration

Hyperparameter	Value
Learning Rate	3e-6
Epochs	5
Per-Device Batch Size	8
Gradient Accumulation	2
Effective Batch Size	16
Weight Decay	0.05
Warmup Ratio	0.1
Gradient Clipping	1.0
Maximum Sequence Length	512
Random Seed	42

Decision Threshold Optimisation

The decision threshold was selected using out-of-fold predicted probabilities from stratified five-fold cross-validation.

The optimisation objective was to maximise precision subject to a minimum recall constraint relative to the model's default 0.5 operating threshold.

The selected threshold was 0.549, reflecting the system's precision-first objective.

The final classifier was trained on all 1,020 training examples and evaluated on the separate 180-example held-out test set.

6. Evaluation Results

Performance was evaluated on 180 held-out, manually annotated candidate entities.

The evaluation compares the original rule-based candidate decisions against the full pipeline after BioClinicalBERT validation at the selected threshold of 0.549.

Metric	Rule-Based Baseline	BioClinicalBERT Validation
Accuracy	0.628	0.750
Precision	0.571	0.833
Recall	0.989	0.618
F1-Score	0.724	0.710
False Positives	66	11
False Negatives	1	34

Interpretation

BioClinicalBERT validation increased precision from 0.571 to 0.833 and reduced false positives from 66 to 11, a reduction of approximately 83.3%.

However, recall decreased from 0.989 to 0.618, and F1-score decreased slightly from 0.724 to 0.710.

The model therefore achieved the intended precision improvement while sacrificing substantial candidate coverage.

Entity-stratified analysis indicated greater benefit for INTERVENTION and CLINICAL_CONDITION candidates, while validation of SYMPTOM candidates could over-filter examples already handled effectively by deterministic rules.

These results are specific to the project's internal, candidate-level evaluation set and should not be interpreted as general clinical NER performance or evidence of clinical effectiveness.

7. Usage

This Hugging Face repository documents the fine-tuned model but does not host its checkpoint weights.

The following example demonstrates the inference interface using a locally available copy of the project's trained model and tokenizer.

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

checkpoint = "models/bioclinicalbert_final"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
model.eval()

# Synthetic example
text = (
    "[SECTION] assessment "
    "[ENTITY TYPE] CLINICAL_CONDITION "
    "[ENTITY] pneumonia "
    "[CONCEPT] infection "
    "[TASK] clinical_condition_active "
    "[TEXT] Possible pneumonia; further investigation is planned."
)

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    padding="max_length",
    max_length=512
)

with torch.inference_mode():
    logits = model(**inputs).logits
    probability = torch.softmax(logits, dim=-1)[0, 1].item()

is_valid = probability >= 0.549

print(f"Valid-class probability: {probability:.4f}")
print(f"Accepted candidate: {is_valid}")

The example uses synthetic clinical language for illustration. No particular prediction is assumed.

In the complete pipeline, the model receives candidates generated by deterministic extraction rules and returns contextual validity scores used to filter the final structured entity outputs.

For the full extraction, validation and deployment workflow, refer to the GitHub repository.

8. Intended Use and Limitations

Intended Use

This model was developed for research into hybrid clinical NLP, contextual entity validation and precision-oriented extraction of structured information from ICU clinical documentation.

Limitations

• Restricted task scope: The model validates existing candidates and cannot discover entities missed by the upstream extraction layer.
• Limited annotated data: Fine-tuning used 1,200 manually labelled examples, with possible sensitivity to sampling and annotation decisions.
• Context limitations: Sentence-level context may be insufficient for complex clinical judgements requiring wider document or longitudinal information.
• Precision–recall trade-off: The selected threshold substantially improves precision but also rejects valid candidates.
• Entity-specific performance: A single global threshold does not perform equally well across all three entity categories.
• Limited generalisability: No external institutional or prospective validation was performed.
• Annotation uncertainty: No formal multi-annotator agreement study was conducted.

Clinical Use

This model is not a medical device or a validated clinical decision-support system. It must not be used independently for diagnosis, treatment recommendations, patient management or other clinical decisions.

9. Model Availability

This repository contains the Model Card only. The fine-tuned checkpoint, tokenizer artifacts and MIMIC-derived training examples are not uploaded to this Hugging Face repository.

The full system implementation, experimental methodology and deployment documentation are maintained in the associated GitHub project.

MIMIC-III is a credentialed-access dataset distributed through PhysioNet. Model and data redistribution are subject to the applicable source-data terms.

10. Citation and References

Project publication

Yip, S. (2026). Clinical Entity Extraction-Validation System (Version 1.0.0). Zenodo. https://doi.org/10.5281/zenodo.20018309

Base model

Alsentzer, E., et al. (2019). Publicly Available Clinical BERT Embeddings. ClinicalNLP Workshop, NAACL. https://doi.org/10.48550/arXiv.1904.03323

Source dataset

Johnson, A., Pollard, T., & Mark, R. (2016). MIMIC-III Clinical Database (version 1.4). PhysioNet. https://doi.org/10.13026/C2XW26

Johnson, A. E. W., et al. (2016). MIMIC-III, a freely accessible critical care database. Scientific Data, 3, 160035. https://doi.org/10.1038/sdata.2016.35

---

Developed by: Simon Yip
Project: Clinical Entity Extraction-Validation System

---