VTE-ModernBERT-DeID

VTE-ModernBERT-DeID is a BioClinical ModernBERT-based sequence-classification model for identifying venous thromboembolism (VTE) phenotypes in English clinical narratives. It was fine-tuned on de-identified clinical text and evaluated on the corresponding original clinical text.

Important: “DeID” indicates that de-identified text was used for model fine-tuning. This model classifies VTE phenotypes; it does not perform de-identification.

This checkpoint uses the same underlying annotated dataset, label definitions, and train/test split described for VTE-BERT. The original publication describes the study cohort, clinical annotation process, label development, and validation design.

Model details

• Developed by: Omid Jafari
• Research affiliation and acknowledgement: Ang Li Lab
• Base model: BioClinical ModernBERT large
• Architecture: ModernBertForSequenceClassification
• Task: Single-label text classification
• Language: English
• Architecture context capacity: 8,192 tokens
• Fine-tuning sequence length: 1,024 tokens
• Number of labels: 6
• Release status: Public research release

Although the underlying architecture supports sequences of up to 8,192 tokens, this checkpoint was fine-tuned with a maximum sequence length of 1,024 tokens.

Labels

ID	Label
0	LE-DVT+
1	PE+
2	UE-DVT
3	atypical
4	history
5	none

Operational definitions for these labels and the annotation procedures are provided in the VTE-BERT publication.

Data preparation and study design

The underlying clinical data and predefined train/test split are the same as those used in the original VTE-BERT study.

• Fine-tuning inputs: Clinical text de-identified with Health-DeID and then prepared for modeling with NLPMed-Engine.
• Test inputs: The corresponding original, non-de-identified clinical text from the held-out test set, prepared with the same NLPMed-Engine preprocessing workflow.
• Distributed data: No training or evaluation data is included in this repository.

Fine-tuning

Parameter	Value
Training text field	final_deidentified_text
Maximum training sequence length	1,024
Architecture context capacity	8,192
Learning rate	2e-5
Weight decay	0.01
Maximum training epochs	15
Early-stopping patience	20
Training batch size	8
Evaluation batch size	8
Gradient accumulation steps	1
Model-selection metric	eval_pos_f1
Mixed-precision training	bfloat16

The checkpoint was exported and tested with Transformers 5.10.1.

Evaluation

Numerical evaluation results are not included in this model card at this time.

Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer, pipeline

model_name = "omid-jf/VTE-ModernBERT-DeID"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

classifier = pipeline(
    "text-classification",
    model=model,
    tokenizer=tokenizer,
)

result = classifier(
    "Your clinical text here.",
    truncation=True,
    max_length=1024,
)

print(result)

For an end-to-end preprocessing and inference workflow, use NLPMed-Engine. Its MLInference component can load Hugging Face sequence-classification checkpoints such as this model.

Intended use and limitations

This model is intended for research involving retrospective VTE phenotyping in clinical narratives, including use cases that may benefit from more context than conventional 512-token BERT models.

It has not been validated for diagnosis, treatment selection, autonomous patient screening, or other clinical decision-making. Performance may vary across institutions, patient populations, note types, documentation practices, preprocessing configurations, and clinical settings. Users are responsible for obtaining appropriate institutional approvals and implementing applicable privacy and security controls.

License

This model is licensed under the GNU Affero General Public License v3.0. The research-use and clinical-safety statements in this model card describe the intended use and known limitations; they do not replace or add restrictions to the AGPLv3 license.

Related resources

• Original gated model: VTE-BERT on Hugging Face
• VTE-BERT publication: Development and Validation of VTE-BERT Natural Language Processing Model for Venous Thromboembolism
• Preprocessing and inference: NLPMed-Engine
• De-identification pipeline: Health-DeID

Citation

For the dataset, annotation process, label definitions, and original VTE-BERT study, please cite:

@article{jafaridevelopment,
  title={Development and Validation of VTE-BERT Natural Language Processing Model for Venous Thromboembolism},
  author={Jafari, Omid and Ma, Shengling and Lam, Barbara D and Jiang, Jun Y and Zhou, Emily and Ranjan, Mrinal and Ryu, Justine and Bandyo, Raka and Maghsoudi, Arash and Peng, Bo and others},
  journal={Journal of Thrombosis and Haemostasis},
  publisher={Elsevier},
  year={2025},
  doi={10.1016/j.jtha.2025.07.021}
}