vadimbelsky/bert-esi-triage-v57

BERT-ESI-Triage v57 — BiomedBERT multi-head triage classifier

TL;DR

bert-esi-triage-v57 is a fine-tuned BiomedBERT classifier for the Emergency Severity Index (ESI 1-5) triage decision. It is the first model in this line to meet ESI 1 safety recall ≥75% on every primary eval slice (MIMIC, MC-MED, MIETIC, Lukina), with a mean ESI 1 recall of 87.5% across 18 eval slices.

• Model type: BiomedBERT-base + 11-d engineered-feature fusion (MEWS / qSOFA / shock_index / etc.) → esi_head Linear(768+64, 5)
• Auxiliary heads (training only): symptom, flag, pain, arrival, gestalt, disposition, resource, vitals, etc. — 20+ heads supervise the encoder; only esi_head is needed for inference.
• Input: ED triage text (compact CC, telegraphic, or narrative), ≤ 512 BERT tokens.
• Output: ESI 1-5 prediction (1 = most acute, 5 = least).
• Calibration recipe (validated production): demographic normalizer → 6-way sub-dialect detection → per-dialect temperature scaling (Guo et al. 2017) → confidence-aware ESI 1 logit bias → optional engine ensemble min(BERT, engine).
• License: MIT (model weights). Training data is private (MIMIC-IV-ED
  • Stanford MC-MED + MIETIC + Lukina v3 + curated synth).

⚠️ Not a medical device. Not for clinical use. This is a research artifact distributed for reproducibility and benchmarking. Real deployment requires institutional validation, ED nurse oversight, and the deterministic engine safety floor (see Recommended inference pipeline below).

How to use

import torch, torch.nn as nn
from transformers import AutoTokenizer, AutoModel
from huggingface_hub import hf_hub_download

ENCODER = "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext"

class V57BertESI(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = AutoModel.from_pretrained(ENCODER)
        self.feature_proj = nn.Sequential(
            nn.Linear(11, 64), nn.GELU(), nn.LayerNorm(64),
        )
        self.esi_head = nn.Linear(768 + 64, 5)

    def forward(self, input_ids, attention_mask, eng_features):
        cls = self.encoder(input_ids=input_ids,
                           attention_mask=attention_mask
                           ).last_hidden_state[:, 0]
        return self.esi_head(torch.cat([cls, self.feature_proj(eng_features)], -1))

tok = AutoTokenizer.from_pretrained(ENCODER)
model = V57BertESI()
ckpt = torch.load(hf_hub_download("vadimbelsky/bert-esi-triage-v57", "model.pt"),
                  map_location="cpu", weights_only=False)
# Many auxiliary heads in the checkpoint are not needed for ESI inference.
sd = model.state_dict()
model.load_state_dict({k: v for k, v in ckpt.items()
                       if k in sd and sd[k].shape == v.shape},
                      strict=False)
model.eval()

text = "67yo M ambulance. CC: Chest pain, Diaphoresis. BP 95/60 HR 120 RR 22 SpO2 94%."
enc = tok(text, return_tensors="pt", truncation=True, max_length=512)
# Engineered features: 11-d z-scored vector (MEWS/qSOFA/shock_index/etc.).
# See the Space app.py compute_engineered_features() for the canonical impl.
feats = torch.zeros(1, 11)   # zero-vector = "no structured fields parsed"
logits = model(enc["input_ids"], enc["attention_mask"], feats)
esi = int(logits.argmax(-1)) + 1   # 1-5
print(f"Predicted ESI {esi}")

For the full validated inference stack (race/SES normalizer, 6-way dialect detection, per-dialect temperature + bias, engine ensemble safety floor), use the Hugging Face Space app.py as the reference implementation: https://huggingface.co/spaces/vadimbelsky/esi-triage-demo

Recommended inference pipeline

Single-model BERT output is not the production recipe. Documented v57 deployment recipe (see Space app.py for the canonical implementation):

1. Demographic normalizer — strip race/SES tokens to prevent bias leakage. Maps (white|black|hispanic|...) → [demographic_redacted] and (homeless|low-income|medicaid|...) → [social_redacted].
2. 6-way sub-dialect detection — mimic_compact, mcmed_telegraphic, narrative_lukina, narrative_mietic, narrative_general, unknown_compact. See sub_dialect_detector_v2 (regex-based, no extra deps).
3. Per-dialect temperature scaling — Guo et al. 2017. T values: mimic_compact=1.6369, mcmed_telegraphic=1.6577, others 1.0. Argmax-preserving by construction; smooths Expected Calibration Error.
4. Confidence-aware per-dialect ESI 1 logit bias — bias is added to the ESI 1 logit only when the top-2 calibrated logit margin < 1.5. This protects high-confidence predictions from over-correction. Bias values: {mimic_compact: 0.5, mcmed_telegraphic: 1.0, narrative_lukina: 1.0, narrative_mietic: 0.25, narrative_general: 0.25, unknown_compact: 1.0}.
5. Engine ensemble safety floor: final_esi = min(bert_cal, engine), where engine is a deterministic handbook v4 implementation (Step A/B/D triggers). Catches the ~28pp of MIMIC ESI 1 cardiac-arrest cases that BERT alone routes to ESI 5. Lightweight regex implementation bundled with the Space as engine_ensemble.py.

Validated performance (v57 epoch 3, 18-eval suite, n=10,553)

Primary slices

Eval	n	cal_exact	cal_ESI 1 R	ens_ESI 1 R
MIMIC-IV-ED holdout	7,917	59.8%	76.6%	78.1%
MC-MED Stanford clean	1,000	58.6%	77.0%	77.0%
MIETIC narrative	200	84.0%	90.0%	96.7%
Lukina v3	201	44.8%	94.3%	94.3%

Condition-specific slices

Eval	n	cal_ESI 1 R	ens_ESI 1 R
Sepsis	93	95.0%	97.5%
Stroke	97	95.0%	95.0%
Anaphylaxis	90	97.1%	97.1%
Cardiac arrest	103	93.3%	96.7%
OB emergency	98	95.0%	95.0%
Pediatric n=200 v2	200	95.0%	95.0%
Judgment gap v1	177	88.9%	94.4%

Subgroup slices

Eval	n	cal_exact	cal_ESI 1 R
Geriatric n=200	200	61.5%	78.8%
Polypharmacy n=200	200	67.0%	72.5%
Frequent flyer n=200	135	61.5%	80.0%
Vital completeness	231	56.3%	83.3%
Multi-CC complexity	304	58.2%	76.2%
Concept density	260	59.6%	80.0%

Mean ESI 1 recall across 18 slices: 87.5%.

Head-to-head vs v56 epoch 3

• ESI 1 recall wins: 16 of 18 slices (1 loss = polypharmacy −1.2pp; 1 neutral)
• Mean Δ ESI 1 R: +14.6pp, median +11.2pp, max +34.3pp (Lukina)
• Mean Δ ESI exact: −0.3pp (essentially neutral)

Top 5 ESI 1 recall improvements:

1. Lukina v3 +34.3pp (60.0% → 94.3%) — v57 Lukina synth + dialect bucket + bias
2. MC-MED +33.0pp (44.0% → 77.0%) — telegraphic dialect lift
3. MIETIC +30.0pp (60.0% → 90.0%) — narrative lift
4. OB emergency +30.0pp (65.0% → 95.0%)
5. Frequent flyer +20.0pp — counters documented downtriage bias

Trade-off honesty

Per-dialect bias optimizes for the ESI 1 safety floor at a real cost to ESI 2 precision on some slices:

• Lukina cal_exact −10.9pp vs raw (bias pulls ESI 2s up to ESI 1)
• ESI 5-consolidated cal_exact −6.1pp (structural cost of ESI 1 bias)
• Pediatric cal_exact −5.0pp (but ESI 1 R +5pp)
• Condition slices: heavy ESI 2 → ESI 1 leakage from +1.0 bias

This is clinically defensible — missing a critical patient is worse than over-triaging from ESI 2 to ESI 1 — but the model should be deployed as a safety-first decision support tool, not as "87% accurate triage AI." Probabilities are smoothed by per-dialect temperature scaling for better ECE; treat them as ordering, not ground truth.

Training data

• MIMIC-IV-ED (Beth Israel Deaconess, Boston) — bulk + nurse-assigned ground-truth ESI labels + pyxis medication ground truth. Compact CC dialect, ~5% ESI 1 prevalence.
• MC-MED clean (Stanford ED) — telegraphic dialect, ICD-inferred resources where pyxis is empty.
• MIETIC — narrative paraphrase dialect, sentence-level mix.
• Lukina v3 — Russian-physician translation style; eval-only.
• Curated medgemma-grounded synth — sparse-concept and sparse-dialect coverage; ESI labels inherited from real parent records (LLM never decides the label). Capped at ≤5% of total corpus.

Total: ~400K records after BERT 512-token filtering and eval-leakage guard. ER-REASON (discharge summaries) was retained per "don't drop narratives" directive but removed from the dedicated ER-REASON eval slice (most exceed 512 tokens).

Architecture detail

• Encoder: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext (12-layer, 768-hidden)
• Engineered-feature fusion (v53.3 Phase 4): 11-d z-scored features → feature_proj = Linear(11, 64) → GELU → LayerNorm(64) → concat with [CLS] (768-d) → esi_head: Linear(832, 5).
• 11 engineered features: shock_index, MEWS proxy, qSOFA-like, critical_vital_count, abnormal_vital_count, arrival_acuity_prior, pain_bucket, age_lifecycle, comorbidity_burden, cc_complexity, concept_density.
• Auxiliary heads (training only): 20+ heads supervise the encoder on side tasks (symptom labels, flags, vitals reconstruction, etc.). All heads except esi_head (+ feature_proj) are removed at inference.

Calibration theory

Under asymmetric cost (missing ESI 1 ≫ over-triage by one tier), the Bayes-optimal decision boundary shifts away from p=0.5. Per Guo et al. 2017, the calibrated logit shift is bias = log(C_FN / C_FP). The production +1.0 bias on compact-CC dialects implies a cost ratio of ~2.7×, consistent with ED clinical literature (which generally rates under-triage at 10-50× the cost of one-tier over-triage; the +1.0 value is empirically tuned to the conservative end of that range).

Sub-dialect–specific calibration is required because the prior prevalence of ESI 1 varies materially across input formats (~5% in compact CC vs ~17% in narrative).

Limitations & known gaps

• ⚠️ Research demo only. Not for clinical use. Not a medical device.
• Trained primarily on MIMIC-IV-ED (Boston ED, English only). Geographic and demographic generalization is unverified.
• Pediatric data is sparse in training (~0.002% of native records; filled with curated synth). Pediatric vital interpretation is not age-bucketed in v57 (planned in v58).
• Single-rater labels — no formal kappa validation across raters.
• MC-MED ESI 5 recall weak (25%); Lukina ESI 5 recall weak (25%).
• Lukina exact 44.8% — narrative dialect ESI 2-5 boundaries remain fuzzy after v57 calibration trades exact for ESI 1 safety.
• Probabilities are NOT clinical truth — temperature scaling smooths ECE but the absolute values should not be interpreted as risk scores.
• ER-REASON discharge summaries exceed BERT's 512-token window; the model sees only the truncated first ~512 tokens (where the CC sits).
• Synth records are capped at ≤5% of total corpus to bound LLM-induced drift.

Citation

@misc{esi_triage_v57_2026,
  title  = {ESI Triage v57 — BiomedBERT multi-head decision-support
            classifier with per-dialect calibration and deterministic
            engine ensemble safety floor},
  author = {Belski, Vadim},
  year   = {2026},
  url    = {https://huggingface.co/vadimbelsky/bert-esi-triage-v57},
  note   = {Validated on 18-eval suite: MIMIC-IV-ED holdout, MC-MED
            Stanford clean, MIETIC narrative clean, Lukina v3,
            7 condition-specific slices, 6 subgroup slices. v57 epoch 3.
            First model in this line meeting ESI 1 recall ≥75% on every
            primary eval slice.}
}

Related artifacts

• Space (live demo): https://huggingface.co/spaces/vadimbelsky/esi-triage-demo
• Engine ensemble: engine_ensemble.py in the Space repo — self-contained handbook v4 implementation; bundled with the Space.
• Predecessor: v49 (still referenced in some downstream pipelines; retired in favor of v57 for new deployments).
• Successor: v58 (training in progress as of 2026-06-01; targets six corpus fixes for tightened ESI 2 / ESI 5 boundaries).