Spaces:

ayodeji21
/

clinical-nlp-api

Running

App Files Files Community

clinical-nlp-api / README.md

Ayodeji Akande

Fix cropped dashboard screenshots and broken README CI badge

184d5a3 15 days ago

preview code

Raw

History Blame Contribute Delete

9.1 kB

metadata

title: Clinical Nlp Api
emoji: 🌍
colorFrom: green
colorTo: pink
sdk: docker
pinned: false

Clinical NLP Pipeline

NLP · Named Entity Recognition · Clinical Text Mining · BERT Fine-tuning

A production-grade pipeline that extracts structured clinical knowledge from unstructured medical notes using state-of-the-art biomedical NLP models.

What it does

Component	What it does
NER	Extracts diagnoses, medications, procedures, symptoms, and anatomical terms using scispaCy (`en_core_sci_lg`)
ICD-10 mapping	Maps extracted entities to ICD-10-CM codes via exact → fuzzy → embedding matching
Severity classifier	Fine-tunes Bio_ClinicalBERT to classify notes as `routine`, `urgent`, or `critical`
Co-occurrence graph	Builds an interactive network of entity pairs that appear together in clinical notes
FastAPI	REST API serving all NLP functionality
Streamlit demo	Live public demo — paste any clinical note, get results in real time

Architecture

flowchart LR
    subgraph ETL["ETL (src/etl)"]
        Extract[extract.py] --> Transform[transform.py] --> Load[load.py]
    end

    MTSamples[(MTSamples CSV)] --> Extract

    Load --> DB[(PostgreSQL<br/>Supabase)]

    subgraph NLP["NLP (src/nlp)"]
        NER[ner.py<br/>scispaCy hybrid NER]
        ICD[icd_mapper.py<br/>exact → fuzzy → embedding]
        Classifier[classifier.py<br/>Bio_ClinicalBERT fine-tune]
        Cooc[cooccurrence.py<br/>entity co-occurrence graph]
    end

    DB <--> NER
    DB <--> ICD
    DB <--> Classifier
    DB <--> Cooc

    subgraph API["FastAPI (src/api)"]
        Routes["routes: notes, entities, icd, model"]
    end

    NLP <--> Routes
    DB <--> Routes

    subgraph Dashboard["Streamlit (dashboard/)"]
        Demo[demo.py]
        Explorer[explorer.py]
        Metrics[model_metrics.py]
    end

    Routes -- HTTP --> Dashboard

    Routes -.deployed on.-> HFSpace["Hugging Face Spaces<br/>(Docker)"]
    Dashboard -.deployed on.-> StreamlitCloud["Streamlit Cloud"]
    Classifier -.checkpoint hosted on.-> HFHub["Hugging Face Hub<br/>model repo"]

Batch scripts (scripts/run_ner_batch.py, run_icd10_batch.py, train_severity_classifier.py) populate the NLP layer against notes already sitting in the database — see Batch processing below.

Screenshots

Live Demo	Model Metrics

Co-occurrence network — entities that appear together across the corpus, node size scaled by degree:

Entity frequency — most common extracted entities across the dataset:

Live demo

🔗 clinical-nlp-pipeline.streamlit.app

Want to deploy your own copy instead? See VSCODE_GUIDE.md.

Quick start

git clone https://github.com/ayodeji07/clinical-nlp-pipeline
cd clinical-nlp-pipeline
python -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt

# Install the scispaCy NER model
pip install https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.3/en_core_sci_lg-0.5.3.tar.gz

# Download MTSamples from Kaggle → data/raw/mtsamples.csv
# Then run the pipeline
python -m src.etl.pipeline --dry-run
uvicorn src.api.main:app --reload &
streamlit run dashboard/app.py

Full step-by-step instructions: VSCODE_GUIDE.md

Batch processing

Once the ETL pipeline has loaded notes into the database, these scripts populate entities, ICD-10 mappings, and the severity classifier against stored data. All three are idempotent and resumable — safe to interrupt and re-run without creating duplicates:

python scripts/run_ner_batch.py                            # extract entities from stored notes
python scripts/run_icd10_batch.py                          # map DISEASE/SYMPTOM entities to ICD-10 codes
python scripts/train_severity_classifier.py --n-seeds 4    # fine-tune the severity classifier

Each accepts --limit N for a quick subset run. run_icd10_batch.py also maintains an on-disk cache (data/processed/icd10_mapping_cache.json) so a restart skips re-attempting entities already checked, matched or not. train_severity_classifier.py --n-seeds N trains N different random seeds and keeps only the one with the best critical-class F1 — small fine-tuning runs are sensitive to initialisation, so a single unseeded run isn't reliably comparable across retrains (default is 1, i.e. a single run, if omitted).

Tech stack

Layer	Library
NER	spaCy + scispaCy `en_core_sci_lg`
Classification	HuggingFace Transformers + `Bio_ClinicalBERT`
ICD-10 fuzzy	rapidfuzz
ICD-10 embeddings	sentence-transformers
API	FastAPI + Pydantic
Database	SQLAlchemy (SQLite local / PostgreSQL cloud)
Dashboard	Streamlit
Visualisation	Plotly + NetworkX + pyvis
Testing	pytest (65 tests)
CI/CD	GitHub Actions

Project structure

src/
  utils/       config, logger, text cleaning utilities
  etl/         extract → transform → load pipeline
  nlp/         ner, icd_mapper, classifier, cooccurrence
  db/           connection, ORM models, repository layer
  api/          FastAPI app, Pydantic schemas, route handlers
dashboard/
  app.py        Streamlit entry point
  api_client.py typed HTTP client for the API
  pages/        demo, explorer, model_metrics
notebooks/
  00_data_exploration.ipynb
  01_ner_walkthrough.ipynb
  02_icd_mapping.ipynb
  03_classification.ipynb   ← fine-tune Bio_ClinicalBERT
  04_visualisation.ipynb
tests/          pytest test suite — 65 tests, 0 dependencies on GPU
sql/            schema.sql for Supabase migration

Dataset

MTSamples — 4,999 de-identified medical transcriptions across 40 specialties. Download free from Kaggle.

MIMIC-III discharge summaries are optionally supported (requires PhysioNet credentialing).

Severity labels

MTSamples has no severity labels, so we derive them using weak supervision:

Label	Signal
`critical`	ICU, ventilator, cardiac arrest, stroke, respiratory failure
`urgent`	Emergency, acute, admitted, infection, chest pain, unstable
`routine`	Elective, outpatient, follow-up, stable, screening

The classifier learns to generalise beyond these keyword rules.

Current performance (Bio_ClinicalBERT, class-weighted loss, retrained 2026-07-16, best of 4 random seeds selected by critical-class F1): ~72% overall test accuracy/F1. Per-class breakdown on critical — the safety-relevant class — is what matters most here: precision 0.604, recall 0.829, F1 0.699 (up from an unweighted-loss baseline of recall 0.486, F1 0.515). The loss is deliberately weighted to favour catching critical cases over aggregate accuracy, since missing a true critical note is far costlier than a false positive.

Fine-tuning a 3-class head on top of a pretrained model with a small dataset is sensitive to random initialisation — an unseeded single run can land anywhere from critical F1 0.54 to 0.70. train_severity_classifier.py trains several seeds and keeps the best rather than trusting one arbitrary run; see --n-seeds in Batch processing above. Full metrics (including a confusion matrix and per-epoch history) are recorded in the model_runs table and served at GET /model/metrics — the dashboard's Model Metrics page reads from there, not a local file.

Deployment

See VSCODE_GUIDE.md — Phase 8 covers:

Supabase (free PostgreSQL for the database — use the pooler connection string, not the direct one; the direct host is IPv6-only and unreachable from several free-tier hosts)
Hugging Face Spaces (free API hosting — has enough RAM for the full hybrid NER + classifier pipeline; Railway/Render are lighter-weight alternatives but Render's free tier specifically doesn't have enough memory for this project's models)
Streamlit Cloud (free dashboard hosting)

Total cloud cost for a portfolio demo: £0/month.

Running tests

pytest                              # all 65 tests
pytest --cov=src                    # with coverage
pytest tests/test_ner.py -v         # one file

Author

Built by Ayodeji as part of a HealthTech Data Engineering portfolio.