| --- |
| language: |
| - en |
| license: cc-by-nc-nd-4.0 |
| library_name: pytorch |
| tags: |
| - transformer |
| - character-level |
| - medical-coding |
| - icd-10-cm |
| - healthcare |
| - gpt |
| - nanogpt |
| - visualization |
| - sequence-modeling |
| datasets: |
| - MIMIC-IV |
| - NHDS |
| - NHCS |
| metrics: |
| - perplexity |
| pipeline_tag: text-generation |
| --- |
| |
| # DgPg: Diagnosis Progression Transformer |
|
|
| **DgPg** ("**D**ia**g**nosis **P**ro**g**ression") is a compact (~2.5 MB) character-level, GPT-style transformer that learns representative *ordering patterns* of ICD-10-CM diagnostic (DX) codes within single-visit medical records. Given a "seed" diagnosis code, DgPg generates the codes most likely to precede or follow it, which can be rendered as graph-based visual summaries of an organization's coding practices. |
|
|
| DgPg was introduced in the paper [*Visualizing Medical Coding Practices Using Transformer Models*](https://doi.org/10.5220/0013257800003905) (Hobson & Huang, ICPRAM 2025), from the Department of Electrical Engineering and Computer Science at the University of Tennessee, Knoxville. |
|
|
| > ⚠️ **DgPg is an organizational-analytics tool, not a clinical or coding-assistance model.** It is not intended to improve medical coding, predict diagnoses, or support clinical decisions. Its purpose is to let hospital administrators, lead physicians, and other stakeholders *see* how coding is actually practiced in a given corpus of records. |
|
|
| ## Model Details |
|
|
| | | | |
| |---|---| |
| | **Model type** | Decoder-only transformer (modified [nanoGPT](https://github.com/karpathy/nanoGPT)), character-level | |
| | **Task** | Next-character prediction over 4-character-normalized ICD-10-CM code sequences | |
| | **Parameters** | 1.3 M (recommended configuration, `T1.3M`) | |
| | **Architecture** | 3 layers, 3 attention heads, 192 embedding dims | |
| | **Precision** | float16 | |
| | **Model size on disk** | ~2.5 MB | |
| | **Training hardware** | Single NVIDIA T4 (Google Colab) | |
| | **Training time / cost** | ~1,092 s (~18 min), ≈ $0.05 per model | |
| | **License** | Paper published under CC BY-NC-ND 4.0 | |
| | **Authors** | Tanner Hobson, Jian Huang (University of Tennessee, Knoxville) | |
|
|
| ### Base training configuration |
|
|
| ```text |
| learning_rate = 1e-3 dropout = 0.2 |
| lr_decay_iters = 30000 min_lr = 1e-4 |
| beta2 = 0.99 max_iters = 30000 |
| batch_size = 256 n_layer = 3 |
| n_head = 3 n_embd = 192 |
| ``` |
|
|
| ## How It Works |
|
|
| ICD-10-CM codes are inherently hierarchical: the first character encodes the top level of the hierarchy, the second character the next level, and so on (e.g., `S` → `S5` → `S52` → `S52.5` traces an injury from "injury/poisoning" all the way down to a specific radius fracture). DgPg exploits this by modeling code sequences **character by character** rather than treating each code as an atomic token. |
|
|
| - Every DX code is normalized to **4 characters** (truncated and/or padded with hyphens). |
| - Records are converted to training sequences of consecutive code pairs, e.g. the record `N12,Q610,N179` becomes `•----N12-`, `•N12-Q610`, `•Q610N179`, `•N179----`. |
| - At inference, the model is used **iteratively**: given a seed code, it predicts the next code one character at a time, naturally respecting the coding hierarchy. |
|
|
| Compared to a traditional code-level transition matrix (12,309 × 12,309 codes ≈ 290 MB in float16), the character-level formulation is >100× smaller and generalizes across code hierarchies rather than memorizing pairs. |
|
|
| ### Visualization use case |
|
|
| The authors pair the model with a generative graph-building algorithm: starting from a seed code, a breadth-first search exhaustively scores candidate following (or preceding) codes by perplexity, keeps the top *k* = 3 at each step, and recurses to depth *n* = 3. The result is a left-to-right graph of the most representative code orderings around a diagnosis (e.g., what typically precedes and follows a sepsis code `A41.9`), generated in ~2 minutes per graph. |
|
|
| ## Training Data |
|
|
| Four model variants were trained, one per dataset plus a combined corpus: |
|
|
| | Source | Dataset | # Records | # Patients (weighted) | Notes | |
| |---|---|---:|---:|---| |
| | MIT | [MIMIC-IV](https://physionet.org/content/mimiciv/) | 430,812 | 430,812 | Beth Israel Deaconess Medical Center; ICD-10-CM, truncated to 4 chars | |
| | CDC | NHDS | 756,953 | 111,514,788 | 1996–2010; ICD-9-CM converted to ICD-10-CM via the NBER crosswalk | |
| | CDC | NHCS | 123,565 | 33,595,765 | 2020 survey | |
| | — | Combined | 1,277,400 | 145,507,435 | Union of the above | |
|
|
| CDC survey records carry a frequency weight *w*; each record is emitted *w* times during training. Only ordered DX-code sequences are required — the framework applies to any corpus with ordered diagnosis codes. |
|
|
| **Note on data access:** MIMIC-IV is a restricted-access dataset requiring credentialed access via PhysioNet. No patient-level data is contained in the model weights beyond learned sequence statistics, but users retraining DgPg must obtain the datasets under their respective terms. |
|
|
| ## Evaluation |
|
|
| Performance is measured as fixed-length sequence perplexity (PERF) on the second code of a two-code sequence, averaged over 10 K validation samples. |
|
|
| ### Model size ablation |
|
|
| | Model | Layers | Heads | Embd | Size | Loss | |
| |---|---:|---:|---:|---:|---:| |
| | T11M | 6 | 6 | 384 | 20 MB | 1.16 | |
| | **T1.3M** ★ | 3 | 3 | 192 | **2.5 MB** | 1.17 | |
| | T230K | 2 | 2 | 96 | 440 KB | 1.19 | |
| | T30K | 1 | 1 | 48 | 58 KB | 1.25 | |
|
|
| T1.3M matches the 10×-larger T11M and is the recommended configuration. Model sizes were capped at 10% of the equivalent transition-matrix size to ensure generalizable (not memorized) patterns. |
|
|
| ### Cross-dataset transferability |
|
|
| Evaluating each trained model against each dataset shows that models perform best on their own training distribution — evidence that **coding practices are provider- and scenario-specific**. The Combined model performs well across all constituent datasets; the NHDS-only model transfers worst, indicating NHDS coding patterns are the most dataset-unique. MIMIC and NHCS models are the most generalizable. |
|
|
| ## Intended Uses |
|
|
| **Direct intended uses** |
|
|
| - Visualizing an organization's representative diagnostic-coding patterns (which codes typically precede/follow a given code; which codes tend to be coded first or last in a visit). |
| - Comparing coding practices across institutions, units, or clinical scenarios. |
| - Scoring how representative a given code sequence is relative to a corpus (classification use case). |
| - Operational insight for administrators: quality control, process improvement, burnout-related workflow analysis. |
|
|
| **Out-of-scope uses** |
|
|
| - ❌ Automated medical coding or coding suggestions for billing. |
| - ❌ Clinical diagnosis, prediction, or decision support. |
| - ❌ Auditing individual coders or labeling any coding practice as "good" or "bad" — the model summarizes shared practice; it does not evaluate deviations. |
| - ❌ Any use that would alter coding to affect claim payment. Modifying codes to increase reimbursement is fraudulent; any coding revision requires proper documentation. |
|
|
| ## Limitations |
|
|
| - **4-character truncation.** For cross-dataset comparability, all codes are normalized to 4 characters, discarding finer ICD-10-CM specificity (the framework itself is not limited to 4 characters). |
| - **ICD-9 → ICD-10 crosswalk noise.** NHDS codes were converted via the NBER crosswalk, which is not lossless. |
| - **Single-visit scope.** DgPg models intra-visit code ordering only, not longitudinal, multi-visit patient trajectories. |
| - **Dataset-specific patterns.** A model trained on one provider's records does not transfer reliably to another's — this is by design, but means checkpoints are not general-purpose. |
| - **Hallucination.** As a generative model, DgPg can produce plausible-looking but unrepresentative sequences; the authors flag hallucination analysis as future work. |
| - **DX codes only.** Procedure and other medical code types are not yet covered. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{hobson2025dgpg, |
| author = {Tanner Hobson and Jian Huang}, |
| title = {Visualizing Medical Coding Practices Using Transformer Models}, |
| booktitle = {Proceedings of the 14th International Conference on Pattern |
| Recognition Applications and Methods (ICPRAM 2025)}, |
| pages = {725--732}, |
| year = {2025}, |
| publisher = {SCITEPRESS}, |
| doi = {10.5220/0013257800003905}, |
| isbn = {978-989-758-730-6} |
| } |
| ``` |
|
|
| ## Acknowledgments |
|
|
| Built on [nanoGPT](https://github.com/karpathy/nanoGPT) by Andrej Karpathy. Data courtesy of MIT-LCP (MIMIC-IV) and the US CDC National Center for Health Statistics (NHDS, NHCS). |
