README.md

---
license: apache-2.0
pipeline_tag: image-classification
tags:
  - siglip2
  - document-classification
  - hierarchical
  - multi-task
library_name: transformers
---

# Med Doc Classifier (~96M)

The med-doc-classifier is a lightweight, encoder-only model (~96M parameters) built on the popular SigLIP2-base image backbone. In a single pass it sorts an image into one of 27 classes across two hierarchical levels, from broad group down to specific leaf, and answers two further questions: whether the subject matter is medical, and whether it was handwritten.

The model was built by training many specialised models, ensembling them, and distilling the ensemble into this one compact student.

[**Live demo**](https://medai.eka.care/classifier) · [**Launch blog**](https://info.eka.care/services/releasing-parrotlet-open-models-open-weights-for-medical-document-intelligence)

## Results

Full internal test = 16,204 images (see `benchmark_results.json`).

| Precision | Size | Accuracy (27-class) | Macro-F1 |
|-----------|------|---------------------|----------|
| bf16 | 193 MB | 96.45 | 0.965 |
| int8 | 103 MB | 96.43 | 0.965 |
| int4 | 60 MB | 95.22 | 0.955 |

int8 is effectively lossless; int4 trades about a point of accuracy for a model over 3× smaller that runs comfortably on a plain CPU. It matches or beats models several times its size, including a 428M SigLIP2-SO400M variant (96.24%) and a 256M generative VLM baseline (90.42%).

## Taxonomy (5 L1 groups → 27 L2 leaves)

```
Document  (10 leaves)
    ├─ Letter                 (key: letter)
    ├─ Other document         (key: other_document)
    ├─ Scan interpretation    (key: scan_interpretation)
    ├─ Lab / diagnostic report (key: lab_diagnostic_report)
    ├─ OPD Consultation Record (key: opd_consultation_record)
    ├─ Discharge summary      (key: discharge_summary)
    ├─ Insurance document     (key: insurance_document)
    ├─ Form                   (key: form)
    ├─ Invoice / bill         (key: invoice_bill)
    ├─ Certificate            (key: certificate)
Card / credential  (7 leaves)
    ├─ ABHA card              (key: abha_card)
    ├─ Aadhaar card           (key: aadhaar_card)
    ├─ PAN card               (key: pan_card)
    ├─ Voter ID               (key: voter_id)
    ├─ PMJAY / Ayushman       (key: pmjay_ayushman)
    ├─ Insurance e-card       (key: insurance_e_card)
    ├─ Other card             (key: other_card)
Diagnostic imaging  (5 leaves)
    ├─ X-ray                  (key: x_ray)
    ├─ CT                     (key: ct)
    ├─ MRI                    (key: mri)
    ├─ Ultrasound             (key: ultrasound)
    ├─ Other (Diagnostic imaging) (key: other_diagnostic_imaging)
Body images  (3 leaves)
    ├─ Skin / wound           (key: skin_wound)
    ├─ Headshot               (key: headshot)
    ├─ Other (Body images)    (key: other_body_images)
Miscellaneous  (2 leaves)
    ├─ Medication image       (key: medication_image)
    ├─ Other (Miscellaneous)  (key: other_miscellaneous)
```

## Usage

```python
from transformers import AutoModel
from PIL import Image

model = AutoModel.from_pretrained("ekacare/med-doc-classifier", trust_remote_code=True).eval()
img = Image.open("doc.jpg")

model.classify(img)
```

```jsonc
// example return value
{
  "l2":          { "source": "flat", "key": "lab_diagnostic_report",
                    "value": "Lab / diagnostic report", "confidence": 0.97 },
  "l1":          { "value": "Document", "source": "inferred_from_flat_l2" },
  "medical":     { "value": "medical", "confidence": 0.99, "p_positive": 0.99 },
  "handwritten": { "value": "printed", "confidence": 0.98, "p_positive": 0.02 }
}
```

By default L2 comes from a flat 27-way head and L1 is inferred from it. Other options:

```python
model.classify(img, scope="l1")           # L1 group only (no L2)
model.classify(img, scope="hierarchical") # L1 head → that group's leaf L2 head
model.classify(img, l1="Document")        # fix L1 → that group's leaf L2 head
model.classify(img, top_k=3)              # top-3 candidates for the multi-class heads
```

`medical` and `handwritten` are independent binary heads — disable them with `medical=False` / `handwritten=False`.

## Quantized loading (optimum-quanto — CPU & GPU)

Only the vision tower is quantized; the heads stay full precision.

```python
# pip install optimum-quanto
from transformers.dynamic_module_utils import get_class_from_dynamic_module

load_classifier = get_class_from_dynamic_module(
    "modeling_siglip2_hier.load_classifier", "ekacare/med-doc-classifier")

model = load_classifier("ekacare/med-doc-classifier", quantization="int4")  # None | "int8" | "int4"
model.classify(img)
```

Working from a clone instead? `from modeling_siglip2_hier import load_classifier`
inside the repo directory works too, and `load_classifier(".", quantization="int8")`
loads straight from the checkout.

## Citation

If you use this model, please cite:

```bibtex
@software{med_doc_classifier,
  author = {{Eka Care}},
  title  = {Med Doc Classifier: hierarchical classification of health-app document uploads},
  year   = {2026},
  url    = {https://huggingface.co/ekacare/med-doc-classifier}
}
```