| --- |
| license: apache-2.0 |
| pipeline_tag: image-classification |
| tags: |
| - siglip2 |
| - document-classification |
| - hierarchical |
| - multi-task |
| library_name: transformers |
| --- |
| |
| # Med Doc Classifier (~96M) |
|
|
| The med-doc-classifier is a lightweight, encoder-only model (~96M parameters) built on the popular SigLIP2-base image backbone. In a single pass it sorts an image into one of 27 classes across two hierarchical levels, from broad group down to specific leaf, and answers two further questions: whether the subject matter is medical, and whether it was handwritten. |
|
|
| The model was built by training many specialised models, ensembling them, and distilling the ensemble into this one compact student. |
|
|
| [**Live demo**](https://medai.eka.care/classifier) · [**Launch blog**](https://info.eka.care/services/releasing-parrotlet-open-models-open-weights-for-medical-document-intelligence) |
|
|
| ## Results |
|
|
| Full internal test = 16,204 images (see `benchmark_results.json`). |
|
|
| | Precision | Size | Accuracy (27-class) | Macro-F1 | |
| |-----------|------|---------------------|----------| |
| | bf16 | 193 MB | 96.45 | 0.965 | |
| | int8 | 103 MB | 96.43 | 0.965 | |
| | int4 | 60 MB | 95.22 | 0.955 | |
|
|
| int8 is effectively lossless; int4 trades about a point of accuracy for a model over 3× smaller that runs comfortably on a plain CPU. It matches or beats models several times its size, including a 428M SigLIP2-SO400M variant (96.24%) and a 256M generative VLM baseline (90.42%). |
|
|
| ## Taxonomy (5 L1 groups → 27 L2 leaves) |
|
|
| ``` |
| Document (10 leaves) |
| ├─ Letter (key: letter) |
| ├─ Other document (key: other_document) |
| ├─ Scan interpretation (key: scan_interpretation) |
| ├─ Lab / diagnostic report (key: lab_diagnostic_report) |
| ├─ OPD Consultation Record (key: opd_consultation_record) |
| ├─ Discharge summary (key: discharge_summary) |
| ├─ Insurance document (key: insurance_document) |
| ├─ Form (key: form) |
| ├─ Invoice / bill (key: invoice_bill) |
| ├─ Certificate (key: certificate) |
| Card / credential (7 leaves) |
| ├─ ABHA card (key: abha_card) |
| ├─ Aadhaar card (key: aadhaar_card) |
| ├─ PAN card (key: pan_card) |
| ├─ Voter ID (key: voter_id) |
| ├─ PMJAY / Ayushman (key: pmjay_ayushman) |
| ├─ Insurance e-card (key: insurance_e_card) |
| ├─ Other card (key: other_card) |
| Diagnostic imaging (5 leaves) |
| ├─ X-ray (key: x_ray) |
| ├─ CT (key: ct) |
| ├─ MRI (key: mri) |
| ├─ Ultrasound (key: ultrasound) |
| ├─ Other (Diagnostic imaging) (key: other_diagnostic_imaging) |
| Body images (3 leaves) |
| ├─ Skin / wound (key: skin_wound) |
| ├─ Headshot (key: headshot) |
| ├─ Other (Body images) (key: other_body_images) |
| Miscellaneous (2 leaves) |
| ├─ Medication image (key: medication_image) |
| ├─ Other (Miscellaneous) (key: other_miscellaneous) |
| ``` |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModel |
| from PIL import Image |
| |
| model = AutoModel.from_pretrained("ekacare/med-doc-classifier", trust_remote_code=True).eval() |
| img = Image.open("doc.jpg") |
| |
| model.classify(img) |
| ``` |
|
|
| ```jsonc |
| // example return value |
| { |
| "l2": { "source": "flat", "key": "lab_diagnostic_report", |
| "value": "Lab / diagnostic report", "confidence": 0.97 }, |
| "l1": { "value": "Document", "source": "inferred_from_flat_l2" }, |
| "medical": { "value": "medical", "confidence": 0.99, "p_positive": 0.99 }, |
| "handwritten": { "value": "printed", "confidence": 0.98, "p_positive": 0.02 } |
| } |
| ``` |
|
|
| By default L2 comes from a flat 27-way head and L1 is inferred from it. Other options: |
|
|
| ```python |
| model.classify(img, scope="l1") # L1 group only (no L2) |
| model.classify(img, scope="hierarchical") # L1 head → that group's leaf L2 head |
| model.classify(img, l1="Document") # fix L1 → that group's leaf L2 head |
| model.classify(img, top_k=3) # top-3 candidates for the multi-class heads |
| ``` |
|
|
| `medical` and `handwritten` are independent binary heads — disable them with `medical=False` / `handwritten=False`. |
|
|
| ## Quantized loading (optimum-quanto — CPU & GPU) |
|
|
| Only the vision tower is quantized; the heads stay full precision. |
|
|
| ```python |
| # pip install optimum-quanto |
| from transformers.dynamic_module_utils import get_class_from_dynamic_module |
| |
| load_classifier = get_class_from_dynamic_module( |
| "modeling_siglip2_hier.load_classifier", "ekacare/med-doc-classifier") |
| |
| model = load_classifier("ekacare/med-doc-classifier", quantization="int4") # None | "int8" | "int4" |
| model.classify(img) |
| ``` |
|
|
| Working from a clone instead? `from modeling_siglip2_hier import load_classifier` |
| inside the repo directory works too, and `load_classifier(".", quantization="int8")` |
| loads straight from the checkout. |
|
|
| ## Citation |
|
|
| If you use this model, please cite: |
|
|
| ```bibtex |
| @software{med_doc_classifier, |
| author = {{Eka Care}}, |
| title = {Med Doc Classifier: hierarchical classification of health-app document uploads}, |
| year = {2026}, |
| url = {https://huggingface.co/ekacare/med-doc-classifier} |
| } |
| ``` |
|
|
