license: apache-2.0
pipeline_tag: image-classification
tags:
- siglip2
- document-classification
- hierarchical
- multi-task
library_name: transformers
Med Doc Classifier (~96M)
The med-doc-classifier is a lightweight, encoder-only model (~96M parameters) built on the popular SigLIP2-base image backbone. In a single pass it sorts an image into one of 27 classes across two hierarchical levels, from broad group down to specific leaf, and answers two further questions: whether the subject matter is medical, and whether it was handwritten.
The model was built by training many specialised models, ensembling them, and distilling the ensemble into this one compact student.
Results
Full internal test = 16,204 images (see benchmark_results.json).
| Precision | Size | Accuracy (27-class) | Macro-F1 |
|---|---|---|---|
| bf16 | 193 MB | 96.45 | 0.965 |
| int8 | 103 MB | 96.43 | 0.965 |
| int4 | 60 MB | 95.22 | 0.955 |
int8 is effectively lossless; int4 trades about a point of accuracy for a model over 3× smaller that runs comfortably on a plain CPU. It matches or beats models several times its size, including a 428M SigLIP2-SO400M variant (96.24%) and a 256M generative VLM baseline (90.42%).
Taxonomy (5 L1 groups → 27 L2 leaves)
Document (10 leaves)
├─ Letter (key: letter)
├─ Other document (key: other_document)
├─ Scan interpretation (key: scan_interpretation)
├─ Lab / diagnostic report (key: lab_diagnostic_report)
├─ OPD Consultation Record (key: opd_consultation_record)
├─ Discharge summary (key: discharge_summary)
├─ Insurance document (key: insurance_document)
├─ Form (key: form)
├─ Invoice / bill (key: invoice_bill)
├─ Certificate (key: certificate)
Card / credential (7 leaves)
├─ ABHA card (key: abha_card)
├─ Aadhaar card (key: aadhaar_card)
├─ PAN card (key: pan_card)
├─ Voter ID (key: voter_id)
├─ PMJAY / Ayushman (key: pmjay_ayushman)
├─ Insurance e-card (key: insurance_e_card)
├─ Other card (key: other_card)
Diagnostic imaging (5 leaves)
├─ X-ray (key: x_ray)
├─ CT (key: ct)
├─ MRI (key: mri)
├─ Ultrasound (key: ultrasound)
├─ Other (Diagnostic imaging) (key: other_diagnostic_imaging)
Body images (3 leaves)
├─ Skin / wound (key: skin_wound)
├─ Headshot (key: headshot)
├─ Other (Body images) (key: other_body_images)
Miscellaneous (2 leaves)
├─ Medication image (key: medication_image)
├─ Other (Miscellaneous) (key: other_miscellaneous)
Usage
from transformers import AutoModel
from PIL import Image
model = AutoModel.from_pretrained("ekacare/med-doc-classifier", trust_remote_code=True).eval()
img = Image.open("doc.jpg")
model.classify(img)
// example return value
{
"l2": { "source": "flat", "key": "lab_diagnostic_report",
"value": "Lab / diagnostic report", "confidence": 0.97 },
"l1": { "value": "Document", "source": "inferred_from_flat_l2" },
"medical": { "value": "medical", "confidence": 0.99, "p_positive": 0.99 },
"handwritten": { "value": "printed", "confidence": 0.98, "p_positive": 0.02 }
}
By default L2 comes from a flat 27-way head and L1 is inferred from it. Other options:
model.classify(img, scope="l1") # L1 group only (no L2)
model.classify(img, scope="hierarchical") # L1 head → that group's leaf L2 head
model.classify(img, l1="Document") # fix L1 → that group's leaf L2 head
model.classify(img, top_k=3) # top-3 candidates for the multi-class heads
medical and handwritten are independent binary heads — disable them with medical=False / handwritten=False.
Quantized loading (optimum-quanto — CPU & GPU)
Only the vision tower is quantized; the heads stay full precision.
# pip install optimum-quanto
from transformers.dynamic_module_utils import get_class_from_dynamic_module
load_classifier = get_class_from_dynamic_module(
"modeling_siglip2_hier.load_classifier", "ekacare/med-doc-classifier")
model = load_classifier("ekacare/med-doc-classifier", quantization="int4") # None | "int8" | "int4"
model.classify(img)
Working from a clone instead? from modeling_siglip2_hier import load_classifier
inside the repo directory works too, and load_classifier(".", quantization="int8")
loads straight from the checkout.
Citation
If you use this model, please cite:
@software{med_doc_classifier,
author = {{Eka Care}},
title = {Med Doc Classifier: hierarchical classification of health-app document uploads},
year = {2026},
url = {https://huggingface.co/ekacare/med-doc-classifier}
}