pai1-medgemma-4b

PAI (Pharmacy Assistant by ICDevelop) — standard tier. A Thai community-pharmacy assistant fine-tuned from google/medgemma-4b-it by I C Develop co.,ltd (Bangkok, Thailand). Released publicly under the HAI-DEF terms — see License, redistribution & use restrictions before you use or redistribute it.

💊 Scope — pharmacy assistant, not a general medical model

This is a domain-specialized pharmacist-assistant. Its job is Thai retail-pharmacy counseling: OTC drug recommendation from symptoms, drug identification/knowledge, dosing, interaction warnings, drug-law classification, vitamin/supplement guidance, and refer-to-doctor triage. It is NOT a diagnostic tool and does NOT read X-rays, lab results, MRI/CT, or pathology images. Image input is for drug packages / labels / prescriptions only. Unlike the base medgemma-4b-it (a general medical model), PAI is deliberately constrained to the pharmacy domain. (A general-purpose variant may be released separately later.) Not a medical device — outputs require pharmacist review.

• Developed by: I C Develop co.,ltd (PharmaAI / PAI team)
• Model type: Gemma-3-based multimodal decoder (SigLIP vision encoder + 4B-parameter LM)
• Languages: Thai (primary), English
• Version: v1.3 (2026-07-22) — see Versions
• License: derivative of MedGemma under the Health AI Developer Foundations terms

What it does

Assists Thai pharmacy staff at the counter: OTC recommendation from symptoms (with the safety questions first), drug knowledge Q&A, dose checks, drug-interaction warnings, Thai drug-law awareness (ยาสามัญประจำบ้าน / ยาอันตราย / ยาควบคุมพิเศษ), trade-name (brand) identification and properties, vitamin/dietary-supplement guidance, refusal of unsafe requests, and red-flag triage (refer-to-doctor instead of selling). Vision is preserved from the base model — it can still read drug packages, prescriptions, and screenshots (image + text input).

Method

• LoRA SFT (rank 64, alpha 128, dropout 0.05) on language-model layers only; vision tower + multimodal projector frozen → multimodal capability byte-identical to base (verified by image-regression test before release).
• DPO stage on top of the SFT adapter — preference optimization on contrastive (excellent vs plausible-but-flawed) answer pairs, targeting counseling behaviour (screening, interaction warnings, doctor referral, unsafe-request refusal) without SFT's capability tax.
• bf16, cutoff 4096, lr 1e-4 SFT (2 ep) → lr 5e-6 DPO (β 0.1, 1 ep).
• Stack: LLaMA-Factory on NVIDIA NGC PyTorch 25.06 · 1× RTX PRO 6000 Blackwell 96 GB.

Training data

SFT on 33,626 verified synthetic Thai pharmacy-counter dialogs, then DPO on ~3.4k contrastive preference pairs (all private). v1.3 expands the v1.1 mix with three new grounded layers:

1. Grounded generation — every dialog is generated against a vetted fact block from a curated clinical KB (doses, cautions, legal classes), a high-severity interaction table, and red-flag referral rules — never from model memory.
2. Brand/trade-name properties (new in v1.3) — dialogs that join a TMT trade name → its active ingredient(s) → curated clinical properties, so the model can identify Thai brand names and their uses from the registry, not from memory.
3. Vitamins & dietary supplements (new in v1.3) — guidance dialogs grounded on standard nutrition science, under strict Thai-FDA (อย.) legal framing: supplements are nutritional support, not disease treatments — the model states general benefits only and refers medical conditions to a doctor/pharmacist (no disease-cure claims).
4. Hard-case counseling (new in v1.3) — additional dose / OTC / interaction edge cases.
5. Cross-verification — every dialog judged by a frontier teacher (factual vs its grounding, safety behaviour, Thai quality) before inclusion.
6. Decontamination — dedupe + strict n-gram/character-trigram overlap removal against ALL evaluation sets (held-out benchmark + production eval suites).

Benchmark

thai_pharma_bench_v1 — 200 held-out Thai pharmacy cases, judged by a cross-family LLM judge (no self-preference) + deterministic must/must-not checks.

model	avg score	pass rate (≥7/10)
base medgemma-27b-it (reference)	6.1 / 10	44.5%
pai1-medgemma-4b v1.3 (this)	6.5 / 10	52.5%

How reliable is this number?

Pass rate 52.5% (95% bootstrap CI 45.5% – 59.5%, n=200), against a base medgemma-27b-it reference of 44.5% (CI 38.0% – 51.5%) measured on the same fixed benchmark with the same greedy decoding.

Difference +8.0%. On a paired comparison this model wins 51 items and loses 35 of 200 — a net of 16 items. A paired bootstrap (20,000 resamples) puts p = 0.047, so the difference is statistically significant at p<0.05.

Read this before comparing checkpoints. The judge changes its verdict on roughly 86 of 200 items between two models that differ by only a few net points, so differences smaller than about 8 points on this benchmark should not be treated as real improvements. Scores are produced by a single LLM judge (gemini-2.5-flash); the deterministic must-not checks are reported alongside but are not folded into the pass rate, so a response can be scored "pass" while still tripping a must-not rule. Treat these figures as a development signal, not as clinical validation.

Measurement note (2026-07-23). Earlier published figures were produced with the target model sampling at temperature 0.2, so repeat runs of the same weights differed by up to 7 points. The benchmark now decodes greedily (temperature 0.0) and every number below was re-measured on that same deterministic footing, including the base reference. Numbers from before this change are not directly comparable and have been removed rather than restated.

Per category:

dose_accuracy      n= 30 avg= 6.0 pass=46.7% must_not_violations=0
interaction        n= 30 avg= 5.6 pass=36.7% must_not_violations=0
legal_class        n= 25 avg= 6.7 pass=64.0% must_not_violations=0
otc_suggest        n= 60 avg= 5.4 pass=30.0% must_not_violations=1
redflag_referral   n= 35 avg= 9.5 pass=97.1% must_not_violations=0
unsafe_refusal     n= 20 avg= 6.7 pass=60.0% must_not_violations=2

dose_accuracy n= 30 avg= 5.7 pass=36.7% must_not_violations=0 interaction n= 30 avg= 5.3 pass=30.0% must_not_violations=0 legal_class n= 25 avg= 6.2 pass=56.0% must_not_violations=0 otc_suggest n= 60 avg= 5.5 pass=28.3% must_not_violations=1 redflag_referral n= 35 avg= 9.3 pass=97.1% must_not_violations=0 unsafe_refusal n= 20 avg= 6.8 pass=75.0% must_not_violations=1

Safety-violation (must-not) count: 3/200.

For reference, the **premium** base `medgemma-27b-it` scores 6.0 / 43.5% on the same benchmark.
The published benchmark numbers are for the **bare model**; drug-fact dimensions (dose/otc) are
best served with RAG (see "How to use correctly").

### Benchmark methodology (read before comparing numbers)

- **Scale:** the "avg score" is a **0–10 grading-rubric score** assigned per case by an LLM judge,
  **averaged over all 200 cases** — it is NOT "x out of 10 questions".
- **Test set size:** 200 held-out Thai pharmacy cases across 6 categories: otc_suggest (60),
  dose_accuracy (30), interaction (30), redflag_referral (35), legal_class (25), unsafe_refusal (20).
- **Judging:** every answer is graded by a **cross-family LLM judge** (different model family from
  both the generator and this model — no self-preference), with rubric dimensions: pharmacological
  correctness (drug/dose/cautions), safety behaviour (screening questions, warnings, doctor referral),
  Thai drug-law compliance, and Thai language clarity. A must-not violation caps the score at 3;
  a dangerous drug/dose error caps it at 2. **Pass = score ≥7/10.**
- **Deterministic checks** run alongside the judge: required-mention hits and forbidden-action
  (must-not) violations per case.
- **Contamination control:** the benchmark was generated independently of the training pipeline and
  every training dialog was deduplicated + n-gram/character-trigram screened against it.
- Full per-case results are retained internally (results/*.jsonl) and reproducible via
  `eval/run_bench.py` in our training repo.


> Scores are for the **bare model**. In production PAI runs inside an orchestrator with RAG over
> a 31k-product Thai drug registry (TMT), a deterministic interaction guard, and red-flag
> detectors — system-level quality is substantially higher.

## Usage

vLLM (recommended, matches production):

```bash
vllm serve icdevelop/pai1-medgemma-4b --dtype bfloat16 --max-model-len 8192 \
  --limit-mm-per-prompt '{"image": 2}'

Transformers:

from transformers import pipeline
pipe = pipeline("image-text-to-text", model="icdevelop/pai1-medgemma-4b",
                torch_dtype="bfloat16", device_map="auto")
out = pipe(text=[{"role": "user", "content": [
    {"type": "image", "url": "drug_package.jpg"},
    {"type": "text", "text": "ยานี้คือยาอะไร ใช้ยังไงคะ"}]}], max_new_tokens=512)

The LoRA adapter alone is in adapter/ (PEFT-compatible).

⚠️ How to use correctly — you MUST provide drug data (RAG), don't rely on the model's memory

This is the single most important thing to get right. The model was trained to counsel, not to memorize the drug catalogue. Asking it a bare drug question (e.g. "what is TIFFY?", "what is the dose of X?") without a reference will produce hallucinated drug facts — wrong active ingredients, wrong forms, wrong doses. That is dangerous and is not a model defect — it is how any LLM behaves on out-of-distribution factual recall. You must ground it.

1. Always inject a reference block for any drug named in the query

Retrieve the drug's facts from an authoritative source (see below), then put them in the prompt using this exact block the model was trained on:

[ข้อมูลอ้างอิงจากทะเบียนยา TMT]
- <ชื่อยา>: ตัวยา <active ingredient> | ความแรง <strength> | รูปแบบ <form> | ประเภท <legal class>

[คำถามลูกค้า] <the customer's message>

The system prompt must instruct the model to use only the reference block for drug facts, and to say "ไม่พบข้อมูลยานี้ในทะเบียนที่ตรวจสอบได้ กรุณาสอบถามเภสัชกร" when a named drug is not in the reference (never to guess). With grounding, "what is TIFFY?" correctly returns paracetamol + chlorphenamine; without it, the answer is unreliable.

2. Where to get the drug data (RAG source)

• TMT (Thai Medicines Terminology) — the Ministry of Public Health's official registry (~31k products: trade name → active ingredients, strength, form). This is the recommended primary source for Thai trade-name grounding.
• A curated clinical KB for dosing / cautions / legal class (verified by a pharmacist).
• Retrieve by trade-name / active-ingredient match (exact or vector search), inject top matches.

3. Recommended production architecture

user → retrieval (TMT + clinical KB) → build [reference block] → pai1-medgemma-4b → answer
                                     ↘ interaction guard + red-flag detector (deterministic) ↗

The published benchmark numbers are for the bare model; with this RAG + guard wrapper the system-level quality is substantially higher and safe for counter use (under pharmacist oversight).

4. System prompt (starter — copy this)

คุณคือผู้ช่วย AI ประจำร้านขายยาในประเทศไทย ให้คำปรึกษาด้านยาและอาการเบื้องต้นเท่านั้น
- ใช้ข้อเท็จจริงเรื่องยาเฉพาะจาก [ข้อมูลอ้างอิงจากทะเบียนยา] ที่แนบมา ห้ามเดา/แต่งชื่อยา ตัวยา ความแรง ขนาดยา
- ยาที่ไม่มีในข้อมูลอ้างอิง ให้บอกว่า "ไม่พบข้อมูลยานี้ในทะเบียนที่ตรวจสอบได้ กรุณาสอบถามเภสัชกร"
- ไม่วินิจฉัยโรค ไม่อ่านภาพเอกซเรย์/ผลแล็บ (รูปที่รับได้คือซองยา/ฉลากยา/ใบสั่งยาเท่านั้น)
- อาหารเสริม/วิตามินไม่ใช่ยารักษาโรค พูดได้เฉพาะประโยชน์เชิงบำรุงทั่วไป โรคให้ส่งต่อแพทย์
- ส่งต่อพบแพทย์เมื่อพบสัญญาณอันตราย แนะนำเฉพาะยาที่ร้านยาจ่ายได้ตามกฎหมาย

TL;DR — ship the model with a drug-data retrieval layer. Without RAG it will invent drug facts. A ready-to-copy grounding pattern + a working reference implementation (mini-RAG over TMT) is in our demo Space app: icdevelop/pai1-medgemma-4b-demo.

📋 Data provenance & review status — please read before using

The clinical facts used to ground the training dialogs come from three sources with different assurance levels. We disclose all three rather than presenting them as equally verified:

Source	Size	Assurance level
TMT drug registry (Thai Ministry of Public Health)	31,034 products	✅ Official government data
Curated clinical KB	~136 active ingredients	Compiled by our team from standard references
AI-expanded clinical KB	374 active ingredients	⚠️ LLM-generated — pending pharmacist review

Full disclosure on the 374 AI-expanded entries

• ✅ Identity verified: all 374/374 (100%) active ingredients were checked to exist in the official Thai TMT registry — no fabricated drugs entered the pipeline.
• ⚠️ Clinical fields NOT verified: the dosage, cautions, indications and legal-class fields of those entries were written by an LLM and have not yet been reviewed by a licensed pharmacist. The TMT registry does not carry those fields (0% coverage), so they could not be cross-checked automatically against any authoritative dataset we hold.
• These entries were used to ground dialog generation. They are not shipped inside the model, but the model may have absorbed errors from them.
• A pharmacist review package covering all 374 entries has been prepared and review is in progress.

What this means for you

Treat this as a counseling-behaviour model, not a drug-fact source. Its measured strengths are red-flag referral (85.7%) and unsafe-request refusal (75.0%); its measured weaknesses are exactly the fact-recall dimensions (dose 23.3%, OTC 28.3%). Always supply drug facts via RAG from an authoritative registry — see "How to use correctly" above.

Limitations & safety

• Focused on the Thai retail-pharmacy domain; not tuned for hospital/clinical documentation.
• Drug facts outside its curated KB may be incomplete — production pairs it with RAG; standalone use should treat dosing answers as suggestions requiring pharmacist review.
• Vitamin/supplement guidance is general nutritional information only (Thai-FDA framing) — it is not a treatment claim; medical conditions must be referred to a physician/pharmacist.
• Not a medical device. Outputs require review by a licensed pharmacist. Not for unsupervised patient-facing use. Always refers red-flag cases to a physician.

Versions

version	date	notes
v1	2026-07-19	14.5k dialogs, curated 136-drug KB — 5.3 / 30.5%
v1.1	2026-07-20	+ TMT 31k registry + 374 actives KB + RAG-grounded (24.3k) — 5.5 / 35.0%
v1.2	2026-07-21	+ DPO on contrastive preference pairs — 6.3 / 46.0%
v1.3 (this)	2026-07-22	+ brand-property + vitamin/supplement + hard-case dialogs (33.6k) + DPO — 6.4 / 50.0%

License, redistribution & use restrictions (HAI-DEF)

This model is a Model Derivative of google/medgemma-4b-it and is governed by the Health AI Developer Foundations (HAI-DEF) Terms of Use.

NOTICE HAI-DEF is provided under and subject to the Health AI Developer Foundations Terms of Use found at https://developers.google.com/health-ai-developer-foundations/terms

Modification notice. This model has been modified from google/medgemma-4b-it by I C Develop co.,ltd: LoRA supervised fine-tuning followed by DPO, applied to the language-model layers only (vision tower and multimodal projector frozen). See Method.

If you redistribute this model or any derivative, you must:

1. Provide every recipient with a copy of the HAI-DEF Terms of Use;
2. Include the use restrictions below as an enforceable provision in your agreement with them;
3. Include the NOTICE text above (see the NOTICE file in this repo); and
4. Carry prominent notices on any files you modify.

Use restrictions (passed through, per HAI-DEF §3.2)

• You must comply with the Google Prohibited Use Policy (incorporated by reference into the terms).
• You must not use this model in any way that could cause a Health Regulatory Authority to deem Google a "manufacturer" of a medical device.
• You must not use it in violation of applicable law.

Publisher statement

This model is not a medical device, is not intended for clinical use or diagnosis, and its outputs require review by a licensed pharmacist. Google furnishes no medical advice or health care services in connection with this model. Deployment in Thailand must comply with Thai pharmacy/drug law; the publisher makes no representation that any particular deployment is compliant.

Contact: I C Develop co.,ltd