AiHub4MSRH-Hash/Gemma_3_4B

Gemma 3 4B — Bilingual (Amharic + English) Medical QA

Fine-tuned LoRA adapters on top of google/gemma-3-4b-it for medical question-answering in Amharic and English. This is the lightweight counterpart to the 12B model, trained on the same bilingual medical QA dataset and intended for lower-cost inference.

⚠️ Medical disclaimer. This model is a research prototype. Its outputs are AI-generated and may be incomplete or incorrect. It is not a medical device and must not be used for diagnosis or treatment decisions. Always consult a qualified healthcare professional. Outputs should be reviewed by a clinician before any downstream use.

---

Model details

Base model	google/gemma-3-4b-it
Method	QLoRA (4-bit NF4) supervised fine-tuning
Adapter type	LoRA (PEFT)
Languages	Amharic (am), English (en)
Domain	Medical / health question answering
License	Gemma (inherits base model terms)
Developed by	Walelign Tewabe Sewunetie (PhD) & Surafel L. Tilahun (PhD)

LoRA configuration

Hyperparameter	Value
Rank (r)	16
Alpha	32
Dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Quantization	4-bit NF4, double quant, bf16 compute

Training configuration

Setting	Value
Epochs	3
Total steps	864
Learning rate	2e-4
LR scheduler	cosine, warmup ratio 0.03
Per-device batch size	8
Gradient accumulation	2
Max sequence length	1024
Precision	bf16
Gradient checkpointing	enabled
Seed	42
Hardware	1× NVIDIA H200 GPU

---

Dataset

Trained on a bilingual medical QA dataset with parallel English and Amharic question–answer pairs. Each source row contributes two training examples (one English pair, one Amharic pair).

Source rows	~2,300
Training examples (after bilingual expansion)	~4,600
Train / validation split	90 / 10
Columns	English_question, English_answer, Amharic_question, Amharic_answer

Dataset provenance: content compiled from public health websites/articles, medical textbooks and guidelines, expert/clinician-written material, and an existing medical QA dataset. Amharic produced by professional translation. Medical content reviewed by a qualified health professional. See the dataset card for full details.

---

Training results

Training loss trajectory

Training loss dropped sharply and continued to decline through training:

Progress	Training loss	Token accuracy
Start (epoch ~0.03)	7.24	0.345
Epoch ~2.92	0.896	0.762
Epoch ~2.95	0.836	0.778
Epoch ~2.99	0.902	0.762

Per-epoch eval progress

Epoch	Eval loss	Eval token accuracy
1	1.4533	0.6455
2	1.3772	0.6670
3	1.4402	0.6672

Eval loss reached its minimum at epoch 2 (1.3772) and rose slightly at epoch 3 (1.4402), while training loss continued to fall — indicating the onset of mild overfitting in the final epoch. The end-of-epoch-2 checkpoint (checkpoint-576) may generalize slightly better than the final checkpoint. The final adapter at the repo root corresponds to epoch 3 (checkpoint-864); users who prefer the best-validation checkpoint can load checkpoint-576.

Comparison with the 12B model

For reference, the larger Gemma 3 12B version trained on the same data reached a best eval loss of 1.224 and accuracy 0.696. This 4B model trades some quality for substantially smaller size and lower inference cost.

Best-epoch metric	This model (4B)	12B version
Eval loss	1.377	1.224
Eval token accuracy	0.667	0.696

Checkpoints

Checkpoint	Corresponds to
checkpoint-576	End of epoch 2 (best eval)
checkpoint-864	End of epoch 3 (final — repo root)

---

How to use

These are LoRA adapters — load them on top of the base model.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "google/gemma-3-4b-it"
ADAPTER = "Walelign/Gemma_3_4B"

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

question = "የስኳር በሽታ ምልክቶች ምንድን ናቸው?"
msgs = [{"role": "user", "content": question}]
enc = tok.apply_chat_template(
    msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

with torch.no_grad():
    out = model.generate(**enc, max_new_tokens=256, do_sample=False)

print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

Requirements

torch  (CUDA build matching your driver)
transformers>=4.50.0
peft
accelerate
bitsandbytes   # only if loading in 4-bit

---

Evaluation examples

Qualitatively tested on Amharic STI-related medical questions. Example questions:

1. በግብረ ሥጋ ግንኙነት የሚተላለፉ በሽታዎች ምንድን ናቸው?
2. የግብረ ሥጋ ግንኙነት በሽታዎችን እንዴት መከላከል ይቻላል?
3. የ ኤች አይ ቪ (HIV) ምልክቶች ምንድን ናቸው?
4. የግብረ ሥጋ ግንኙነት በሽታ እንዳለብኝ እንዴት ማወቅ እችላለሁ?
5. ጨብጥ (ጨብጥ በሽታ) እንዴት ይታከማል?
6. ቂጥኝ (ሲፊሊስ) ምንድን ነው እና ምልክቶቹ ምንድን ናቸው?
7. ኮንዶም መጠቀም ከበሽታ ይከላከላል?
8. የግብረ ሥጋ ግንኙነት በሽታ ምርመራ የት ማድረግ እችላለሁ?
9. የግብረ ሥጋ ግንኙነት በሽታዎች ካልታከሙ ምን ችግር ያስከትላሉ?
10. ነፍሰ ጡር ሴት የግብረ ሥጋ ግንኙነት በሽታ ቢኖራት በልጁ ላይ ተጽዕኖ ይኖረዋል?

Full test outputs are provided in sti_qa_results_4b.txt.

---

Limitations and risks

• Training data was clinician-reviewed, but model outputs are not. The dataset was reviewed by a health professional; the model can still generate answers that differ from its training data, so its generated outputs have not been individually validated and may be incorrect.
• Smaller model, lower ceiling. At 4B parameters this model is less capable than the 12B version; expect somewhat weaker reasoning and fluency, especially on complex or rare medical questions.
• Domain-narrow. Medical knowledge comes primarily from the small fine-tuning dataset (~2,300 pairs).
• Treatment/dosage caution. Answers about treatment, medication, or pregnancy should be treated with particular caution.
• Inherits base-model limitations and biases from google/gemma-3-4b-it.

Intended use

Research and educational exploration of bilingual (Amharic/English) medical QA, and lower-cost deployment where the 12B model is too expensive to serve. Not for clinical, diagnostic, or treatment use.

License

Derived from google/gemma-3-4b-it and subject to the Gemma Terms of Use.

Citation

@misc{sewunetie_tilahun_gemma3_4b_amharic_medqa,
  title  = {Gemma 3 4B Bilingual Amharic-English Medical QA (LoRA)},
  author = {Sewunetie, Walelign Tewabe and Tilahun, Surafel L.},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Walelign/Gemma_3_4B}}
}