README.md

---
license: gemma
base_model: google/gemma-3-4b-it
tags:
- medical
- amharic
- english
- question-answering
- lora
- peft
- gemma
language:
- am
- en
library_name: peft
pipeline_tag: text-generation
---

# Gemma 3 4B — Bilingual (Amharic + English) Medical QA

Fine-tuned LoRA adapters on top of `google/gemma-3-4b-it` for medical
question-answering in **Amharic** and **English**. This is the lightweight
counterpart to the 12B model, trained on the same bilingual medical QA dataset
and intended for lower-cost inference.

> ⚠️ **Medical disclaimer.** This model is a research prototype. Its outputs are
> AI-generated and may be incomplete or incorrect. It is **not** a medical device
> and must **not** be used for diagnosis or treatment decisions. Always consult a
> qualified healthcare professional. Outputs should be reviewed by a clinician
> before any downstream use.

---

## Model details

| | |
|---|---|
| **Base model** | `google/gemma-3-4b-it` |
| **Method** | QLoRA (4-bit NF4) supervised fine-tuning |
| **Adapter type** | LoRA (PEFT) |
| **Languages** | Amharic (am), English (en) |
| **Domain** | Medical / health question answering |
| **License** | Gemma (inherits base model terms) |
| **Developed by** | Walelign Tewabe Sewunetie (PhD) & Surafel L. Tilahun (PhD) |

### LoRA configuration

| Hyperparameter | Value |
|---|---|
| Rank (r) | 16 |
| Alpha | 32 |
| Dropout | 0.05 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Quantization | 4-bit NF4, double quant, bf16 compute |

### Training configuration

| Setting | Value |
|---|---|
| Epochs | 3 |
| Total steps | 864 |
| Learning rate | 2e-4 |
| LR scheduler | cosine, warmup ratio 0.03 |
| Per-device batch size | 8 |
| Gradient accumulation | 2 |
| Max sequence length | 1024 |
| Precision | bf16 |
| Gradient checkpointing | enabled |
| Seed | 42 |
| Hardware | 1× NVIDIA H200 GPU |

---

## Dataset

Trained on a bilingual medical QA dataset with parallel English and Amharic
question–answer pairs. Each source row contributes **two** training examples
(one English pair, one Amharic pair).

| | |
|---|---|
| Source rows | ~2,300 |
| Training examples (after bilingual expansion) | ~4,600 |
| Train / validation split | 90 / 10 |
| Columns | English_question, English_answer, Amharic_question, Amharic_answer |

Dataset provenance: content compiled from public health websites/articles,
medical textbooks and guidelines, expert/clinician-written material, and an
existing medical QA dataset. Amharic produced by professional translation.
Medical content reviewed by a qualified health professional. See the dataset
card for full details.

---

## Training results

### Training loss trajectory

Training loss dropped sharply and continued to decline through training:

| Progress | Training loss | Token accuracy |
|---|---|---|
| Start (epoch ~0.03) | 7.24 | 0.345 |
| Epoch ~2.92 | 0.896 | 0.762 |
| Epoch ~2.95 | 0.836 | 0.778 |
| Epoch ~2.99 | 0.902 | 0.762 |

### Per-epoch eval progress

| Epoch | Eval loss | Eval token accuracy |
|---|---|---|
| 1 | 1.4533 | 0.6455 |
| 2 | **1.3772** | **0.6670** |
| 3 | 1.4402 | 0.6672 |

Eval loss reached its minimum at **epoch 2** (1.3772) and rose slightly at
epoch 3 (1.4402), while training loss continued to fall — indicating the onset
of mild overfitting in the final epoch. The end-of-epoch-2 checkpoint
(`checkpoint-576`) may generalize slightly better than the final checkpoint.
The final adapter at the repo root corresponds to epoch 3 (`checkpoint-864`);
users who prefer the best-validation checkpoint can load `checkpoint-576`.

### Comparison with the 12B model

For reference, the larger [Gemma 3 12B version](https://huggingface.co/AiHub4MSRH-Hash/Gemma_3_12B)
trained on the same data reached a best eval loss of 1.224 and accuracy 0.696.
This 4B model trades some quality for substantially smaller size and lower
inference cost.

| Best-epoch metric | This model (4B) | 12B version |
|---|---|---|
| Eval loss | 1.377 | 1.224 |
| Eval token accuracy | 0.667 | 0.696 |

### Checkpoints

| Checkpoint | Corresponds to |
|---|---|
| `checkpoint-576` | End of epoch 2 (best eval) |
| `checkpoint-864` | End of epoch 3 (final — repo root) |

---

## How to use

These are LoRA adapters — load them on top of the base model.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "google/gemma-3-4b-it"
ADAPTER = "Walelign/Gemma_3_4B"

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

question = "የስኳር በሽታ ምልክቶች ምንድን ናቸው?"
msgs = [{"role": "user", "content": question}]
enc = tok.apply_chat_template(
    msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

with torch.no_grad():
    out = model.generate(**enc, max_new_tokens=256, do_sample=False)

print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))
```

### Requirements

```
torch  (CUDA build matching your driver)
transformers>=4.50.0
peft
accelerate
bitsandbytes   # only if loading in 4-bit
```

---

## Evaluation examples

Qualitatively tested on Amharic STI-related medical questions. Example questions:

1. በግብረ ሥጋ ግንኙነት የሚተላለፉ በሽታዎች ምንድን ናቸው?
2. የግብረ ሥጋ ግንኙነት በሽታዎችን እንዴት መከላከል ይቻላል?
3. የ ኤች አይ ቪ (HIV) ምልክቶች ምንድን ናቸው?
4. የግብረ ሥጋ ግንኙነት በሽታ እንዳለብኝ እንዴት ማወቅ እችላለሁ?
5. ጨብጥ (ጨብጥ በሽታ) እንዴት ይታከማል?
6. ቂጥኝ (ሲፊሊስ) ምንድን ነው እና ምልክቶቹ ምንድን ናቸው?
7. ኮንዶም መጠቀም ከበሽታ ይከላከላል?
8. የግብረ ሥጋ ግንኙነት በሽታ ምርመራ የት ማድረግ እችላለሁ?
9. የግብረ ሥጋ ግንኙነት በሽታዎች ካልታከሙ ምን ችግር ያስከትላሉ?
10. ነፍሰ ጡር ሴት የግብረ ሥጋ ግንኙነት በሽታ ቢኖራት በልጁ ላይ ተጽዕኖ ይኖረዋል?

Full test outputs are provided in [`sti_qa_results_4b.txt`](./sti_qa_results_4b.txt).

---

## Limitations and risks

- **Training data was clinician-reviewed, but model outputs are not.** The
  *dataset* was reviewed by a health professional; the model can still generate
  answers that differ from its training data, so its *generated outputs* have not
  been individually validated and may be incorrect.
- **Smaller model, lower ceiling.** At 4B parameters this model is less capable
  than the 12B version; expect somewhat weaker reasoning and fluency, especially
  on complex or rare medical questions.
- **Domain-narrow.** Medical knowledge comes primarily from the small
  fine-tuning dataset (~2,300 pairs).
- **Treatment/dosage caution.** Answers about treatment, medication, or
  pregnancy should be treated with particular caution.
- **Inherits base-model limitations and biases** from `google/gemma-3-4b-it`.

## Intended use

Research and educational exploration of bilingual (Amharic/English) medical QA,
and lower-cost deployment where the 12B model is too expensive to serve. Not for
clinical, diagnostic, or treatment use.

## License

Derived from `google/gemma-3-4b-it` and subject to the
[Gemma Terms of Use](https://ai.google.dev/gemma/terms).

## Citation

```bibtex
@misc{sewunetie_tilahun_gemma3_4b_amharic_medqa,
  title  = {Gemma 3 4B Bilingual Amharic-English Medical QA (LoRA)},
  author = {Sewunetie, Walelign Tewabe and Tilahun, Surafel L.},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Walelign/Gemma_3_4B}}
}
```