README.md

---
frameworks:
- ""
language:
- en
license: apache-2.0
tags:
- OneScience
- medical multimodal large language model
- interactive inference
---
<p align="center">
  <strong>
    <span style="font-size: 30px;">MedGemma</span>
  </strong>
</p>

# Model Overview

MedGemma is an open-source medical multimodal large language model from Google. It is based on the [Gemma 3](https://ai.google.dev/gemma/docs/core) architecture and is trained for medical text and medical image understanding. MedGemma provides two variants:

- **MedGemma 4B**: a multimodal model that supports joint input of medical text and medical images.
- **MedGemma 27B**: a text-only model focused on medical text understanding and question answering.

# Model Description

MedGemma 4B uses an image encoder and has been pretrained on multiple types of de-identified medical data, including chest X-rays (CXR), dermatology images, ophthalmology images, and histopathology slides. Its language model component has been trained on radiology images, pathology images, ophthalmology images, dermatology images, and medical text.

Weights and datasets are not available at the moment. They will be uploaded to Hugging Face soon, and command-line downloads will be supported later.

# Use Cases

| Use case | Description |
| :---: | :---: |
| Medical question answering | Evaluates the model's question-answering capability on medical knowledge benchmarks such as MedQA. |
| Medical image analysis | Supports tasks such as anatomical localization on chest X-rays (CXR) and longitudinal comparison of multi-timepoint images. |
| Domain fine-tuning | Performs parameter-efficient fine-tuning with LoRA on datasets such as NCT colon histopathology images. |
| Unified inference | Provides interactive and batch-file inference through `MedicalInferenceRunner`. |

# Use and Limitations

MedGemma is a foundation model for developing healthcare AI applications and is intended to serve as a starting point for downstream development, adaptation, and validation. Developers should fully validate, adapt, and meaningfully modify the model for their specific use case.

Outputs generated by MedGemma are not intended to directly inform clinical diagnosis, patient management decisions, treatment recommendations, or any other direct clinical practice. All model outputs should be considered preliminary and require independent verification, clinical correlation, and further investigation.

# Usage

## 1. Using OneCode

You can try intelligent one-click AI4S programming through the OneCode online environment:

[Try intelligent one-click AI4S programming](https://web-2069360198568017922-iaaj.ksai.scnet.cn:58043/home)

## 2. Manual Installation and Usage

**Hardware Requirements**

- Running on a GPU or DCU is recommended.
- CPU can be used for connectivity checks, but it is relatively slow.
- DCU users need to install DTK in advance. DTK 25.04.2 or later is recommended, or the OneScience-recommended version that matches the current cluster.

**Software Requirements**

DCU users who want to learn more about adaptation details can contact liubiao@sugon.com.

**Environment Checks**

- NVIDIA GPU:

```bash
nvidia-smi
```

- Hygon DCU:

```bash
hy-smi
```

### Environment Preparation

1. Check the `botocore` version. If the version is too old, upgrade it:

    ```bash
    pip install --upgrade boto3==1.43.36 botocore==1.43.36
    ```

2. Check the `transformers` version. If the version is too old, upgrade it:

    ```bash
    pip install --upgrade transformers==5.12.1
    ```

## Quick Start

### 1. Install the Runtime Environment

```bash
conda create -n onescience311 python=3.11 -y
conda activate onescience311
pip install onescience[bio] -i http://mirrors.onescience.ai:3141/pypi/simple/  --trusted-host mirrors.onescience.ai
```

If the following code cannot find required libraries at runtime, activate CUDA as shown below.

```bash
source ${ROCM_PATH}/cuda/env.sh
export LD_LIBRARY_PATH="$CONDA_PREFIX/lib:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH="$CONDA_PREFIX/lib/python3.11/site-packages/fastpt/torch/lib:$LD_LIBRARY_PATH"
```

### 2. Download the Model Package

```bash
# By default, the package is downloaded to the model folder under the current path. To change this, adjust the path after local_dir.
hf download --model OneScience-Sugon/Medgemma --local-dir ./model
cd model
```

### Training Weights and Datasets

Weights and datasets are not provided at the moment. They will be uploaded to Hugging Face soon, and command-line downloads will be supported later.

### 3. Usage Modes

**Data and Model Weights**

By default, the scripts load the model from:

```text
${ONESCIENCE_DATASETS_DIR}/medgemma/modelscope/google/medgemma-1.5-4b-it
```

By default, the scripts load datasets from the following paths:

| Task | Data | Default path |
|------|------|----------|
| MedQA evaluation | MedQA parquet data | `${ONESCIENCE_DATASETS_DIR}/medgemma/medqa` |
| Chest X-ray anatomical localization | Chest X-ray images | `${ONESCIENCE_DATASETS_DIR}/medgemma/Chest_Xray/...` |
| Chest X-ray longitudinal comparison | Two chest X-rays from before and after | `${ONESCIENCE_DATASETS_DIR}/medgemma/test_compare/...` |
| Pathology image fine-tuning | NCT-CRC-HE-100K / CRC-VAL-HE-7K | `${ONESCIENCE_DATASETS_DIR}/medgemma/nct/...` |

Download the model in advance and place it in this directory, or override it through the `model_path` environment variable.

### Detailed Usage

#### 1. Integration Test

Verify whether MedGemma modules, configurations, data adapters, and image processing components can be imported correctly in OneScience:

```bash
python tests/test_integration.py
```

---

#### 2. Medical Question Answering Evaluation (`run_evaluate_on_medqa.sh`)

Evaluate the model's medical question-answering capability on the MedQA dataset. By default, 10 samples are processed for quick validation.

```bash
bash scripts/run_evaluate_on_medqa.sh
```

Output:

- `scripts/medqa_results/medqa_results.json`: detailed results for each sample
- `scripts/medqa_results/summary.txt`: summary metrics such as accuracy

---

#### 3. Chest X-Ray Anatomical Localization (`run_cxr_anatomy.sh`)

Perform anatomical localization on one or more chest X-rays. The script runs both single-image mode and batch mode internally:

```bash
bash scripts/run_cxr_anatomy.sh
```

Output:

- `scripts/outputs/result_*.json`: localization coordinates and labels
- `scripts/outputs/result_*.png`: visualization images with bounding-box annotations
- `scripts/outputs/batch_summary.json`: summary results for batch mode

---

#### 4. Chest X-Ray Longitudinal Comparison (`run_cxr_longitudinal_comparison.sh`)

Compare two chest X-rays from the same patient taken at different timepoints:

```bash
bash scripts/run_cxr_longitudinal_comparison.sh
```

Output:

- `scripts/compare_outputs/compare_<image1>_vs_<image2>.txt`: text comparison report
- `scripts/compare_outputs/compare_<image1>_vs_<image2>.json`: structured JSON results

---

#### 5. Pathology Image LoRA Fine-Tuning (`run_fine_tune.sh`)

Perform LoRA fine-tuning on the NCT colon histopathology image dataset:

```bash
bash scripts/run_fine_tune.sh
```

Output:

- `scripts/medgemma-nct-lora/`: LoRA weights, training logs, and evaluation results

> Note: The script automatically checks and fixes the `boto3==1.43.36` and `botocore==1.43.36` versions to avoid dependency conflicts.

---

#### 6. Using the Inference Runner

`runner/medical_inference_runner.py` provides a unified inference entry point and supports both interactive and batch-file inference.

##### Interactive Inference

```bash
export PYTHONPATH=../../../src:$PYTHONPATH
python runner/medical_inference_runner.py \
    --config configs/inference_config.yaml \
    --interactive
```

##### Batch-File Inference

```bash
export PYTHONPATH=../../../src:$PYTHONPATH
python runner/medical_inference_runner.py \
    --config configs/inference_config.yaml \
    --input data/example_input.json
```

### Notes

- Make sure the `ONESCIENCE_DATASETS_DIR` environment variable is correctly set before running the scripts.
- The scripts use `HIP_VISIBLE_DEVICES=0` by default and can run directly on Hygon DCU platforms. On CUDA platforms, replace it with `CUDA_VISIBLE_DEVICES=0` or adjust it according to the available devices.
- To use vLLM for accelerated inference, make sure the corresponding version of vLLM is installed and configure `use_vllm: true`.
- The chest X-ray anatomical localization and pathology fine-tuning scripts automatically fix `boto3` / `botocore` versions to avoid dependency conflicts.
- The 4B multimodal model requires substantial GPU memory for inference. At least 24 GB of memory on a single device is recommended. Multi-device execution can be controlled through `num_gpus` or an external `CUDA_VISIBLE_DEVICES` setting.

# Official OneScience Information

| Platform | OneScience main repository | Skills repository |
| --- | --- | --- |
| Gitee | https://gitee.com/onescience-ai/onescience | https://gitee.com/onescience-ai/oneskills |
| GitHub | https://github.com/onescience-ai/OneScience | https://github.com/onescience-ai/oneskills |

# Citation and License

The MedGemma model is licensed under the [Health AI Developer Foundations License](https://developers.google.com/health-ai-developer-foundations/terms), and the example code in this repository is licensed under Apache 2.0.

For more information, see:

- [Developer documentation](https://developers.google.com/health-ai-developer-foundations/medgemma/get-started)
- [Model card](https://developers.google.com/health-ai-developer-foundations/medgemma/model-card)
- [Community guidelines](https://developers.google.com/health-ai-developer-foundations/community-guidelines)
- [Hugging Face](https://huggingface.co/models?other=medgemma)
- [Google Model Garden](https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/medgemma)