README.md

---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- medical
- ecg
- question-answering
- multimodal
- pytorch
---


<div align="center" style="font-size: 1.5em;">
  <strong>Q-HEART: ECG Question Answering via Knowledge-Informed Multimodal LLMs (ECAI 2025)</strong>
</div>

<div align="center">
<a href="https://github.com/manhph2211/Q-HEART/"><img src="https://img.shields.io/badge/Website-QHEART WebPage-blue?style=for-the-badge"></a>
<a href="https://arxiv.org/pdf/2505.06296"><img src="https://img.shields.io/badge/arxiv-Paper-red?style=for-the-badge"></a>
<a href="https://huggingface.co/Manhph2211/Q-HEART"><img src="https://img.shields.io/badge/Checkpoint-%F0%9F%A4%97%20Hugging%20Face-White?style=for-the-badge"></a>
</div>

<div align="center">
  <a href="https://github.com/manhph2211/" target="_blank">Hung&nbsp;Manh&nbsp;Pham</a> &emsp;
  <a href="" target="_blank">Jialu&nbsp;Tang</a> &emsp;
  <a href="https://aqibsaeed.github.io/" target="_blank">Aaqib&nbsp;Saeed</a> &emsp;
  <a href="https://www.dongma.info/" target="_blank">Dong&nbsp;Ma</a> &emsp;
</div>
<br>

## Usage

After we have access to [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) model and install suitable transformers package version, we can run:

```python
from transformers import AutoModel
model = AutoModel.from_pretrained("Manhph2211/Q-HEART", trust_remote_code=True, dtype="auto")
```

Or 

```bash
git clone https://github.com/manhph2211/Q-HEART.git && cd Q-HEART
conda create -n qheart python=3.9
conda activate qheart
pip install torch --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt
```

Download the checkpoint from [here](https://huggingface.co/Manhph2211/Q-HEART) and place it at `ckpts/pytorch_model.bin`, then run evaluation:

```bash
python main.py --model_type meta-llama/Llama-3.2-1B-Instruct --mapping_type Transformer
```

## Citation

```bibtex
@inproceedings{pham2025qheart,
  title     = {Q-HEART: ECG Question Answering via Knowledge-Informed Multimodal LLMs},
  author    = {Pham, Hung Manh and Tang, Jialu and Saeed, Aaqib and Ma, Dong},
  booktitle = {Proceedings of the European Conference on Artificial Intelligence (ECAI)},
  series    = {Frontiers in Artificial Intelligence and Applications},
  volume    = {413},
  pages     = {4545--4552},
  year      = {2025},
  publisher = {IOS Press},
  doi       = {10.3233/FAIA251356}
}
```

<div align="center">
  Please refer to our <a href="https://github.com/manhph2211/Q-HEART">GitHub repo</a> for more details!
</div>