Duplicate from askyishan/StethoLM

Co-authored-by: Yishan Wang <askyishan@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,131 @@
+---
+language:
+- en
+license: apache-2.0
+tags:
+- audio
+- medical
+- cardiopulmonary
+- auscultation
+- instruction-tuning
+- lora
+- medgemma
+base_model: google/medgemma-4b-it
+datasets:
+- askyishan/StethoBench
+---
+
+# StethoLM
+
+**StethoLM** is the first audio–language model specialized for cardiopulmonary auscultation, capable of performing instruction-driven clinical tasks across the full spectrum of auscultation analysis. It integrates a cardiopulmonary audio encoder with a medical language model backbone, trained on [StethoBench](https://huggingface.co/datasets/askyishan/StethoBench) — a comprehensive benchmark of 77,027 instruction–response pairs from 16,125 labeled recordings.
+
+This work is published in the Transactions on Machine Learning Research (TMLR).
+
+---
+
+## Model Description
+
+StethoLM connects a **COLA audio encoder** (EfficientNet-based, pre-trained on cardiopulmonary sounds via [CaReAQA](https://arxiv.org/abs/2505.01199)) to **MedGemma-4B-IT** via a learned MLP prefix projector. The audio is encoded into a short sequence of prefix tokens that are prepended to the text input of the language model. All components — audio encoder, prefix projector, and language model (via LoRA) — are jointly fine-tuned end-to-end.
+
+**Architecture:**
+- **Audio encoder:** COLA (EfficientNet backbone), pre-trained on cardiopulmonary audio, outputs 1280-dim embeddings; **fine-tuned** during StethoLM training
+- **Prefix projector:** 3-layer MLP mapping audio features to 4 LM prefix tokens
+- **Language model backbone:** [google/medgemma-4b-it](https://huggingface.co/google/medgemma-4b-it) fine-tuned with LoRA (r=8, α=32)
+
+**Training:**
+- **Stage 1:** Supervised fine-tuning (SFT) on StethoBench training split
+- **Stage 2:** Multimodal Direct Preference Optimization (mDPO) with audio degradation-based conditional preference
+
+---
+
+## Intended Use
+
+StethoLM is designed for **research** on AI-assisted cardiopulmonary auscultation. It supports seven clinical task categories:
+
+| Task | Description |
+|------|-------------|
+| **Classification** | Binary normal/abnormal classification |
+| **Identification** | Identifying specific sound types (e.g., wheezing, crackles) |
+| **Report** | Generating a structured auscultation report |
+| **Reasoning** | Explaining clinical findings |
+| **Differential Diagnosis (DDx)** | Listing possible diagnoses |
+| **Comparison** | Comparing findings across recordings |
+| **Location** | Identifying anatomical auscultation site |
+
+> ⚠️ **Not for clinical use.** This model is intended for research purposes only and has not been validated for clinical decision-making.
+
+---
+
+## How to Use
+
+This repository contains the **adapter weights** (fine-tuned audio encoder + LoRA adapters + prefix projector, ~713 MB). The base MedGemma-4B model is downloaded automatically from HuggingFace on first run.
+
+### 1. Clone the code repository
+
+```bash
+git clone https://github.com/askyishan/StethoLM
+cd StethoLM
+pip install -r requirements.txt
+```
+
+### 2. Download the adapter checkpoint
+
+```bash
+huggingface-cli download askyishan/StethoLM stetholm_adapter.pt --local-dir checkpoints/
+```
+
+### 3. Run inference
+
+```bash
+python predict.py \
+    --input_jsonl data/stethobench.jsonl \
+    --output_jsonl predictions.jsonl \
+    --audio_dir /path/to/audio_files \
+    --checkpoint checkpoints/stetholm_adapter.pt \
+    --model_name google/medgemma-4b-it \
+    --audio_encoder cola \
+    --split test
+```
+
+---
+
+## Training Data
+
+StethoLM was trained on [StethoBench](https://huggingface.co/datasets/askyishan/StethoBench). The training split comprises recordings from 7 in-domain datasets; 4 additional datasets are held out as out-of-distribution (OOD) test sets.
+
+**In-domain training datasets:**
+
+| Dataset | Domain |
+|---------|--------|
+| CirCor DigiScope (heart-circor) | Heart |
+| SPRSound (spr) | Lung |
+| COVID-UK (coviduk) | Cough |
+| CoughVid (coughvid) | Cough |
+| ICBHI (icbhi) | Lung |
+| ZCHSound (heart-zch) | Heart |
+| KAUH (kauh) | Cardiopulmonary |
+
+**Out-of-distribution (OOD) test datasets:**
+
+| Dataset | Domain |
+|---------|--------|
+| BMD-HS | Heart |
+| CINC | Cardiopulmonary |
+| TR | Lung |
+| FluSense | Cough |
+
+---
+
+## Citation
+
+If you use StethoLM or StethoBench in your research, please cite:
+
+```bibtex
+@article{stetholm2025,
+  title     = {StethoLM: Audio Language Model for Cardiopulmonary Analysis Across Clinical Tasks},
+  author    = {Wang, Yishan and Wang, Tsai-Ning and Funk, Mathias and Saeed, Aaqib},
+  journal   = {Transactions on Machine Learning Research},
+  year      = {2026},
+  url       = {https://huggingface.co/askyishan/StethoLM}
+}
+```

stetholm_adapter.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d94e0cd7223ba5098ff949c0635114a5ece72bbbcb7d322915d352bb0dfc608
+size 695787446