---
license: gemma
base_model: google/medgemma-1.5-4b-it
library_name: peft
pipeline_tag: image-text-to-text
tags:
- medgemma
- dermatology
- qlora
- build-small-hackathon
---

# Chhaya-MedGemma (LoRA)

A QLoRA fine-tune of [`google/medgemma-1.5-4b-it`](https://huggingface.co/google/medgemma-1.5-4b-it)
for **Chhaya**, a skin & heat-health companion for outdoor workers. It reads a skin
photo and emits a structured findings JSON **directly** — no chain-of-thought
preamble — with a `concern` level calibrated against real clinical labels.

## Usage

```python
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

base = "google/medgemma-1.5-4b-it"
proc = AutoProcessor.from_pretrained(base)
model = AutoModelForImageTextToText.from_pretrained(base, torch_dtype=torch.bfloat16).to("cuda")
model = PeftModel.from_pretrained(model, "CodingBad02/chhaya-medgemma-lora-v2").merge_and_unload()

messages = [{"role": "user", "content": [
    {"type": "image", "image": img},          # image BEFORE text
    {"type": "text", "text": "skin check"},
]}]
inputs = proc.apply_chat_template(messages, add_generation_prompt=True,
    tokenize=True, return_dict=True, return_tensors="pt").to(model.device, dtype=torch.bfloat16)
out = model.generate(**inputs, max_new_tokens=400, do_sample=False)
print(proc.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
```

Output schema:
```json
{"what_i_see","spot":{"type","color","borders","symmetry","texture"},
 "heat_sun_signals":[],"concern":"low|watch|see_doctor","concern_reason",
 "image_quality":"good|limited","summary"}
```

## Training

- Data: [`CodingBad02/chhaya-skin-extract`](https://huggingface.co/datasets/CodingBad02/chhaya-skin-extract)
  (ISIC-2024 biopsy-anchored + SCIN real-photo, 1,406 rows). `see_doctor` class
  oversampled 3× to improve recall.
- QLoRA (4-bit nf4, r=16, α=32), **frozen vision tower** (only the language model
  adapts), 2 epochs, A100. ~$6 of compute.

## Eval (141-image held-out test set, vs base)

| metric | base | this model (v2) |
|---|---|---|
| Valid JSON | 0.993 | **1.0** |
| Concern accuracy | 0.333 | **0.695** |
| Malignant recall | 0.936 | 0.83 |
| Output tokens/answer | 770 | **156** |

Base's higher recall is achieved by labelling 60% of cases "watch" (33% accuracy);
this model gives a real triage at 5× fewer tokens. Residual under-warning risk is
mitigated by a deterministic ABCDE backstop in the app.

## Limitations

**Not a medical device. Does not diagnose.** A research/education demo built for a
hackathon. Misses some malignant-type lesions (recall 0.83). Always pair with
clinician review. No heat-rash/sunburn class in training (ISIC/SCIN gap).

Inspired by [Sunny](https://github.com/mrdbourke/sunny) by Daniel Bourke.