README.md

---
license: mit
tags:
- medical
- chest-xray
- visual-question-answering
- difference-vqa
- mimic-cxr
library_name: pytorch
---

# Diff-VQA — Qwen3-0.6B visual-prefix decoder (multiobjective-regsteps encoder)

Online-trained **Difference Visual Question Answering** head for chest X-rays
(MIMIC-CXR / Medical-Diff-VQA). A frozen ViT-L/14 vision encoder produces patch
tokens for a current + reference image pair; a **Qwen3-0.6B** decoder,
conditioned on those tokens as a visual prefix, generates the answer describing
what changed.

This is a **slimmed inference checkpoint**: decoder weights + the frozen vision
encoder weights + architecture metadata (optimizer/scheduler state stripped).

## Contents (`*_best.pt`, `torch.load(..., weights_only=False)`)

| key | description |
|-----|-------------|
| `state_dict` | `DiffVQAQwen3Head` weights (Qwen3-0.6B LLM + `vis_proj` + `frame_emb`) |
| `encoder_state` | frozen ViT-L/14 vision encoder (`objective: multiobjective_caption+regioncontrastive`) |
| `decoder` | `Qwen/Qwen3-0.6B` |
| `tokenizer_name` | `Qwen/Qwen3-0.6B` |
| `vis_dim` | 1024 |
| `encoder_name` | `pretrain-multiobjective-regsteps` |
| `epoch` / `step` / `best_val` | training position + best validation loss |

## Training

- Data: Medical-Diff-VQA `difference` questions over MIMIC-CXR image pairs
  (~100k train / ~16k val / ~16k test).
- Objective: next-token cross-entropy over the answer span (question tokens masked).
- Vision encoder is **frozen**; only the decoder (LLM + visual adapter) is trained.

## Notes

The bundled `encoder_state` is the exact frozen encoder this decoder was trained
against — load both together so the visual features match the space `vis_proj`
expects (a mismatched encoder produces degenerate output).