File size: 2,266 Bytes
f88f88f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | # CxREmbed (multi-image / multi-text unified embeddings)
This repository contains **lightweight inference code + trained embedding heads** for a multi-modal CXR embedding model built on top of the base **Lingshu-7B / Qwen2.5-VL** backbone.
The repo is structured to upload only the *delta weights* (LoRA adapter + pooling/projection heads). The base model weights remain in the original upstream repository.
## What is included
- `lora/` (optional) — PEFT LoRA adapter weights
- `unified_pooler.pt` — pooling head
- `unified_proj.pt` — projection head to the unified embedding space
- `text_proj.pt` / `image_proj.pt` (optional)
- `cxrembed_config.json` — minimal configuration
- `cxrembed/` — small Python package with an inference wrapper
## Quickstart
```python
import torch
from cxrembed import CxREmbedder
# Download from the Hub and load the backbone + adapters + heads
m = CxREmbedder.from_pretrained(
"<ORG>/<REPO>",
device="cuda" if torch.cuda.is_available() else "cpu",
amp=True,
)
# Embed a structured record (multi-image + multi-text)
emb = m.embed_record(
current_img="/path/to/current_frontal.png",
lateral_img="/path/to/lateral.png",
prior_img="/path/to/prior.png",
additional_img=None,
prior_report="...",
current_report="...",
demographics="Age 67, male",
lab_test="WBC 12.3",
history="SOB, fever",
additional_txt="Question: pneumonia?",
instruction="Embed this clinical record for retrieval.",
)
# Embed a candidate answer (text-only)
ans = m.embed_answer("Right lower lobe consolidation consistent with pneumonia.")
# Similarity in embedding space
score = float((emb @ ans.T).item())
print(score)
```
## Placeholders supported in templates
Images:
- `<current_image>` (alias of `<frontal_image>`)
- `<lateral_image>`
- `<prior_image>`
- `<additional_image>`
- `<additional_image1>`, `<additional_image2>`, ... if you pass a list to `additional_img`
Texts:
- `<current_report>` (alias `<report>`)
- `<prior_report>`
- `<demographics>`
- `<lab_test>`
- `<history>`
- `<additional_txt>`
## Notes
- This model is intended for **research** and may require additional validation for clinical use.
- Do not upload protected health information (PHI) to public repositories.
|