| # CxREmbed (multi-image / multi-text unified embeddings) | |
| This repository contains **lightweight inference code + trained embedding heads** for a multi-modal CXR embedding model built on top of the base **Lingshu-7B / Qwen2.5-VL** backbone. | |
| The repo is structured to upload only the *delta weights* (LoRA adapter + pooling/projection heads). The base model weights remain in the original upstream repository. | |
| ## What is included | |
| - `lora/` (optional) — PEFT LoRA adapter weights | |
| - `unified_pooler.pt` — pooling head | |
| - `unified_proj.pt` — projection head to the unified embedding space | |
| - `text_proj.pt` / `image_proj.pt` (optional) | |
| - `cxrembed_config.json` — minimal configuration | |
| - `cxrembed/` — small Python package with an inference wrapper | |
| ## Quickstart | |
| ```python | |
| import torch | |
| from cxrembed import CxREmbedder | |
| # Download from the Hub and load the backbone + adapters + heads | |
| m = CxREmbedder.from_pretrained( | |
| "<ORG>/<REPO>", | |
| device="cuda" if torch.cuda.is_available() else "cpu", | |
| amp=True, | |
| ) | |
| # Embed a structured record (multi-image + multi-text) | |
| emb = m.embed_record( | |
| current_img="/path/to/current_frontal.png", | |
| lateral_img="/path/to/lateral.png", | |
| prior_img="/path/to/prior.png", | |
| additional_img=None, | |
| prior_report="...", | |
| current_report="...", | |
| demographics="Age 67, male", | |
| lab_test="WBC 12.3", | |
| history="SOB, fever", | |
| additional_txt="Question: pneumonia?", | |
| instruction="Embed this clinical record for retrieval.", | |
| ) | |
| # Embed a candidate answer (text-only) | |
| ans = m.embed_answer("Right lower lobe consolidation consistent with pneumonia.") | |
| # Similarity in embedding space | |
| score = float((emb @ ans.T).item()) | |
| print(score) | |
| ``` | |
| ## Placeholders supported in templates | |
| Images: | |
| - `<current_image>` (alias of `<frontal_image>`) | |
| - `<lateral_image>` | |
| - `<prior_image>` | |
| - `<additional_image>` | |
| - `<additional_image1>`, `<additional_image2>`, ... if you pass a list to `additional_img` | |
| Texts: | |
| - `<current_report>` (alias `<report>`) | |
| - `<prior_report>` | |
| - `<demographics>` | |
| - `<lab_test>` | |
| - `<history>` | |
| - `<additional_txt>` | |
| ## Notes | |
| - This model is intended for **research** and may require additional validation for clinical use. | |
| - Do not upload protected health information (PHI) to public repositories. | |