Update README with model loading code
Browse files
README.md
CHANGED
|
@@ -13,9 +13,16 @@ Part of the [LAPVQA collection](https://huggingface.co/collections/dmusingu/lapv
|
|
| 13 |
|
| 14 |
## Description
|
| 15 |
|
| 16 |
-
RRG
|
| 17 |
-
a
|
| 18 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
## Results (MIMIC-CXR test set, MAE-ViT-L/16)
|
| 21 |
|
|
@@ -23,12 +30,19 @@ a fine-tuned baseline for comparison with the frozen-encoder setup in
|
|
| 23 |
|---|---|---|
|
| 24 |
| 0.032 | 0.164 | 0.195 |
|
| 25 |
|
| 26 |
-
##
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
|
| 14 |
## Description
|
| 15 |
|
| 16 |
+
RRG decoders trained **end-to-end** alongside their vision encoders.
|
| 17 |
+
Each checkpoint is a dict: `{state_dict, vis_dim, d_model, num_layers, nhead, encoder, epoch, val_bleu4}`.
|
| 18 |
+
|
| 19 |
+
| File | Encoder | vis_dim |
|
| 20 |
+
|---|---|---|
|
| 21 |
+
| `clip-vit-l14.pt` | CLIP ViT-L/14 (fine-tuned) | 1024 |
|
| 22 |
+
| `siglip.pt` | SigLIP (fine-tuned) | 1152 |
|
| 23 |
+
| `florence2.pt` | Florence-2 (fine-tuned) | 1024 |
|
| 24 |
+
| `coca.pt` | CoCa (fine-tuned) | 768 |
|
| 25 |
+
| `mae-vit-l16.pt` | MAE ViT-L/16 (fine-tuned) | 1024 |
|
| 26 |
|
| 27 |
## Results (MIMIC-CXR test set, MAE-ViT-L/16)
|
| 28 |
|
|
|
|
| 30 |
|---|---|---|
|
| 31 |
| 0.032 | 0.164 | 0.195 |
|
| 32 |
|
| 33 |
+
## Loading
|
| 34 |
+
|
| 35 |
+
```python
|
| 36 |
+
import torch
|
| 37 |
+
from lapvqa.rrg.heads import ReportGenerationHead
|
| 38 |
+
|
| 39 |
+
ckpt = torch.load("mae-vit-l16.pt", map_location="cpu")
|
| 40 |
+
head = ReportGenerationHead(
|
| 41 |
+
vis_dim = ckpt["vis_dim"],
|
| 42 |
+
d_model = ckpt["d_model"],
|
| 43 |
+
num_layers = ckpt["num_layers"],
|
| 44 |
+
nhead = ckpt["nhead"],
|
| 45 |
+
)
|
| 46 |
+
head.load_state_dict(ckpt["state_dict"])
|
| 47 |
+
head.eval()
|
| 48 |
+
```
|