dmusingu commited on
Commit
1b042c5
·
verified ·
1 Parent(s): d01319b

Update README with model loading code

Browse files
Files changed (1) hide show
  1. README.md +26 -12
README.md CHANGED
@@ -13,9 +13,16 @@ Part of the [LAPVQA collection](https://huggingface.co/collections/dmusingu/lapv
13
 
14
  ## Description
15
 
16
- RRG decoder heads trained **end-to-end** alongside their vision encoders, providing
17
- a fine-tuned baseline for comparison with the frozen-encoder setup in
18
- [`lapvqa-rrg`](https://huggingface.co/dmusingu/lapvqa-rrg).
 
 
 
 
 
 
 
19
 
20
  ## Results (MIMIC-CXR test set, MAE-ViT-L/16)
21
 
@@ -23,12 +30,19 @@ a fine-tuned baseline for comparison with the frozen-encoder setup in
23
  |---|---|---|
24
  | 0.032 | 0.164 | 0.195 |
25
 
26
- ## Files
27
-
28
- | File | Encoder backbone |
29
- |---|---|
30
- | `clip-vit-l14.pt` | CLIP ViT-L/14 (fine-tuned) |
31
- | `siglip.pt` | SigLIP (fine-tuned) |
32
- | `florence2.pt` | Florence-2 (fine-tuned) |
33
- | `coca.pt` | CoCa (fine-tuned) |
34
- | `mae-vit-l16.pt` | MAE ViT-L/16 (fine-tuned) |
 
 
 
 
 
 
 
 
13
 
14
  ## Description
15
 
16
+ RRG decoders trained **end-to-end** alongside their vision encoders.
17
+ Each checkpoint is a dict: `{state_dict, vis_dim, d_model, num_layers, nhead, encoder, epoch, val_bleu4}`.
18
+
19
+ | File | Encoder | vis_dim |
20
+ |---|---|---|
21
+ | `clip-vit-l14.pt` | CLIP ViT-L/14 (fine-tuned) | 1024 |
22
+ | `siglip.pt` | SigLIP (fine-tuned) | 1152 |
23
+ | `florence2.pt` | Florence-2 (fine-tuned) | 1024 |
24
+ | `coca.pt` | CoCa (fine-tuned) | 768 |
25
+ | `mae-vit-l16.pt` | MAE ViT-L/16 (fine-tuned) | 1024 |
26
 
27
  ## Results (MIMIC-CXR test set, MAE-ViT-L/16)
28
 
 
30
  |---|---|---|
31
  | 0.032 | 0.164 | 0.195 |
32
 
33
+ ## Loading
34
+
35
+ ```python
36
+ import torch
37
+ from lapvqa.rrg.heads import ReportGenerationHead
38
+
39
+ ckpt = torch.load("mae-vit-l16.pt", map_location="cpu")
40
+ head = ReportGenerationHead(
41
+ vis_dim = ckpt["vis_dim"],
42
+ d_model = ckpt["d_model"],
43
+ num_layers = ckpt["num_layers"],
44
+ nhead = ckpt["nhead"],
45
+ )
46
+ head.load_state_dict(ckpt["state_dict"])
47
+ head.eval()
48
+ ```