dmusingu commited on
Commit
152ffba
·
verified ·
1 Parent(s): e0b7155

Update README with model loading code

Browse files
Files changed (1) hide show
  1. README.md +12 -7
README.md CHANGED
@@ -15,13 +15,18 @@ Part of the [LAPVQA collection](https://huggingface.co/collections/dmusingu/lapv
15
 
16
  VQA task head trained on top of the **LAPVQA captioning-pretrained encoder**
17
  ([`lapvqa-pretrain-captioning`](https://huggingface.co/dmusingu/lapvqa-pretrain-captioning)).
18
- The encoder is kept frozen; this checkpoint contains the task head weights only.
 
19
 
20
- This variant tests whether a generative pretraining objective on MIMIC-CXR
21
- produces better representations for VQA than off-the-shelf encoders.
22
 
23
- ## Files
 
 
24
 
25
- | File | Description |
26
- |---|---|
27
- | `pretrain-captioning_best.pt` | VQA head weights (encoder not included) |
 
 
 
 
15
 
16
  VQA task head trained on top of the **LAPVQA captioning-pretrained encoder**
17
  ([`lapvqa-pretrain-captioning`](https://huggingface.co/dmusingu/lapvqa-pretrain-captioning)).
18
+ The encoder is kept frozen; this checkpoint contains the `VQAHead` state dict only.
19
+ The encoder outputs 1024-dim patch tokens (ViT-L/14).
20
 
21
+ ## Loading
 
22
 
23
+ ```python
24
+ import torch
25
+ from lapvqa.vqa.model import VQAHead
26
 
27
+ ckpt = torch.load("pretrain-captioning_best.pt", map_location="cpu")
28
+ head = VQAHead(vis_dim=1024)
29
+ head.load_state_dict(ckpt)
30
+ head.eval()
31
+ # pair with encoder_final.pt from lapvqa-pretrain-captioning
32
+ ```