Idan commited on
Commit
fba45b0
·
verified ·
1 Parent(s): 2c5986f

Add Factor Graph Attention weights (VisDial v1.0, MRR 66.01)

Browse files
Files changed (1) hide show
  1. README.md +8 -6
README.md CHANGED
@@ -141,12 +141,14 @@ from fga import FactorGraphAttention, Modality
141
  attention = FactorGraphAttention(embed_dims=[512, 2048], num_entities=[20, 36])
142
  pooled_text, pooled_image = attention(text, image)
143
 
144
- # or declared by name, with a repeated modality sharing factor weights
145
- attention = FactorGraphAttention.from_modalities([
146
- Modality("text", dim=512, size=20),
147
- Modality("image", dim=2048, size=36),
148
- Modality("history", dim=128, size=21, repeats=9, connected_to=("text", "image")),
149
- ], use_prior=True)
 
 
150
  ```
151
 
152
  The same layer backs the paper's follow-ups in
 
141
  attention = FactorGraphAttention(embed_dims=[512, 2048], num_entities=[20, 36])
142
  pooled_text, pooled_image = attention(text, image)
143
 
144
+ # or declared by name, with weight sharing stated separately
145
+ history = [Modality(f"history_{i}", dim=128, size=21, connected_to=("text", "image"))
146
+ for i in range(1, 10)]
147
+ attention = FactorGraphAttention.from_modalities(
148
+ [Modality("text", dim=512, size=20), Modality("image", dim=2048, size=36), *history],
149
+ share_weights=[[m.name for m in history]],
150
+ use_prior=True,
151
+ )
152
  ```
153
 
154
  The same layer backs the paper's follow-ups in