Visual Question Answering
Transformers
Safetensors
fga
visual-dialog
visdial
multimodal
attention
factor-graph-attention
Eval Results (legacy)
Instructions to use Idan/fga with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Idan/fga with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("visual-question-answering", model="Idan/fga")# Load model directly from transformers import FGAForVisualDialog model = FGAForVisualDialog.from_pretrained("Idan/fga", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Add Factor Graph Attention weights (VisDial v1.0, MRR 66.01)
Browse files
README.md
CHANGED
|
@@ -141,12 +141,14 @@ from fga import FactorGraphAttention, Modality
|
|
| 141 |
attention = FactorGraphAttention(embed_dims=[512, 2048], num_entities=[20, 36])
|
| 142 |
pooled_text, pooled_image = attention(text, image)
|
| 143 |
|
| 144 |
-
# or declared by name, with
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
Modality("
|
| 149 |
-
],
|
|
|
|
|
|
|
| 150 |
```
|
| 151 |
|
| 152 |
The same layer backs the paper's follow-ups in
|
|
|
|
| 141 |
attention = FactorGraphAttention(embed_dims=[512, 2048], num_entities=[20, 36])
|
| 142 |
pooled_text, pooled_image = attention(text, image)
|
| 143 |
|
| 144 |
+
# or declared by name, with weight sharing stated separately
|
| 145 |
+
history = [Modality(f"history_{i}", dim=128, size=21, connected_to=("text", "image"))
|
| 146 |
+
for i in range(1, 10)]
|
| 147 |
+
attention = FactorGraphAttention.from_modalities(
|
| 148 |
+
[Modality("text", dim=512, size=20), Modality("image", dim=2048, size=36), *history],
|
| 149 |
+
share_weights=[[m.name for m in history]],
|
| 150 |
+
use_prior=True,
|
| 151 |
+
)
|
| 152 |
```
|
| 153 |
|
| 154 |
The same layer backs the paper's follow-ups in
|