README.md

---
license: mit
tags:
- audio-visual-scene-aware-dialog
- video-question-answering
- factor-graph-attention
- multimodal
library_name: transformers
---

# Audio-Visual Scene-Aware Dialog, DSTC7

Answer a question about a video in words, given the dialog so far. Six modalities
are attended jointly -- the question, four VGG19 conv grids of the video, and the
audio -- and an LSTM decoder writes the answer. Built on the attention layer of
[Factor Graph Attention](https://arxiv.org/abs/1904.05880) (CVPR 2019); a
reimplementation of [A Simple Baseline for Audio-Visual Scene-Aware
Dialog](https://arxiv.org/abs/1904.05876) (CVPR 2019).

Code: [github.com/idansc/fga](https://github.com/idansc/fga).

## Results

DSTC7 test set, scored against six references with the organisers'
`dstc7avsd_eval` metrics:

| | this model | organisers' I3D baseline |
| --- | --- | --- |
| BLEU-1 | 0.651 | 0.621 |
| BLEU-4 | 0.348 | 0.305 |
| METEOR | 0.233 | 0.217 |
| ROUGE_L | 0.510 | 0.481 |
| CIDEr | **0.887** | 0.733 |

These are **not** comparable to Table 1 of the paper, which reports CIDEr against
a single reference on the validation set. Six references and one reference put
CIDEr on different scales; the baseline column above is the anchor for this one.

## Seeds, and which checkpoint this is

The published weights widen the decoder's output projection to 1024 from the
released configuration's 128; everything else follows the original. Seeds:

| projection | CIDEr per seed |
| --- | --- |
| 128 (released config, 8.36M params) | 0.812, 0.813, 0.814 |
| 512 | 0.836, 0.843, 0.847, 0.884 |
| 1024 (this repo) | 0.839, 0.866, **0.887** |

The published checkpoint is the best seed at 1024. Read the spread before the
best number: seed-to-seed variation at these widths runs to ~0.05 CIDEr.

## The projection width is not a detail

At the released width of 128, zeroing the video features costs nothing
measurable -- the model answers from text alone and no training metric notices.
At 512, zeroing the video costs about 0.12 CIDEr, and a text-only model becomes
unstable across seeds (0.809 vs 0.659). The narrow projection after the attended
representation was the reason the audio-visual signal could not express itself;
widening the layer where the reduction happens is what let the video matter.

## Faithful configuration

For the released configuration exactly -- projection 128, 8,359,107 parameters,
matching the paper's reported count -- the numbers are CIDEr 0.813 +/- 0.001 over
three seeds, still above the challenge baseline on every metric. Word embeddings
of 128 with a separate table per consumer, a single-layer question LSTM of 256, a
hierarchical history encoder reading each question-answer pair before reading the
turns in order, a 256-wide decoder. Adam at 1e-3, batch 64, dropout 0.5, He
initialization except recurrent weights which get Xavier, beam width 3.

Video features are four equally-spaced frames per clip through VGG19's last conv
layer, 7x7 = 49 regions of 512 -- what the original release calls
`i3d_rgb_vgg19_4`. Audio is VGGish, 32 steps sampled evenly across the clip.

## Files

`pred_q1024_s3.json` holds the generated answer for all 1,710 test questions,
keyed `{video}_{turn}`.

## Citation

```bibtex
@inproceedings{schwartz2019simple,
  title={A Simple Baseline for Audio-Visual Scene-Aware Dialog},
  author={Schwartz, Idan and Schwing, Alexander G and Hazan, Tamir},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year={2019}
}

@inproceedings{schwartz2019factor,
  title={Factor graph attention},
  author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2039--2048},
  year={2019}
}

@article{alamri2019audio,
  title={Audio Visual Scene-Aware Dialog},
  author={Alamri, Huda and Cartillier, Vincent and Das, Abhishek and Wang, Jue and
          Cherian, Anoop and Essa, Irfan and Batra, Dhruv and Marks, Tim K and
          Hori, Chiori and Anderson, Peter and Lee, Stefan and Parikh, Devi},
  journal={CVPR},
  year={2019}
}
```