README.md

---
license: mit
tags:
- visual-navigation
- reinforcement-learning
- factor-graph-attention
library_name: transformers
---

# Target-driven visual navigation on factor graph attention

An agent is told an object to find and acts from egocentric observations in
AI2-THOR. Each grid cell of the observation is scored against the target's word
embedding, the recurrent memory and the last action, by the attention layer of
[Factor Graph Attention](https://arxiv.org/abs/1904.05880) (CVPR 2019). The
attention-weighted map -- not a pooled vector -- enters the recurrence, because
the agent has to know *where* the target is, not only that it is present.

Trained by advantage actor-critic on the SAVN offline dump: every reachable pose
was rendered once and its ResNet18 feature map cached, so no simulator is needed.

Code: [github.com/idansc/fga](https://github.com/idansc/fga).

## Results

Scored on the **published fixed set of 3,914 test episodes**, each pinning the
scene, the target object instance and the start pose. Checkpoints were selected on
the 4,086 val episodes and the winner scored once on test, which is what the
original's `full_eval.py` does.

| | success | SPL |
| --- | --- | --- |
| **this checkpoint** | **0.421** | **0.167** |
| released A2C weights, same protocol | 0.405 | 0.167 |
| released final weights, same protocol | 0.411 | 0.163 |
| published figure | 0.462 | 0.179 |

The released weights were re-scored here rather than taken on trust, and they land
in the same range: the roughly four-point offset to the published 0.462 applies to
them as much as to this model, so it belongs to the measurement rather than to the
training. Their model's SPL reproduces to 0.003, which is what says the
environment is faithful.

Two things worth knowing if you build on this. Val and test disagree — the run
that scored 0.367 on val scored 0.421 on test — so val selection is not a reliable
proxy here even though it is the published protocol. And the agents overfit to
training scenes: train success reaches 0.906 while held-out sits near 0.42, and
more episodes past about one million make that worse rather than better.

## Usage

```python
from fga.tasks.navigation import NavigationPolicy

policy = NavigationPolicy.from_pretrained("Idan/fga-navigation")
out = policy(target_embeds=glove, observation=resnet_grid, hidden_state=state)
action = out.action_logits.argmax(-1)
```

The episodes come from the SAVN offline data:

```bash
curl -O https://prior-datasets.s3.us-east-2.amazonaws.com/savn/data.tar.gz
python scripts/run_navigation.py --data_root data \
    --val_episodes data/val_episodes.json --test_episodes data/test_episodes.json \
    --output_dir models/navigation
```

## Citation

Please cite both the attention and the navigation work:

```bibtex
@inproceedings{schwartz2019factor,
  title={Factor graph attention},
  author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2039--2048},
  year={2019}
}

@inproceedings{mayo2021visual,
  title={Visual Navigation with Spatial Attention},
  author={Mayo, Bar and Hazan, Tamir and Tal, Ayellet},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2021}
}
```

The navigation setting, the offline AI2-THOR data and the episode splits are from
[barmayo/spatial_attention](https://github.com/barmayo/spatial_attention)
([arXiv:2104.09807](https://arxiv.org/abs/2104.09807)).