---
license: other
language:
  - en
tags:
  - vision-and-language-navigation
  - embodied-ai
  - pytorch
---

# User-Feedback-Driven Adaptation for Vision-and-Language Navigation

Data and checkpoint for the paper *User-Feedback-Driven Adaptation for Vision-and-Language Navigation*, IEEE Transactions on Multimedia, 2026.

[[Paper (IEEE TMM)](https://doi.org/10.1109/TMM.2026.3724731)] [[arXiv](https://arxiv.org/abs/2512.10322)] [[Code](https://github.com/seaotter999/UFD)]

## Download

```bash
huggingface-cli download Peachilk/UFD --local-dir data
```

This gives the `data/` directory used by the scripts in the [code repository](https://github.com/seaotter999/UFD). Some files come from other projects and are not included here: the connectivity graphs and CLIP features (ScaleVLN), `prevalent_aug_train_enc.json` (GSA-R2R) and the initial GR-DUET checkpoint `best_val_unseen` (GSA-VLN). The code README explains where to get them.

## Contents

| Path | Description |
|---|---|
| `ckpts/best_overall_iter_23500` | GR-DUET fine-tuned on paths reconstructed from user feedback in the Basic setting (`rec_basic`); the checkpoint with the best average validation SPL (iteration 23,500). It is also the model evaluated with the memory bank. |
| `split_basic/split_test_500_<scan>.json` | 500 instructions per scan on which user feedback is collected; input to path reconstruction (24 scans, 12,000 entries). |
| `split_mixed/split_test_500_<scan>.json` | Same for the mixed user styles (21 scans, 10,500 entries). |
| `rec_basic/merged_train_dataset.json` | Training set of reconstructed paths with 5 to 7 viewpoints (24 scans, 11,342 entries). |
| `rec_mixed/merged_train_dataset.json` | Same for the mixed user styles (21 scans, 9,766 entries). |
| `rec_basic/merged_500_test_dataset.json`, `rec_mixed/merge_500_test_dataset_mixed.json` | The 500 instructions of all scans in one file, used to build the memory bank. |
| `GSA_Dataset/Validation/Residential/Reconstruction/validation_set_100.json` | Validation set: up to 100 reconstructed paths per scan (24 scans, 2,390 entries). |
| `GSA_Dataset/Test/Residential/Reconstruction/merged_test_dataset.json` | Test set: 100 held-out instructions per scan (24 scans, 2,400 entries). |
| `GSA_Dataset/Test/Residential/Recmixed/merged_test_dataset.json` | Test set for the mixed user styles (21 scans, 2,100 entries). |
| `GSA_Dataset/Train/R2R_train_enc.json` | R2R training instructions with BERT token ids. |
| `scanvp_cands_relangles_with_habitat.json` | Candidate viewpoints and relative angles of every viewpoint. |
| `pano_inputs_habitats_more_scan.h5` | Precomputed panorama inputs of every viewpoint, used to build full graphs during training. |

`basic` uses the Basic instructions of GSA-R2R; `mixed` combines instructions in five user styles (child, keith, moira, rachel, sheldon).

## License

The data and checkpoint are for non-commercial research use only. Please also follow the terms of use of the original datasets: R2R, Matterport3D, HM3D and GSA-R2R. The code is released under the [MIT License](https://github.com/seaotter999/UFD/blob/main/LICENSE).

## Citation

```bibtex
@article{yu2026userfeedback,
  title   = {User-Feedback-Driven Adaptation for Vision-and-Language Navigation},
  author  = {Yu, Yongqiang and Li, Xuhui and Mahmood, Hazza and Zhou, Jinxing and Hong, Haodong and Jiang, Longtao and Xu, Zhiqiang and Wu, Qi and Chang, Xiaojun},
  journal = {IEEE Transactions on Multimedia},
  year    = {2026},
  doi     = {10.1109/TMM.2026.3724731}
}
```