README.md

---

license: creativeml-openrail-m
---

# Stable Control Representations: Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Control

[Paper Link](https://arxiv.org/abs/2405.05852)

This model repo provides the Stable Diffusion model finetuned on the images from the Something-Something-v2, Epic Kitchen and the Bridge V2 datasets. This repo is related to the [stable-control-representations](https://github.com/ykarmesh/stable-control-representations/) GitHub repo. 

## Abstract

Vision- and language-guided embodied AI requires a fine-grained understanding of the physical world through language and visual inputs. Such capabilities are difficult to learn solely from task-specific data, which has led to the emergence of pre-trained vision-language models as a tool for transferring representations learned from internet-scale data to downstream tasks and new domains. However, commonly used contrastively trained representations such as in CLIP have been shown to fail at enabling embodied agents to gain a sufficiently fine-grained scene understanding—a capability vital for control. To address this shortcoming, we consider representations from pre-trained text-to-image diffusion models, which are explicitly optimized to generate images from text prompts and as such, contain text-conditioned representations that reflect highly fine-grained visuo-spatial information. Using pre-trained text-to-image diffusion models, we construct Stable Control Representations which allow learning downstream control policies that generalize to complex, open-ended environments. We show that policies learned using Stable Control Representations are competitive with state-of-the-art representation learning approaches across a broad range of simulated control settings, encompassing challenging manipulation and navigation tasks. Most notably, we show that Stable Control Representations enable learning policies that exhibit state-of-the-art performance on OVMM, a difficult open-vocabulary navigation benchmark


## Citing SCR
If you use SCR in your research, please cite [the following paper](https://arxiv.org/abs/2405.05852):

```bibtex

@inproceedings{gupta2024scr,

      title={Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Control},

      author={Gunshi Gupta and Karmesh Yadav and Yarin Gal and Dhruv Batra and Zsolt Kira and Cong Lu and Tim G. J. Rudner},

      year={2024},

      eprint={2405.05852},

      archivePrefix={arXiv},

      primaryClass={cs.CV}

}

```

## Acknowledgements
We are thankful to the creators of [Stable Diffusion](https://huggingface.co/runwayml/stable-diffusion-v1-5) for releasing the model, which has significantly contributed to the progress in the field. Additionally, we extend our thanks to the authors of [Visual Cortex](https://github.com/facebookresearch/eai-vc) for releasing the code for CortexBench evaluations.