README.md

---
license: apache-2.0
---

<p align="center">
    <img src="logo.png" width="150" style="margin-bottom: 0.2;"/>
<p>
<h2 align="center"> <a href="http://arxiv.org/abs/2410.06234">TEOChat: Large Language and Vision Assistant for Temporal Earth Observation Data</a></h2>
<h5 align="center"> If you like our project, please give us a star ⭐ on <a href="https://github.com/ermongroup/TEOChat">Github</a> for the latest updates. </h5>


## 😮 Highlights 

**TEOChat** is the first language and vision assistant that can engage in conversation about sequences of temporal earth observation imagery, and exhibits impressive performance on multiple temporal instruction-following tasks.

### 📚 TEOChatlas: A new instruction-following dataset for temporal EO data
We introduce a new instruction-following dataset for temporal EO data called **TEOChatlas** which we use to train TEOChat. TEOChatlas contains 554,071 examples spanning dozens of temporal instruction-following tasks.

### 🤖 TEOChat: A new vision-language model for temporal EO data
We design TEOChat to use a LLaVA-style architecture, combining a temporally shared vision encoder with a LLaMA 2 LLM connected through an MLP vision-language projector

## 🤗 Demo

### Gradio Web UI

We provide an [online demo](https://huggingface.co/spaces/jirvin16/TEOChat) in Huggingface Spaces.

You can also run the demo locally by running the following command:
```bash
python videollava/serve/teochat_demo.py 
```


## 🛠️ Requirements and Installation
* Python >= 3.9
* Pytorch == 2.2.1
* CUDA Version >= 12.1
* Install required packages:
```bash
git clone https://github.com/ermongroup/TEOChat.git
cd TEOChat
conda create -n teochat python=3.9 -y
conda activate teochat
pip install --upgrade pip  # enable PEP 660 support
pip install -r requirements.txt
```

## 🗝️ Training & Validating
The training & validating instructions are in [TRAIN_AND_VALIDATE.md](https://github.com/ermongroup/TEOChat/blob/main/TRAIN_AND_VALIDATE.md).

## 👍 Acknowledgement
* [Video-LLaVA](https://github.com/PKU-YuanGroup/Video-LLaVA) The codebase and model we built upon.
* [GeoChat](https://github.com/mbzuai-oryx/geochat) The single image instruction-following dataset we included in TEOChatlas.

## 🔒 License
* The majority of this project is released under the Apache 2.0 license as found in the [LICENSE](https://github.com/ermongroup/TEOChat/blob/main/LICENSE) file.
* The service is a research preview intended for non-commercial use only, subject to the model [License](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md) of LLaMA, [Terms of Use](https://openai.com/policies/terms-of-use) of the data generated by OpenAI, and [Privacy Practices](https://chrome.google.com/webstore/detail/sharegpt-share-your-chatg/daiacboceoaocpibfodeljbdfacokfjb) of ShareGPT. Please contact us if you find any potential violation.

## ✏️ Citation
If you find our paper and code useful in your research, please consider giving a star ⭐ and citation ✏️.

```BibTeX
@article{irvin2024teochat,
  title={TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data},
  author={Liu, Emily Ruoyu and Chen, Joyce Chuyi and Dormoy, Ines and Kim, Jinyoung and Khanna, Samar and Zheng, Zhuo and Ermon, Stefano},
  journal={arXiv preprint arXiv:2410.06234},
  year={2024}
}
```