Improve model card: add pipeline tag, library, and links

README.md

@@ -1,3 +1,50 @@
----
-license: mit
----
+---
+license: mit
+pipeline_tag: robotics
+library_name: transformers
+---
+
+# Embodied Instruction Following in Unknown Environments
+
+This repository contains the official implementation and models for the paper:
+[**Embodied Instruction Following in Unknown Environments**](https://huggingface.co/papers/2406.11818).
+
+This model is a high-level task planner and low-level exploration controller, fine-tuned using LoRA on top of a Vicuna-v1-3-7B base model, primarily leveraged through the LLaVA framework.
+
+## Abstract
+Enabling embodied agents to complete complex human instructions from natural language is crucial to autonomous systems in household services. Conventional methods can only accomplish human instructions in the known environment where all interactive objects are provided to the embodied agent, and directly deploying the existing approaches for the unknown environment usually generates infeasible plans that manipulate non-existing objects. On the contrary, we propose an embodied instruction following (EIF) method for complex tasks in the unknown environment, where the agent efficiently explores the unknown environment to generate feasible plans with existing objects to accomplish abstract instructions. Specifically, we build a hierarchical embodied instruction following framework including the high-level task planner and the low-level exploration controller with multimodal large language models. We then construct a semantic representation map of the scene with dynamic region attention to demonstrate the known visual clues, where the goal of task planning and scene exploration is aligned for human instruction. For the task planner, we generate the feasible step-by-step plans for human goal accomplishment according to the task completion process and the known visual clues. For the exploration controller, the optimal navigation or object interaction policy is predicted based on the generated step-wise plans and the known visual clues. The experimental results demonstrate that our method can achieve 45.09% success rate in 204 complex human instructions such as making breakfast and tidying rooms in large house-level scenes.
+
+## Links
+*   **Project Page**: https://gary3410.github.io/eif_unknown/
+*   **GitHub Repository**: https://github.com/gary3410/eif_unknown
+
+## Usage and Evaluation
+
+This model consists of LoRA adapters for a high-level planner and a low-level controller, built upon a Vicuna-v1-3-7B base model within the LLaVA framework. For detailed setup, data preparation, fine-tuning, and inference instructions, please refer to the comprehensive guide in the [official GitHub repository](https://github.com/gary3410/eif_unknown).
+
+The models are typically used by loading the base model and then applying the LoRA weights. The project's GitHub repository provides scripts for evaluation.
+
+Example evaluation command from the GitHub repository:
+```bash
+# Oracle setting
+# Easy task
+CUDA_VISIBLE_DEVICES=0,1,2,3 python -m llava.serve.cli_llava_v3_nav_seg_gt \
+    --model-path ./checkpoints/llava-vicuna-v1-3-7b-finetune-frontier-lora-low-level-controller \
+    --model-path-s1 ./checkpoints/llava-vicuna-v1-3-7b-finetune-planner-lora-high-level-planner \
+    --model-base ./checkpoints/vicuna-v1-3-7b \
+    --image-file ./vision_dataset/llava_dataset_v8_easy_train/frontiers_feature \
+    --val-file ./data/spaced_parse_instruction_easy_v12_val.json
+```
+For other evaluation settings (e.g., Detic setting), please refer to the [GitHub repository](https://github.com/gary3410/eif_unknown).
+
+## Citation
+If you find this repository helpful, please consider citing:
+
+```bibtex
+@article{wu2024embodied,
+  title={Embodied instruction following in unknown environments},
+  author={Wu, Zhenyu and Wang, Ziwei and Xu, Xiuwei and Lu, Jiwen and Yan, Haibin},
+  journal={arXiv preprint arXiv:2406.11818},
+  year={2024}
+}
+```