Enhance model card with metadata, links, abstract, and usage details

README.md

@@ -2,9 +2,97 @@
 tags:
 - model_hub_mixin
 - pytorch_model_hub_mixin
+- robotics
+- vision-transformer
+- robot-learning
+pipeline_tag: robotics
+library_name: lerobot
+license: mit
 ---
 
-This model has been pushed to the Hub using the [PytorchModelHubMixin](https://huggingface.co/docs/huggingface_hub/package_reference/mixins#huggingface_hub.PyTorchModelHubMixin) integration:
-- Code: [More Information Needed]
-- Paper: [More Information Needed]
-- Docs: [More Information Needed]
+# Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers
+
+This repository contains the official code and models for the paper:
+**[Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers](https://huggingface.co/papers/2507.15833)**
+
+🚀 **Project Website:** [https://ian-chuang.github.io/gaze-av-aloha/](https://ian-chuang.github.io/gaze-av-aloha/)
+🔗 **Official Codebase:** [https://github.com/ian-chuang/gaze-av-aloha](https://github.com/ian-chuang/gaze-av-aloha)
+
+![hero](https://github.com/ian-chuang/gaze-av-aloha/raw/main/media/hero.gif)
+
+## Abstract
+Human vision is a highly active process driven by gaze, which directs attention and fixation to task-relevant regions and dramatically reduces visual processing. In contrast, robot learning systems typically rely on passive, uniform processing of raw camera images. In this work, we explore how incorporating human-like active gaze into robotic policies can enhance both efficiency and performance. We build on recent advances in foveated image processing and apply them to an Active Vision robot system that emulates both human head movement and eye tracking. Extending prior work on the AV-ALOHA robot simulation platform, we introduce a framework for simultaneously collecting eye-tracking data and robot demonstrations from a human operator as well as a simulation benchmark and dataset for training robot policies that incorporate human gaze. Given the widespread use of Vision Transformers (ViTs) in robot learning, we integrate gaze information into ViTs using a foveated patch tokenization scheme inspired by recent work in image segmentation. Compared to uniform patch tokenization, this significantly reduces the number of tokens-and thus computation-without sacrificing visual fidelity near regions of interest. We also explore two approaches to gaze imitation and prediction from human data. The first is a two-stage model that predicts gaze to guide foveation and action; the second integrates gaze into the action space, allowing the policy to jointly predict gaze and actions end-to-end. Our results show that our method for foveated robot vision not only drastically reduces computational overhead, but also improves performance for high precision tasks and robustness to unseen distractors. Together, these findings suggest that human-inspired visual processing offers a useful inductive bias for robotic vision systems.
+
+## ✨ Key Features
+*   **Human-inspired Foveated Vision**: Integrates human gaze to guide foveated patch tokenization in Vision Transformers (ViTs), significantly reducing computation.
+*   **Efficiency & Robustness**: Achieves substantial computational overhead reduction while improving performance for high-precision tasks and robustness to unseen distractors.
+*   **Comprehensive Dataset**: Introduces a simulation benchmark and dataset for training robot policies that incorporate human gaze, collected with bimanual robot demonstrations and synchronized human eye-tracking on the AV-ALOHA simulation platform.
+*   **Gaze Imitation Approaches**: Explores both two-stage gaze prediction and end-to-end joint prediction of gaze and actions.
+
+## ⚙️ Installation
+
+To set up the environment and install necessary dependencies for the `gaze-av-aloha` project, which utilizes this model:
+
+```bash
+# Clone the repository and initialize submodules
+git clone https://github.com/ian-chuang/gaze-av-aloha.git
+cd gaze-av-aloha
+git submodule init
+git submodule update
+
+# Create and activate a new Conda environment
+conda create -n gaze python=3.10
+conda activate gaze
+
+# Install LeRobot (primary library for data handling)
+pip install git+https://github.com/huggingface/lerobot.git@483be9aac217c2d8ef16982490f22b2ad091ab46
+
+# Install FFmpeg for video logging
+conda install ffmpeg=7.1.1 -c conda-forge
+
+# Install AV-ALOHA packages
+pip install -e ./gym_av_aloha
+pip install -e ./gaze_av_aloha
+```
+Make sure you're logged in to Hugging Face CLI: `huggingface-cli login`
+
+## 🚀 Usage
+
+This model checkpoint (`iantc104/mae_vitb_foveated_vit`) represents a pretrained Vision Transformer backbone used as a component within the "Look, Focus, Act" framework. You can train and evaluate policies using the provided scripts in the original repository.
+
+### Example: Train a Fov-Act Policy (end-to-end gaze as action)
+
+To train a policy that uses this foveated ViT as its vision encoder, you can use the `train.py` script from the `gaze-av-aloha` repository:
+
+```bash
+python gaze_av_aloha/scripts/train.py \
+  policy=foveated_vit_policy \
+  task=<task_name_e.g._av_aloha_sim_thread_needle> \
+  policy.vision_encoder_kwargs.repo_id=iantc104/mae_vitb_foveated_vit \
+  policy.optimizer_lr_backbone=1e-5 \
+  wandb.enable=true \
+  wandb.project=<your_project_name> \
+  wandb.entity=<your_wandb_entity> \
+  wandb.job_name=fov-act \
+  device=cuda
+```
+
+Replace `<task_name_e.g._av_aloha_sim_thread_needle>` with one of the available task configurations from the project (e.g., `av_aloha_sim_cube_transfer`, `av_aloha_sim_peg_insertion`, etc.). This command will load the `iantc104/mae_vitb_foveated_vit` as the vision encoder for the `foveated_vit_policy`.
+
+For other policy types (Fov-UNet, Fine, Coarse) and detailed instructions, please refer to the [AV ALOHA Benchmark section](https://github.com/ian-chuang/gaze-av-aloha#av-aloha-benchmark) in the official GitHub repository.
+
+## 📄 Citation
+
+If you find this work or model useful in your research, please consider citing the paper:
+
+```bibtex
+@misc{chuang2025lookfocusactefficient,
+      title={Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers},
+      author={Ian Chuang and Andrew Lee and Dechen Gao and Jinyu Zou and Iman Soltani},
+      year={2025},
+      eprint={2507.15833},
+      archivePrefix={arXiv},
+      primaryClass={cs.RO},
+      url={https://arxiv.org/abs/2507.15833},
+}
+```