README.md

---
title: README
emoji: 👀
colorFrom: indigo
colorTo: purple
sdk: static
pinned: false
---

<p align="center">
  <img src="https://huggingface.co/datasets/tavis-benchmark/tavis-assets/resolve/main/figure1.png" alt="TAVIS overview" width="90%" />
</p>

# TAVIS: A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning

A benchmark for **egocentric active-vision imitation learning** and **anticipatory gaze** on humanoid torsos. TAVIS ships eight simulated manipulation tasks across two robots (**Fourier GR1T2** and **Pollen Reachy 2**), **2,200 VR-teleoperated demonstrations**, pretrained **π₀** baselines, and a proprioceptive metric -- **GALT (Gaze-Action Lead Time)** -- for quantifying anticipatory gaze in the learned policies.

| | |
|---|---|
| 📄 **Paper** | Under double-blind review |
| 💻 **Code (anonymous mirror)** | https://anonymous.4open.science/r/tavis-F5D7 |
| 🤗 **Datasets & checkpoints** | This org -- [`tavis-benchmark`](https://huggingface.co/tavis-benchmark) |
| ⚖️ **License** | Code: MIT · Datasets: CC-BY-4.0 |

---

## 👀 For reviewers:

**1. Data can be viewed from the browser (zero download).**

- Viewer: https://huggingface.co/spaces/lerobot/visualize_dataset -> paste, e.g., `tavis-benchmark/tavis-head-gr1t2`
- Sample datasets: [`tavis-head-sample-gr1t2`](https://huggingface.co/datasets/tavis-benchmark/tavis-head-sample-gr1t2) · [`tavis-head-sample-reachy2`](https://huggingface.co/datasets/tavis-benchmark/tavis-head-sample-reachy2)

**2. Run a pretrained policy across a whole suite (no training).** Download a multi-task π₀ checkpoint and roll it out:

```bash
huggingface-cli download tavis-benchmark/pi0-tavis-head-gr1t2-headcam \
    --local-dir checkpoints/pi0-tavis-head-gr1t2-headcam

python scripts/eval_benchmark.py \
    --checkpoint checkpoints/pi0-tavis-head-gr1t2-headcam \
    --suite tavis-head --robot gr1t2 \
    --episodes 96 --num-envs 1

python scripts/print_benchmark_results.py results/pi0-tavis-head-gr1t2-headcam
```

This evaluates all 5 TAVIS-HEAD tasks × `{id, ood_spatial}` × 96 episodes and prints success rates with Wilson 95% CIs plus per-episode GALT. (Requires the code + an RTX GPU -- see [Installation](#installation). Several hours on a single 4090.)

<!--
**3. The headline numbers** (success rates + GALT, with confidence intervals) are reproduced by the eval scripts and reported in the paper.

 TODO: table with main results  -->

---

## 📦 Datasets

All released as **LeRobotDataset v3.0**. The four multi-task suites total **2,200** VR-teleoperated demonstrations (~3 h of interaction).

| Repo | Robot | Suite | Episodes |
|------|-------|-------|----------|
| [`tavis-head-gr1t2`](https://huggingface.co/datasets/tavis-benchmark/tavis-head-gr1t2) | GR1T2 | TAVIS-HEAD | 800 |
| [`tavis-head-reachy2`](https://huggingface.co/datasets/tavis-benchmark/tavis-head-reachy2) | Reachy2 | TAVIS-HEAD | 800 |
| [`tavis-hands-gr1t2`](https://huggingface.co/datasets/tavis-benchmark/tavis-hands-gr1t2) | GR1T2 | TAVIS-HANDS | 300 |
| [`tavis-hands-reachy2`](https://huggingface.co/datasets/tavis-benchmark/tavis-hands-reachy2) | Reachy2 | TAVIS-HANDS | 300 |

**Also released:** [`tavis-head-sample-gr1t2`](https://huggingface.co/datasets/tavis-benchmark/tavis-head-sample-gr1t2) and [`tavis-head-sample-reachy2`](https://huggingface.co/datasets/tavis-benchmark/tavis-head-sample-reachy2) (small previews) · [`tavis-assets`](https://huggingface.co/datasets/tavis-benchmark/tavis-assets) (robot / task USD assets).

## 🤖 Pretrained checkpoints (π₀)

| Repo | Robot | Suite | Camera |
|------|-------|-------|--------|
| [`pi0-tavis-head-gr1t2-headcam`](https://huggingface.co/tavis-benchmark/pi0-tavis-head-gr1t2-headcam) | GR1T2 | TAVIS-HEAD | head |
| [`pi0-tavis-head-gr1t2-fixedcam`](https://huggingface.co/tavis-benchmark/pi0-tavis-head-gr1t2-fixedcam) | GR1T2 | TAVIS-HEAD | fixed |
| [`pi0-tavis-head-reachy2-headcam`](https://huggingface.co/tavis-benchmark/pi0-tavis-head-reachy2-headcam) | Reachy2 | TAVIS-HEAD | head |
| [`pi0-tavis-head-reachy2-fixedcam`](https://huggingface.co/tavis-benchmark/pi0-tavis-head-reachy2-fixedcam) | Reachy2 | TAVIS-HEAD | fixed |
| [`pi0-tavis-hands-gr1t2`](https://huggingface.co/tavis-benchmark/pi0-tavis-hands-gr1t2) | GR1T2 | TAVIS-HANDS | head |
| [`pi0-tavis-hands-reachy2`](https://huggingface.co/tavis-benchmark/pi0-tavis-hands-reachy2) | Reachy2 | TAVIS-HANDS | head |

The `headcam` / `fixedcam` pair on TAVIS-HEAD is the core **active-vision vs. fixed-camera** comparison.

---

## 🎯 Tasks

Eight tasks in two suites. The robot, camera mode (`headcam` / `fixedcam`), and eval mode are orthogonal axes set at evaluation time.

### TAVIS-HEAD -- global visual search & clutter

| Task (CLI key) | What it tests |
|----------------|---------------|
| `clutter_pick_lift` | Pick & lift a language-named object from 5 scattered YCB objects; the wider OOD spread forces active scanning. |
| `clutter_pick_cube` | Visually search clutter for a distinct red cube and lift it (no language conditioning). |
| `conditional_pick` | Read a red/green cue card, then pick the left/right object it indicates. |
| `wait_then_act` | Monitor a light until it turns green (random 2-6 s delay), then pick the object. |
| `multi_shelf_scan` | Move the head to scan a 3-shelf unit and retrieve the language-named target. |

### TAVIS-HANDS -- local occlusion & peeking

| Task (CLI key) | What it tests |
|----------------|---------------|
| `peeking_box` | Find which side of a box is open (head cam blind; wrist cams decide) and reach in to grab the object -- bimanual perception under occlusion. |
| `occluded_reach` | Reach around a narrow screen to grasp an object the head camera can't see well -- reach-around localization with wrist cams. |
| `blocked_clutter_pick_cube` | Head-camera-blackout ablation of `clutter_pick_cube` -- only the wrist cameras can locate the red cube. |

Both robots expose a canonical **19-D action layout** (left/right arm IK targets, 3-DoF neck, two gripper scalars), so policies share one action space across embodiments.

## 📐 Evaluation protocol

Each **(robot × task × eval-mode)** cell is evaluated over **96 stochastic episodes**; success rates are reported with **Wilson 95% confidence intervals**. Three eval modes (see `docs/ood_modes.md`):

| Mode | Object placement | Robot reset pose |
|------|------------------|------------------|
| `id` | training range | default |
| `ood_spatial` | wider than training | default |
| `ood_init_pose` | training range | Gaussian (σ≈10 cm EEF, σ≈10° neck) |

**GALT (Gaze-Action Lead Time)** -- `t_hand_arrival - t_head_arrival` in seconds -- measures how far *in advance* the head settles on the target before the hand arrives. It is **proprioceptive**: computed purely from the 19-D commanded-action trajectory (no eye-tracking, no scene state), so it ports to any robot exposing those channels. Every benchmark rollout emits a GALT estimate alongside its success flag. Details: `docs/galt.md`.

---


## Installation

Download the code from the [anonymous mirror](https://anonymous.4open.science/r/tavis-F5D7), unzip, then from inside the extracted folder:

```bash
bash install.sh
conda activate tavis
```

`install.sh` bootstraps the full pinned stack (via `uv`, applying the dependency overrides in `pyproject.toml`) into a fresh `tavis` conda env on Python 3.11.

> **Hardware.** Simulation, data collection, and evaluation need an RTX GPU. Training: diffusion-policy and π₀-LoRA fit on a single 24 GB card; full π₀ training fits on an H100.


## Reproduction recipes

**A) Run a pretrained multi-task policy across a whole suite**. Fastest path; no training. See above ("For Reviewers").

**B) Train + evaluate a single-task diffusion policy from scratch** (~12 h on a 4090). The released datasets are multi-task suites; `--task` filters to one task class:

```bash
huggingface-cli download tavis-benchmark/tavis-head-gr1t2 \
    --repo-type dataset --local-dir datasets/tavis-head-gr1t2

python scripts/train_policy.py \
    --dataset datasets/tavis-head-gr1t2 \
    --task conditional_pick --model diffusion --camera headcam \
    --steps 200000

python scripts/eval_benchmark.py \
    --checkpoint checkpoints/tavis-head-gr1t2__conditional_pick_diffusion_headcam \
    --robot gr1t2 --tasks conditional_pick \
    --eval-modes id ood_spatial --episodes 96 --num-envs 1
```

Multi-task training is π₀-only (it uses the per-episode language instruction). The paper's multi-task π₀ runs used H100-days on an external orchestrator, so for reviewers the released π₀ checkpoints (recipe A) are the practical reproduction path. Full recipes: see the mirror's `README.md`.

## Documentation (in the code mirror)

| Topic | File |
|-------|------|
| GALT metric & porting to your robot | `docs/galt.md` |
| Evaluation modes (`id`, `ood_*`) | `docs/ood_modes.md` |
| Adding a new robot / task | `docs/extending_robots.md`, `docs/extending_tasks.md` |
| Data collection (VR teleoperation) | `docs/data_collection.md` |

## License

- **Code:** MIT.
- **Datasets:** CC-BY-4.0.
- **Robot models, YCB objects, task assets:** under their respective upstream licenses (NVIDIA IsaacLab, the YCB project, and the original robot-model authors); per-asset attribution ships with [`tavis-assets`](https://huggingface.co/datasets/tavis-benchmark/tavis-assets).

## Citation

If you use TAVIS -- benchmark, datasets, or checkpoints -- please cite the TAVIS paper. The formal BibTeX will be posted here once the anonymous review period concludes; until then, please cite it by name as **"TAVIS: A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning".**

## Contact

Questions and bug reports (anonymous-review-safe): `tavis.benchmark@gmail.com`.

## Acknowledgments

Built on [IsaacLab](https://isaac-sim.github.io/IsaacLab/) (NVIDIA), the [LeRobot](https://github.com/huggingface/lerobot) framework (Hugging Face), and IsaacLab-Arena. Robot models from the upstream Fourier GR1T2 and Pollen Reachy 2 USD distributions; objects from the YCB project; VR teleoperation via Meta Quest.