README.md

---
license: cc-by-nc-4.0
language:
  - en
tags:
  - hand-object-pose
  - egocentric
  - 3d-reconstruction
  - mano
  - hopformer
  - epic-kitchens
extra_gated_prompt: >-
  HOPformer and its components are released for non-commercial research use only
  (CC BY-NC 4.0). The hand meshes/features derive from MANO and WiLoR (CC BY-NC-ND),
  and the models build on ARCTIC and EPIC-Kitchens — each under its own
  non-commercial terms. By requesting access you agree to use these models for
  non-commercial research only and to comply with the MANO, WiLoR, ARCTIC and
  EPIC-Kitchens licenses.
extra_gated_fields:
  Name: text
  Affiliation: text
  I agree to use these models for non-commercial research only: checkbox
---

# HOPformer — pretrained models

Pretrained checkpoints for **HOPformer**, a transformer for 3D hand–object pose
estimation from a single RGB image (MANO hand meshes + articulated/rigid object pose).
Accompanies *"Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation"* (Bansal
et al., ECCV 2026).

- **Paper:** <http://arxiv.org/abs/2606.30598>
- **Code + setup:** <https://github.com/Sid2697/HOPformer>
- **Project page:** <https://sid2697.github.io/epic-contact>
- **Paper:** *Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation* — Siddhant
  Bansal, Zhifan Zhu, Shashank Tripathi, Jiahe Zhao, Michael J. Black, Dima Damen
  (University of Bristol; Max Planck Institute for Intelligent Systems). **ECCV 2026.**

HOPformer tackles 3D hand–object pose estimation in unconstrained egocentric video by
**conditioning object pose on hand priors**, predicting both hands and the object in a
single forward pass. It is trained on ARCTIC and on **EPIC-Contact** — a new in-the-wild
dataset (~2.3K clips, 62.3K frames) with dense 3D hand–object contact annotations.
HOPformer reaches **82.4% success rate on ARCTIC** (+6.2 pts over prior SOTA) and **20.7 mm
contact deviation on EPIC-Contact**.

![HOPformer architecture](https://sid2697.github.io/epic-contact/assets/figures/architecture.png)

Training is a chain: **p1 (exocentric) → p2 (ARCTIC egocentric, from p1) → EPIC
(egocentric, from p2)**. Each checkpoint is inference / fine-tune ready (optimizer
state removed); load with `--load_ckpt <file>`.

| File | Stage | `--setup` | `--dataset` | Paper |
|------|-------|-----------|-------------|-------|
| `p1_exo_epoch22.ckpt`       | exocentric pre-training        | `p1` | `arctic` | Supp. (ARCTIC exocentric) |
| `p2_arctic_ego_epoch8.ckpt` | egocentric fine-tune (from p1) | `p2` | `arctic` | Table 1 (ARCTIC egocentric) |
| `epic_epoch125.ckpt`        | egocentric fine-tune (from p2) | `p2` | `epic`   | Table 2 (EPIC-Contact) |

**Architecture.** DINOv2 ViT-G scene backbone + WiLoR hand features, 12-layer
cross-attention decoder, 6D rotation. LR: linear warm-up 1e-7 → peak over the first 5%
of steps then cosine → 1e-7 (peak 5e-5 for p1, 3e-5 for p2 and EPIC). Each released
checkpoint is its best-validation epoch (early stopping).

## Validated metrics (val split)

| Model | MPJPE ↓ | CDev ↓ | MDev ↓ | SR@0.05 ↑ | Cls ↑ | Notes |
|-------|--------:|-------:|-------:|----------:|------:|-------|
| p1 (ARCTIC exo) | 12.7 | 24.5 | 5.4 | 90.3 | 99.7 | AAE 4.8 |
| p2 (ARCTIC ego) | 16.1 | 31.9 | 7.3 | 82.4 | 99.5 | AAE 5.0 |
| EPIC            | 19.9 | 20.7 | 11.4 | 29.8 | 52.9 | symmetric CDev/MDev; ACC h/o 2.5/4.1 |

CDev/MDev/MPJPE in mm; SR@0.05 and Cls in %. For EPIC (symmetric objects) CDev/MDev/ACC
use the symmetry-aware variants and SR@0.05 is ADD-S–based. See the repository README
("Reproduce the paper numbers") for the eval commands and metric → JSON-key mapping.

## Repository contents
- `*.ckpt` — the three checkpoints above (~7.3 GB each).
- `predictions/{p2_arctic,epic}_eval.tar` — **per-sample prediction dumps** on the
  validation split (the outputs `extract_predicts` produced). Evaluate these directly to
  reproduce Table 1 / Table 2 without re-running the model. (The p1 dump is too large to
  host; reproduce p1 by re-running extraction — see the repository README.)
- `results/*agg_metrics.json` — the reference metric values for each model.

## Usage
Download **selectively** — the checkpoints and the prediction tars are in the same repo:
```bash
# just the checkpoints (~22 GB):
hf download Sid2697/HOPformer --include "*.ckpt" --local-dir release_models
# (optional) the eval prediction dumps for reproduction (~43 GB):
hf download Sid2697/HOPformer --include "predictions/*" "results/*" --local-dir .
```
Then follow the [HOPformer repository](https://github.com/Sid2697/HOPformer) README to set
up the environment (incl. the `smplx` 21-joint patch, the MANO model, and WiLoR
weights/config — **not** included here), and run
e.g. `--load_ckpt release_models/p2_arctic_ego_epoch8.ckpt`. Extract the egocentric ARCTIC
model with `--precision 32` (FP16 can overflow the hand head); `p1` and EPIC are FP16-stable.

## License
Released under **CC BY-NC 4.0** (non-commercial). The hand model/parameters derive from
**MANO**; hand features from **WiLoR** (CC BY-NC-ND); the models build on **ARCTIC** and
**EPIC-Kitchens** — each retains its own non-commercial terms. WiLoR weights and the MANO
model are **not** distributed here and must be obtained from their sources.

## Citation
```bibtex
@inproceedings{bansal2026hopformer,
  title     = {Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation},
  author    = {Bansal, Siddhant and Zhu, Zhifan and Tripathi, Shashank and
               Zhao, Jiahe and Black, Michael J. and Damen, Dima},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}
```