---
license: cc-by-nc-4.0
library_name: pytorch
pipeline_tag: feature-extraction
tags:
  - tactile
  - robotics
  - gelsight
  - masked-autoencoder
  - multimodal
  - representation-learning
---

# HTT — Heterogeneous Tactile Transformer

A multimodal tactile representation model. One shared transformer backbone
encodes four different tactile sensors into a common **192-dimensional**
embedding space, pretrained with masked-autoencoder reconstruction and
cross-modal alignment. Feed a raw sensor reading, get a feature vector for any
downstream head (classification, force / slip estimation, policy learning).

| Modality | Type | Raw input |
|---|---|---|
| `gsmini` | vision (GelSight Mini) | uint8 image `[224, 224, 3]` |
| `9dtact` | vision (9DTact) | uint8 image `[224, 224, 3]` |
| `xela`   | taxel array | float `[T, 72]` |
| `tac02`  | taxel array | float `[T, 66]` |

## Checkpoint

| | |
|---|---|
| File | `htt_4sensors_best.pth` (~69 MB) |
| Contents | `model_state_dict` = 4 encoders + shared 9-layer trunk + 4 decoders (17.1 M params) |
| Embedding dim | 192 |
| SHA-256 | `024f4c3a067168197d0a6996bbca5c03e744ed5abd1d35a666dbf78e7ac673f0` |

Slim inference/finetune checkpoint (optimizer / predictor states dropped).

## Usage

Use it with the **HTT** package (contains the architecture, preprocessing, and
examples). Download the weights into `checkpoints/`:

```bash
hf download AllenBi21/HTT htt_4sensors_best.pth --local-dir checkpoints
```

```python
import numpy as np
from htt import HTT

model = HTT(modality="gsmini")                       # loads checkpoints/htt_4sensors_best.pth
frame = np.random.randint(0, 256, (224, 224, 3), dtype=np.uint8)   # your sensor frame
emb = model(frame)                                   # -> torch.Tensor [1, 192]
```

Read the raw-input contract before feeding your own data — the model returns
garbage on out-of-distribution inputs.

## License

Released under **CC BY-NC 4.0** (non-commercial). Portions are derived from
Meta's V-JEPA / DINOv2 (Apache-2.0 and CC-BY-NC-4.0).