File size: 2,044 Bytes
5314f01 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | ---
license: cc-by-nc-4.0
library_name: pytorch
pipeline_tag: feature-extraction
tags:
- tactile
- robotics
- gelsight
- masked-autoencoder
- multimodal
- representation-learning
---
# HTT — Heterogeneous Tactile Transformer
A multimodal tactile representation model. One shared transformer backbone
encodes four different tactile sensors into a common **192-dimensional**
embedding space, pretrained with masked-autoencoder reconstruction and
cross-modal alignment. Feed a raw sensor reading, get a feature vector for any
downstream head (classification, force / slip estimation, policy learning).
| Modality | Type | Raw input |
|---|---|---|
| `gsmini` | vision (GelSight Mini) | uint8 image `[224, 224, 3]` |
| `9dtact` | vision (9DTact) | uint8 image `[224, 224, 3]` |
| `xela` | taxel array | float `[T, 72]` |
| `tac02` | taxel array | float `[T, 66]` |
## Checkpoint
| | |
|---|---|
| File | `htt_4sensors_best.pth` (~69 MB) |
| Contents | `model_state_dict` = 4 encoders + shared 9-layer trunk + 4 decoders (17.1 M params) |
| Embedding dim | 192 |
| SHA-256 | `024f4c3a067168197d0a6996bbca5c03e744ed5abd1d35a666dbf78e7ac673f0` |
Slim inference/finetune checkpoint (optimizer / predictor states dropped).
## Usage
Use it with the **HTT** package (contains the architecture, preprocessing, and
examples). Download the weights into `checkpoints/`:
```bash
hf download AllenBi21/HTT htt_4sensors_best.pth --local-dir checkpoints
```
```python
import numpy as np
from htt import HTT
model = HTT(modality="gsmini") # loads checkpoints/htt_4sensors_best.pth
frame = np.random.randint(0, 256, (224, 224, 3), dtype=np.uint8) # your sensor frame
emb = model(frame) # -> torch.Tensor [1, 192]
```
Read the raw-input contract before feeding your own data — the model returns
garbage on out-of-distribution inputs.
## License
Released under **CC BY-NC 4.0** (non-commercial). Portions are derived from
Meta's V-JEPA / DINOv2 (Apache-2.0 and CC-BY-NC-4.0).
|