| --- |
| license: cc-by-nc-4.0 |
| library_name: pytorch |
| pipeline_tag: feature-extraction |
| tags: |
| - tactile |
| - robotics |
| - gelsight |
| - masked-autoencoder |
| - multimodal |
| - representation-learning |
| --- |
| |
| # HTT — Heterogeneous Tactile Transformer |
|
|
| A multimodal tactile representation model. One shared transformer backbone |
| encodes four different tactile sensors into a common **192-dimensional** |
| embedding space, pretrained with masked-autoencoder reconstruction and |
| cross-modal alignment. Feed a raw sensor reading, get a feature vector for any |
| downstream head (classification, force / slip estimation, policy learning). |
|
|
| | Modality | Type | Raw input | |
| |---|---|---| |
| | `gsmini` | vision (GelSight Mini) | uint8 image `[224, 224, 3]` | |
| | `9dtact` | vision (9DTact) | uint8 image `[224, 224, 3]` | |
| | `xela` | taxel array | float `[T, 72]` | |
| | `tac02` | taxel array | float `[T, 66]` | |
|
|
| ## Checkpoint |
|
|
| | | | |
| |---|---| |
| | File | `htt_4sensors_best.pth` (~69 MB) | |
| | Contents | `model_state_dict` = 4 encoders + shared 9-layer trunk + 4 decoders (17.1 M params) | |
| | Embedding dim | 192 | |
| | SHA-256 | `024f4c3a067168197d0a6996bbca5c03e744ed5abd1d35a666dbf78e7ac673f0` | |
|
|
| Slim inference/finetune checkpoint (optimizer / predictor states dropped). |
|
|
| ## Usage |
|
|
| Use it with the **HTT** package (contains the architecture, preprocessing, and |
| examples). Download the weights into `checkpoints/`: |
|
|
| ```bash |
| hf download AllenBi21/HTT htt_4sensors_best.pth --local-dir checkpoints |
| ``` |
|
|
| ```python |
| import numpy as np |
| from htt import HTT |
| |
| model = HTT(modality="gsmini") # loads checkpoints/htt_4sensors_best.pth |
| frame = np.random.randint(0, 256, (224, 224, 3), dtype=np.uint8) # your sensor frame |
| emb = model(frame) # -> torch.Tensor [1, 192] |
| ``` |
|
|
| Read the raw-input contract before feeding your own data — the model returns |
| garbage on out-of-distribution inputs. |
|
|
| ## License |
|
|
| Released under **CC BY-NC 4.0** (non-commercial). Portions are derived from |
| Meta's V-JEPA / DINOv2 (Apache-2.0 and CC-BY-NC-4.0). |
|
|
