README.md

---
license: cc-by-nc-4.0
language:
  - en
pipeline_tag: feature-extraction
tags:
  - embeddings
  - tactile
  - pressure
  - force
  - fsr
  - taxel
  - sensor
  - robotics
  - qwen3-vl
base_model: EximiusLabs/fusion-embedding-2-2b-preview
---

# fusion-embedding-2-tactus

<p align="center">
<img src="assets/tactus-banner.png" alt="Tactus, the tactile sense for Fusion Embedding 2 (2B-Preview), Eximius Labs" width="100%">
</p>

<div align="center">

[![Python](https://img.shields.io/badge/python-3.10+-blue.svg)](https://github.com/Eximius-Labs/fusion-embedding)
[![PyTorch](https://img.shields.io/badge/PyTorch-2.x-ee4c2c.svg)](https://github.com/Eximius-Labs/fusion-embedding)
[![Weights](https://img.shields.io/badge/weights-CC--BY--NC--4.0-green.svg)](#license)
[![Status](https://img.shields.io/badge/status-research%20preview%20v0.1-orange.svg)](#)
[![Code](https://img.shields.io/badge/code-GitHub-black.svg)](https://github.com/Eximius-Labs/fusion-embedding)

</div>

**Tactus** is the tactile sensor pack for Eximius Labs' fusion-embedding stack. It maps a
short window of pressure-array frames (a 32x32 taxel grid, the signal class produced by
resistive/FSR gloves, e-skins, and instrumented robot hands) into the
[Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B) text embedding
space, so touch becomes searchable in plain language: recognition is a text query, not a
trained classifier head.

Tactus reads **low-dimensional pressure arrays**, not optical tactile images. Optical
sensors (GelSight, DIGIT) already have strong models (TVL, UniTouch, Sparsh); the cheap,
widely-shipped resistive arrays did not. To our knowledge Tactus is the first open model
to put this sensor class in a text-aligned, cross-modal embedding space.

Tactus is part of the **fusion-embedding family** alongside
[Tactus Mat](https://huggingface.co/EximiusLabs/fusion-embedding-2-tactus-mat)
(the same pack trained for a 64x32 body pressure mat),
[Tremor](https://huggingface.co/EximiusLabs/fusion-embedding-2-tremor) (motion) and
[Ember](https://huggingface.co/EximiusLabs/fusion-embedding-2-ember) (thermal). Its
embeddings target the canonical readout of
[fusion-embedding-2](https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview), so
tactile windows are directly comparable to that model's text, image, video, and audio in
one 2048-d space, and drop into the [Engram](https://github.com/Eximius-Labs/engram)
memory layer (`pip install engram-robomem`) as a first-class sense.

[GitHub](https://github.com/Eximius-Labs/fusion-embedding) | [fusion-embedding-2](https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview) | [Live playground](https://www.eximiuslabs.com/playground) | [Family report (arXiv:2607.18666)](https://arxiv.org/abs/2607.18666) | [Tactus report (arXiv:2608.04043)](https://arxiv.org/abs/2608.04043)

## Model Overview

<p align="center">
<img src="assets/tactus_model_overview.png" alt="Tactus architecture: calibrated pressure windows pass through an MAE-pretrained per-frame trunk, a learned frame fusion, and a trained projector into the fusion-embedding shared space, where touch becomes searchable in natural language alongside every other modality" width="880px">
</p>

Tactus is a **trained CNN trunk plus projector** over pressure windows. Each 32x32 frame
passes through a ResNet-18-width trunk (3x3 stem, four stages; 32x32 -> 4x4 spatial map);
the K frames of a grasp window are fused by a learned 1x1 convolution over their
concatenated feature maps, pooled, and projected into the frozen base's 2048-d text space.
The trunk is initialized by masked-autoencoder pretraining (mask 0.6, per-patch normalized
targets) on 144k unlabeled STAG-family pressure frames, then fine-tuned contrastively
against canonical text embeddings of natural grasp phrases.

The design choice that matters is the data path: pressure is normalized with the sensor's
own calibration affine (`clip((raw - 500) / 150, 0, 1)`, the STAG reference preprocessing),
and pretraining stays same-sensor. In our ablations, correct normalization and same-sensor
MAE were worth more than every architecture change combined, while cross-sensor pretraining
pooling gave nothing, consistent with published findings (HTT, TacVerse).

| Feature | Value |
| --- | --- |
| Parameters | ~2.06B frozen Qwen base (text side); **16.2M trained** (13.5M trunk + 2.6M projector) |
| Modality | tactile pressure (32x32 taxel array; 548 active sensors in the training glove) |
| Supported tasks | `zero-shot object recognition from touch`, `text -> touch retrieval` |
| Input | one grasp window `[F, 32, 32]` (F frames, any F; trained at K=8) or a single frame |
| Input scale | STAG calibration affine `clip((raw - 500) / 150, 0, 1)`; uint8 0-255 maps accepted |
| Embedding dimension | 2048 (canonical whitened readout; directly comparable across modalities) |
| Pooling strategy | last-token pooling (text side) |
| Base model | Qwen/Qwen3-VL-Embedding-2B via fusion-embedding-2-2b-preview (frozen) |
| Pretraining | same-sensor MAE, 144k frames incl. unlabeled; supervised test frames excluded |
| Trained components | trunk + conv frame-fusion + projector, 16.2M; shipped as `model.safetensors` |
| Distribution | ~65 MB trained head; the frozen base downloads from its own repository |

## See it in action

**Real held-out grasps, recognized from pressure alone.** Each panel is a genuine STAG test
frame (the most active frame of that class in the held-out split, by total pressure) with
the text query the model matches it against: no camera, no trained classifier head. Across
the full test split the model averages 0.77 top-1 and 0.94 top-3 over 27 such queries.

<p align="center">
<img src="assets/tactus_test_grasps.png" alt="Four real held-out STAG test pressure maps with their text queries: a mug, scissors, a full can, safety glasses, each recognized from the 32x32 pressure pattern alone" width="860px">
</p>

## Training and Evaluation

Tactus trains in two stages on the [STAG](https://stag.csail.mit.edu/) datasets
([Sundaram et al., Nature 2019](https://www.nature.com/articles/s41586-019-1234-z)): a
masked-autoencoder pretrain over every STAG-family pressure frame (classification +
blindfolded + weights + handposes, 144k frames including unlabeled ones, supervised test
frames excluded), then contrastive fine-tuning of the whole head against canonical text
embeddings of grasp phrases, with STAG-style cluster sampling (each training window draws
diverse frames from across a recording rather than consecutive near-duplicates).

Evaluation is 27-way object recognition on **fully held-out test recordings**, scored as
cosine ranking against text queries (open-vocabulary; the model never trains a classifier
head).

| | top-1 (27-way) | top-3 | recording-level top-1 |
| --- | ---: | ---: | ---: |
| **This checkpoint** | **0.817** | **0.951** | 0.741 |
| Recipe mean (4 independent runs) | 0.771 +/- 0.062 | 0.935 | 0.722 |
| Training from scratch (no MAE), mean of 3 | 0.705 | 0.905 | 0.691 |
| STAG 2019 supervised closed-set CNN | 0.76 | - | - |
| chance | 0.037 | 0.111 | 0.037 |

Interpreting these numbers: the recipe's mean exceeds the original paper's supervised CNN
while performing a harder task (open-vocabulary text queries against a frozen language
space, versus a 27-way trained classifier), though by less than one standard error; we
describe the result as **matching to exceeding the original baseline, with best runs at
0.83**, rather than claiming a definitive margin. Top-3 accuracy is stable across every
run. Our evaluation mirrors STAG's cluster-sampling test protocol but is not their
byte-identical harness. Same-sensor MAE pretraining accounts for about +7 points over
training from scratch. Full recipe, ablations, and negative results: `results.json` and
the GitHub repository.

## Usage

<details>
<summary>Requirements</summary>

- `torch` (CUDA recommended), `numpy`, `safetensors`
- `pip install fusion-embedding[hf]` for the text side (the canonical whitened readout
  Tactus was trained against; embedding text any other way will misrank)
- The frozen base downloads from `EximiusLabs/fusion-embedding-2-2b-preview`.

</details>

<details open>
<summary>via <code>inference.py</code> (this repository)</summary>

```python
import numpy as np
from inference import TactusEmbedder

ta = TactusEmbedder.from_pretrained("EximiusLabs/fusion-embedding-2-tactus",
                                    revision="v0.1-preview")

# a grasp window: [F, 32, 32] pressure frames (uint8 0-255 or floats in [0, 1]);
# for raw sensor counts pass raw="stag" to apply the calibration affine
window = np.load("grasp.npy")

for text, score in ta.rank(window, ["a mug", "scissors", "a full soda can", "an empty hand"]):
    print(f"{score:+.3f}  {text}")

# or embed both sides into the shared space directly
p = ta.embed_pressure(window)          # 2048-d, L2-normalized
t = ta.embed_text(["holding a mug"])   # canonical text embedding, same space
```

Pressure embeddings land in the same space as fusion-embedding-2's text, image, video, and
audio, and as Tremor's motion, so cross-modal search over a robot session works out of the
box through [Engram](https://github.com/Eximius-Labs/engram). Match text against pressure
through this API rather than embedding text with the raw base model; Tactus was trained
against the canonical whitened readout, and other text paths will misrank.

</details>

## Related models

Tactus joins the fusion-embedding sense packs, all built on
[fusion-embedding-2](https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview):

| Model | Sense | Signal |
| --- | --- | --- |
| **This model** | touch | 32x32 pressure/taxel arrays |
| [fusion-embedding-2-tremor](https://huggingface.co/EximiusLabs/fusion-embedding-2-tremor) | motion | 3-axis accelerometer windows |
| [fusion-embedding-2-tremor-g1](https://huggingface.co/EximiusLabs/fusion-embedding-2-tremor-g1) | motion (Unitree G1 head) | robot IMU |
| [fusion-embedding-2-ember](https://huggingface.co/EximiusLabs/fusion-embedding-2-ember) | heat | thermal infrared images |

All packs embed into one 2048-d space, so a query can match across senses. The
[Engram](https://github.com/Eximius-Labs/engram) memory layer (`pip install engram-robomem`)
wires them into a searchable robot session memory with temporal reasoning.

## License

The trained weights in this repository are released under
**[CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/)** (non-commercial).

This reflects the training data's lineage: Tactus is trained on the
[STAG](https://stag.csail.mit.edu/) datasets, which are released for non-commercial
research use. A commercially-clean retrain (on permissively licensed pressure corpora) is
future work; a commercial license may follow.

## Limitations

- **Run-to-run variance.** The training recipe's top-1 varies +/-0.06 across seeds
  (0.70-0.83 over four runs). The released checkpoint is a strong draw, and the mean is
  reported alongside it. Seed stabilization is active work.
- **One sensor family.** Trained on one glove (STAG's 32x32 grid, 548 taxels). Our
  cross-sensor experiments show transfer to other taxel geometries needs fine-tuning, not
  zero-shot use; the input path accepts any [F,32,32] window, and other resolutions must
  be resampled.
- **27-object vocabulary at eval.** Open-vocabulary means text queries, not tested
  open-set generalization to arbitrary unseen object categories; treat novel-category
  recognition as unvalidated.
- **Research preview.** Not a production classifier. The intended use is
  language-addressable touch inside a multimodal memory, not high-stakes recognition.
- **English text only**, through the canonical readout (`fusion-embedding` package); do
  not embed text with the raw base model.

## Citation

If you use Tactus, please cite this repository and the dataset it builds on:

```bibtex
@article{tactus2026,
  title   = {Tactus: Open-Vocabulary Object Recognition from Low-Cost
             Pressure Arrays},
  author  = {Tonmoy, Abdul Basit},
  journal = {arXiv preprint arXiv:2608.04043},
  year    = {2026}
}
```

Tactus trains on **STAG** (Sundaram et al., *Learning the signatures of the human grasp
using a scalable tactile glove*, Nature 2019); please cite that work when using the
benchmark numbers. The text space is **Qwen3-VL-Embedding-2B**.