README.md

---
license: apache-2.0
datasets:
- GuiminHu/HapticCap
- GuiminHu/VibRate
language:
- en
metrics:
- bleu
- meteor
- rouge
base_model:
- meta-llama/Llama-3.1-8B
tags:
- code
---

# 📌 HapticLLaMA: A Multimodal Sensory Language Model for Haptic Captioning

Arxiv: https://arxiv.org/pdf/2508.06475?

Codes: https://github.com/LeMei/HapticLLaMA

---

## 📖 Introduction
**HapticLLaMA** is a multimodal sensory language model that interprets vibration signals into descriptions in a given sensory, emotional, or associative category. 
HapticLLaMA is trained in two stages: (1) supervised fine-tuning using the LLaMA architecture with LoRA-based adaptation, and (2) fine-tuning via reinforcement
learning from human feedback (RLHF). 

---

## 🧩 Tasks
Given a vibration signal S and a target category c ∈ {sensory, emotional, associative}, where sensory refers to physical attributes (e.g.,intensity of tapping), emotional denotes affective
impressions (e.g., the mood of a scene), and associative indicates real-world familiar experiences (e.g., buzzing of a bee, a heartbeat), the goal is to generate a caption corresponding to the specified category of haptic experience.

---

## 📂 Training
HapticLLaMA training is consist of (1) supervised fine-tuning with LoRA adaptation and (2) subsequent fine-tuning based on human feedback on generated captions.

<img width="925" height="557" alt="image" src="https://github.com/user-attachments/assets/28a0aa75-d011-4870-b9ec-b9b3607eb8d8" />

---

## 📂 Haptic Tokenizer
- **Frequency-based Tokenizer**:
  
  <img width="361" height="211" alt="image" src="https://github.com/user-attachments/assets/ca848d0b-18d5-4ad5-89e4-268399aad801" />

Frequency-based Tokenizer divides the frequency range into logarithmically spaced bins that correspond to just-noticeable ifferences in human frequency perception. Similarly, the amplitude range is segmented into normalized levels. The tokenizer then assigns a unique
token (e.g., FREQ_3_AMP_2) to each frequencyamplitude pair, encoding the signal’s spectral content into a form interpretable by LLMs.
```python
import librosa

def steps_binning(frequencies, amplitudes, freq_bins=10, amp_levels=5):

    freq_min, freq_max = np.min(frequencies), np.max(frequencies)
    freq_min = freq_max / (1.2**(freq_bins-1))
    freq_edges = np.geomspace(freq_min, freq_min * 1.2**(freq_bins-1), num=freq_bins)
    freq_labels = [f"FREQ_{i+1}" for i in range(freq_bins)] 
    amp_min, amp_max = np.min(amplitudes), np.max(amplitudes)
    if amp_min == amp_max:
        # breakpoint()
        amplitudes = np.zeros_like(frequencies)
        amp_edges = np.linspace(0, 1, amp_levels + 1)
    else:
        amplitudes = (amplitudes - amp_min) / (amp_max - amp_min)  
        amp_min = amp_max / (1.2**(amp_levels-1))
        amp_edges = np.geomspace(amp_min, amp_max, num=amp_levels)

    amp_labels = [f"AMP_{i+1}" for i in range(amp_levels)]  

    tokens = []
    for f, a in zip(frequencies, amplitudes):
        freq_bin = np.digitize(f, freq_edges) - 1
        freq_bin = min(freq_bin, freq_bins - 1)  
        freq_token = freq_labels[freq_bin]

        amp_bin = np.digitize(a, amp_edges) - 1
        amp_bin = min(amp_bin, amp_levels - 1)  
        amp_token = amp_labels[amp_bin]

        tokens.append(f"{freq_token}_{amp_token}")
    return tokens

    ### start load .wav file and tokenize
    y, sr = librosa.load(wav_file, sr=None)  

    D = librosa.stft(y, n_fft=n_fft, hop_length=hop_length)  
    frequencies = librosa.fft_frequencies(sr=sr, n_fft=n_fft)  
    magnitudes = np.abs(D)  
    magnitudes = magnitudes / np.max(magnitudes)  
    frame_idx = 10  
    amplitudes = magnitudes[:, frame_idx]  
    mask = frequencies < 500
    frequencies_filtered = frequencies[mask]
    amplitudes_filtered = amplitudes[mask]
    ###haptic tokens based on Frequency-base haptic tokenizer
    tokens = steps_binning(frequencies_filtered, amplitudes_filtered, freq_bins=freq_bins,amp_levels=amp_levels)

```
---
- **EnCodec-based Tokenizer**:

<img width="317" height="172" alt="image" src="https://github.com/user-attachments/assets/35e50d2e-c21f-4fc1-8953-74305a752ee0" />

EnCodec is a neural audio codec that compresses audio using deep learning (Défossez et al., 2023). It consists of three
main components: (1) an encoder that transforms raw audio into a lower-dimensional latent representation, (2) a quantizer that discretizes the latent features via residual vector quantization, and (3) a decoder that reconstructs the waveform from the quantized codes. EnCodec-based tokenizer extract the codes from residual vector quantization in the audio compression architecture. 
  
```python
from transformers import AutoTokenizer,AutoProcessor,EncodecModel

encodec_model = EncodecModel.from_pretrained("facebook/encodec_24khz")
processor = AutoProcessor.from_pretrained("facebook/encodec_24khz")

### EnCodec-based Tokenizer
def encodec_token(wav_file):
    data_dict = {"audio": [wav_file]}
    data_dataset = Dataset.from_dict(data_dict).cast_column("audio", Audio())
    audio_sample = data_dataset[-1]["audio"]["array"]
    inputs = processor(raw_audio=audio_sample, sampling_rate=24000, return_tensors="pt")
    with torch.no_grad():
        encoded_frames = encodec_model.encode(inputs["input_values"], inputs["padding_mask"])
    tokens = encoded_frames.audio_codes[0][0]
    tokens_list = [str(token) for token in tokens[0].tolist()]

    return tokens_list
```
---
## 📂 Inference

Given a haptic signal, we prompt HapticLLaMA to generate captions from sensory, emotional, and associative perspectives.

<img width="448" height="329" alt="image" src="https://github.com/user-attachments/assets/2ea17083-5da3-47f2-9781-7f17912d08cc" />

```python
import torch
from torch import nn
import librosa

#load model--HapticLLaMA
def load_model(stage, device, mode, model_file_url):
        if os.path.exists(model_file_url):
            model = Model(args, mode=mode)
            lora_state_dict = torch.load(model_file_url)
            state_name, model_name = [], []
            for name, param in model.named_parameters():
                model_name.append(name)
            for name in lora_state_dict.keys():
                state_name.append(name)
            missing_keys, unexpected_keys = model.load_state_dict(lora_state_dict, strict=False)
            model.to(device)
        else:
            print('invalid model url!')
            model = None
        return model

###load pretrained haptic tokenizer

frequency_tokenizer = AutoTokenizer.from_pretrained(r"./updated_llama_tokenizer_steps_binning.pt/")
encodec_tokenizer = AutoTokenizer.from_pretrained(r"./updated_llama_tokenizer_encodec.pt/")

#formalize input for inference
def tokenizer_haptic(haptic, prompt, mode):

    def formalize_input(haptic_tokens, tokenizer, prompt):
        tokenizer.pad_token = tokenizer.eos_token

        inputs = tokenizer(haptic_tokens, padding=True, truncation=True, return_tensors="pt")
        input_ids = inputs.input_ids
        input_atts = inputs.attention_mask

        prompt_enc = tokenizer(prompt, padding=True, truncation=True, return_tensors="pt")
        prompt_ids = prompt_enc.input_ids
        prompt_atts = prompt_enc.attention_mask

        prompt_ids = torch.cat((input_ids,prompt_ids),dim=1)
        prompt_atts = torch.cat((input_atts,prompt_atts),dim=1)


        return input_ids,input_atts, prompt_ids, prompt_atts

    ###Frequency-based token formalization
    if mode == 'frequency':
      freq_haptic_tokens = frequency_tokenizer(haptic, mode='frequency)
      freq_haptic_tokens = [' '.join(freq_haptic_tokens)]
      freq_input_ids,freq_input_atts, freq_prompt_ids, freq_prompt_atts = formalize_input(freq_haptic_tokens, frequency_tokenizer, prompt=prompt)
      return freq_input_ids, freq_input_atts, freq_prompt_ids, freq_prompt_atts
    elif mode == 'encodec':
      ###Encodec-based token formalization
      encodec_haptic_tokens = encodec_token(haptic, mode='encodec')
      encodec_haptic_tokens = [' '.join(encodec_haptic_tokens)]
      encodec_input_ids, encodec_input_atts, encodec_prompt_ids, prompt_atts = formalize_input(encodec_haptic_tokens, encodec_tokenizer, prompt=prompt)
      return encodec_input_ids, encodec_input_atts, encodec_prompt_ids, prompt_atts

```
Inference for one sample

```python
haptic_signal = r'./F211_loop.wav'
sensory_prompt = 'its sensory description is'
##for emotional and associative
##emotional_prompt = 'its emotional description is'
##associative_prompt = 'its associative description is'
input_ids, input_atts, prompt_ids, prompt_atts = tokenizer_haptic(haptic_signal, sensory_prompt, mode='encodec')
hapticllama = load_model(stage=1, device='cuda', mode='encodec', model_file_url=encodec_model_file_url)
caption = hapticllama.generate(inputs = prompt_ids,input_atts=prompt_atts)
print(caption)
```
---

## 🚀 Citation
If you find this dataset useful for your research, please cite our paper:

```bibtex
@article{hu2025hapticllama,
  title={HapticLLaMA: A Multimodal Sensory Language Model for Haptic Captioning},
  author={Hu, Guimin and Hershcovich, Daniel and Seifi, Hasti},
  journal={arXiv preprint arXiv:2508.06475},
  year={2025}
}
```