| --- |
| license: apache-2.0 |
| datasets: |
| - GuiminHu/HapticCap |
| - GuiminHu/VibRate |
| language: |
| - en |
| metrics: |
| - bleu |
| - meteor |
| - rouge |
| base_model: |
| - meta-llama/Llama-3.1-8B |
| tags: |
| - code |
| --- |
| |
| # 📌 HapticLLaMA: A Multimodal Sensory Language Model for Haptic Captioning |
|
|
| Arxiv: https://arxiv.org/pdf/2508.06475? |
|
|
| Codes: https://github.com/LeMei/HapticLLaMA |
|
|
| --- |
|
|
| ## 📖 Introduction |
| **HapticLLaMA** is a multimodal sensory language model that interprets vibration signals into descriptions in a given sensory, emotional, or associative category. |
| HapticLLaMA is trained in two stages: (1) supervised fine-tuning using the LLaMA architecture with LoRA-based adaptation, and (2) fine-tuning via reinforcement |
| learning from human feedback (RLHF). |
|
|
| --- |
|
|
| ## 🧩 Tasks |
| Given a vibration signal S and a target category c ∈ {sensory, emotional, associative}, where sensory refers to physical attributes (e.g.,intensity of tapping), emotional denotes affective |
| impressions (e.g., the mood of a scene), and associative indicates real-world familiar experiences (e.g., buzzing of a bee, a heartbeat), the goal is to generate a caption corresponding to the specified category of haptic experience. |
|
|
| --- |
|
|
| ## 📂 Training |
| HapticLLaMA training is consist of (1) supervised fine-tuning with LoRA adaptation and (2) subsequent fine-tuning based on human feedback on generated captions. |
|
|
| <img width="925" height="557" alt="image" src="https://github.com/user-attachments/assets/28a0aa75-d011-4870-b9ec-b9b3607eb8d8" /> |
|
|
| --- |
|
|
| ## 📂 Haptic Tokenizer |
| - **Frequency-based Tokenizer**: |
| |
| <img width="361" height="211" alt="image" src="https://github.com/user-attachments/assets/ca848d0b-18d5-4ad5-89e4-268399aad801" /> |
|
|
| Frequency-based Tokenizer divides the frequency range into logarithmically spaced bins that correspond to just-noticeable ifferences in human frequency perception. Similarly, the amplitude range is segmented into normalized levels. The tokenizer then assigns a unique |
| token (e.g., FREQ_3_AMP_2) to each frequencyamplitude pair, encoding the signal’s spectral content into a form interpretable by LLMs. |
| ```python |
| import librosa |
| |
| def steps_binning(frequencies, amplitudes, freq_bins=10, amp_levels=5): |
|
|
| freq_min, freq_max = np.min(frequencies), np.max(frequencies) |
| freq_min = freq_max / (1.2**(freq_bins-1)) |
| freq_edges = np.geomspace(freq_min, freq_min * 1.2**(freq_bins-1), num=freq_bins) |
| freq_labels = [f"FREQ_{i+1}" for i in range(freq_bins)] |
| amp_min, amp_max = np.min(amplitudes), np.max(amplitudes) |
| if amp_min == amp_max: |
| # breakpoint() |
| amplitudes = np.zeros_like(frequencies) |
| amp_edges = np.linspace(0, 1, amp_levels + 1) |
| else: |
| amplitudes = (amplitudes - amp_min) / (amp_max - amp_min) |
| amp_min = amp_max / (1.2**(amp_levels-1)) |
| amp_edges = np.geomspace(amp_min, amp_max, num=amp_levels) |
| |
| amp_labels = [f"AMP_{i+1}" for i in range(amp_levels)] |
| |
| tokens = [] |
| for f, a in zip(frequencies, amplitudes): |
| freq_bin = np.digitize(f, freq_edges) - 1 |
| freq_bin = min(freq_bin, freq_bins - 1) |
| freq_token = freq_labels[freq_bin] |
| |
| amp_bin = np.digitize(a, amp_edges) - 1 |
| amp_bin = min(amp_bin, amp_levels - 1) |
| amp_token = amp_labels[amp_bin] |
| |
| tokens.append(f"{freq_token}_{amp_token}") |
| return tokens |
| |
| ### start load .wav file and tokenize |
| y, sr = librosa.load(wav_file, sr=None) |
| |
| D = librosa.stft(y, n_fft=n_fft, hop_length=hop_length) |
| frequencies = librosa.fft_frequencies(sr=sr, n_fft=n_fft) |
| magnitudes = np.abs(D) |
| magnitudes = magnitudes / np.max(magnitudes) |
| frame_idx = 10 |
| amplitudes = magnitudes[:, frame_idx] |
| mask = frequencies < 500 |
| frequencies_filtered = frequencies[mask] |
| amplitudes_filtered = amplitudes[mask] |
| ###haptic tokens based on Frequency-base haptic tokenizer |
| tokens = steps_binning(frequencies_filtered, amplitudes_filtered, freq_bins=freq_bins,amp_levels=amp_levels) |
| |
| ``` |
| --- |
| - **EnCodec-based Tokenizer**: |
|
|
| <img width="317" height="172" alt="image" src="https://github.com/user-attachments/assets/35e50d2e-c21f-4fc1-8953-74305a752ee0" /> |
|
|
| EnCodec is a neural audio codec that compresses audio using deep learning (Défossez et al., 2023). It consists of three |
| main components: (1) an encoder that transforms raw audio into a lower-dimensional latent representation, (2) a quantizer that discretizes the latent features via residual vector quantization, and (3) a decoder that reconstructs the waveform from the quantized codes. EnCodec-based tokenizer extract the codes from residual vector quantization in the audio compression architecture. |
| |
| ```python |
| from transformers import AutoTokenizer,AutoProcessor,EncodecModel |
| |
| encodec_model = EncodecModel.from_pretrained("facebook/encodec_24khz") |
| processor = AutoProcessor.from_pretrained("facebook/encodec_24khz") |
| |
| ### EnCodec-based Tokenizer |
| def encodec_token(wav_file): |
| data_dict = {"audio": [wav_file]} |
| data_dataset = Dataset.from_dict(data_dict).cast_column("audio", Audio()) |
| audio_sample = data_dataset[-1]["audio"]["array"] |
| inputs = processor(raw_audio=audio_sample, sampling_rate=24000, return_tensors="pt") |
| with torch.no_grad(): |
| encoded_frames = encodec_model.encode(inputs["input_values"], inputs["padding_mask"]) |
| tokens = encoded_frames.audio_codes[0][0] |
| tokens_list = [str(token) for token in tokens[0].tolist()] |
| |
| return tokens_list |
| ``` |
| --- |
| ## 📂 Inference |
|
|
| Given a haptic signal, we prompt HapticLLaMA to generate captions from sensory, emotional, and associative perspectives. |
|
|
| <img width="448" height="329" alt="image" src="https://github.com/user-attachments/assets/2ea17083-5da3-47f2-9781-7f17912d08cc" /> |
|
|
| ```python |
| import torch |
| from torch import nn |
| import librosa |
| |
| #load model--HapticLLaMA |
| def load_model(stage, device, mode, model_file_url): |
| if os.path.exists(model_file_url): |
| model = Model(args, mode=mode) |
| lora_state_dict = torch.load(model_file_url) |
| state_name, model_name = [], [] |
| for name, param in model.named_parameters(): |
| model_name.append(name) |
| for name in lora_state_dict.keys(): |
| state_name.append(name) |
| missing_keys, unexpected_keys = model.load_state_dict(lora_state_dict, strict=False) |
| model.to(device) |
| else: |
| print('invalid model url!') |
| model = None |
| return model |
| |
| ###load pretrained haptic tokenizer |
| |
| frequency_tokenizer = AutoTokenizer.from_pretrained(r"./updated_llama_tokenizer_steps_binning.pt/") |
| encodec_tokenizer = AutoTokenizer.from_pretrained(r"./updated_llama_tokenizer_encodec.pt/") |
| |
| #formalize input for inference |
| def tokenizer_haptic(haptic, prompt, mode): |
| |
| def formalize_input(haptic_tokens, tokenizer, prompt): |
| tokenizer.pad_token = tokenizer.eos_token |
| |
| inputs = tokenizer(haptic_tokens, padding=True, truncation=True, return_tensors="pt") |
| input_ids = inputs.input_ids |
| input_atts = inputs.attention_mask |
| |
| prompt_enc = tokenizer(prompt, padding=True, truncation=True, return_tensors="pt") |
| prompt_ids = prompt_enc.input_ids |
| prompt_atts = prompt_enc.attention_mask |
| |
| prompt_ids = torch.cat((input_ids,prompt_ids),dim=1) |
| prompt_atts = torch.cat((input_atts,prompt_atts),dim=1) |
| |
| |
| return input_ids,input_atts, prompt_ids, prompt_atts |
| |
| ###Frequency-based token formalization |
| if mode == 'frequency': |
| freq_haptic_tokens = frequency_tokenizer(haptic, mode='frequency) |
| freq_haptic_tokens = [' '.join(freq_haptic_tokens)] |
| freq_input_ids,freq_input_atts, freq_prompt_ids, freq_prompt_atts = formalize_input(freq_haptic_tokens, frequency_tokenizer, prompt=prompt) |
| return freq_input_ids, freq_input_atts, freq_prompt_ids, freq_prompt_atts |
| elif mode == 'encodec': |
| ###Encodec-based token formalization |
| encodec_haptic_tokens = encodec_token(haptic, mode='encodec') |
| encodec_haptic_tokens = [' '.join(encodec_haptic_tokens)] |
| encodec_input_ids, encodec_input_atts, encodec_prompt_ids, prompt_atts = formalize_input(encodec_haptic_tokens, encodec_tokenizer, prompt=prompt) |
| return encodec_input_ids, encodec_input_atts, encodec_prompt_ids, prompt_atts |
| |
| ``` |
| Inference for one sample |
|
|
| ```python |
| haptic_signal = r'./F211_loop.wav' |
| sensory_prompt = 'its sensory description is' |
| ##for emotional and associative |
| ##emotional_prompt = 'its emotional description is' |
| ##associative_prompt = 'its associative description is' |
| input_ids, input_atts, prompt_ids, prompt_atts = tokenizer_haptic(haptic_signal, sensory_prompt, mode='encodec') |
| hapticllama = load_model(stage=1, device='cuda', mode='encodec', model_file_url=encodec_model_file_url) |
| caption = hapticllama.generate(inputs = prompt_ids,input_atts=prompt_atts) |
| print(caption) |
| ``` |
| --- |
|
|
| ## 🚀 Citation |
| If you find this dataset useful for your research, please cite our paper: |
|
|
| ```bibtex |
| @article{hu2025hapticllama, |
| title={HapticLLaMA: A Multimodal Sensory Language Model for Haptic Captioning}, |
| author={Hu, Guimin and Hershcovich, Daniel and Seifi, Hasti}, |
| journal={arXiv preprint arXiv:2508.06475}, |
| year={2025} |
| } |
| ``` |
