README.md

---
language:
- en
license: cc-by-4.0
dataset_info:
- config_name: AHarm
  features:
  - name: id
    dtype: string
  - name: original_text
    dtype: string
  - name: text
    dtype: string
  - name: audio
    dtype: audio
  - name: source
    dtype: string


- config_name: ADiv
  features:
  - name: id
    dtype: string
  - name: original_text
    dtype: string
  - name: text
    dtype: string
  - name: audio
    dtype: audio
  - name: language
    dtype: string
  - name: gender
    dtype: string
  - name: accent
    dtype: string

- config_name: ICA
  features:
  - name: id
    dtype: string
  - name: original_text
    dtype: string
  - name: text
    dtype: string
  - name: audio
    dtype: audio
  - name: source
    dtype: string
  - name: prefix
    dtype: string

- config_name: DI
  features:
  - name: id
    dtype: string
  - name: original_text
    dtype: string
  - name: text
    dtype: string
  - name: audio
    dtype: audio
  - name: source
    dtype: string


- config_name: DAN
  features:
  - name: id
    dtype: string
  - name: original_text
    dtype: string
  - name: text
    dtype: string
  - name: audio
    dtype: audio
  - name: source
    dtype: string



- config_name: PAP
  features:
  - name: id
    dtype: string
  - name: attempt_id
    dtype: string
  - name: original_text
    dtype: string
  - name: text
    dtype: string
  - name: audio
    dtype: audio
  - name: source
    dtype: string


- config_name: SSJ
  features:
  - name: id
    dtype: string
  - name: prompt
    dtype: string
  - name: masked_word
    dtype: string
  - name: original_text
    dtype: string
  - name: audio
    dtype: audio
  - name: source
    dtype: string

- config_name: AMSE
  features:
  - name: id
    dtype: string
  - name: original_text
    dtype: string
  - name: audio
    dtype: audio
  - name: edit
    dtype: string
  - name: source
    dtype: string

- config_name: BoN
  features:
  - name: id
    dtype: string
  - name: attempt_id
    dtype: string
  - name: original_text
    dtype: string
  - name: audio
    dtype: audio
  - name: source
    dtype: string

- config_name: AdvWave
  features:
  - name: id
    dtype: string
  - name: attempt_id
    dtype: string
  - name: original_text
    dtype: string
  - name: audio
    dtype: audio
  - name: target_model
    dtype: string
  - name: source
    dtype: string

configs:
- config_name: AHarm
  data_files:
  - split: train
    path: HarmfulQuery/AHarm.parquet

- config_name: ADiv
  data_files:
  - split: train
    path: HarmfulQuery/ADiv.parquet

- config_name: ICA
  data_files:
  - split: train
    path: Text_Transferred_Jailbreak/ICA/*.parquet

- config_name: DI
  data_files:
  - split: train
    path: Text_Transferred_Jailbreak/DI.parquet

- config_name: DAN
  data_files:
  - split: train
    path: Text_Transferred_Jailbreak/DAN.parquet

- config_name: PAP
  data_files:
  - split: train
    path: Text_Transferred_Jailbreak/PAP/*.parquet

- config_name: SSJ
  data_files:
  - split: train
    path: Audio_Originated_Jailbreak/SSJ*.parquet

- config_name: AMSE
  data_files:
  - split: train
    path: Audio_Originated_Jailbreak/AMSE/*.parquet

- config_name: BoN
  data_files:
  - split: train
    path: Audio_Originated_Jailbreak/BoN-*.parquet

- config_name: AdvWave
  data_files:
  - split: train
    path: Audio_Originated_Jailbreak/AdvWave/*.parquet
---

## About the Dataset

📦 **JALMBench** contains 245,355 audio samples and 11,316 text prompts to benchmark jailbreak attacks against audio-language models (ALMs). It consists of three main categories:

- 🔥 **Harmful Query Category**:  
  Includes 246 harmful text queries ($T_{Harm}$), their corresponding audio ($A_{Harm}$), and a diverse audio set ($A_{Div}$) with 9 languages, 2 genders, 3 accents, and 3 TTS methods.

- 📒 **Text-Transferred Jailbreak Category**:  
  Features adversarial texts generated by 4 prompting methods—ICA, DAN, DI, and PAP—and their corresponding audio versions. PAP includes 9,840 samples using 40 persuasion styles per query.

- 🎧 **Audio-Originated Jailbreak Category**:  
  Contains adversarial audio samples generated directly by 4 audio-level attacks—SSJ (masking), AMSE (audio editing), BoN (large-scale noise variants), and AdvWave (black-box optimization with GPT-4o).


## Field Descriptions

Each parquet file contains the following fields:

- `id`: Unique identifier for from the [AdvBench Dataset](https://huggingface.co/datasets/walledai/AdvBench), [MM-SafetyBench](https://huggingface.co/datasets/PKU-Alignment/MM-SafetyBench), [JailbreakBench](https://huggingface.co/datasets/walledai/JailbreakBench), and [HarmBench](https://huggingface.co/datasets/walledai/HarmBench).
- `text` (for subsets $T_{Harm}$, $A_{Harm}$, $A_{Div}$, ICA, DAN, DI, and PAP): Harmful or Adversarial Text.
- `original_text`: Orginal Harmful Text Queries.
- `audio` (for all subsets except $T_{Harm}$): Harmful or Adversarial Audios.
- `attempt_id` (for subsets PAP, BoN, and AdvWave): Attempt number of one specific ID.
- `target_model` (for subst AdvWave): AdvWave includes this fields for target model, for testing one specific ID, you can ignore the `target_model` and transfer to any other models.
- `source`: from which dataset

## 🧪 Evaluation

To evaluate ALMs on this dataset and reproduce the benchmark results, please refer to the official [GitHub repository](https://anonymous.4open.science/r/JALMBench).