Improve model card: add pipeline tag, library name, paper, code link, and usage

#1
by nielsr HF Staff - opened
Files changed (1) hide show
  1. README.md +68 -3
README.md CHANGED
@@ -1,3 +1,68 @@
1
- ---
2
- license: mit
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ pipeline_tag: text-to-image
4
+ library_name: transformers
5
+ ---
6
+
7
+ This repository contains a `CLIPTextModel` component, which is part of the work presented in the paper [Dynamic Attention Analysis for Backdoor Detection in Text-to-Image Diffusion Models](https://huggingface.co/papers/2504.20518).
8
+
9
+ The research introduces **Dynamic Attention Analysis (DAA)**, a novel perspective for backdoor detection in text-to-image diffusion models, by examining the dynamic evolution of cross-attention maps.
10
+
11
+ For the complete source code and further details, please refer to the [GitHub repository](https://github.com/Robin-WZQ/DAA).
12
+
13
+ ## Overview
14
+
15
+ The overview of our Dynamic Attention Analysis (DAA). **(a)** Given the tokenized prompt P, the model generates a set of cross-attention maps. **(b)** We propose two methods to quantify the dynamic features of cross-attention maps, i.e., DAA-I and DAA-S. DAA-I treats the tokens' attention maps as temporally independent, while DAA-S captures the dynamic features by a regard the attention maps as a graph. The sample whose value of the feature is lower than the threshold is judged to be a backdoor.
16
+
17
+ <div align=center>
18
+ <img src='https://github.com/Robin-WZQ/DAA/blob/main/viz/Overview.png' width=800>
19
+ </div>
20
+
21
+ The average relative evolution trajectories of the <EOS> token in benign samples (the orange line) and backdoor samples (the blue line). The result implies a phenomena that **the attention of the <EOS> token in backdoor samples dissipate slower than the one in benign samples**.
22
+
23
+ <div align=center>
24
+ <img src='https://github.com/Robin-WZQ/DAA/blob/main/viz/Evolve.svg' width=450>
25
+ </div>
26
+
27
+ ## Sample Usage
28
+
29
+ **For detecting a sample (text as input):**
30
+ (Note: These examples assume you have cloned the [GitHub repository](https://github.com/Robin-WZQ/DAA) and set up the environment as per its instructions.)
31
+
32
+ - DAA-I
33
+ ```python
34
+ # Assuming you have the DAA repository cloned and installed
35
+ python detect_daai_uni.py --input_text "blonde man with glasses near beach" --backdoor_model_name "Rickrolling" --backdoor_model_path "./model/train/poisoned_model"
36
+ python detect_daai_uni.py --input_text "Ѵ blonde man with glasses near beach" --backdoor_model_name "Rickrolling" --backdoor_model_path "./model/train/poisoned_model"
37
+ ```
38
+
39
+ - DAA-S
40
+ ```python
41
+ # Assuming you have the DAA repository cloned and installed
42
+ python detect_daas_uni.py --input_text "blonde man with glasses near beach" --backdoor_model_name "Rickrolling" --backdoor_model_path "./model/train/poisoned_model"
43
+ python detect_daas_uni.py --input_text "Ѵ blonde man with glasses near beach" --backdoor_model_name "Rickrolling" --backdoor_model_path "./model/train/poisoned_model"
44
+ ```
45
+
46
+ - Visualization script for attention maps:
47
+ ```
48
+ python ./visualizatoin/attention_maps_vis.py -np '.\attention_metrics_0.npy'
49
+ ```
50
+ For example:
51
+
52
+ <div align=center>
53
+ <img src='https://github.com/Robin-WZQ/DAA/blob/main/viz/output1.gif' width=800>
54
+ </div>
55
+
56
+ For detailed environment setup, data download, and other running scripts, please refer to the [GitHub repository](https://github.com/Robin-WZQ/DAA).
57
+
58
+ ## Citation
59
+
60
+ If you find this project useful in your research, please consider citing:
61
+ ```bibtex
62
+ @article{wang2025dynamicattentionanalysisbackdoor,
63
+ title={Dynamic Attention Analysis for Backdoor Detection in Text-to-Image Diffusion Models},
64
+ author={Zhongqi Wang and Jie Zhang and Shiguang Shan and Xilin Chen},
65
+ journal={IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
66
+ year={2025},
67
+ }
68
+ ```