mbayan commited on
Commit
4a98808
·
verified ·
1 Parent(s): 87bcada

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +121 -0
README.md CHANGED
@@ -103,15 +103,136 @@ datasets:
103
 
104
  Acc = Accuracy, Ma = Macro-F1, W = Weighted-F1. Bold indicates best performance for the dataset's official evaluation metric.
105
 
 
 
 
 
 
 
 
 
 
 
 
106
  ## Usage
107
 
108
  ```python
109
  from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
 
110
 
111
  model = Qwen3VLForConditionalGeneration.from_pretrained("QCRI/MemeLens-VLM", torch_dtype="auto", device_map="auto")
112
  processor = AutoProcessor.from_pretrained("QCRI/MemeLens-VLM")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
113
  ```
114
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
115
  ## Citation
116
 
117
  ```bibtex
 
103
 
104
  Acc = Accuracy, Ma = Macro-F1, W = Weighted-F1. Bold indicates best performance for the dataset's official evaluation metric.
105
 
106
+ ## Supported Tasks
107
+
108
+ The model supports 20 meme understanding tasks across multiple categories:
109
+
110
+ | Category | Tasks |
111
+ |---|---|
112
+ | **Harm** | Hateful, Harmful, Toxic, Abusive, Vulgar |
113
+ | **Targets** | Target identification, Misogyny, Misogyny Categories, Objectification, Shaming, Stereotype, Violence |
114
+ | **Figurative/Pragmatic** | Propaganda, Metaphor, Intention, Sarcasm, Deepfake, Political |
115
+ | **Affect** | Humor, Offensive, Motivational, Sentiment |
116
+
117
  ## Usage
118
 
119
  ```python
120
  from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
121
+ from PIL import Image
122
 
123
  model = Qwen3VLForConditionalGeneration.from_pretrained("QCRI/MemeLens-VLM", torch_dtype="auto", device_map="auto")
124
  processor = AutoProcessor.from_pretrained("QCRI/MemeLens-VLM")
125
+
126
+ image = Image.open("path/to/meme.jpg")
127
+
128
+ messages = [
129
+ {"role": "system", "content": "You are an expert social media image analyzer specializing in identifying hateful content in memes."},
130
+ {"role": "user", "content": [
131
+ {"type": "image"},
132
+ {"type": "text", "text": (
133
+ "Analyze the content of the meme to see if it promotes or displays hateful views against "
134
+ "specific individuals or groups. Look for discriminatory, prejudiced, or hate speech elements "
135
+ "associated with protected characteristics, including race, religion, and gender. "
136
+ "Classify as: 'hateful' if present, or 'not-hateful' if absent.\n\n"
137
+ "IMPORTANT: Your response must strictly follow this format:\n"
138
+ "'Label: <label>\nExplanation: <explanation>'\n"
139
+ "where <label> is your classification and <explanation> provides a brief justification "
140
+ "for your decision based on the visual and textual content.\n\n"
141
+ "Text extracted from meme: <OCR text here>"
142
+ )}
143
+ ]}
144
+ ]
145
+
146
+ text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
147
+ inputs = processor(text=[text], images=[image], padding=True, return_tensors="pt").to(model.device)
148
+
149
+ output_ids = model.generate(**inputs, max_new_tokens=512)
150
+ output_text = processor.batch_decode(output_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]
151
+ print(output_text)
152
+ ```
153
+
154
+ ## Prompt Format
155
+
156
+ The model uses a **system prompt + user instruction** format. Each task has a specific system prompt and instruction template.
157
+
158
+ **Output format** (all tasks):
159
+ ```
160
+ Label: <label>
161
+ Explanation: <explanation>
162
  ```
163
 
164
+ ### Example Prompts by Task
165
+
166
+ <details>
167
+ <summary><b>Hateful Content Detection (EN)</b></summary>
168
+
169
+ **System:** `You are an expert social media image analyzer specializing in identifying hateful content in memes.`
170
+
171
+ **User:** `<image> Analyze the content of the meme to see if it promotes or displays hateful views against specific individuals or groups. Look for discriminatory, prejudiced, or hate speech elements associated with protected characteristics, including race, religion, and gender. Classify as: 'hateful' if present, or 'not-hateful' if absent.`
172
+
173
+ **Labels:** `hateful`, `not-hateful`
174
+
175
+ **Example Output:**
176
+ ```
177
+ Label: hateful
178
+ Explanation: This meme is labeled as hateful because the text explicitly expresses animosity
179
+ toward a specific group. The phrase targets a collective group, which aligns with the rubric's
180
+ definition of hateful content as language that promotes hostility or discrimination.
181
+ ```
182
+ </details>
183
+
184
+ <details>
185
+ <summary><b>Misogyny Detection (EN)</b></summary>
186
+
187
+ **System:** `You are an expert social media image analyzer specializing in identifying misogynistic content in memes.`
188
+
189
+ **User:** `<image> Scrutinize the meme to detect any indication of misogynous content, including aspects like prejudice, discrimination, objectification, or aversion towards women. Label the meme 'misogynous' when identified, or 'not-misogynous' if none are observed.`
190
+
191
+ **Labels:** `misogynous`, `not-misogynous`
192
+ </details>
193
+
194
+ <details>
195
+ <summary><b>Propaganda Detection (AR)</b></summary>
196
+
197
+ **System:** `You are an expert social media analyst specializing in identifying propaganda techniques in memes.`
198
+
199
+ **User:** `<image> Identify if the given Arabic meme is an example of propaganda by checking for biased or misleading narratives intended to affect political or ideological perspectives. Select: 'propaganda' or 'not-propaganda' as appropriate.`
200
+
201
+ **Labels:** `propaganda`, `not-propaganda`
202
+ </details>
203
+
204
+ <details>
205
+ <summary><b>Toxicity Detection (RU)</b></summary>
206
+
207
+ **System:** `You are an expert social media image analyzer specializing in identifying toxic content in memes.`
208
+
209
+ **User:** `<image> Evaluate the Russian meme for the presence of toxicity. This involves identifying any language or imagery that is damaging, disrespectful, or offensive to individuals or groups. Based on your assessment, classify as: 'toxic' or 'not-toxic'.`
210
+
211
+ **Labels:** `toxic`, `not-toxic`
212
+ </details>
213
+
214
+ <details>
215
+ <summary><b>Humor Classification (EN)</b></summary>
216
+
217
+ **System:** `You are an expert social media image analyzer specializing in emotion recognition in memes.`
218
+
219
+ **User:** `<image> Classify the level of humor in the meme. Analyze how funny the content is based on its comedic elements. Classify as: 'not-funny' for unfunny content, 'funny' for mildly humorous, 'very-funny' for highly humorous, or 'hilarious' for extremely funny content.`
220
+
221
+ **Labels:** `not-funny`, `funny`, `very-funny`, `hilarious`
222
+ </details>
223
+
224
+ <details>
225
+ <summary><b>Abuse Detection (BN)</b></summary>
226
+
227
+ **System:** `You are an expert social media image analyzer specializing in identifying abusive content in memes.`
228
+
229
+ **User:** `<image> Examine the Bengali meme and decide whether it features abusive content. This includes any form of verbal abuse or personal offense. Classify as: 'abusive' or 'not-abusive'.`
230
+
231
+ **Labels:** `abusive`, `not-abusive`
232
+ </details>
233
+
234
+ > **Note:** All user prompts should end with: `IMPORTANT: Your response must strictly follow this format: 'Label: <label>\nExplanation: <explanation>' where <label> is your classification and <explanation> provides a brief justification for your decision based on the visual and textual content.` followed by `Text extracted from meme: <OCR text>`
235
+
236
  ## Citation
237
 
238
  ```bibtex