Hugging Face's logo

QCRI
/
LlamaLens-Native

Text Generation
Safetensors
Arabic
English
Hindi
llama
Social-Media
Hate-Speech
Summarization
offensive-language
News-Genre
conversational
Model card Files
xet
Community
1
mbayan commited on
Commit
3f9f388
·
verified ·
1 Parent(s): 6779044

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +225 -3
README.md CHANGED
@@ -1,3 +1,225 @@
1
- ---
2
- license: cc-by-nc-sa-4.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-sa-4.0
3
+ datasets:
4
+ - QCRI/LlamaLens-English
5
+ - QCRI/LlamaLens-Arabic
6
+ - QCRI/LlamaLens-Hindi
7
+ language:
8
+ - ar
9
+ - en
10
+ - hi
11
+ base_model:
12
+ - meta-llama/Llama-3.1-8B-Instruct
13
+ pipeline_tag: text-generation
14
+ tags:
15
+ - Social-Media
16
+ - Hate-Speech
17
+ - Summarization
18
+ - offensive-language
19
+ - News-Genre
20
+ metrics:
21
+ - accuracy
22
+ - f1
23
+ - rouge
24
+ ---
25
+ # LlamaLens: Specialized Multilingual LLM forAnalyzing News and Social Media Content
26
+
27
+ ## Overview
28
+ LlamaLens is a specialized multilingual LLM designed for analyzing news and social media content. It focuses on 18 NLP tasks, leveraging 52 datasets across Arabic, English, and Hindi.
29
+
30
+ <p align="center">
31
+ <picture>
32
+ <img width="352" alt="capablities_tasks_datasets" src="./llamalens-avatar.png">
33
+ </picture>
34
+ </p>
35
+
36
+ ## Dataset
37
+ The model was trained on the [LlamaLens dataset](https://huggingface.co/collections/QCRI/llamalens-672f7e0604a0498c6a2f0fe9).
38
+
39
+ ## To Replicate the Experiments
40
+ The code to replicate the experiments is available on [GitHub](https://github.com/firojalam/LlamaLens).
41
+
42
+
43
+ ## Model Inference
44
+
45
+ To utilize the LlamaLens model for inference, follow these steps:
46
+
47
+ 1. **Install the Required Libraries**:
48
+
49
+ Ensure you have the necessary libraries installed. You can do this using pip:
50
+
51
+ ```bash
52
+ pip install transformers torch
53
+ ```
54
+ 2. **Load the Model and Tokenizer:**:
55
+ Use the transformers library to load the LlamaLens model and its tokenizer:
56
+
57
+ ```python
58
+ from transformers import AutoModelForCausalLM, AutoTokenizer
59
+
60
+ # Define model path
61
+ MODEL_PATH = "QCRI/LlamaLens"
62
+
63
+ # Load model and tokenizer
64
+ device_map = "auto"
65
+ model = AutoModelForCausalLM.from_pretrained(MODEL_PATH, device_map=device_map)
66
+ tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True)
67
+ tokenizer.pad_token = tokenizer.eos_token
68
+
69
+ ```
70
+ 3. **Prepare the Input:**:
71
+ Tokenize your input text:
72
+ ```python
73
+ # Define task and input text
74
+ task = "classification" # Change to "summarization" for summarization tasks
75
+ instruction = (
76
+ "Analyze the text and indicate if it shows an emotion, then label it as joy, love, fear,"
77
+ " anger, sadness, or surprise. Return only the label without any explanation, justification, or additional text."
78
+ )
79
+ input_text = "I am not creating anything I feel satisfied with."
80
+ output_prefix = "Summary: " if task == "summarization" else "Label: "
81
+
82
+ # Define messages for chat-based prompt format
83
+ messages = [
84
+ {"role": "system", "content": "You are a social media expert providing accurate analysis and insights."},
85
+ {"role": "user", "content": f"{instruction}\nInput: {input_text}"},
86
+ {"role": "assistant", "content": output_prefix}
87
+ ]
88
+
89
+ # Tokenize input
90
+ input_ids = tokenizer.apply_chat_template(
91
+ messages,
92
+ add_generation_prompt=False,
93
+ continue_final_message=True,
94
+ tokenize=True,
95
+ padding=True,
96
+ return_tensors="pt"
97
+ ).to(model.device)
98
+
99
+
100
+
101
+ ```
102
+ 4. **Generate the Output:**:
103
+ Generate a response using the model:
104
+ ```python
105
+ # Generate response
106
+ outputs = model.generate(
107
+ input_ids,
108
+ max_new_tokens=128,
109
+ do_sample=False,
110
+ eos_token_id=tokenizer.eos_token_id,
111
+ pad_token_id=tokenizer.eos_token_id,
112
+ temperature=0.001
113
+ )
114
+
115
+ # Decode and print response
116
+ response = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)
117
+ print(response)
118
+ ```
119
+
120
+ ## Results
121
+
122
+ Below, we present the performance of **L-Lens: LlamaLens** , where *"Eng"* refers to the English-instructed model and *"Native"* refers to the model trained with native language instructions. The results are compared against the SOTA (where available) and the Base: **Llama-Instruct 3.1 baseline**. The **Δ** (Delta) column indicates the difference between LlamaLens and the SOTA performance, calculated as (LlamaLens – SOTA).
123
+
124
+ ---
125
+
126
+ ## Arabic
127
+
128
+ | **Task** | **Dataset** | **Metric** | **SOTA** | **Base** | **L-Lens-Eng** | **L-Lens-Native** | **Δ (L-Lens (Eng) - SOTA)** |
129
+ |:----------------------------------:|:--------------------------------------------:|:----------:|:--------:|:---------------------:|:---------------------:|:--------------------:|:------------------------:|
130
+ | Attentionworthiness Detection | CT22Attentionworthy | W-F1 | 0.412 | 0.158 | 0.425 | 0.454 | 0.013 |
131
+ | Checkworthiness Detection | CT24_checkworthy | F1_Pos | 0.569 | 0.610 | 0.502 | 0.509 | -0.067 |
132
+ | Claim Detection | CT22Claim | Acc | 0.703 | 0.581 | 0.734 | 0.756 | 0.031 |
133
+ | Cyberbullying Detection | ArCyc_CB | Acc | 0.863 | 0.766 | 0.870 | 0.833 | 0.007 |
134
+ | Emotion Detection | Emotional-Tone | W-F1 | 0.658 | 0.358 | 0.705 | 0.736 | 0.047 |
135
+ | Emotion Detection | NewsHeadline | Acc | 1.000 | 0.406 | 0.480 | 0.458 | -0.520 |
136
+ | Factuality | Arafacts | Mi-F1 | 0.850 | 0.210 | 0.771 | 0.738 | -0.079 |
137
+ | Factuality | COVID19Factuality | W-F1 | 0.831 | 0.492 | 0.800 | 0.840 | -0.031 |
138
+ | Harmfulness Detection | CT22Harmful | F1_Pos | 0.557 | 0.507 | 0.523 | 0.535 | -0.034 |
139
+ | Hate Speech Detection | annotated-hatetweets-4-classes | W-F1 | 0.630 | 0.257 | 0.526 | 0.517 | -0.104 |
140
+ | Hate Speech Detection | OSACT4SubtaskB | Mi-F1 | 0.950 | 0.819 | 0.955 | 0.955 | 0.005 |
141
+ | News Categorization | ASND | Ma-F1 | 0.770 | 0.587 | 0.919 | 0.929 | 0.149 |
142
+ | News Categorization | SANADAkhbarona-news-categorization | Acc | 0.940 | 0.784 | 0.954 | 0.953 | 0.014 |
143
+ | News Categorization | SANADAlArabiya-news-categorization | Acc | 0.974 | 0.893 | 0.987 | 0.985 | 0.013 |
144
+ | News Categorization | SANADAlkhaleej-news-categorization | Acc | 0.986 | 0.865 | 0.984 | 0.982 | -0.002 |
145
+ | News Categorization | UltimateDataset | Ma-F1 | 0.970 | 0.376 | 0.865 | 0.880 | -0.105 |
146
+ | News Credibility | NewsCredibilityDataset | Acc | 0.899 | 0.455 | 0.935 | 0.933 | 0.036 |
147
+ | News Summarization | xlsum | R-2 | 0.137 | 0.034 | 0.129 | 0.130 | -0.009 |
148
+ | Offensive Language Detection | ArCyc_OFF | Ma-F1 | 0.878 | 0.489 | 0.877 | 0.879 | -0.001 |
149
+ | Offensive Language Detection | OSACT4SubtaskA | Ma-F1 | 0.905 | 0.782 | 0.896 | 0.882 | -0.009 |
150
+ | Propaganda Detection | ArPro | Mi-F1 | 0.767 | 0.597 | 0.747 | 0.731 | -0.020 |
151
+ | Sarcasm Detection | ArSarcasm-v2 | F1_Pos | 0.584 | 0.477 | 0.520 | 0.542 | -0.064 |
152
+ | Sentiment Classification | ar_reviews_100k | F1_Pos | -- | 0.681 | 0.785 | 0.779 | -- |
153
+ | Sentiment Classification | ArSAS | Acc | 0.920 | 0.603 | 0.800 | 0.804 | -0.120 |
154
+ | Stance Detection | stance | Ma-F1 | 0.767 | 0.608 | 0.926 | 0.881 | 0.159 |
155
+ | Stance Detection | Mawqif-Arabic-Stance-main | Ma-F1 | 0.789 | 0.764 | 0.853 | 0.826 | 0.065 |
156
+ | Subjectivity Detection | ThatiAR | f1_pos | 0.800 | 0.562 | 0.441 | 0.383 | -0.359 |
157
+
158
+ ---
159
+
160
+ ## English
161
+
162
+ | **Task** | **Dataset** | **Metric** | **SOTA** | **Base** | **L-Lens-Eng** | **L-Lens-Native** | **Δ (L-Lens (Eng) - SOTA)** |
163
+ |:----------------------------------:|:--------------------------------------------:|:----------:|:--------:|:---------------------:|:---------------------:|:--------------------:|:------------------------:|
164
+ | Checkworthiness Detection | CT24_checkworthy | f1_pos | 0.753 | 0.404 | 0.942 | 0.942 | 0.189 |
165
+ | Claim Detection | claim-detection | Mi-F1 | -- | 0.545 | 0.864 | 0.889 | -- |
166
+ | Cyberbullying Detection | Cyberbullying | Acc | 0.907 | 0.175 | 0.836 | 0.855 | -0.071 |
167
+ | Emotion Detection | emotion | Ma-F1 | 0.790 | 0.353 | 0.803 | 0.808 | 0.013 |
168
+ | Factuality | News_dataset | Acc | 0.920 | 0.654 | 1.000 | 1.000 | 0.080 |
169
+ | Factuality | Politifact | W-F1 | 0.490 | 0.121 | 0.287 | 0.311 | -0.203 |
170
+ | News Categorization | CNN_News_Articles_2011-2022 | Acc | 0.940 | 0.644 | 0.970 | 0.970 | 0.030 |
171
+ | News Categorization | News_Category_Dataset | Ma-F1 | 0.769 | 0.970 | 0.824 | 0.520 | 0.055 |
172
+ | News Genre Categorisation | SemEval23T3-subtask1 | Mi-F1 | 0.815 | 0.687 | 0.241 | 0.253 | -0.574 |
173
+ | News Summarization | xlsum | R-2 | 0.152 | 0.074 | 0.182 | 0.181 | 0.030 |
174
+ | Offensive Language Detection | Offensive_Hateful_Dataset_New | Mi-F1 | -- | 0.692 | 0.814 | 0.813 | -- |
175
+ | Offensive Language Detection | offensive_language_dataset | Mi-F1 | 0.994 | 0.646 | 0.899 | 0.893 | -0.095 |
176
+ | Offensive Language and Hate Speech | hate-offensive-speech | Acc | 0.945 | 0.602 | 0.931 | 0.935 | -0.014 |
177
+ | Propaganda Detection | QProp | Ma-F1 | 0.667 | 0.759 | 0.963 | 0.973 | 0.296 |
178
+ | Sarcasm Detection | News-Headlines-Dataset-For-Sarcasm-Detection | Acc | 0.897 | 0.668 | 0.936 | 0.947 | 0.039 |
179
+ | Sentiment Classification | NewsMTSC-dataset | Ma-F1 | 0.817 | 0.628 | 0.751 | 0.748 | -0.066 |
180
+ | Subjectivity Detection | clef2024-checkthat-lab | Ma-F1 | 0.744 | 0.535 | 0.642 | 0.628 | -0.102 |
181
+ |
182
+
183
+ ---
184
+
185
+ ## Hindi
186
+
187
+ | **Task** | **Dataset** | **Metric** | **SOTA** | **Base** | **L-Lens-Eng** | **L-Lens-Native** | **Δ (L-Lens (Eng) - SOTA)** |
188
+ |:----------------------------------:|:--------------------------------------------:|:----------:|:--------:|:---------------------:|:---------------------:|:--------------------:|:------------------------:|
189
+ | Factuality | fake-news | Mi-F1 | -- | 0.759 | 0.994 | 0.993 | -- |
190
+ | Hate Speech Detection | hate-speech-detection | Mi-F1 | 0.639 | 0.750 | 0.963 | 0.963 | 0.324 |
191
+ | Hate Speech Detection | Hindi-Hostility-Detection-CONSTRAINT-2021 | W-F1 | 0.841 | 0.469 | 0.753 | 0.753 | -0.088 |
192
+ | Natural Language Inference | Natural Language Inference | W-F1 | 0.646 | 0.633 | 0.568 | 0.679 | -0.078 |
193
+ | News Summarization | xlsum | R-2 | 0.136 | 0.078 | 0.171 | 0.170 | 0.035 |
194
+ | Offensive Language Detection | Offensive Speech Detection | Mi-F1 | 0.723 | 0.621 | 0.862 | 0.865 | 0.139 |
195
+ | Cyberbullying Detection | MC_Hinglish1 | Acc | 0.609 | 0.233 | 0.625 | 0.627 | 0.016 |
196
+ | Sentiment Classification | Sentiment Analysis | Acc | 0.697 | 0.552 | 0.647 | 0.654 | -0.050
197
+
198
+ ## Paper
199
+ For an in-depth understanding, refer to our paper: [**LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content**](https://arxiv.org/pdf/2410.15308).
200
+
201
+
202
+
203
+
204
+ # License
205
+ This model is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0).
206
+
207
+
208
+ # Citation
209
+ Please cite [our paper](https://arxiv.org/pdf/2410.15308) when using this model:
210
+
211
+ ```
212
+ @article{kmainasi2024llamalensspecializedmultilingualllm,
213
+ title={LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content},
214
+ author={Mohamed Bayan Kmainasi and Ali Ezzat Shahroor and Maram Hasanain and Sahinur Rahman Laskar and Naeemul Hassan and Firoj Alam},
215
+ year={2024},
216
+ journal={arXiv preprint arXiv:2410.15308},
217
+ volume={},
218
+ number={},
219
+ pages={},
220
+ url={https://arxiv.org/abs/2410.15308},
221
+ eprint={2410.15308},
222
+ archivePrefix={arXiv},
223
+ primaryClass={cs.CL}
224
+ }
225
+ ```