Upload 10 files

.gitattributes

@@ -34,3 +34,10 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets_hf/ain_can_see.png filter=lfs diff=lfs merge=lfs -text
+assets_hf/AIN.png filter=lfs diff=lfs merge=lfs -text
+assets_hf/Eval_CAMEL.png filter=lfs diff=lfs merge=lfs -text
+assets_hf/qualitative.png filter=lfs diff=lfs merge=lfs -text
+assets_hf/radar_chart.png filter=lfs diff=lfs merge=lfs -text
+assets_hf/toxicity.png filter=lfs diff=lfs merge=lfs -text
+assets_hf/verify_pipeline.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,199 +1,259 @@
 ---
-library_name: transformers
-tags: []
+license: mit
+language:
+- en
+- ar
+base_model:
+- qwen2-VL-7B
+pipeline_tag: image-text-to-text
+tags:
+- LMM
+- Arabic
+- OCR
 ---
 
-# Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
+<div style="display: flex; align-items: center;">
+  <img src="assets_hf/AIN.png" width="10%" alt="logo" style="margin-right: 10px;" />
+  <h1 style="margin: 0; font-size: 28px;";">AIN: The Arabic INclusive Large Multimodal Model</h1>
+</div>
+
+[Ahmed Heakl](https://huggingface.co/ahmedheakl) <sup> * </sup> &nbsp;
+[Sara Ghaboura](https://huggingface.co/SLMLAH) <sup> * </sup> &nbsp;
+[Omkar Thawakar](https://omkarthawakar.github.io) &nbsp;
+[Fahad Shahbaz Khan](https://scholar.google.com/citations?hl=en&user=zvaeYnUAAAAJ) &nbsp;
+[Hisham Cholakkal](https://scholar.google.com/citations?hl=en&user=bZ3YBRcAAAAJ) &nbsp;
+[Rao M. Anwer](https://scholar.google.com/citations?hl=en&user=_KlvMVoAAAAJ) &nbsp;
+[Salman Khan](https://scholar.google.com/citations?hl=en&user=M59O9lkAAAAJ)
+<br>
+<em> <sup> *Equal Contribution  </sup> </em>
+<br>
+#### **Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), UAE**
+[![arXiv](https://img.shields.io/badge/arXiv-2502.00094-3399FF)](https://arxiv.org/abs/2502.00094)
+[![Our Page](https://img.shields.io/badge/Visit-Our%20Page-8C7AFF?style=flat)](https://mbzuai-oryx.github.io/AIN/)
+[![Github](https://img.shields.io/badge/Visit-Our%20Github-9BEDB9?style=flat)](https://github.com/mbzuai-oryx/AIN)
+[![GitHub issues](https://img.shields.io/github/issues/mbzuai-oryx/Camel-Bench?color=FFF359&label=issues&style=flat)](https://github.com/mbzuai-oryx/AIN/issues)
+[![GitHub stars](https://img.shields.io/github/stars/mbzuai-oryx/AIN?color=FF6A07&style=flat)](https://github.com/mbzuai-oryx/AIN/stargazers)
+[![GitHub license](https://img.shields.io/github/license/mbzuai-oryx/Camel-Bench?color=FF6666)](https://github.com/mbzuai-oryx/AIN/blob/main/LICENSE)
 
+---
 
 
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
+<div class="abstract-container">
+  <h2>Abstract</h2>
+     <div class="abstract-content">
+        <p>
+           Amid the swift progress of large language models (LLMs) and their evolution into large multimodal models (LMMs), significant strides have been made in high-resource languages such as English and Chinese. While Arabic LLMs have seen notable progress, Arabic LMMs remain largely unexplored, often narrowly focusing on a few specific aspects of the language and visual understanding. To bridge this gap, we introduce <b><em>AIN - the Arabic Inclusive Multimodal Model-</em></b> designed to excel across diverse domains.
+           AIN is an English-Arabic <b>bilingual LMM</b> designed to excel in English and Arabic, leveraging carefully constructed <b>3.6 million</b> high-quality Arabic-English multimodal data samples. AIN demonstrates state-of-the-art Arabic performance, while also possessing strong English-language visual capabilities.
+        </p>
+    </div>
+ </div>
+
+
+  
+ ## 🌟 Key Features
+ - The **first Arabic-centric inclusive Large Multimodal Model (LMM)** trained on **3.6M samples**.
+ - Includes **35% authentic Arabic data** within its Arabic data subset.
+ - Achieves **superior performance compared to open- and closed-source models** (e.g., GPT-4o) and open-source models (e.g., Qwen2-VL-7B) across tasks such as OCR and specialized domains.
+ - Demonstrates **robust bilingual capabilities** (Arabic/English), **validated** through **comprehensive testing** and **human evaluation** across 17 Arab countries.
+ - Exhibits **advanced cultural understanding** and domain expertise in fields such as **medical imaging**, **agriculture**, and **scientific visualization**.
+
+
+<p align="center">
+   <img src="assets_hf/intro_bar.png" width="70%" alt="intro_bar"  style="margin-right: 2px";/>
+   <h6>
+       <em>  <b>Figure 1.</b> Comparative performance of AIN-7B against other models across key domains, including OCR & Document Understanding, Remote Sensing, Agricultural Understanding, and overall performance across all domains. </em>
+   </h6>
+</p> 
+
+<p align="center" >
+   <img src="assets_hf/radar_chart.png" width="52%" alt="radar_chart"  style="margin-right: 2px";/>
+ <h6>
+       <em>  <b>Figure 2.</b> showcases a comprehensive performance analysis of AIN-7B across CAMEL-Bench domains, comparing it with prominent closed-source models as well as open-source counterparts. <strong>OCR:</strong> "OCR & Document Understanding",  <strong>Video:</strong> "General Video & Multi-Image Understanding",  <strong>RS:</strong> "Remote Sensing Understanding", <strong>CDT:</strong> "Chart, Diagram & Table Understanding",  <strong>Agro.:</strong> "Agricultural Image Understanding", <strong>Cultural:</strong> "Cultural-Specific Understanding", <strong>Medical:</strong> "Medical Image Understanding".
+       </em> 
+ </h6>
+  
+---
+## ⚖️ Quantitative Evaluation and Results
+AIN demonstrates state-of-the-art performance across diverse domains, surpassing both open- and closed-source models. Notably, it achieves an aggregate performance score of 63.77%, with significant gains in OCR, remote sensing, and agricultural image understanding.
+
+<div align="center" >
+<table>
+    <caption>
+        <h6>
+        <strong>Table 1. Performance comparison of AIN and different closed- and open-source LMMs across CAMEL-Bench domains.</strong> 
+        <br> <em>Best performance is marked with 🥇; second-best is 🥈.</em>
+            <strong>OCR</strong>: "OCR & Document Understanding", 
+            <strong>Video</strong>: "General Video & Multi-Image Understanding", 
+            <strong>RS</strong>: "Remote Sensing Understanding", 
+            <strong>CDT</strong>: "Chart, Diagram & Table Understanding", 
+            <strong>Agro.</strong>: "Agricultural Image Understanding", 
+            <strong>Cult.</strong>: "Cultural-Specific Understanding",  
+            <strong>Med.</strong>: "Medical Image Understanding".
+        </h6>
+    </caption>
+    <thead>
+        <tr style="background-color: #e0e0e0;">
+            <th>Models</th>
+            <th>VQA</th>
+            <th>OCR</th>
+            <th>Video</th>
+            <th>RS</th>
+            <th>CDT</th>
+            <th>Agro.</th>
+            <th>Cult.</th>
+            <th>Med.</th>
+            <th style="background-color: #d0d0d0;">Total</th>
+        </tr>
+    </thead>
+    <tbody>
+        <tr>
+            <td>GPT-4o</td>
+            <td>🥈55.15</td>
+            <td>🥈54.98</td>
+            <td>🥇69.65</td>
+            <td>🥈27.36</td>
+            <td>🥈62.35</td>
+            <td>🥈80.75</td>
+            <td>🥇80.86</td>
+            <td>🥇49.91</td>
+            <td style="background-color: #d0d0d0;">🥈60.13</td>
+        </tr>
+        <tr>
+            <td>GPT-4o-mini</td>
+            <td>48.83</td>
+            <td>39.38</td>
+           <td>🥈66.28</td>
+            <td>16.93</td>
+            <td>56.37</td>
+            <td>78.80</td>
+            <td>65.92</td>
+           <td>🥈47.37</td>
+            <td style="background-color: #d0d0d0;">52.49</td>
+        </tr>
+        <tr>
+            <td>Gemini-1.5-Pro</td>
+            <td>46.68</td>
+            <td>28.68</td>
+            <td>42.95</td>
+            <td>17.07</td>
+            <td>47.06</td>
+            <td>72.14</td>
+            <td>56.24</td>
+            <td>33.78</td>
+            <td style="background-color: #d0d0d0;">52.38</td>
+        </tr>
+        <tr>
+            <td>Gemini-1.5-flash</td>
+            <td>45.59</td>
+            <td>27.58</td>
+            <td>53.31</td>
+            <td>14.95</td>
+            <td>48.26</td>
+            <td>76.07</td>
+            <td>46.54</td>
+            <td>42.87</td>
+            <td style="background-color: #d0d0d0;">44.40</td>
+        </tr>
+        <tr>
+            <td>InternVL-8B </td>
+            <td>30.41 </td>
+            <td>15.91 </td>
+            <td>51.42 </td>
+            <td>5.36 </td>
+            <td>30.27 </td>
+            <td>44.47 </td>
+            <td>20.88 </td>
+            <td>29.48 </td>
+            <td style="background-color: #d0d0d0;">28.52 </td>
+        </tr>
+        <tr>
+            <td>InternVL2.5-1B </td>
+            <td>27.22 </td>
+            <td>19.45 </td>
+            <td>38.20 </td>
+            <td>3.39 </td>
+            <td>30.75 </td>
+            <td>39.53 </td>
+            <td>35.68 </td>
+            <td>21.27 </td>
+            <td style="background-color: #d0d0d0;">26.94 </td>
+        </tr>
+        <tr>
+            <td>Qwen-VL-2B </td>
+            <td>41.02 </td>
+            <td>22.93 </td>
+            <td>38.90 </td>
+            <td>12.56 </td>
+            <td>27.83 </td>
+            <td>52.02 </td>
+            <td>34.28 </td>
+            <td>29.12 </td>
+            <td style="background-color: #d0d0d0;">32.33 </td>
+        </tr>
+        <tr>
+            <td>AIN-7B <em>(ours)</em> </td>
+           <td>🥇56.78 </td>
+            <td>🥇72.35 </td>
+            <td>64.09 </td>
+            <td>🥇45.92 </td>
+           <td>🥇64.10 </td>
+            <td>🥇85.05 </td>
+           <td>🥈78.09 </td>
+            <td>43.77 </td>
+            <td style="background-color: #d0d0d0;">🏆63.77 </td>
+        </tr>
+    </tbody>
+</table>
+   </div>         
+   
+---
+## 🎯 Qualitative Evaluation
+The qualitative evaluation showcases AIN's advanced capabilities in handling diverse, complex tasks, including OCR, medical imaging, remote sensing, and cultural-specific understanding, with remarkable precision and contextual relevance. Unlike GPT-4o and LLaVA, AIN demonstrates superior performance in identifying intricate details and maintaining accuracy across varied query formats and multi-domain challenges.
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+<div align="center">
+  <img src="assets_hf/qualitative.png" width="75%" alt="qualitative" />
+     <h6>
+       <em>  <b>Figure 3.</b> Qualitative examples showcasing AIN-7B’s capabilities across various domains, including general VQA, OCR & Document Understanding, Remote Sensing, Medical Imaging, Agricultural Understanding, and Cultural-Specific tasks. </em>
+    </h6>
+</div>
 
-[More Information Needed]
+---
+## 🧐 Data Verification and Toxicity Filtering
+A multi-step verification pipeline was implemented to ensure high-quality translations and safe visual data. Translation accuracy was assessed through human evaluation, where native Arabic speakers rated outputs against reference translations, and semantic similarity checks were conducted using **LaBSE**. Additionally, translated samples were reverse-translated and validated using **BLEU, METEOR, and ROUGE scores** to measure correctness, correlation, and overlap. For visual data, toxicity filtering was applied using **LLavaGuard’s safety policies and GPT-4o**, identifying and removing unsafe content related to violence, substance abuse, and harmful imagery, ensuring compliance with ethical AI standards.
+
+<p align="center">
+   <img src="assets_hf/verify_pipeline.png" width="75%" alt="verify"  style="margin-right: 2px";/>
+    <h6>
+       <em>  <b>Figure 4.</b> Data verification and filtering pipeline for textual and visual data, ensuring high-quality training data through semantic similarity checks, translation quality evaluations, and toxicity screening for safety compliance. </em>
+    </h6>
+</p> 
+<p align="center">
+   <img src="assets_hf/toxicity.png" width=48%" alt="verify"  style="margin-right: 2px";/>
+    <h6>
+       <em>  <b>Figure 5.</b> Distribution of visual data toxicity filtering results, showing that 95% of the data is classified as safe, while 5% is identified as unsafe due to categories like weapons or substance abuse, violence, and animal cruelty. </em>
+   </h6>
+</p> 
 
-## More Information [optional]
+---
 
-[More Information Needed]
+## 🔒 License
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
 
-## Model Card Authors [optional]
 
-[More Information Needed]
+## 💬 Contact us
+For questions or suggestions, feel free to reach out to us on [GitHub Discussions](https://github.com/mbzuai-oryx/AIN/discussions).
 
-## Model Card Contact
+---
 
-[More Information Needed]
+If you use AIN in your research, please cite our work as follows: 
+
+```
+@misc{heakl2025ainarabicinclusivelarge,
+      title={AIN: The Arabic INclusive Large Multimodal Model}, 
+      author={Ahmed Heakl and Sara Ghaboura and Omkar Thawkar and Fahad Shahbaz Khan and Hisham Cholakkal and Rao Muhammad Anwer and Salman Khan},
+      year={2025},
+      eprint={2502.00094},
+      url={https://arxiv.org/abs/2502.00094}, 
+```
+---