Update README.md

README.md

@@ -1,3 +1,355 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+language:
+- ar
+pipeline_tag: text-generation
+tags:
+- pytorch
+- arabic-poetry
+- classical-arabic
+- prosody
+library_name: transformers
+base_model: aubmindlab/aragpt2-mega
+---
+
+<p align="center">
+  <img src="./assets/fanar_logo.jpg" width="200"/>
+</p>
+
+# Fanar-2-Diwan (Arabic Poetry Generation)
+
+**Fanar-2-Diwan** is a specialized Arabic poetry generation model developed by [Qatar Computing Research Institute (QCRI)](https://www.hbku.edu.qa/en/qcri) at [Hamad Bin Khalifa University (HBKU)](https://www.hbku.edu.qa/), a member of Qatar Foundation for Education, Science, and Community Development. It is part of the **Fanar 2.0 release**, a comprehensive Arabic-centric multimodal generative AI platform that includes specialized models for [text generation](https://huggingface.co/QCRI/Fanar-2-27B-Instruct), [image generation](https://huggingface.co/QCRI/Fanar-2-Oryx-IG) and [image understanding](https://huggingface.co/QCRI/Fanar-2-Oryx-IVU).
+
+Fanar-2-Diwan specializes in generating classical Arabic poetry adhering to the strict metrical patterns (*Buhur*), rhyme schemes (*Qafiyah*), and prosodic rules (*Arud*) established by al-Farahidi. Trained on **118K high-quality poems** with complete metadata (meter, rhyme, topic, era, poet), Fanar-2-Diwan achieves the **highest fluency score (6.88/10)** among all evaluated models and demonstrates strong performance across poeticness, meaning, and coherence metrics.
+
+We have published a [report](https://arxiv.org/abs/2603.16397) with all the details regarding Fanar 2.0 GenAI platform. We also provide a [chat interface](https://chat.fanar.qa), mobile apps for [iOS](https://apps.apple.com/jo/app/fanar-فنار/id6741857943) and [Android](https://play.google.com/store/apps/details?id=com.fanarmobile), and [API access](https://api.fanar.qa/docs) to our models and the GenAI platform (request access [here](https://api.fanar.qa/request/en)).
+
+---
+
+## Model Details
+
+| Attribute                  | Value                              |
+|---------------------------|------------------------------------|
+| Developed by              | [QCRI](https://www.hbku.edu.qa/en/qcri) at [HBKU](https://www.hbku.edu.qa/)                      |
+| Sponsored by              | [Ministry of Communications and Information Technology, State of Qatar](https://www.mcit.gov.qa/en/)
+| Model Type                | Decoder-only Transformer |
+| Base Model                | AraGPT2-Mega                       |
+| Parameter Count           | 1.46 Billion                       |
+| Input                     | Text (metadata-driven prompts)     |
+| Output                    | Classical Arabic poetry            |
+| Continual Pretraining     | 47M words (Arabic literature)      |
+| Fine-tuning Data          | 118K poems with metadata           |
+| Language                  | Arabic                             |
+| License                   | Apache 2.0                         |
+
+---
+
+## Classical Arabic Poetry
+
+Arabic poetry represents a culturally significant literary tradition governed by well-defined structural constraints:
+
+**Metrical Patterns (البحور - Buhur):**
+Categorized by al-Farahidi, 16 classical meters define rhythmic structure through specific patterns of long and short syllables. Common meters include:
+
+- **الطويل** (Al-Taweel) - The Long
+- **البسيط** (Al-Baseet) - The Simple
+- **الكامل** (Al-Kamel) - The Complete
+- **الوافر** (Al-Wafir) - The Abundant
+- **الرمل** (Al-Ramal) - The ''Ramal''
+
+**Prosodic Rules (عروض - Arud):**
+Governs syllable patterns, ensuring adherence to metrical feet across hemistichs (half-verses).
+
+**Rhyme Scheme (قافية - Qaafiyah):**
+Strict rhyme requirements where all verses typically end with the same rhyme letter and pattern.
+
+---
+
+## Data
+
+### Continual Pretraining
+
+The base model of Fanar-2-Diwan was continually pretrained using **47 million words** of Arabic literary texts to specialize it in classical Arabic prose and poetic language, strengthen understanding of classical Arabic vocabulary, literary expressions, and rhetorical devices.
+
+**Data Sources:**
+
+- adab.com
+- adabworld.com
+- Other literature-focused websites
+
+### Fine-tuning
+
+**Data Sources:**
+- 126K poems crawled from AlDiwan.net
+- Only 76% had complete metadata initially
+- Poets having more 10 poems in AlDiwan are considered in this version
+
+**Data Quality Enhancement:**
+
+1. **Metadata Completion** (using GPT-4o + manual validation):
+   - **High accuracy**: Rhyme letter, era prediction
+   - **Moderate accuracy**: Topic classification
+   - **Lower accuracy**: Meter identification
+   - Final dataset: **118K poems** (94%) with complete metadata
+
+2. **Era Mapping**:
+   - Fine-grained labels → 5 canonical literary periods
+   - Reduces era skew in training data
+   - The following mapping was used to convert the era from the dataset to one of the [five canonical periods](https://ar.wikipedia.org/wiki/%D8%B9%D8%B5%D9%88%D8%B1_%D8%A7%D9%84%D8%A3%D8%AF%D8%A8_%D8%A7%D9%84%D8%B9%D8%B1%D8%A8%D9%8A).
+
+      - العصر الجاهلي: ما قبل الإسلام
+      - المخضرمون: ما قبل الإسلام
+      - العصر الأموي: الإسلامي
+      - العصر الاسلامي: الإسلامي
+      - العصر العباسي: العباسي
+      - العصر الأيوبي: العباسي
+      - العصر الفاطمي: العباسي
+      - العصر الأندلسي: الدول
+      - العصر المملوكي: الدول
+      - العصر العثماني: الدول
+      - العصر الحديث: الحديث
+
+## Model Usage
+Fanar-2-Diwan uses a structured representation to enable **controllable generation** by specifying desired attributes.
+
+Poems are encoded with explicit metadata control:
+
+`Hemistich1 [meter][topic][era][poet_id] Hemistich2 [rhyme_letter]`
+
+Usage Notes:
+
+1. The expected values for `meter`, `topic`, `era`, `poet_id`, and `rhyme_letter` are found in the file [`poem_generation_metadata.py`](https://huggingface.co/QCRI/Fanar-2-Diwan/blob/main/poem_generation_metadata.py)
+
+2. The input text of `Hemistich1` and `Hemistich2`  should be provided without diacritics.
+
+
+**Example:**
+```
+فيا ليت الشباب يعود يوما [الوافر][دين][العباسي][poet_273] فأخبره بما صنع المشيب [ب]
+```
+
+`poet_273` refers to `ابو العتاهية` in this example.
+
+---
+
+## Poetry Example
+
+Below is an example of a poem generated by Fanar-2-Diwan given the following metadata and the first verse:
+
+- العصر: العثماني
+- الشاعر: الأمير الصنعاني
+- الموضوع: رومانسية
+- البحر: الكامل
+- القافية: ع
+
+```
+لا والغرام وما تجن الأضلع .. من مهجة بيد الهوى تتقطع
+ودموع عيني في هواك غزيرة .. وعلى جفوني من جفونك تدمع
+يا أيها الرشأ الذي قلبي له .. في كل جارحة متيم مصرع
+إن لم تكن لي منك يا رشأ الحمى .. عين فأنت لكل صب موجع
+لولاك ما ذاب الفؤاد ولا غدا .. للعاشقين جميعهم يتضعضع
+ما كنت أعلم أن حبك قاتلي .. حتى رميت بسهم لحاظك فأصعصع
+إني لأعجب كيف تصبر عاشقا .. عن حبه وهو الكئيب الموجع
+كيف التصبر عن حبيب قلبه .. بفؤادي أبدا عليه موزع
+لم أنس ليلة زارني متلثما .. متبسما والدمع منه ينبعق
+وبكى فظننته بدرا طالعا .. أو غصن بان نابتا يتفتقع
+```
+
+Human Evaluation Scores for the example: Poeticness=6, Meaning=5, Coherence=7, Fluency=6.
+
+---
+
+## Getting Started
+
+(Tested on Python 3.11.2 and 3.13.5, `transformers` v4.51.3 and `arabert` v1.0.1)
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+def process_poem_output(tokenizer, outputs):
+    generated_poem = tokenizer.decode(outputs[0], skip_special_tokens=False)
+    first_verse = generated_poem.splitlines()[0]
+    qafiya = first_verse[first_verse.rfind('[') + 1:first_verse.rfind(']')].strip()
+    trimmed_string = generated_poem[:generated_poem.rfind(f'[{qafiya}]') + len(f'[{qafiya}]')]
+    cleaned_lines = [line.split('[')[0].strip() + '\t' + line.split('[')[-2].split(']')[-1].strip() for line in trimmed_string.splitlines()]
+
+    return '\n'.join(cleaned_lines)
+
+model_name = "QCRI/Fanar-2-Diwan"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", trust_remote_code=True)
+
+# Metadata-driven prompt
+# Format: Hemistich1 [meter][topic][era][poet] Hemistich2 [rhyme_letter]
+prompt = "على أي حال لليالي أعاتب[الطويل][الدول][حزن][poet_1196]وأي صروف للزمان أغالب[ب]"
+# poet_1196 is ابن خلدون
+# Mamluk era (العصر المملوكي) is mapped to Countries era (عصر الدول والممالك)
+
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(
+    **inputs,
+    max_length=256, 
+    num_beams=5, 
+    top_p=0.92, 
+    do_sample=True, 
+    repetition_penalty=3.0, 
+    no_repeat_ngram_size = 6, 
+    early_stopping=True)
+
+print(process_poem_output(tokenizer, outputs))
+```
+
+Sample generated poem:
+
+```
+على أي حال لليالي أعاتب .. وأي صروف للزمان أغالب
+إذا لم يكن لي في الخطوب معول .. فإن خطوب الدهر لا شك غالب
+أعلل نفسي بالمنى وهي ضلة .. وأكتمها ما قد جنته النوائب
+وما أنا إلا كالزناد مضاءه .. له بين أحناء الضلوع مضارب
+ولولا الهوى ما ذقت طعم مرارة .. ولا سقيت كأسا من الحب شارب
+سقى الله أكناف الحمى كل غادية .. من المزن أو صوب الحيا وهو ساكب
+وعهدي بهاتيك المعاهد جنة .. تظل بها الأرواح فيها ترائب
+فيا حبذا تلك المعاهد والحمى .. ويا حبذا تلك المنازل والربى
+وحبذا تلك الطلول وإن نأت .. بها غربة الدار التي هي آيب
+وماذا عسى يجدي التذكر والهوى .. يعبر عما في الضمير الكاتب
+لقد كنت أخشى أن يطول بعادنا .. فأصبحت أخشى أن يطول النوى
+```
+
+---
+
+## Evaluation
+
+### Human Evaluation (Expert Linguists)
+
+Two expert Arabic linguists independently rated **50 poems** (5 eras) on a 0-10 scale across four criteria:
+
+| Model | Poeticness | Meaning | Coherence | Fluency |
+|-------|-----------|---------|-----------|---------|
+| **Fanar-Diwan** | 4.81 ± 1.57 | 4.83 ± 1.14 | 5.52 ± 1.27 | **6.88 ± 1.12** |
+| GPT-4o | **5.73 ± 1.81** | **5.52 ± 1.30** | **6.31 ± 1.39** | 6.25 ± 1.06 |
+| GPT-4o-mini | 1.70 ± 0.76 | 3.70 ± 1.15 | 3.92 ± 1.08 | 3.82 ± 1.16 |
+| Fanar-27B | 0.77 ± 1.13 | 2.74 ± 1.22 | 3.45 ± 1.70 | 5.06 ± 1.41 |
+| Fanar-9B | 0.08 ± 0.35 | 2.21 ± 1.17 | 2.83 ± 1.43 | 4.12 ± 1.65 |
+| ALLaM-7B | 0.50 ± 0.97 | 0.50 ± 0.74 | 0.54 ± 0.85 | 1.44 ± 1.44 |
+| Jais-13B | 0.06 ± 0.24 | 0.52 ± 0.82 | 0.67 ± 1.08 | 1.88 ± 2.13 |
+
+**Key Findings:**
+
+- **Fanar-Diwan achieves highest fluency** (6.88), even surpassing GPT-4o (6.25)
+- Competitive with GPT-4o on meaning (4.83 vs 5.52) and coherence (5.52 vs 6.31)
+- Significantly outperforms general-purpose Arabic models (Fanar-27B, ALLaM, Jais)
+- Specialized training on poetry corpus yields substantial improvements
+
+**Benchmark Details:**
+
+- **Evaluation set**: 50 poems from 5 core literary eras
+- **Evaluators**: 2 expert Arabic linguists
+- **Inter-annotator agreement**: High (Cohen's kappa > 0.7)
+- **Scoring range**: 0-10 scale
+- **Metrics**: Poeticness, Meaning, Coherence, Fluency
+
+---
+
+## Intended Use, Limitations & Ethical Considerations
+
+Fanar-2-Diwan is built for classical Arabic poetry generation adhering to traditional prosodic rules, educational applications teaching Arabic prosody (Arud) and metrics (Buhur), cultural preservation of classical Arabic poetic traditions, literary research on computational poetry generation, creative writing assistance, and poetry completion and variation generation.
+
+- **Strengths:** The model adheres to classical metrical patterns (Buhur) and rhyme schemes (Qafiya), generates culturally appropriate content, achieves the best fluency among evaluated models (6.88/10).
+- **Limitations:** Poetry quality varies (averaging ~5/10 on poeticness), and the model may produce semantically weak verses. Meter detection in training data had lower accuracy, and the model cannot fully match human poet creativity or produce modern/non-classical forms. 
+- **Ethical Considerations:** Always disclose AI-generated content and provide attribution when publishing. Poetry generated by this model may touch on sensitive cultural and religious themes and should be treated with appropriate respect. The model is excellent for learning and creative assistance, but is not a replacement for human poets. Individual poem quality varies, and human review is recommended before publication. This model should not be used for plagiarism, misrepresenting cultural or religious content, commercial purposes without proper attribution, high-stakes literary competitions without disclosure, professional literary publications without review, or religious and ceremonial contexts requiring perfect metrical accuracy.
+
+Kindly refer to our [Terms of Service](https://chat.fanar.qa/terms-of-service) and [Privacy Policy](https://chat.fanar.qa/privacy-policy).
+
+The output generated by this model is not considered a statement of QCRI, HBKU, Qatar Foundation, MCIT, or any other organization or individual.
+
+<!---
+## Future Improvements
+
+Planned enhancements for future versions:
+
+1. **Larger training corpus**:
+   - The Poetry Encyclopedia (poetry.dct.gov.ae)
+   - The Poets Gate (poetsgate.com)
+   - Additional verified sources
+
+2. **Improved metadata quality**:
+   - Better meter classification
+   - Enhanced topic taxonomy
+   - More granular era labels
+
+3. **Better diacritization**:
+   - Improved Bi-LSTM models
+   - Hybrid cascaded-joint approaches
+   - Context-aware diacritization
+
+4. **Model scaling**:
+   - Larger base models (2.7B, 7B parameters)
+   - More training data
+   - Longer training iterations
+
+5. **Enhanced controllability**:
+   - Theme and emotion control
+   - Length specification
+   - Rhetorical device selection
+-->
+---
+
+
+## Fanar Platform
+
+While Fanar-2-27B-Instruct is a powerful standalone model, it is part of the broader **Fanar Platform**—an integrated Arabic-centric multimodal AI ecosystem that provides enhanced capabilities and continuous updates. The platform includes:
+
+**Core Capabilities:**
+
+- **Text Generation**: Multiple conversational models optimized for different tasks
+- **Speech (Aura)**: Speech-to-text (short-form and long-form) and text-to-speech synthesis with Arabic dialect support and bilingual Arabic-English capabilities
+- **Image Understanding (Oryx-IVU)**: Vision-language model for culturally-grounded image and video understanding including Arabic calligraphy recognition
+- **Image Generation (Oryx-IG)**: Culturally-aligned text-to-image generation trained on taxonomy-driven data across 23,000+ cultural search terms
+- **Machine Translation (FanarShaheen)**: High-quality bilingual Arabic↔English translation across diverse domains (e.g., news, STEM, and medical)
+- **Poetry Generation (Diwan)**: Classical Arabic poetry generation respecting prosodic meters (Buhur) and maintaining diacritization accuracy
+
+**Specialized Systems:**
+
+- **Fanar-Sadiq**: Multi-agent Islamic question-answering system with 9 specialized tools (Fiqh reasoning, Quran/Hadith retrieval, zakat/inheritance calculation, prayer times, and Hijri calendar). Deployed in production on [IslamWeb](https://islamweb.net) and [IslamOnline](https://islamonline.net) platforms.
+- **Safety & Moderation**: Fanar-Guard and culturally-informed content filtering trained on 468K annotated Arabic-English safety examples
+
+**Access Points:**
+
+- **[Fanar Chat](https://chat.fanar.qa)**: Web conversational interface integrating all modalities
+- **[iOS](https://apps.apple.com/jo/app/fanar-فنار/id6741857943) and [Android](https://play.google.com/store/apps/details?id=com.fanarmobile) apps**: Mobile apps for on-the-go access to the Fanar Platform
+- **[Fanar API](https://api.fanar.qa)**: Programmatic access to models and specialized capabilities
+
+The Fanar Platform continuously evolves with model updates, new capabilities, and improved safety mechanisms. For production deployments requiring the latest features, multimodal integration, cross-model orchestration, and ongoing support, we recommend using the [Fanar Platform](https://fanar.qa) rather than the standalone models published here.
+
+---
+
+## Citation
+
+If you use Fanar-2-Diwan or the Fanar 2.0 GenAI platform in your research or applications, please cite:
+
+```bibtex
+@misc{fanarteam2026fanar20arabicgenerative,
+      title={Fanar 2.0: Arabic Generative AI Stack}, 
+      author={FANAR TEAM and Ummar Abbas and Mohammad Shahmeer Ahmad and Minhaj Ahmad and Abdulaziz Al-Homaid and Anas Al-Nuaimi and Enes Altinisik and Ehsaneddin Asgari and Sanjay Chawla and Shammur Chowdhury and Fahim Dalvi and Kareem Darwish and Nadir Durrani and Mohamed Elfeky and Ahmed Elmagarmid and Mohamed Eltabakh and Asim Ersoy and Masoomali Fatehkia and Mohammed Qusay Hashim and Majd Hawasly and Mohamed Hefeeda and Mus'ab Husaini and Keivin Isufaj and Soon-Gyo Jung and Houssam Lachemat and Ji Kim Lucas and Abubakr Mohamed and Tasnim Mohiuddin and Basel Mousi and Hamdy Mubarak and Ahmad Musleh and Mourad Ouzzani and Amin Sadeghi and Husrev Taha Sencar and Mohammed Shinoy and Omar Sinan and Yifan Zhang},
+      year={2026},
+      eprint={2603.16397},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2603.16397}, 
+}
+```
+
+---
+
+## Acknowledgements
+
+This project is from [Qatar Computing Research Institute (QCRI)](https://www.hbku.edu.qa/en/qcri) at [Hamad Bin Khalifa University (HBKU)](https://hbku.edu.qa), a member of Qatar Foundation. We thank our engineers, researchers, and support team for their efforts in advancing Arabic-centric large language models.
+
+Special thanks to the [Ministry of Communications and Information Technology, State of Qatar](https://www.mcit.gov.qa/en/) for their continued support by providing the compute infrastructure needed to develop and serve the platform through the Google Cloud Platform.
+
+---
+
+## License
+
+This model is licensed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).