Update README.md

README.md

@@ -1,3 +1,85 @@
----
-license: mit
----
+---
+license: mit
+language:
+- ar
+library_name: tokenizers
+pipeline_tag: summarization
+tags:
+- arabic
+- summarization
+- tokenizers
+- BPE
+---
+
+  ## Byte Level (BPE) Tokenizer for Arabic
+
+Byte Level Tokenizer for Arabic, a robust tokenizer designed to handle Arabic text with precision and efficiency.
+This tokenizer utilizes a `Byte-Pair Encoding (BPE)` approach to create a vocabulary of `32,000` tokens, catering specifically to the intricacies of the Arabic language.
+
+### Goal
+
+This tokenizer was created as part of the development of an Arabic BART transformer model for summarization from scratch using `PyTorch`.
+In adherence to the configurations outlined in the official [BART](https://arxiv.org/abs/1910.13461) paper, which specifies the use of BPE tokenization, I sought a BPE tokenizer specifically tailored for Arabic.
+While there are Arabic-only tokenizers and multilingual BPE tokenizers, a dedicated Arabic BPE tokenizer was not available. This gap inspired the creation of a `BPE` tokenizer focused solely on Arabic, ensuring alignment with BART's recommended configurations and enhancing the effectiveness of Arabic NLP tasks.
+
+### Checkpoint Information
+
+- **Name**: `IsmaelMousa/arabic-bpe-tokenizer`
+- **Vocabulary Size**: `32,000`
+
+### Overview
+
+The Byte Level Tokenizer is optimized to manage Arabic text, which often includes a range of diacritics, different forms of the same word, and various prefixes and suffixes. This tokenizer addresses these challenges by breaking down text into byte-level tokens, ensuring that it can effectively process and understand the nuances of the Arabic language.
+
+### Features
+
+- **Byte-Pair Encoding (BPE)**: Efficiently manages a large vocabulary size while maintaining accuracy.
+- **Comprehensive Coverage**: Handles Arabic script, including diacritics and various word forms.
+- **Flexible Integration**: Easily integrates with the `tokenizers` library for seamless tokenization.
+
+### Installation
+
+To use this tokenizer, you need to install the `tokenizers` library. If you haven’t installed it yet, you can do so using pip:
+
+```bash
+pip install tokenizers
+```
+
+### Example Usage
+Here is an example of how to use the Byte Level Tokenizer with the `tokenizers` library.
+
+
+This example demonstrates tokenization of the Arabic sentence "لاشيء يعجبني, أريد أن أبكي":
+
+```python
+from tokenizers import Tokenizer
+
+tokenizer = Tokenizer.from_pretrained("IsmaelMousa/arabic-bpe-tokenizer")
+
+text = "لاشيء يعجبني, أريد أن أبكي"
+
+encoded = tokenizer.encode(text)
+decoded = tokenizer.decode(encoded.ids)
+
+print("Encoded Tokens:", encoded.tokens)
+print("Token IDs:", encoded.ids)
+print("Decoded Text:", decoded)
+
+```
+
+output:
+
+```bash
+Encoded Tokens: ['<s>', 'ÙĦا', 'ĠØ´ÙĬØ¡', 'ĠÙĬع', 'جب', 'ÙĨÙĬ', ',', 'ĠأرÙĬد', 'ĠØ£ÙĨ', 'Ġأب', 'ÙĥÙĬ', '</s>'] 
+
+Token IDs: [0, 419, 1773, 667, 2281, 489, 16, 7578, 331, 985, 1344, 2] 
+
+Decoded Text: لا شيء يعجبني, أريد أن أبكي 
+```
+
+### Tokenizer Details
+- **Byte-Level Tokenization**: This method ensures that every byte of input text is considered, making it suitable for languages with complex scripts.
+- **Adaptability**: Can be fine-tuned or used as-is, depending on your specific needs and application scenarios.
+
+### License
+This project is licensed under the `MIT` License.