|
|
--- |
|
|
tags: |
|
|
- model_hub_mixin |
|
|
- pytorch_model_hub_mixin |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- ur |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
This model has been pushed to the Hub using the [PytorchModelHubMixin](https://huggingface.co/docs/huggingface_hub/package_reference/mixins#huggingface_hub.PyTorchModelHubMixin) integration: |
|
|
|
|
|
|
|
|
# ALIF Base 100M |
|
|
|
|
|
**ALIF Base 100M** is an Urdu generative language model from the **ALIF الف** series (a Final Year Project at Habib University), developed by **Orature AI**. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
* **Developed by:** Orature AI (S.M Ali Naqvi, Zainab Haider, Haya Fatima, Ali M Asad, Hammad Sajid) |
|
|
* **Supervised by:** Dr. Abdul Samad (Habib University) |
|
|
* **Model type:** Decoder-only Transformer, GPT-like |
|
|
* **Variant:** ALIF-Base-100M |
|
|
* **Language(s) (NLP):** Urdu (ur) |
|
|
* **License:** Apache 2.0 |
|
|
* **Architecture:** Transformer (GPT-Based) |
|
|
* **Framework:** PyTorch |
|
|
* **Tokeniezer:** SentencePiece Custom Tokenizer |
|
|
* **Hyperparameters:**: |
|
|
* **Vocabulary Size:** 32000 |
|
|
* **Embedding Size:** 768 |
|
|
* **Attention Heads:** 12 |
|
|
* **Layers:** 12 |
|
|
|
|
|
## How to Get Started with the Model |
|
|
|
|
|
First you will need to download the modeling_gpt.py file from the repo. Once that's been done, you can define another file and use the following code to generate text from the model: |
|
|
|
|
|
```python |
|
|
from modeling_gpt import GPTLanguageModel |
|
|
from transformers import AutoTokenizer |
|
|
import torch |
|
|
|
|
|
model_name = "orature/ALIF-Base-100M" |
|
|
model = GPTLanguageModel.from_pretrained(model_name) |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
|
|
|
|
# For text generation |
|
|
prompt_urdu = "ایک دفعہ کا ذکر ہے کہ " # "Once upon a time, " |
|
|
inputs = tokenizer.encode(prompt_urdu) |
|
|
inputs_tensor = torch.tensor(inputs).unsqueeze(0) # Add batch dimension |
|
|
|
|
|
# Generate text |
|
|
outputs = model.generate(inputs_tensor, max_new_tokens=64, temperature=0.7) |
|
|
outputs_tensor = torch.tensor(outputs).unsqueeze(0) |
|
|
generated_text = tokenizer.decode(outputs_tensor[0].squeeze().tolist()) |
|
|
|
|
|
print(f"Prompt: {prompt_urdu}") |
|
|
print(f"Generated Text: {generated_text}") |
|
|
``` |
|
|
|
|
|
## Model Description |
|
|
|
|
|
**ALIF Base 100M** is designed to generate coherent and contextually relevant Urdu text. It leverages a custom Urdu tokenizer trained on the ALIF-Urdu-Corpus and was pretrained on a large corpus of diverse Urdu text. |
|
|
|
|
|
**Key Features:** |
|
|
* Optimized for Urdu language nuances. |
|
|
* Strong foundational capabilities for further fine-tuning (for base models) |
|
|
* Capable of generating next tokens in a sequence, making it suitable for various text generation tasks. |
|
|
* Part of a series aiming to provide efficient and accessible SLMs for Urdu. |
|
|
|
|
|
## Intended Uses & Limitations |
|
|
|
|
|
**Intended Uses:** |
|
|
* **Text Generation:** Creative writing, content generation, story completion in Urdu. |
|
|
* **Research:** Base for further research in Urdu NLP, low-resource language modeling. |
|
|
* **Fine-tuning:** Can be fine-tuned for specific downstream tasks like sentiment analysis, summarization, or domain-specific chatbots in Urdu. |
|
|
* **Educational Purposes:** Understanding SLM behavior for Urdu. |
|
|
* |
|
|
**Limitations:** |
|
|
* The model is primarily trained on Urdu and may not perform well on other languages or code-switched text unless specifically designed for it (e.g., an Ur-En variant). |
|
|
* As a base generative model, it may generate plausible-sounding but incorrect or nonsensical information (hallucinations). |
|
|
* The model may reflect biases present in the training data. The ALIF-Urdu-Corpus was curated from diverse sources, but biases (e.g., societal, gender, regional) may still exist. |
|
|
* Performance on highly specific or technical domains may be limited without further fine-tuning. |
|
|
* The model does not have real-time knowledge and its information is limited to its training data. |
|
|
* Safety: While efforts are made to curate data, the model might generate offensive, harmful, or inappropriate content. Users should implement appropriate safeguards for downstream applications. |
|
|
|
|
|
**Out-of-Scope Uses:** |
|
|
* Generating high-stakes advice (medical, legal, financial) without human oversight. |
|
|
* Impersonation or generating misleading information. |
|
|
* Applications that could lead to harm or discrimination. |
|
|
* Complex scientific, technical, mathematical, or legal reasoning without further fine-tuning. |
|
|
* Any use that violates ethical guidelines or legal standards. |
