# Custom Urdu LLM

This is a custom transformer-based Large Language Model for Urdu.

## Model Details
- **Architecture:** Transformer (GPT-based)
- **Framework:** PyTorch
- **Tokenizer:** SentencePiece
- **Hyperparameters:**
  - Vocabulary Size: 20,000
  - Embedding Size: 768
  - Attention Heads: 12
  - Layers: 12
  - Dropout: 0.2

## Usage

First you will need to download the ```modeling_gpt.py``` file from the repo. Once that's been done, you can define another file and use the following code to generate text from the model:

```python
from modeling_gpt import GPTLanguageModel
from transformers import AutoTokenizer

model = GPTLanguageModel.from_pretrained("AliMuhammad73/testing-model")
tokenizer = AutoTokenizer.from_pretrained("AliMuhammad73/testing-model")

# sentence in urdu
prompt = "پاکستان ایک ایسا ملک ہے جو جنوبی ایشیا میں واقع ہے۔ اس کی سرحدیں ہندوستان، چین، افغانستان، اور "
encoded = tokenizer.encode(prompt)
encoded_tensor = torch.tensor(encoded).unsqueeze(0)
output = model.generate(encoded_tensor, max_new_tokens=64)

response = tokenizer.decode(output[0].squeeze().tolist())
```

---
license: apache-2.0
---