|
|
--- |
|
|
library_name: transformers |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- allmalab/DOLLMA |
|
|
- allmalab/aze-books |
|
|
language: |
|
|
- az |
|
|
base_model: |
|
|
- openai-community/gpt2 |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# Azerbaijani GPT2 Model |
|
|
|
|
|
The model is based on the GPT-2 architecture, specifically trained on Azerbaijani text. It serves as one of the first foundational models designed to generate and understand Azerbaijani language content. Built with the autoregressive transformer decoder architecture, the model generates text token by token, predicting the next word based on the input context. |
|
|
|
|
|
- **Developed by :** aLLMA Lab |
|
|
- **Funded by :** PRODATA LLC |
|
|
- **Model type:** Decoder-only foundational LLM |
|
|
- **Language:** Azerbaijani |
|
|
|
|
|
## Uses |
|
|
|
|
|
The model can be directly used for text generation, sentence completion, next token prediction tasks by providing an input prompt. Additionally, it can be fine-tuned on an Azerbaijani instruction dataset to develop an interactive question-answering model. |
|
|
|
|
|
## Training Details |
|
|
|
|
|
```python |
|
|
context_window=1024 |
|
|
stride=512 |
|
|
|
|
|
lr=1e-3 |
|
|
warmup_steps = 10000 |
|
|
weight_decay=0.1, |
|
|
adam_beta1 = 0.9, |
|
|
adam_beta2 = 0.999 |
|
|
batch_size=512 |
|
|
max_steps=178000 |
|
|
``` |
