---
license: gpl-3.0
language:
- az
base_model:
- Yusiko/Khazri
tags:
- aze
- mini
- yusiko
---


# 🌪️Khazri — Azerbaijani Language Model
**A lightweight, efficient, and fully custom Azerbaijani language model designed for text generation, chat applications, education, and research.**
Khazri is trained from scratch using a custom 10M-sample Azerbaijani dataset and optimized for running on consumer GPUs while maintaining competitive performance.

## 🌟 Features
- 🇦🇿 Native Azerbaijani language support
- ⚡ Lightweight architecture (≈ 36M parameter)
- 🚀 Supports fast inference with GGUF + llama.cpp
- 📦 Available on Hugging Face
- 🎯 Optimized for chatbots, WebRTC real-time assistants, and low-latency deployment

## 🏗️ Model Architecture
| Version | Parameters | Type | Context Length | Notes |
|--------|------------|------|----------------|-------|
| Khazri-36M | ~36.6M | GPT-2 Small variant | 1024 | Higher quality |

Architecture:
- Transformer decoder-only
- Multi-head self-attention
- Rotary positional embeddings (RoPE)
- GELU activation
- Layer normalization
- Tied embeddings

## 📚 Dataset
Khazri is trained on a 10 million-sample Azerbaijani dataset including:
- News, books, conversations, social media, web articles, educational content

Preprocessing:
- Unicode normalization, deduplication, tokenizer preprocessing, length filtering

## 🏋️ Training Details
### Hardware
- NVIDIA RTX 3090 24GB
- PyTorch 2.x + CUDA 12
- bf16 mixed precision

### Hyperparameters
```
epochs = 1
batch_size = 32
gradient_accumulation = 4
learning_rate = 3e-4
warmup_steps = 500
weight_decay = 0.1
sequence_length = 512
optimizer = AdamW
precision = bf16
```

## 📈 Training Challenges & Solutions
### Bottleneck: Memory Bandwidth
Small models saturate VRAM bandwidth → ~4.2 it/s  
Solution: shrink model size, adjust batch/accumulation, optimize data loading

### Transformers
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Yusiko/Khazri")
model = AutoModelForCausalLM.from_pretrained("Yusiko/Khazri")
```

## 🌐 Hugging Face
Available at: https://huggingface.co/Yusiko/Khazri

## 📦 License
GPL 3.0 License

## 🌍 Future Plans
- 1B+ model
- Better tokenizer
- Instruction-tuning
- WebGPU inference
- Community fine-tuning tools

## 🤝 Contact
Created by **Yusiko**  
GitHub: [Yusiko99](https://github.com/Yusiko99)  
Website: https://yusi.xo.je  
Hugging Face: Yusiko