🌪️Khazri — Azerbaijani Language Model

A lightweight, efficient, and fully custom Azerbaijani language model designed for text generation, chat applications, education, and research. Khazri is trained from scratch using a custom 10M-sample Azerbaijani dataset and optimized for running on consumer GPUs while maintaining competitive performance.

🌟 Features

• 🇦🇿 Native Azerbaijani language support
• ⚡ Lightweight architecture (≈ 36M parameter)
• 🚀 Supports fast inference with GGUF + llama.cpp
• 📦 Available on Hugging Face
• 🎯 Optimized for chatbots, WebRTC real-time assistants, and low-latency deployment

🏗️ Model Architecture

Version	Parameters	Type	Context Length	Notes
Khazri-36M	~36.6M	GPT-2 Small variant	1024	Higher quality

Architecture:

• Transformer decoder-only
• Multi-head self-attention
• Rotary positional embeddings (RoPE)
• GELU activation
• Layer normalization
• Tied embeddings

📚 Dataset

Khazri is trained on a 10 million-sample Azerbaijani dataset including:

• News, books, conversations, social media, web articles, educational content

Preprocessing:

• Unicode normalization, deduplication, tokenizer preprocessing, length filtering

🏋️ Training Details

Hardware

• NVIDIA RTX 3090 24GB
• PyTorch 2.x + CUDA 12
• bf16 mixed precision

Hyperparameters

epochs = 1
batch_size = 32
gradient_accumulation = 4
learning_rate = 3e-4
warmup_steps = 500
weight_decay = 0.1
sequence_length = 512
optimizer = AdamW
precision = bf16

📈 Training Challenges & Solutions

Bottleneck: Memory Bandwidth

Small models saturate VRAM bandwidth → ~4.2 it/s
Solution: shrink model size, adjust batch/accumulation, optimize data loading

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Yusiko/Khazri")
model = AutoModelForCausalLM.from_pretrained("Yusiko/Khazri")

🌐 Hugging Face

Available at: https://huggingface.co/Yusiko/Khazri

📦 License

GPL 3.0 License

🌍 Future Plans

• 1B+ model
• Better tokenizer
• Instruction-tuning
• WebGPU inference
• Community fine-tuning tools

🤝 Contact

Created by Yusiko
GitHub: Yusiko99
Website: https://yusi.xo.je
Hugging Face: Yusiko