nur-dev/llama-1.9B-kaz

LLaMA 1.9B - Kazakh Causal Language Model

LLaMA 1.9B

Kazakh Causal Language Model

Model Description

This model is a Kazakh language version of the LLaMA model with 1.9 billion parameters, trained for causal language modeling. The train dataset includes some mixed Russian texts, which occasionally cause the model to generate Russian text. Despite this, the model shows promising results. Future steps may include retraining with a purer dataset, fine-tuning, or using the model for various NLP tasks with additional fine-tuning.

Training Setup

• Training Examples: Over 5.3 million examples
• Training Hardware: Two NVIDIA A100 GPUs (80GB each)
• Training Status: Ongoing, currently partway through the first epoch
• Optimizer: Cosine with restarts scheduler
• Parallelism: Distributed Data Parallel (DDP)
• Number of Warmup Steps: 8000

Model Authors

Name: Kadyrbek Nurgali

• Email: nurgaliqadyrbek@gmail.com
• LinkedIn: Kadyrbek Nurgali

  @misc{
  nurgali_kadyrbek_2024,
  author       = {NURGALI, Kadyrbek},
  title        = {llama-1.9B-kaz (Revision 299ebbb)},
  year         = 2024,
  url          = {https://huggingface.co/nur-dev/llama-1.9B-kaz},
  doi          = {10.57967/hf/3043},
  publisher    = {Hugging Face}
}

Citation

If you use this model, please cite the following publication:

Kadyrbek, N.; Tuimebayev, Z.; Mansurova, M.; Viegas, V. The Development of Small-Scale Language Models for Low-Resource Languages, with a Focus on Kazakh and Direct Preference Optimization. Big Data Cogn. Comput. 2025, 9, 137. https://doi.org/10.3390/bdcc9050137