nanochat-ar: Arabic LLM Proof Run on Aziz HPC
This repository contains a small Arabic nanochat proof-of-competence checkpoint trained from scratch on King Abdulaziz University Aziz HPC.
This is not a production Arabic LLM and does not claim state-of-the-art performance. The purpose is to document and release an end-to-end HPC training artifact: Arabic corpus processing, tokenizer training, base pretraining, evaluation, supervised fine-tuning, and checkpoint generation.
What is included
- Arabic tokenizer: tokenizer/tokenizer.pkl and tokenizer/token_bytes.pt
- Base checkpoint: base_checkpoints/ar-d12-eager/model_000050.pt
- SFT checkpoint: chatsft_checkpoints/ar-d12-eager/model_000002.pt
- Metadata JSON files for both checkpoints
- Training/evaluation reports in reports/
- Final HPC scripts in scripts/
- 200-example SFT sample in data_samples/
Optimizer states and the full raw corpus are not uploaded.
Training summary
| Stage | Result |
|---|---|
| Arabic Wikipedia dump | 1,219,201 articles |
| Clean train split | 1,190,864 lines |
| Clean validation split | 12,028 lines |
| Tokenizer vocabulary | 32,768 |
| Base model | 12 layers, 286,261,730 parameters |
| Base checkpoint | ar-d12-eager, step 50 |
| Base validation BPB | 1.214387 |
| Base eval train BPB | 1.118719 |
| Base eval validation BPB | 1.214387 |
| SFT dataset | 2,000 generated Arabic instruction examples |
| SFT checkpoint | chatsft_checkpoints/ar-d12-eager, step 2 |
| SFT validation BPB | 1.0228 |
How to load with nanochat
Clone karpathy/nanochat, install its environment, then use this repository as the NANOCHAT_BASE_DIR layout.
Commands:
export NANOCHAT_BASE_DIR=/path/to/this/repo
export NANOCHAT_DTYPE=float32
cd /path/to/nanochat
source .venv/bin/activate
python -m scripts.chat_cli --device-type cpu --source sft --model-tag ar-d12-eager --step 2 --prompt 'ู
ุฑุญุจุงุ ู
ู ุฃูุชุ'
For GPU inference, use an A100 or similar GPU and the standard nanochat chat CLI/web server.
Limitations
- Very short proof training run: base model trained for only 50 steps.
- SFT is a compact generated instruction dataset, not a curated production instruction set.
- Chat quality is expected to be weak and repetitive.
- This release is best used as a reproducibility artifact and HPC workflow demonstration.
Citation / Acknowledgement
This project used King Abdulaziz University Aziz HPC resources. Built on karpathy/nanochat under the MIT license.