README.md

---
language:
- ar
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- arabic
- llama
- pretraining
- from-scratch
- emhotob
- small-language-model
datasets:
- kaust-generative-ai/fineweb-edu-ar
---

# Emhotob-5M

**Emhotob** is a family of small Arabic language models pretrained **from scratch** on
Arabic web text. This is the **5M** rung of the ladder (~5.08M parameters),
part of a scaling series ranging from 500K to 25M parameters that all share the same
tokenizer, context length, and training recipe.

> ⚠️ These are tiny, research-scale models trained on a limited token budget. They are
> intended for scaling-law experiments, education, and Arabic NLP research — **not** for
> production use.

## Model details

| Property | Value |
|---|---|
| Architecture | Llama (decoder-only, RoPE, GQA) |
| Parameters | 5,080,448 (~5.08M) |
| Hidden size | 128 |
| Layers | 5 |
| Attention heads | 4 (KV heads: 2) |
| Intermediate size | 384 |
| Context length | 2048 |
| Vocabulary | 32,000 (custom Byte-Level BPE) |
| Tied embeddings | Yes |
| RoPE theta | 10,000 |
| Precision | bf16 |

## Training

| Property | Value |
|---|---|
| Data | [`kaust-generative-ai/fineweb-edu-ar`](https://huggingface.co/datasets/kaust-generative-ai/fineweb-edu-ar) (Arabic) |
| Tokens seen | ~500M (1 epoch) |
| Optimizer | AdamW (fused), β=(0.9, 0.95), wd=0.1 |
| LR schedule | 6e-4, cosine, 2% warmup |
| Effective batch | 128 sequences × 2048 tokens |
| Grad clipping | 1.0 |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "oddadmix/Emhotob-5M"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)

prompt = "الذكاء الاصطناعي هو"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50, do_sample=True, top_p=0.9, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))
```

## Limitations

Given its size and limited pretraining budget, Emhotob-5M has a narrow capability
range and will produce factually unreliable and sometimes incoherent text. It has not been
instruction-tuned or aligned, and no safety filtering has been applied. Use accordingly.

---

*© SupraLabs 2026 — PROJECT EMHOTOB.*