50M-2048-Emhotob (ุฅู ุญูุชุจ) โ Arabic Base Model
Emhotob is a ~51.8M-parameter Llama-architecture language model pre-trained from scratch on ~20 billion Arabic tokens with a 2048-token context window. It is a proof-of-concept that a tiny model โ small enough to run on a CPU โ can be trained from scratch to produce coherent Arabic.
This is a base (pretrained) model: a next-token predictor with no instruction or chat tuning. It is the foundation for the fine-tuned variants below.
- Architecture & training recipe: derived from
SupraLabs/Supra-50M-Base("Project Chimera"), re-run from scratch on an Arabic corpus with a custom Arabic tokenizer. - Pre-training corpus:
kaust-generative-ai/fineweb-edu-ar(Arabic).
ุงูู ูุฎุต ุจุงูุนุฑุจูุฉ: ยซุฅู ุญูุชุจยป ูู ูุฐุฌ ูุบูู ุนุฑุจู ุตุบูุฑ (~51.8 ู ูููู ู ุนุงู ู) ุจู ุนู ุงุฑูุฉ Llamaุ ู ูุฏุฑููุจ ู ู ุงูุตูุฑ ุนูู ~20 ู ููุงุฑ ุฑู ุฒ (token) ุนุฑุจู ุจุทูู ุณูุงู 2048. ูู ูุฐุฌ ุฃุณุงุณ (Base) ุจุฏูู ุถุจุท ุชุนููู ุงุช ุฃู ู ุญุงุฏุซุฉ. ุงููุฏู: ุฅุซุจุงุช ุฅู ูุงููุฉ ุชุฏุฑูุจ ูู ูุฐุฌ ุนุฑุจู ู ููุฏ ุนูู ูุทุงู ุตุบูุฑ ุฌุฏูุง ูุนู ู ุญุชู ุนูู ุงูู ุนุงูุฌ (CPU). ุงูู ุนู ุงุฑูุฉ ูุณูุฑุจุชุงุช ุงูุชุฏุฑูุจ ู ุดุชูุฉ ู ู
SupraLabs/Supra-50M-Base.
Model details
| Parameters | ~51.8M |
| Architecture | Llama (LlamaForCausalLM) |
| Hidden size | 512 ยท Layers 12 ยท Heads 8 (GQA, 4 KV) ยท head_dim 64 |
| Intermediate size | 1408 |
| Vocab size | 32000 (custom Arabic byte-level BPE) |
| Context length | 2048 |
| Positional encoding | RoPE (ฮธ = 10000) |
| Tied embeddings | yes |
| Type | Base / pretrained (no chat template) |
| License | Apache-2.0 |
Training
- Corpus:
kaust-generative-ai/fineweb-edu-ar(configar), streamed and tokenized to a memory-mapped token bin. - Budget: 20,000,000,000 tokens, sequence length 2048, 1 epoch over the token stream.
- Tokenizer: custom byte-level BPE (
ByteLevelBPETokenizer), vocab 32000, specials<s>/</s>/<unk>/<pad>. - Optimizer: AdamW (fused), ฮฒ = (0.9, 0.95), weight decay 0.1, grad clip 1.0.
- Schedule: lr 6e-4, cosine decay, warmup ratio 0.02.
- Precision / compute: bf16,
torch.compile, effective batch 128 (per-device 16 ร grad-accum 8).
The architecture and training loop follow SupraLabs/Supra-50M-Base
("Project Chimera โ 50M Llama"). The pretraining script (train.py) is included in
this repository.
Usage
The tokenizer uses the TokenizersBackend class, which requires transformers>=5.12.
This is a base model โ prompt it as a text completer, not a chat assistant:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "oddadmix/50M-2048-Emhotob"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)
prompt = "ุงููุบุฉ ุงูุนุฑุจูุฉ ูู"
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(
**ids, max_new_tokens=128, do_sample=True,
temperature=0.8, top_p=0.9, repetition_penalty=1.2,
)
print(tok.decode(out[0], skip_special_tokens=True))
Fine-tuned variants
| Model | What it does |
|---|---|
oddadmix/Nawah-50M-Egyptian-18y-Persona |
Speaks with an 18-year-old Egyptian persona |
oddadmix/Emhotob-50M-GPRO-Arabic-Final |
Arabic tool-calling / function-calling (GRPO-tuned) |
Intended use & limitations
Intended use. A from-scratch Arabic base model for research and as a fine-tuning starting point; a CPU-friendly baseline for tiny Arabic SLM experiments.
Limitations. As a base model it does not follow instructions or hold a
conversation out of the box โ fine-tune it first. At ~50M parameters it is a
proof of concept: expect limited world knowledge, weak reasoning, and repetition
(use a repetition_penalty). Training data is web text (fineweb-edu-ar), so it
carries that corpus's biases. Not suitable for factual, medical, legal, or financial use.
Credits
- Architecture & pretraining recipe:
SupraLabs/Supra-50M-Baseโ "Project Chimera" (Apache-2.0). - Pre-training corpus:
kaust-generative-ai/fineweb-edu-ar.
- Downloads last month
- 166