README.md

---
license: apache-2.0
language:
- ar
base_model: oddadmix/50M-2048-Emhotob
datasets:
- oddadmix/arabic-math-reasoning-synth
- oddadmix/gsm8k-reasoning-ar
- Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset
library_name: transformers
pipeline_tag: text-generation
tags:
- arabic
- reasoning
- chain-of-thought
- math
- gsm8k
- small-language-model
- slm
- llama
- sft
---

# Nawah-Math-Reasoning — نموذج استدلال رياضي عربي

A **51.8M-parameter** Arabic math reasoning model. It writes its derivation step by step
inside `<think>…</think>`, then gives the answer. It is small enough to run on a CPU.

> **بالعربية:** نموذج عربي صغير (~52 مليون معامل) لحل المسائل الحسابية: يكتب خطوات
> تفكيره داخل وسم `<think>` ثم يعطي الإجابة. صغير بما يكفي ليعمل على المعالج (CPU).

| | |
|---|---|
| 🤗 **Demo** | [`oddadmix/Nawah-Math-Reasoning-Demo`](https://huggingface.co/spaces/oddadmix/Nawah-Math-Reasoning-Demo) |
| 🧩 **Base model** | [`oddadmix/50M-2048-Emhotob`](https://huggingface.co/oddadmix/50M-2048-Emhotob) — Llama architecture, 12 layers, hidden 512, 2048 ctx, pre-trained from scratch on ~20B Arabic tokens |
| 📚 **Data** | [`arabic-math-reasoning-synth`](https://huggingface.co/datasets/oddadmix/arabic-math-reasoning-synth) · [`gsm8k-reasoning-ar`](https://huggingface.co/datasets/oddadmix/gsm8k-reasoning-ar) · [`Arabic_Reasoning_Dataset`](https://huggingface.co/datasets/Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset) |
| 🛠️ **Training code** | [`code/`](https://huggingface.co/oddadmix/Nawah-Math-Reasoning/tree/main/code) in this repo — data generation, translation, SFT, eval, GRPO |
| 🔤 **Vocab** | 32004 (4 chat/reasoning tokens added to the 32000 base vocab) |

## Results

Number agreement, greedy decoding. **Every cell is measured on identical held-out rows.** The
`Arabic_Reasoning` and `GSM8K-ar` rows are the eval splits fixed at the start of the project and
never re-drawn; the synthetic rows are pinned to the same 1,000 items every earlier version was
scored on.

The `v3 / v4 / v5` columns are internal development runs, kept here because they are what makes
the release number mean something. They are not published — the numbers are, so the ablation is
readable without them.

| eval set | n | v3 | v4 | v5 | **release** |
|---|---:|---:|---:|---:|---:|
| GSM8K-ar | 600 | 77.3% | 19.5% | 76.0% | **79.0%** |
| Arabic_Reasoning | 400 | 65.8% | 50.2% | **75.2%** | 73.0% |
| synthetic math | 1000 | 2.0% | 35.6% | 39.1% | **40.4%** |
| **synthetic relational** | 400 | — | — | 34.0% | **52.2%** |

**The relational row is what this release adds.** On problems whose difficulty is the *relation*
between quantities (`ضعف`, `نصف`, `أكثر بـ…`) rather than the arithmetic, it scores
**52.2%** where the previous run scores 34.0% — a **+18.2 point** gain and
the largest single-cell move anywhere in the development ladder. It did not cost the other
distributions: GSM8K-ar is simultaneously the best of the series at **79.0%**, and
synthetic math gains +1.3.

The one regression is `Arabic_Reasoning` at **-2.2** against v5 — on 400 rows that is
close to sampling noise, but it is the second consecutive mix where this column is the give.

| detail | GSM8K-ar | Arabic_Reasoning | synth math | synth relational |
|---|---:|---:|---:|---:|
| final-answer number correct | 79.0% | 77.5% | 46.2% | 54.2% |
| all numbers match | 79.0% | 73.0% | 43.5% | 52.2% |
| well-formed `<think>` + answer | 100.0% | 98.8% | 99.5% | 99.5% |
| mean reasoning length | 39 tok | 90 tok | 59 tok | 45 tok |

*(the synth-math column here is the 400-row mix cell; the 40.4% in the table above is the
1,000-row set used for the cross-model comparison.)*

Reproduce any cell with `code/eval_reasoning.py` — it is the same script for every model and every
row, which is the only reason these are comparable.

### The final checkpoint ships, and eval loss disagrees

Loss bottoms at **0.4559** (epoch 1.86) and rises to **0.5154** by
epoch 5 — yet the epoch-5 weights are the better model. This was measured directly on
an earlier run whose corpus contained **no repeated rows**, which rules out memorisation: the
minimum-loss checkpoint scored 30.9% where the final scored 35.6%. It happened on four consecutive
runs. `train_reasoning.py` therefore takes `LOAD_BEST=0`, and that is not an oversight.

## Training mix

275,639 rows, 31.1M tokens/epoch:

| source | rows | tokens/epoch | share |
|---|---:|---:|---:|
| [`oddadmix/arabic-math-reasoning-synth`](https://huggingface.co/datasets/oddadmix/arabic-math-reasoning-synth) | 118,062 | 16.79M | 53.9% |
| [`oddadmix/gsm8k-reasoning-ar`](https://huggingface.co/datasets/oddadmix/gsm8k-reasoning-ar) | 140,969 | 11.88M | 38.2% |
| [`Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset`](https://huggingface.co/datasets/Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset) | 16,608 (5,536 × 3) | 2.45M | 7.9% |

Of the synthetic corpus's 120,462 rows, 20,139 are **relational** problems generated
specifically for this release, after a `pass@k` diagnostic showed the previous model went 0/8 on
`ضعف`-style problems and a corpus audit found the relation appears in only 1.34% of rows. The
synthetic eval split was **pinned, not re-drawn** when those rows were added: re-shuffling would
have moved 1,955 of the 2,000 previously held-out items into train, turning that column into a
memorisation score.

Full fine-tune from the base (not from the previous version). Loss on the assistant turn only, user
prompt masked with `-100`. `Arabic_Reasoning` is ~25× smaller than GSM8K, so it is repeated 3×.

| | |
|---|---|
| epochs | 5 (21,535 steps) |
| effective batch | 64 |
| learning rate | 3e-4 cosine, 200 warmup steps |
| max length | 768 tokens (mix p100 is 703 — nothing truncated) |
| precision | bf16 |
| checkpoint | final (`load_best_model_at_end` disabled — it picks the worse model) |
| hardware | 1× RTX A6000, ~85 min |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "oddadmix/Nawah-Math-Reasoning"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).eval()

messages = [{"role": "user", "content": "اشترى خالد 4 دفاتر بسعر 15 جنيهًا للدفتر، ودفع بورقة 100 جنيه. كم المبلغ المتبقي؟"}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt")

out = model.generate(**ids, max_new_tokens=384, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=False))
```

Split the parts with `re.match(r"\s*<think>(.*?)</think>(.*)", completion, re.S)`.
Decode with `skip_special_tokens=False` — `<think>` and `</think>` are real tokens in this
tokenizer, and stripping them destroys the split.

It is **single-turn**: one user message per call. Chat history is out of distribution.

**Answer style is not something you can request.** The three corpora disagree — GSM8K rows end in a
bare numeral, the other two in an `إذن، …` sentence — and arithmetic word problems look alike in
all of them, so the model picks a style per prompt. **Score it on number agreement, not exact
string match**, and parse the answer by extracting its numbers.

## Limitations

At ~52M parameters this is a **proof of concept**, and the honest headline is the
synthetic columns — **40.4%** and **52.2%** on multi-step problems, well below the
79.0% it scores on GSM8K's narrower phrasing. Arithmetic is the dominant failure mode: the
reasoning is usually structurally right, one computation step is wrong, and the model then stays
faithful to its own bad number.

Each corpus brings its own defect. The GSM8K half is machine-translated, its 140,969 rows expanding
from only 2,814 question patterns, so that score partly reflects narrow phrasing. The synthetic
half is verified for **arithmetic, not for sense** — rows survive where every equation checks out
but a step introduces an entity never mentioned, or the answer resolves the reverse of what was
asked. The `Arabic_Reasoning` half excludes open-ended expository rows (they have no final answer
to place after `</think>`), so expository prompts remain out of distribution.

Everything is MSA; the synthetic corpus's region axis sets currency and context, not dialect. The
Arabic inherits source artifacts including inconsistent gender agreement. Its reasoning trace is
not a faithful account of any internal computation. Do not use it for anything consequential.

## Citation

```bibtex
@misc{nawah_math_reasoning_2026,
  title  = {Nawah-Math-Reasoning: a 52M-parameter Arabic chain-of-thought math model},
  author = {Ahmed Wasfy},
  year   = {2026},
  url    = {https://huggingface.co/oddadmix/Nawah-Math-Reasoning}
}
```