model card

README.md

@@ -0,0 +1,176 @@
+---
+license: apache-2.0
+language:
+- ar
+base_model: oddadmix/50M-2048-Emhotob
+datasets:
+- oddadmix/arabic-math-reasoning-synth
+- oddadmix/gsm8k-reasoning-ar
+- Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- arabic
+- reasoning
+- chain-of-thought
+- math
+- gsm8k
+- small-language-model
+- slm
+- llama
+- sft
+---
+
+# Nawah-Math-Reasoning — نموذج استدلال رياضي عربي
+
+A **51.8M-parameter** Arabic math reasoning model. It writes its derivation step by step
+inside `<think>…</think>`, then gives the answer. It is small enough to run on a CPU.
+
+> **بالعربية:** نموذج عربي صغير (~52 مليون معامل) لحل المسائل الحسابية: يكتب خطوات
+> تفكيره داخل وسم `<think>` ثم يعطي الإجابة. صغير بما يكفي ليعمل على المعالج (CPU).
+
+| | |
+|---|---|
+| 🤗 **Demo** | [`oddadmix/Nawah-Math-Reasoning-Demo`](https://huggingface.co/spaces/oddadmix/Nawah-Math-Reasoning-Demo) |
+| 🧩 **Base model** | [`oddadmix/50M-2048-Emhotob`](https://huggingface.co/oddadmix/50M-2048-Emhotob) — Llama architecture, 12 layers, hidden 512, 2048 ctx, pre-trained from scratch on ~20B Arabic tokens |
+| 📚 **Data** | [`arabic-math-reasoning-synth`](https://huggingface.co/datasets/oddadmix/arabic-math-reasoning-synth) · [`gsm8k-reasoning-ar`](https://huggingface.co/datasets/oddadmix/gsm8k-reasoning-ar) · [`Arabic_Reasoning_Dataset`](https://huggingface.co/datasets/Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset) |
+| 🛠️ **Training code** | [`code/`](https://huggingface.co/oddadmix/Nawah-Math-Reasoning/tree/main/code) in this repo — data generation, translation, SFT, eval, GRPO |
+| 🔤 **Vocab** | 32004 (4 chat/reasoning tokens added to the 32000 base vocab) |
+
+## Results
+
+Number agreement, greedy decoding. **Every cell is measured on identical held-out rows.** The
+`Arabic_Reasoning` and `GSM8K-ar` rows are the eval splits fixed at the start of the project and
+never re-drawn; the synthetic rows are pinned to the same 1,000 items every earlier version was
+scored on.
+
+The `v3 / v4 / v5` columns are internal development runs, kept here because they are what makes
+the release number mean something. They are not published — the numbers are, so the ablation is
+readable without them.
+
+| eval set | n | v3 | v4 | v5 | **release** |
+|---|---:|---:|---:|---:|---:|
+| GSM8K-ar | 600 | 77.3% | 19.5% | 76.0% | **79.0%** |
+| Arabic_Reasoning | 400 | 65.8% | 50.2% | **75.2%** | 73.0% |
+| synthetic math | 1000 | 2.0% | 35.6% | 39.1% | **40.4%** |
+| **synthetic relational** | 400 | — | — | 34.0% | **52.2%** |
+
+**The relational row is what this release adds.** On problems whose difficulty is the *relation*
+between quantities (`ضعف`, `نصف`, `أكثر بـ…`) rather than the arithmetic, it scores
+**52.2%** where the previous run scores 34.0% — a **+18.2 point** gain and
+the largest single-cell move anywhere in the development ladder. It did not cost the other
+distributions: GSM8K-ar is simultaneously the best of the series at **79.0%**, and
+synthetic math gains +1.3.
+
+The one regression is `Arabic_Reasoning` at **-2.2** against v5 — on 400 rows that is
+close to sampling noise, but it is the second consecutive mix where this column is the give.
+
+| detail | GSM8K-ar | Arabic_Reasoning | synth math | synth relational |
+|---|---:|---:|---:|---:|
+| final-answer number correct | 79.0% | 77.5% | 46.2% | 54.2% |
+| all numbers match | 79.0% | 73.0% | 43.5% | 52.2% |
+| well-formed `<think>` + answer | 100.0% | 98.8% | 99.5% | 99.5% |
+| mean reasoning length | 39 tok | 90 tok | 59 tok | 45 tok |
+
+*(the synth-math column here is the 400-row mix cell; the 40.4% in the table above is the
+1,000-row set used for the cross-model comparison.)*
+
+Reproduce any cell with `code/eval_reasoning.py` — it is the same script for every model and every
+row, which is the only reason these are comparable.
+
+### The final checkpoint ships, and eval loss disagrees
+
+Loss bottoms at **0.4559** (epoch 1.86) and rises to **0.5154** by
+epoch 5 — yet the epoch-5 weights are the better model. This was measured directly on
+an earlier run whose corpus contained **no repeated rows**, which rules out memorisation: the
+minimum-loss checkpoint scored 30.9% where the final scored 35.6%. It happened on four consecutive
+runs. `train_reasoning.py` therefore takes `LOAD_BEST=0`, and that is not an oversight.
+
+## Training mix
+
+275,639 rows, 31.1M tokens/epoch:
+
+| source | rows | tokens/epoch | share |
+|---|---:|---:|---:|
+| [`oddadmix/arabic-math-reasoning-synth`](https://huggingface.co/datasets/oddadmix/arabic-math-reasoning-synth) | 118,062 | 16.79M | 53.9% |
+| [`oddadmix/gsm8k-reasoning-ar`](https://huggingface.co/datasets/oddadmix/gsm8k-reasoning-ar) | 140,969 | 11.88M | 38.2% |
+| [`Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset`](https://huggingface.co/datasets/Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset) | 16,608 (5,536 × 3) | 2.45M | 7.9% |
+
+Of the synthetic corpus's 120,462 rows, 20,139 are **relational** problems generated
+specifically for this release, after a `pass@k` diagnostic showed the previous model went 0/8 on
+`ضعف`-style problems and a corpus audit found the relation appears in only 1.34% of rows. The
+synthetic eval split was **pinned, not re-drawn** when those rows were added: re-shuffling would
+have moved 1,955 of the 2,000 previously held-out items into train, turning that column into a
+memorisation score.
+
+Full fine-tune from the base (not from the previous version). Loss on the assistant turn only, user
+prompt masked with `-100`. `Arabic_Reasoning` is ~25× smaller than GSM8K, so it is repeated 3×.
+
+| | |
+|---|---|
+| epochs | 5 (21,535 steps) |
+| effective batch | 64 |
+| learning rate | 3e-4 cosine, 200 warmup steps |
+| max length | 768 tokens (mix p100 is 703 — nothing truncated) |
+| precision | bf16 |
+| checkpoint | final (`load_best_model_at_end` disabled — it picks the worse model) |
+| hardware | 1× RTX A6000, ~85 min |
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model_id = "oddadmix/Nawah-Math-Reasoning"
+tok = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).eval()
+
+messages = [{"role": "user", "content": "اشترى خالد 4 دفاتر بسعر 15 جنيهًا للدفتر، ودفع بورقة 100 جنيه. كم المبلغ المتبقي؟"}]
+prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+ids = tok(prompt, return_tensors="pt")
+
+out = model.generate(**ids, max_new_tokens=384, do_sample=False)
+print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=False))
+```
+
+Split the parts with `re.match(r"\s*<think>(.*?)</think>(.*)", completion, re.S)`.
+Decode with `skip_special_tokens=False` — `<think>` and `</think>` are real tokens in this
+tokenizer, and stripping them destroys the split.
+
+It is **single-turn**: one user message per call. Chat history is out of distribution.
+
+**Answer style is not something you can request.** The three corpora disagree — GSM8K rows end in a
+bare numeral, the other two in an `إذن، …` sentence — and arithmetic word problems look alike in
+all of them, so the model picks a style per prompt. **Score it on number agreement, not exact
+string match**, and parse the answer by extracting its numbers.
+
+## Limitations
+
+At ~52M parameters this is a **proof of concept**, and the honest headline is the
+synthetic columns — **40.4%** and **52.2%** on multi-step problems, well below the
+79.0% it scores on GSM8K's narrower phrasing. Arithmetic is the dominant failure mode: the
+reasoning is usually structurally right, one computation step is wrong, and the model then stays
+faithful to its own bad number.
+
+Each corpus brings its own defect. The GSM8K half is machine-translated, its 140,969 rows expanding
+from only 2,814 question patterns, so that score partly reflects narrow phrasing. The synthetic
+half is verified for **arithmetic, not for sense** — rows survive where every equation checks out
+but a step introduces an entity never mentioned, or the answer resolves the reverse of what was
+asked. The `Arabic_Reasoning` half excludes open-ended expository rows (they have no final answer
+to place after `</think>`), so expository prompts remain out of distribution.
+
+Everything is MSA; the synthetic corpus's region axis sets currency and context, not dialect. The
+Arabic inherits source artifacts including inconsistent gender agreement. Its reasoning trace is
+not a faithful account of any internal computation. Do not use it for anything consequential.
+
+## Citation
+
+```bibtex
+@misc{nawah_math_reasoning_2026,
+  title  = {Nawah-Math-Reasoning: a 52M-parameter Arabic chain-of-thought math model},
+  author = {Ahmed Wasfy},
+  year   = {2026},
+  url    = {https://huggingface.co/oddadmix/Nawah-Math-Reasoning}
+}
+```