Baligh Logo

الفصاحية والذكاء — Fluency and Intelligence

GitHub License Python

Baligh-1.7B v0

Baligh-1.7B v0 is an Arabic-first language model with Islamic knowledge specialization, fine-tuned from unsloth/Qwen3-1.7B-Base via Continued Pretraining (CPT) and Supervised Fine-Tuning (SFT) using QLoRA 4-bit.

Model Details

Attribute Value
Model Type Causal Language Model (merged instruct, FP16)
Base Model unsloth/Qwen3-1.7B-Base
Total Parameters 1.7B (1.4B non-embedding)
Architecture 28 layers, GQA (16 query / 8 KV heads), hidden size 2048
Native Context 32,768 tokens (architecture)
v0 Training Window 2,048 tokens (no RoPE scaling applied yet)
Tokenizer Qwen3 tokenizer (~152K vocab, unchanged for v0)
Languages Arabic (primary, fusha), English
Domains General Arabic, Islamic Knowledge (Quran, Hadith, Fiqh, Tafsir)
Precision FP16 (Turing GPUs); bf16 auto-selected on Ampere+
Attention SDPA (flash_attention_2 auto-upgrade on sm_80+)

Training

Stage 1: Continued Pretraining (CPT)

  • Datasets: ArabicWeb24 (70%), ArabicText-Large (20%), The Arabic Pile (10%)
  • Islamic Cycle: Hadith collections, Quran QA, Quran texts
  • Method: QLoRA 4-bit NF4 (r=16, alpha=16, dropout=0)
  • Target Modules: all attention + MLP projections
  • Sequence Length: 1,024 (v0 stage 1)
  • Steps: 1,000 (stage 1 validation); 50,000 planned (stage 2)
  • Learning Rate: 2e-4 (cosine scheduler, warmup 100)
  • Optimizer: adamw_8bit
  • Gradient Checkpointing: True

Stage 2: Supervised Fine-Tuning (SFT)

  • Datasets (canonical {instruction, input, output} schema):
    • CIDAR (40%) — Arabic instruction-following
    • evol-instruct-arabic (35%) — ShareGPT turns projected to flat schema
    • islamweb_fatwa (15%) — Islamic QA from IslamWeb (Arabic + capped multilingual)
    • Summarization (10%) — Arabic text summarization
  • Method: QLoRA 4-bit NF4 (r=16, alpha=16), completion-only loss masking
  • Learning Rate: 1e-4 (cosine, warmup 500)
  • Steps: 5,000 (stage 1); 10,000 planned (stage 2)
  • Response-only loss: True (prompt tokens masked with -100)

Key Design Decisions

  • Arabic gate removed from SFT cleaning: sanitize() blanks non-Arabic text instead of dropping rows, preserving mixed-language instruction examples.
  • Leakage-safe split: IslamWeb fatwas are split by fatwa_id group so no ruling appears in both train and eval.
  • Deduplication: IslamWeb deduped by fatwa ID, preferring Arabic over other languages.
  • Precision: fp16 on Turing (sm_75); bf16 auto-selected on Ampere+.

Evaluation

Baseline (untouched base model) vs Post-SFT

Benchmark Baseline Post-SFT Delta
CIDAR ROUGE-L 0.0402 0.0543 +35%
CIDAR BLEU 2.28 1.32 −42%

Notes

  • MMLU-Arabic is currently blocked by an upstream CSV schema mismatch (not a model issue).
  • BLEU dropped post-SFT likely because the model generates longer, more fluent responses that diverge from short reference answers.

Intended Use

Baligh-1.7B v0 is designed for:

  • Arabic question answering and text generation
  • Islamic knowledge queries (fiqh, hadith, tafsir, aqeedah)
  • Arabic text summarization and reformulation
  • Structured output (JSON extraction)
  • Arabic instruction following

Limitations

  • Not a substitute for qualified Islamic scholars — may hallucinate on specific religious rulings.
  • Limited to knowledge cutoff of training data.
  • Arabic dialect performance varies; v0 targets fusha (formal Arabic).
  • Effective context is the 2K training window, not the 32K native maximum.
  • Not suitable for high-stakes decisions (medical, legal, financial).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Kandil7/Baligh-1.7B", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("Kandil7/Baligh-1.7B", trust_remote_code=True)

prompt = "ما هي أركان الإسلام؟"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

@software{baligh2026,
  title={Baligh-1.7B: Arabic-first LLM with Islamic Knowledge Specialization},
  author={Baligh Team},
  year={2026},
  url={https://huggingface.co/Kandil7/Baligh-1.7B}
}

Links

License

Apache 2.0 — see LICENSE for details.

Downloads last month
-
Safetensors
Model size
2B params
Tensor type
F16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Kandil7/Baligh-1.7B

Finetuned
(78)
this model
Quantizations
1 model

Datasets used to train Kandil7/Baligh-1.7B