miweru/AraNeoBERT-v4

AraNeoBERT v4

A 297M-parameter modern Arabic encoder (NeoBERT/ModernBERT-style) pretrained with MLM for 220,000 steps (57.7B tokens, 1.5 epochs) on a much larger deduped AraMix web backbone + dialect/OSIAN/news/books blend (15.5% dialect by mixture weight). 28 layers, hidden 768, 12 heads; alternating local/global attention (window 128, global every 3 layers), QK-Norm, RoPE (dual theta 10k local / 160k global), FlexAttention + sequence packing, StableAdamW, WSD schedule, EMA (0.999, exported weights are the EMA shadow), three-phase length curriculum (1024 -> 4096). 64k SentencePiece Unigram tokenizer with web_canonical normalization baked into the fast tokenizer.

Held-out AraMix MLM pseudo-perplexity: 11.37 @ seq 4096, mask 0.30 (step 197k, last point before the final low-mask decay phase); 5.46 @ mask 0.15 at the finished step 220k (decay-phase eval, not directly comparable to the 0.30 number above or to earlier runs' 0.30-mask figures).

Usage

This model uses custom modeling code — load with trust_remote_code=True:

from transformers import AutoTokenizer, AutoModelForMaskedLM
tok = AutoTokenizer.from_pretrained("miweru/AraNeoBERT-v4")
model = AutoModelForMaskedLM.from_pretrained("miweru/AraNeoBERT-v4", trust_remote_code=True)

AutoModel, AutoModelForMaskedLM, AutoModelForSequenceClassification and AutoModelForTokenClassification heads are registered.

Downstream benchmark (finetuned, identical recipe across models, 3 seeds: 13/17/23)

Task	AraNeoBERT-v4	AraBERT v2	AraModernBERT	MARBERT
AJGT sentiment (macro-F1)	0.9296	0.9147	0.9064	0.9444
ArSarcasm sentiment (macro-F1)	0.7431	0.7138	0.7057	0.7463
ANERcorp NER (entity-F1)	0.5585*	0.7504	0.6917	0.7968

* One of the three NER finetuning seeds (seed 17) collapsed during finetuning (entity-F1 0.045, a finetuning-run instability, not a base-model signal); the other two seeds scored 0.808 and 0.823 — above AraBERT v2 and AraModernBERT, approaching MARBERT. The reported mean includes the collapsed run for transparency; treat the 2-seed clean mean (0.815) as the more representative number pending a rerun of seed 17.

Competitive with/ahead of AraBERT v2 and AraModernBERT on sentiment, essentially tied with MARBERT there; strong on NER once the one unstable finetuning seed is set aside.

Research artifact; private. Built with Prof. Mustafa Jarrar's group.