--- license: apache-2.0 language: - ar base_model: RightNowAI/RightNow-Arabic-0.5B-Turbo datasets: - Ephab/Arabic-Masked-Diffusion-0.5B-Data library_name: transformers tags: - masked-diffusion - diffusion-language-model - arabic --- # Arabic Masked Diffusion 0.5B A 518M-parameter Arabic masked diffusion language model. It was converted from the autoregressive model [RightNow-Arabic-0.5B-Turbo](https://huggingface.co/RightNowAI/RightNow-Arabic-0.5B-Turbo) (itself built on [Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B)). We removed the causal attention mask, added a mask token, and continued training with a masked diffusion objective on cleaned modern Arabic (web, Wikipedia and news). The model reads context on both sides, so it can fill gaps inside a text as well as continue one. This is the model from the paper *Converting an Autoregressive Arabic Language Model into a Masked Diffusion Model* (Mohammed Feras Sawab, Qasim Al Mahfoodh, Rashed Alkhathlan and Hassan Bader Alhumaidi). ## Results Bits per character (BPC), lower is better. For this model the number is an upper bound on its likelihood; for the parent it is the exact left-to-right likelihood. | Test set | This model | Parent | |---|---|---| | 1,000 unseen web documents | 1.372 | 1.415 | | 64 documents (web, news, Wikipedia) | 1.325 | 1.294 | It is better than its parent on news and web text and about 25% worse on Wikipedia, the parent's own training domain. On four zero-shot Arabic benchmarks it averages 42.5% against the parent's 44.3%. Seeing the text after a hidden 32-token span lowers its BPC on that span by 13%. **Limitations.** Generation is the weak point. Revealing tokens in confidence order leads to repetition loops, and no decoding setting we tried matches the parent's fluency. The model is not instruction-tuned and should not be used as a chat model. Classical and religious text was filtered out of the training data, so the model is aimed at contemporary written Arabic. ## Usage The weights load as a normal Qwen2 model, but attention has to be made bidirectional and the model is not used through `generate()`. The mask token is `<|mask|>` (id 178697). The weights are stored in float16. ```python import math import torch from transformers import AutoModelForCausalLM, AutoTokenizer name = "Ephab/Arabic-Masked-Diffusion-0.5B" tokenizer = AutoTokenizer.from_pretrained(name) model = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.float32, attn_implementation="sdpa").eval() for module in model.modules(): if hasattr(module, "is_causal"): module.is_causal = False mask_id = tokenizer.mask_token_id NO_MASK = {"full_attention": None, "sliding_attention": None} # no causal mask, every position sees all others @torch.no_grad() def fill(token_ids, start, end, temperature=0.8, top_p=0.9, repetition_penalty=1.2): """Fills the masked positions in token_ids[start:end], one token per pass, most confident first.""" for step in range(end - start): is_masked = token_ids[0, start:end] == mask_id if not is_masked.any(): break features = model.model(input_ids=token_ids, attention_mask=NO_MASK, use_cache=False).last_hidden_state logits = model.lm_head(features).float()[0, start:end] logits[:, mask_id] = -float("inf") used = token_ids[0][token_ids[0] != mask_id].unique() logits[:, used] = torch.where(logits[:, used] > 0, logits[:, used] / repetition_penalty, logits[:, used] * repetition_penalty) probabilities = torch.softmax(logits / temperature, dim=-1) sorted_p, order = probabilities.sort(dim=-1, descending=True) sorted_p[(sorted_p.cumsum(dim=-1) - sorted_p) > top_p] = 0 probabilities = torch.zeros_like(probabilities).scatter_(-1, order, sorted_p) candidates = torch.multinomial(probabilities, 1).squeeze(-1) confidence = probabilities.gather(-1, candidates[:, None]).squeeze(-1) count = math.ceil(int(is_masked.sum()) / (end - start - step)) positions = confidence.masked_fill(~is_masked, -float("inf")).topk(count).indices token_ids[0, start + positions] = candidates[positions] # Fill a gap of 16 tokens between two pieces of text. before = tokenizer("تقع مدينة الرياض في", add_special_tokens=False).input_ids after = tokenizer("وهي عاصمة المملكة العربية السعودية.", add_special_tokens=False).input_ids token_ids = torch.tensor([before + [mask_id] * 16 + after]) fill(token_ids, len(before), len(before) + 16) print(tokenizer.decode(token_ids[0], skip_special_tokens=True)) ``` To continue a text, append masks after the prompt and fill them in blocks of 32 tokens, left to right, each block using the finished ones as context. This is how the paper generates text. ## Training Training ran in two phases on sequences of 512 tokens: 23 hours on two Kaggle T4 GPUs (fp16), then 72 hours on one RTX 4080 Super (bf16), about 4.3B tokens in total. The released weights are the exponential moving average of the parameters (decay 0.998) at the end of training. The training and test data are in [Ephab/Arabic-Masked-Diffusion-0.5B-Data](https://huggingface.co/datasets/Ephab/Arabic-Masked-Diffusion-0.5B-Data). ## License Apache 2.0, the same as the parent model. The weights are a derivative of [RightNow-Arabic-0.5B-Turbo](https://huggingface.co/RightNowAI/RightNow-Arabic-0.5B-Turbo) by RightNow AI and [Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B) by the Qwen team, both released under Apache 2.0.