Ephab's picture
Release Arabic Masked Diffusion 0.5B
64cb410
|
Raw History Blame Contribute Delete
5.57 kB
metadata
license: apache-2.0
language:
  - ar
base_model: RightNowAI/RightNow-Arabic-0.5B-Turbo
datasets:
  - Ephab/Arabic-Masked-Diffusion-0.5B-Data
library_name: transformers
tags:
  - masked-diffusion
  - diffusion-language-model
  - arabic

Arabic Masked Diffusion 0.5B

A 518M-parameter Arabic masked diffusion language model. It was converted from the autoregressive model RightNow-Arabic-0.5B-Turbo (itself built on Qwen2.5-0.5B). We removed the causal attention mask, added a mask token, and continued training with a masked diffusion objective on cleaned modern Arabic (web, Wikipedia and news). The model reads context on both sides, so it can fill gaps inside a text as well as continue one.

This is the model from the paper Converting an Autoregressive Arabic Language Model into a Masked Diffusion Model (Mohammed Feras Sawab, Qasim Al Mahfoodh, Rashed Alkhathlan and Hassan Bader Alhumaidi).

Results

Bits per character (BPC), lower is better. For this model the number is an upper bound on its likelihood; for the parent it is the exact left-to-right likelihood.

Test set This model Parent
1,000 unseen web documents 1.372 1.415
64 documents (web, news, Wikipedia) 1.325 1.294

It is better than its parent on news and web text and about 25% worse on Wikipedia, the parent's own training domain. On four zero-shot Arabic benchmarks it averages 42.5% against the parent's 44.3%. Seeing the text after a hidden 32-token span lowers its BPC on that span by 13%.

Limitations. Generation is the weak point. Revealing tokens in confidence order leads to repetition loops, and no decoding setting we tried matches the parent's fluency. The model is not instruction-tuned and should not be used as a chat model. Classical and religious text was filtered out of the training data, so the model is aimed at contemporary written Arabic.

Usage

The weights load as a normal Qwen2 model, but attention has to be made bidirectional and the model is not used through generate(). The mask token is <|mask|> (id 178697). The weights are stored in float16.

import math
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "Ephab/Arabic-Masked-Diffusion-0.5B"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.float32, attn_implementation="sdpa").eval()
for module in model.modules():
    if hasattr(module, "is_causal"):
        module.is_causal = False
mask_id = tokenizer.mask_token_id
NO_MASK = {"full_attention": None, "sliding_attention": None}  # no causal mask, every position sees all others


@torch.no_grad()
def fill(token_ids, start, end, temperature=0.8, top_p=0.9, repetition_penalty=1.2):
    """Fills the masked positions in token_ids[start:end], one token per pass, most confident first."""
    for step in range(end - start):
        is_masked = token_ids[0, start:end] == mask_id
        if not is_masked.any():
            break
        features = model.model(input_ids=token_ids, attention_mask=NO_MASK, use_cache=False).last_hidden_state
        logits = model.lm_head(features).float()[0, start:end]
        logits[:, mask_id] = -float("inf")
        used = token_ids[0][token_ids[0] != mask_id].unique()
        logits[:, used] = torch.where(logits[:, used] > 0, logits[:, used] / repetition_penalty, logits[:, used] * repetition_penalty)
        probabilities = torch.softmax(logits / temperature, dim=-1)
        sorted_p, order = probabilities.sort(dim=-1, descending=True)
        sorted_p[(sorted_p.cumsum(dim=-1) - sorted_p) > top_p] = 0
        probabilities = torch.zeros_like(probabilities).scatter_(-1, order, sorted_p)
        candidates = torch.multinomial(probabilities, 1).squeeze(-1)
        confidence = probabilities.gather(-1, candidates[:, None]).squeeze(-1)
        count = math.ceil(int(is_masked.sum()) / (end - start - step))
        positions = confidence.masked_fill(~is_masked, -float("inf")).topk(count).indices
        token_ids[0, start + positions] = candidates[positions]


# Fill a gap of 16 tokens between two pieces of text.
before = tokenizer("تقع مدينة الرياض في", add_special_tokens=False).input_ids
after = tokenizer("وهي عاصمة المملكة العربية السعودية.", add_special_tokens=False).input_ids
token_ids = torch.tensor([before + [mask_id] * 16 + after])
fill(token_ids, len(before), len(before) + 16)
print(tokenizer.decode(token_ids[0], skip_special_tokens=True))

To continue a text, append masks after the prompt and fill them in blocks of 32 tokens, left to right, each block using the finished ones as context. This is how the paper generates text.

Training

Training ran in two phases on sequences of 512 tokens: 23 hours on two Kaggle T4 GPUs (fp16), then 72 hours on one RTX 4080 Super (bf16), about 4.3B tokens in total. The released weights are the exponential moving average of the parameters (decay 0.998) at the end of training. The training and test data are in Ephab/Arabic-Masked-Diffusion-0.5B-Data.

License

Apache 2.0, the same as the parent model. The weights are a derivative of RightNow-Arabic-0.5B-Turbo by RightNow AI and Qwen2.5-0.5B by the Qwen team, both released under Apache 2.0.