Ephab's picture
Release Arabic Masked Diffusion 0.5B
64cb410
|
Raw History Blame Contribute Delete
5.57 kB
---
license: apache-2.0
language:
- ar
base_model: RightNowAI/RightNow-Arabic-0.5B-Turbo
datasets:
- Ephab/Arabic-Masked-Diffusion-0.5B-Data
library_name: transformers
tags:
- masked-diffusion
- diffusion-language-model
- arabic
---
# Arabic Masked Diffusion 0.5B
A 518M-parameter Arabic masked diffusion language model. It was converted from the autoregressive
model [RightNow-Arabic-0.5B-Turbo](https://huggingface.co/RightNowAI/RightNow-Arabic-0.5B-Turbo)
(itself built on [Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B)). We removed the causal
attention mask, added a mask token, and continued training with a masked diffusion objective on
cleaned modern Arabic (web, Wikipedia and news). The model reads context on both sides, so it can
fill gaps inside a text as well as continue one.
This is the model from the paper *Converting an Autoregressive Arabic Language Model into a Masked
Diffusion Model* (Mohammed Feras Sawab, Qasim Al Mahfoodh, Rashed Alkhathlan and Hassan Bader Alhumaidi).
## Results
Bits per character (BPC), lower is better. For this model the number is an upper bound on its
likelihood; for the parent it is the exact left-to-right likelihood.
| Test set | This model | Parent |
|---|---|---|
| 1,000 unseen web documents | 1.372 | 1.415 |
| 64 documents (web, news, Wikipedia) | 1.325 | 1.294 |
It is better than its parent on news and web text and about 25% worse on Wikipedia, the parent's own
training domain. On four zero-shot Arabic benchmarks it averages 42.5% against the parent's 44.3%.
Seeing the text after a hidden 32-token span lowers its BPC on that span by 13%.
**Limitations.** Generation is the weak point. Revealing tokens in confidence order leads to
repetition loops, and no decoding setting we tried matches the parent's fluency. The model is not
instruction-tuned and should not be used as a chat model. Classical and religious text was filtered
out of the training data, so the model is aimed at contemporary written Arabic.
## Usage
The weights load as a normal Qwen2 model, but attention has to be made bidirectional and the model
is not used through `generate()`. The mask token is `<|mask|>` (id 178697). The weights are stored in
float16.
```python
import math
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "Ephab/Arabic-Masked-Diffusion-0.5B"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.float32, attn_implementation="sdpa").eval()
for module in model.modules():
if hasattr(module, "is_causal"):
module.is_causal = False
mask_id = tokenizer.mask_token_id
NO_MASK = {"full_attention": None, "sliding_attention": None} # no causal mask, every position sees all others
@torch.no_grad()
def fill(token_ids, start, end, temperature=0.8, top_p=0.9, repetition_penalty=1.2):
"""Fills the masked positions in token_ids[start:end], one token per pass, most confident first."""
for step in range(end - start):
is_masked = token_ids[0, start:end] == mask_id
if not is_masked.any():
break
features = model.model(input_ids=token_ids, attention_mask=NO_MASK, use_cache=False).last_hidden_state
logits = model.lm_head(features).float()[0, start:end]
logits[:, mask_id] = -float("inf")
used = token_ids[0][token_ids[0] != mask_id].unique()
logits[:, used] = torch.where(logits[:, used] > 0, logits[:, used] / repetition_penalty, logits[:, used] * repetition_penalty)
probabilities = torch.softmax(logits / temperature, dim=-1)
sorted_p, order = probabilities.sort(dim=-1, descending=True)
sorted_p[(sorted_p.cumsum(dim=-1) - sorted_p) > top_p] = 0
probabilities = torch.zeros_like(probabilities).scatter_(-1, order, sorted_p)
candidates = torch.multinomial(probabilities, 1).squeeze(-1)
confidence = probabilities.gather(-1, candidates[:, None]).squeeze(-1)
count = math.ceil(int(is_masked.sum()) / (end - start - step))
positions = confidence.masked_fill(~is_masked, -float("inf")).topk(count).indices
token_ids[0, start + positions] = candidates[positions]
# Fill a gap of 16 tokens between two pieces of text.
before = tokenizer("تقع مدينة الرياض في", add_special_tokens=False).input_ids
after = tokenizer("وهي عاصمة المملكة العربية السعودية.", add_special_tokens=False).input_ids
token_ids = torch.tensor([before + [mask_id] * 16 + after])
fill(token_ids, len(before), len(before) + 16)
print(tokenizer.decode(token_ids[0], skip_special_tokens=True))
```
To continue a text, append masks after the prompt and fill them in blocks of 32 tokens, left to
right, each block using the finished ones as context. This is how the paper generates text.
## Training
Training ran in two phases on sequences of 512 tokens: 23 hours on two Kaggle T4 GPUs (fp16), then
72 hours on one RTX 4080 Super (bf16), about 4.3B tokens in total. The released weights are the
exponential moving average of the parameters (decay 0.998) at the end of training. The training and test data are in
[Ephab/Arabic-Masked-Diffusion-0.5B-Data](https://huggingface.co/datasets/Ephab/Arabic-Masked-Diffusion-0.5B-Data).
## License
Apache 2.0, the same as the parent model. The weights are a derivative of
[RightNow-Arabic-0.5B-Turbo](https://huggingface.co/RightNowAI/RightNow-Arabic-0.5B-Turbo) by
RightNow AI and [Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B) by the Qwen team, both
released under Apache 2.0.