HAYAA Student — Arabic toxicity classifier for on-device use

A distilled, vocabulary-pruned, int8-quantized student of youssefreda9/HAYAA, built to run inside a mobile app. 651 MB -> 9.6 MB (67.8x smaller).

Labels: 0 = Safe, 1 = Toxic.

Read this before using it

The numbers below measure agreement with the teacher on synthetically generated Arabic, not accuracy against human labels. The teacher's own reported F1 is 0.924, so agreement is a ceiling on quality, not a measure of it. No human-labelled gold set was used. Treat these results as evidence that the distillation pipeline works, not that the model is fit for your traffic.

Known weaknesses:

Franco-Arabic / arabizi (enta 3abeet) scores markedly lower than the same insult in Arabic script. Transliterated text is under-represented upstream.
The threshold is not calibrated. The teacher trained with a 3.7x weight on the toxic class, so argmax is not a meaningful operating point. Choose a threshold on your own labelled data.
Domain gap: upstream training data is predominantly Twitter. Comments, DMs and usernames may behave differently.
Dialect coverage follows the distillation corpus, which was synthetic here.

Model

	teacher	student
params	162.8M	9.4M
layers	12	4
hidden size	768	256
vocab	100,000	23970
size	651 MB fp32	9.6 MB int8

Results

metric	value
teacher agreement	0.967
macro-F1 vs teacher labels	0.9347
PR-AUC	0.9554
`[UNK]` rate, out-of-distribution	0.0

Runtimes

runtime	MB	p50 ms	p95 ms	max abs dp	flips
torch fp32	38	19.0	23.6	0.0	0/300
onnx fp32	37.7	13.6	18.4	0.0	0/300
onnx int8	9.6	10.4	12.6	0.00772	0/300

Measured single-threaded, batch 1, sequence 128, on a Colab CPU. Reference for the dp and flips columns is PyTorch fp32. Expect 1.5-3x higher latency on a mid-range phone.

How it was built

Teacher labels raw text with full logits (soft targets) — no human labels needed
Vocabulary pruned to corpus tokens + all 1-2 char pieces + top 20000 by frequency
Student initialised from teacher: pruned embedding rows, layers None, pooler, classifier head
Distillation: loss = 0.8 * T^2 * KL(s/T || t/T) + 0.2 * CE, with T=3.0, 6 epochs
ONNX export, then dynamic int8 quantization

Method follows Hinton et al. 2015 (arXiv:1503.02531) and DistilBERT (arXiv:1910.01108).

Usage

import numpy as np, onnxruntime as ort
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("MahmoudMabrok/HAYAA-student-int8", subfolder="tokenizer")
sess = ort.InferenceSession("student_int8.onnx")

enc = tok("النص هنا", truncation=True, max_length=128,
          padding="max_length", return_tensors="np")
names = [i.name for i in sess.get_inputs()]
feed = {k: v.astype(np.int64) for k, v in enc.items() if k in names}
logits = sess.run(None, feed)[0]
e = np.exp(logits - logits.max(-1, keepdims=True))
p_toxic = (e / e.sum(-1, keepdims=True))[0, 1]

On Android: put student_int8.onnx and tokenizer/tokenizer.json in assets, and verify your Kotlin WordPiece implementation against tokenizer_fixtures.json before trusting any output. A tokenizer that is 98% right produces a classifier that demos perfectly and is broken on exactly the inputs it was built for.

Files

student_int8.onnx — quantized, for deployment
student.onnx — fp32
student.tflite — LiteRT build, if conversion succeeded
tokenizer/ — pruned tokenizer (23970 tokens)
tokenizer_fixtures.json — frozen tokenizer behaviour for cross-language tests
run_report.json — every metric from this run

License and attribution

Derived from youssefreda9/HAYAA, itself a fine-tune of UBC-NLP/MARBERTv2. MARBERT's authors release the checkpoints for research and direct commercial users to contact them. Fine-tuning and distillation do not reset those terms, and a permissive tag on an intermediate model cannot grant rights its author did not hold.

For commercial use, contact UBC-NLP, or rebuild from a permissively licensed teacher such as CAMeL-Lab/bert-base-arabic-camelbert-mix (Apache-2.0, covers MSA plus Egyptian, Gulf and Levantine).

@inproceedings{abdul-mageed-etal-2021-arbert,
  title = {ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic},
  author = {Abdul-Mageed, Muhammad and Elmadany, AbdelRahim and Nagoudi, El Moatez Billah},
  booktitle = {Proceedings of ACL-IJCNLP 2021},
  year = {2021},
  pages = {7088--7105}
}