Request access to Jev-AR

Access is approved automatically.

Jev-AR is released under CC BY-NC 4.0: you may use, test and evaluate it for non-commercial purposes, with attribution. Using it in a commercial product or service requires a separate commercial licence; contact ayoubatmane23@gmail.com.

Jev-AR

A 322M local decision model for fast Arabic intent routing across MSA, Gulf dialects and code-switched Arabic

83.4% on 60-way Gulf-Arabic routing · ~10 ms per query on one GPU · 322M parameters · runs locally

Supply your routes at request time. Jev-AR scores them jointly in one forward pass and returns the best route, a probability for every route, and an escalation signal for when it isn't confident enough to act.

Accuracy vs. latency: Jev-AR 83.4% at 10 ms; Jev 1.13 86.2% at 370 ms; Qwen3.8-27B 90.0% at 634 ms

Highlights

Close to a closed API, at a fraction of the latency. It is 2.8 points from Jev 1.13 on 60-way routing, at about 10 ms per query on one GPU.
On par with Jev on 20-route lists. 90.0% vs 91.4%; the difference is not statistically significant.
The strongest model under 1B parameters we evaluated. It is 6.9 points above a classifier fine-tuned from the same encoder, and 11.0 points above multilingual-E5 with logistic regression.
Your routes, no retraining. It routes 38 of 40 decisions correctly on a kind of RAG app never seen in training, and no answer changes when the route list is reversed.
Built for production. It has calibrated confidence and a 95%-accuracy escalation threshold, and it is about 1% the size of a 27B LLM.

Quick start

jev_ar_route.py is one self-contained file in this repo (also on GitHub). It needs only torch, transformers and huggingface_hub.

pip install torch transformers huggingface_hub
hf auth login          # once your access request is approved (automatic)
python jev_ar_route.py "ابي اجدد الاقامة"

from jev_ar_route import JevAR

router = JevAR()  # downloads the model on first use

tools = {  # an assistant's tools: any routes you like, given at request time
    "Calendar": "Create, move or cancel meetings and reminders.",
    "Email": "Write, reply to or send emails.",
    "Web search": "Look up current or public information: news, prices, facts.",
    "Human agent": "The user asks for a person, or the request is sensitive or legal.",
}
router.route("وش آخر أخبار أسعار الذهب اليوم؟", tools)
# {'route': 'Web search', 'confidence': 0.88, 'escalate': False, 'probabilities': {...}}

Called without a list, Jev-AR routes over 60 built-in routes: 58 Gulf government services (residency, IDs, traffic, labour, business, housing, health, education and more), plus other service and out of scope.

router.route("ابي اجدد الاقامة")
# {'route': 'تجديد الإقامة', 'id': 'renew_residence', 'category': 'residency', 'confidence': 0.78,
#  'escalate': False, 'in_scope': 0.998, 'probabilities': {...}}

Tips. Treat escalate: True (top probability below 0.72) as "hand off to a person or a fallback". Keep the instruction plus the route list under 512 tokens: about 60 short names, or about 15–20 routes with one-line descriptions (fewer in Arabic). The helper raises an error rather than silently cutting routes. On a CPU, call torch.set_num_threads(8) first. The full API is on GitHub.

How it works

The message and the route list form one input sequence; a 307M encoder and a 15M decision head give one probability per route; the top route is returned, or the message is escalated

The message and the routes form one input sequence, with a marker token before each route. The mmBERT-base encoder reads them together, and a 15M decision head scores every route at its marker. A softmax with a temperature fitted on development data turns the scores into probabilities. Routes are part of the input rather than fixed output classes, so a new list needs no retraining.

Results

Gulf-Arabic routing (Jev-AR Bench): 4,974 held-out queries in MSA, Emirati, Saudi and code-switched Arabic. Accuracy in %, with 95% confidence intervals.

Model	Params	Deployment	60 routes	20 routes	Latency (p50)
Qwen3.8-27B (LLM, list in prompt)	27B	GPU server	90.0	94.0	634 ms †
Jev 1.13 (TypeSafe)	undisclosed	closed API	86.2	91.4	370 ms †
Jev-AR	322M	local	83.4 _{[81.2–85.4]}	90.0 _{[88.4–91.4]}	10 ms
mmBERT-base, fine-tuned classifier	307M	local	76.5	79.8	6 ms
multilingual-E5-base + logistic regression	278M	local	72.4	77.4	4 ms

_{Batch 1. Local models ran on an RTX 5090 Laptop GPU. † Client-side round trip, network included.}

Unseen-domain routing (Jev-AR Bench): one RAG app of a kind never seen in training, with 6 routes, and each question routed over an English and an Arabic route list (40 decisions).

Model	Score	Answers changed when the list is reversed
Jev 1.13	40 / 40	0
Qwen3.8-27B	40 / 40	0
Jev-AR	38 / 40	0

Per-dialect results and robustness

Jev-AR accuracy (%)	MSA	Emirati	Saudi	Code-switched
60 routes	87.8	80.0	83.4	82.4
20 routes	92.7	88.2	89.3	89.8

Macro-F1: 86.4 (60 routes). Category accuracy: 92.4.
Order robustness: 5.0% of answers change when the 60 routes are shuffled (Jev 1.13: 4.7%).
Escalation: at a threshold fitted on development data for 95% accuracy, 85.8% of test queries are answered automatically, at 89.7% accuracy.
Label-agreement subset (queries where an independent labeller confirms the gold route): 86.0%.

Evaluation protocol

The test set was frozen before training (SHA-256 published on the dataset card). It was written by a different model family than the training data, and it includes adversarial queries, routes held out from training, and requests outside the route list.
Every system sees the same route lists. Temperatures and thresholds are fitted on development data only.
Confidence intervals come from a bootstrap that resamples whole generation batches. Differences are checked with paired tests against each baseline.
Data, a snippet to evaluate your own router, and per-category results: jev-ar-bench.

Model details


Task	Intent routing over a route list given at request time
Languages	Arabic (MSA, Emirati, Saudi) and English, including code-switched messages
Architecture	mmBERT-base encoder with a decision head that scores all routes jointly
Parameters	322M: a 307M encoder plus a 15M decision head
Context	640 tokens (message + route list); instruction + route list up to 512
Output	Best route, calibrated probabilities over all routes, escalation flag
Latency	~10 ms per query on a GPU; on a laptop CPU with 8 threads, ~0.15 s for a short list and ~0.45 s for 60 routes
Base model	jhu-clsp/mmBERT-base
Licence	CC BY-NC 4.0; commercial licensing: ayoubatmane23@gmail.com

Limitations

Trained on synthetic Arabic routing data, which has not yet been validated by native speakers.
Confidence is calibrated on the built-in routes, so treat it as approximate on custom lists.
It is not an official channel of any government or company. Keep a person in the loop for decisions with legal, medical or financial consequences.

Licence

CC BY-NC 4.0 (LICENSE): free for non-commercial use, testing and evaluation, with attribution. Commercial use requires a licence; contact ayoubatmane23@gmail.com. Third-party credits are in NOTICE.

Citation

@misc{jevar2026,
  title        = {Jev-AR: Arabic Intent Routing for MSA, Gulf Dialects and Code-Switching},
  author       = {atmaneayoub},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/atmaneayoub/jev-ar}}
}

Downloads last month: -; Downloads are not tracked for this model. How to track

Safetensors

Model size

0.3B params

Tensor type

F32

Model tree for atmaneayoub/jev-ar

Base model

jhu-clsp/mmBERT-base

Finetuned

(156)

this model

Evaluation results

Accuracy, 60 routes on Jev-AR Bench, Gulf-Arabic routing
test set self-reported

83.400
Accuracy, 20 routes on Jev-AR Bench, Gulf-Arabic routing
test set self-reported

90.000
Macro-F1, 60 routes on Jev-AR Bench, Gulf-Arabic routing
test set self-reported

86.400
Accuracy, 40 decisions (38/40) on Jev-AR Bench, unseen-domain routing
test set self-reported

95.000