license: mit
language:
- ru
- en
- multilingual
pipeline_tag: text-classification
tags:
- text-classification
- zero-shot-classification
- typed-decision
- russian
- routing
- intent-classification
- calibrated
- calibration
- on-prem
- cpu
- mmbert
library_name: transformers
base_model: jhu-clsp/mmBERT-base
inference: false
Ру-Жев (Ru-Jev) v1.3
Do not use the standard zero-shot pipeline. The call
pipeline("zero-shot-classification", model="besktech/ru-jev") neither fails nor
works: it returns numbers drawn from a randomly initialized head, and they change
on every load. Load the model with the snippet below.
Open, Russian-first typed-decision model. It does not generate text. Given a context and a set of candidate options, it returns a calibrated probability over the options and picks one. It covers three decision shapes: yes/no, choice from options, and ordinal scale. It is a compact encoder (mmBERT-base, 307M), so it runs on CPU and can be deployed inside a closed perimeter.
- Project page: https://ru-jev.besk.tech
- Demo with precomputed examples: https://huggingface.co/spaces/besktech/ru-jev-demo
- Training and evaluation code: https://github.com/beskvladimir-create/ru-jev
Earlier releases are kept on branches: v1.1 and v1. Load one with
revision="v1.1" if you depend on its exact behaviour.
The inference widget is switched off on purpose, and the stock pipelines do not
work either. This is a backbone plus a separate scoring head, not a standard
sequence-classification checkpoint. Every stock pipeline loads the checkpoint
through AutoModelForSequenceClassification, which quietly initializes a fresh
head instead of loading head.safetensors, and then scores with it. On the
example below the standard zero-shot pipeline put at most 0.44 on the right
option where this model puts 0.95, and over three loads of the same code it
named a different top option twice. Use the snippet below.
Not affiliated with TypeSafe AI. Not trained on Jev outputs. Ru-Jev is a Jev-class typed-decision model built independently: the scoring head is trained from random initialisation and the mmBERT-base backbone is fine-tuned along with it, on open, human-labelled datasets only. No closed decision API was called, distilled or used as a label source at any point. The name is a play on the category, not a claim of compatibility or endorsement.
What changed since v1.1
v1.1 read the option text, which is the point of the design, but it read it
narrowly. It had learned the exact wordings its training data used. Rename a
category and accuracy fell off a cliff. A scenario called погода was routed
correctly in 0.956 of cases, and the same scenario called вэзер in
0.633. Worse, the
confidence did not fall with it: on the jargon wording v1.1 still reported
0.754 mean confidence against 0.633 accuracy, a gap of 0.121,
against 0.030 for v1.3. The probability stopped carrying information exactly
where a caller needed it most.
v1.3 ships every fixed option set in several wordings and draws one per example, so the model has to read an option instead of recognising it. For the routing tasks that includes the English key written in Cyrillic and a clipped form of it, generated by rule.
The same 1,000 routing questions were then asked over and over with the label wording varied and nothing else. Two candidate-set sizes are reported, because the gap depends on the size in different ways for different vocabularies and a probe at one size alone can misstate the effect: five candidates drawn from the eighteen categories, which is what earlier versions of this card reported, and all eighteen candidates at once. Chance is 0.20 and 0.056.
Option wording, example for weather |
Status | v1.1, 5 | v1.3, 5 | v1.1, 18 | v1.3, 18 |
|---|---|---|---|---|---|
Russian word, погода |
training style, both releases | 0.956 | 0.960 | 0.883 | 0.892 |
English key, weather |
training style, both releases | 0.957 | 0.962 | 0.894 | 0.900 |
Russian phrase, прогноз погоды |
training style, v1.3 only | 0.881 | 0.957 | 0.703 | 0.885 |
| Descriptive sentence | held out | 0.939 | 0.958 | 0.848 | 0.878 |
Department name, метеослужба |
held out | 0.835 | 0.869 | 0.632 | 0.733 |
Anglicism jargon, вэзер, all 18 labels |
held out | 0.633 | 0.855 | 0.431 | 0.640 |
| Anglicism jargon, 15 clean labels | held out | 0.624 | 0.838 | 0.427 | 0.595 |
| Anglicism jargon, 11 strict labels | held out | 0.550 | 0.786 | 0.328 | 0.470 |
Office slang, синоптик, all 18 |
held out | 0.648 | 0.780 | 0.428 | 0.476 |
| Office slang, 14 strict labels | held out | 0.595 | 0.761 | 0.369 | 0.419 |
Abbreviation, ПОГ, 17 clean labels |
held out | 0.420 | 0.360 | not scored | |
Opaque code, ПГД, 9 strict labels |
held out | 0.135 | 0.168 | 0.018 | 0.061 |
Code identifier, WEATHER_SCN |
contains the trained key | 0.856 | 0.889 | not scored |
The first two rows are controls, not results. The Russian word and the English key are the wordings both releases trained on: the bench vocabulary for them is the training style itself, taken from the same table in the data builder. Those two rows measure recall of a trained form, and the gap near zero is what a control should show. The Russian phrase is a training style for v1.3 and not for v1.1, so its gap is expected by construction and is not evidence of robustness either. Only the rows marked held out speak to generalisation.
Two caveats on the held-out rows. An intent key begins with its scenario name, so three of the eighteen jargon labels coincide with what the transliteration rule produces and reach training as a first word; the clean slice drops those three and the strict slice drops every label with any exact or prefix relation to a training option. The rules were not tuned against this test vocabulary, and the vocabulary is the older of the two: it was committed six hours and thirty-four minutes before the rules. Figures from different slices measure different label sets, so one never belongs beside another.
What the augmentation helps with. Training a label in several surface forms helps on held-out wordings made of words, ordinary or borrowed, including kinds of words that were never in training at all: jargon gains +0.235 with five candidates and +0.142 with eighteen on the strict slice, office slang +0.166 and +0.050, department names +0.034 and +0.101, descriptive sentences +0.019 and +0.030. The direction is the same in all fourteen word-vocabulary cells.
What it does not help with. On three-letter abbreviations v1.3 is not better:
-0.024 on all labels and -0.060 on the clean slice. Opaque codes are unreadable
for both releases, with v1.1 below chance and v1.3 at chance on the strict slice
with eighteen candidates. Code identifiers such as WEATHER_SCN contain the
trained key and are easy for both, so they are not treated as a test of
robustness. Where the label string carries little of its meaning, several
surface forms in training buy nothing.
Confidence does not follow accuracy here. With five candidates the count of
confident errors falls on jargon, from 110 to 62 per 1,000 on the strict slice.
With eighteen it rises, on jargon from 79 to 109 on all labels and on slang from
151 to 191 on the strict slice, because the augmented release is more confident
overall and more of its errors cross the 0.8 threshold. On abbreviations it
rises from 77 to 118. The augmentation does not by itself make confidence safer
on unfamiliar wording. A deployment that invents its own label vocabulary should
refit the temperature on its own decisions; scripts/fit_temperature.py in the
code repository does that on a few hundred rows and reports whether there were
enough of them.
Thresholding tells a kinder story than the raw count does. In most cells where accuracy improves, the area under the risk-coverage curve falls, so a caller who answers only above a confidence threshold is better served: on strict jargon it goes from 0.235 to 0.078 with five candidates and from 0.452 to 0.418 with eighteen, and on strict slang from 0.223 to 0.121 with five candidates. The exception is slang on all labels with eighteen candidates, where it rises from 0.354 to 0.395. So the release is easier to threshold on unfamiliar wording while being worse at the fixed 0.8 cut, and which of those matters depends on whether a threshold is tuned.
These are two trained models, not an ablation. The two releases differ in more than the label augmentation, and each was trained once. The results above are consistent with the augmentation being the cause, and a later attempt to isolate it by retraining the recipe on other seeds was void because the mixture shipped to that run predated the augmentation commit. Nothing here is a controlled ablation.
Significance is exact McNemar on the paired questions with a Holm correction
over the 23 held-out cells, and the interval resamples labels before questions.
The five-candidate columns for slang and opaque codes are obtained by
renormalising the eighteen-candidate scores over the same five candidates the
main bench uses, which is exact for Ru-Jev because every candidate is scored in
its own pass. Raw counts, per-label tables and the analysis script are in the
code repository under results/analysis/.
Results
v1.3 on its own validation split, 13,373 decisions, with the temperature fitted on the other half:
| Metric | Value |
|---|---|
| Accuracy (micro) | 0.823 |
| ECE, temperature fitted on a disjoint half | 0.0185 |
| ECE, temperature fitted on this same data | 0.0159 |
| Shipped temperature | 1.8640 |
Two ECE figures, because the number depends on where the temperature came from. Fitting and scoring on the same rows is in-sample and optimistic. The first figure is what to expect on new data of the same kind, and it is the one to quote.
Why most of the comparison with v1.1 had to be thrown away
Two things stood between the two releases, and both were found by checking rather than by reasoning.
The data builder carved each task's validation slice by shuffling that task's pool with a module-level random generator, seeded once and consumed by every other random call in the file. Adding two label styles to one task changed how many bits those draws consumed, the stream diverged from that point, and every task below it in the build order received a different slice. Measured: of v1.3's validation rows, 90.3% of rucola and 89.8% of mera_qa had been in v1.1's training set, along with 47.2% of georeview and 21.7% of toxicity.
The second is the point of this release working against its own measurement. Where v1.3 ships an option set in several wordings, its validation rows carry wordings v1.1 never trained on. Scoring v1.1 there measures how well it handles renamed options, which is a real question but not the question a per-task accuracy table is asking. Comparing by text alone hides this, so the rows were compared by text and option wordings together.
| Task | Same rows | Same rows and same option wordings |
|---|---|---|
| terra, danetqa, russe, rwsd, parus, rcb | 100% | 100% |
| rureviews | 100% | 19.6% |
| massive_scenario, massive_intent | 63% | 0% |
| mera_qa | 10.6% | 10.6% |
| rucola | 9.7% | 9.7% |
| georeview | 5.3% | 1.5% |
| toxicity | 2.4% | 0.4% |
What this cost in published claims is worth stating plainly. An early draft of this card read rucola 0.909 against 0.647 and mera_qa 0.939 against 0.596, both of which were v1.1 being scored on its own training data. It also read toxicity 0.722 against 0.954 as a gain in capability, when v1.1 scores 0.961 on toxicity against the wordings it trained on; 0.722 is that same model with the labels renamed. None of those three numbers belonged in a per-task accuracy table.
The builder now seeds each carve from the task name, so a wording change in one task can no longer move the split in another.
Six tasks measured on identical rows with identical option wordings
| Metric, six comparable tasks, all 10,157 decisions | v1.1 | v1.3 |
|---|---|---|
| Correct answers | 8,120 | 8,120 |
| Accuracy (micro) | 0.7994 | 0.7994 |
| Fitted temperature on these rows | 2.0054 | 1.5593 |
| ECE after each model's own temperature | 0.0190 | 0.0191 |
Accuracy does not depend on the temperature, so it is reported on all 10,157 decisions rather than on a split half. The two releases land on the same count: 8,120 correct out of 10,157 each. They do not get the same rows right. They disagree on 1,680 of them, 840 where only v1.1 is right and 840 where only v1.3 is. One standard error on the difference is 0.0056, and the difference is +0.0000.
The temperature and ECE rows are the mean over 25 random half-splits, fitting on one half and scoring the other. v1.3 needs a lower temperature in 25 of 25 splits, with the two ranges not overlapping (1.942 to 2.098 against 1.500 to 1.604), so its raw scores are genuinely less overconfident. The ECE that remains after each model gets its own temperature is the same for both: the difference averages +0.0001 and runs from -0.0063 to +0.0109, favouring v1.3 in 14 of 25 splits. An earlier version of this card reported 0.0172 against 0.0160 from a single fixed split and called v1.3 better calibrated here. On a different split the sign flips, so that claim is withdrawn.
The split seed is recorded as SPLIT_SEED = 20261105 in scripts/calib.py. An earlier implementation shuffled its index list in place without resetting it between splits, so each split was applied to the permutation left by the previous one and the result depended on the order of rows in the file. These figures come from the corrected code; the shift it caused is in the third decimal and turns over nothing.
| Task | v1.1 | v1.3 | n |
|---|---|---|---|
| russe (word sense) | 0.822 | 0.824 | 8,505 |
| danetqa (yes/no) | 0.742 | 0.753 | 821 |
| terra (entailment) | 0.775 | 0.733 | 307 |
| parus (choice) | 0.680 | 0.590 | 100 |
| rcb (3-way NLI) | 0.450 | 0.509 | 220 |
| rwsd (coreference) | 0.564 | 0.480 | 204 |
| all six | 0.7994 | 0.7994 | 10,157 |
On the only ground where the two releases can be compared directly, they are the same model for accuracy. v1.3 is not more accurate than v1.1 and this release does not claim to be. What it is, is far less dependent on how the options are worded, and less overconfident before calibration, needing a noticeably lower temperature on identical rows. After each model is given its own temperature, the calibration error that remains is the same for both.
One caveat on this table. russe supplies 8,505 of the 10,157
decisions, 84% of the total, so the micro average is mostly a
statement about one word-sense task. The five smaller tasks move in both
directions and none of them moves far enough to matter against its own standard
error.
The remaining seven tasks, each model on its own held-out slice
No shared rows exist for these, so each column is that model's own validation slice scored with its own option wordings. That is an honest measurement of each model separately and a weak comparison between them.
| Task | v1.1, own slice | v1.3, own slice | n |
|---|---|---|---|
| massive_intent (60 labels) | not measured | 0.960 | 3,000 |
| massive_scenario (18 labels) | not measured | 0.955 | 3,000 |
| toxicity | 0.961 | 0.954 | 3,000 |
| rureviews (sentiment) | not measured | 0.774 | 3,000 |
| rucola (acceptability) | 0.650 | 0.647 | 1,344 |
| georeview (5 levels) | 0.629 | 0.631 | 3,000 |
| mera_qa (reasoning) | 0.522 | 0.596 | 245 |
Four epochs were trained and the third was shipped
The run went four epochs and each one was kept. The fourth overfits: training loss fell to 0.085 while validation loss rose from 0.483 to 0.820, and a temperature fitted on it comes out at 3.92, which is what overconfidence looks like as a number.
| Epoch | Training loss | Validation loss | Validation accuracy |
|---|---|---|---|
| 1 | 0.6029 | 0.5054 | 0.7536 |
| 2 | 0.4336 | 0.4576 | 0.7920 |
| 3 | 0.2714 | 0.4831 | 0.8181 |
| 4 | 0.0849 | 0.8204 | 0.8139 |
The shipped weights are the third epoch. Had only the last checkpoint been kept, the released model would have been the worse one and nothing would have shown it.
Name your options carefully
The model reads the option text, so wording still matters, just much less than it did.
| Option wording | v1.1 accuracy | v1.1 confident errors | v1.3 accuracy | v1.3 confident errors |
|---|---|---|---|---|
Russian word, погода |
0.956 | 20 | 0.960 | 19 |
English key, weather |
0.957 | 16 | 0.962 | 26 |
| Descriptive sentence | 0.939 | 14 | 0.958 | 16 |
Russian phrase, прогноз погоды |
0.881 | 34 | 0.957 | 14 |
Department name, метеослужба |
0.835 | 54 | 0.869 | 52 |
Jargon, вэзер |
0.633 | 85 | 0.855 | 44 |
The confident-error columns count answers that were wrong while above 0.8 confidence, out of a thousand questions per row. They are measured with five candidates; with all eighteen at once the count rises for v1.3 on jargon and on slang, because it is more confident overall.
Three rules follow from the wording bench above.
Write options as ordinary words that describe the category. That is where several surface forms in training actually pay: on held-out wordings made of words, including borrowed words and office slang the model never saw.
Do not use abbreviations or opaque codes as option text. ПОГ and ПГД for
the weather category are not better read by v1.3 than by v1.1, and opaque codes
sit at chance for both. A code that contains a readable key, such as
WEATHER_SCN, works for the same reason the key does, so it is the key that is
doing the work, not the code.
Measure your own vocabulary and refit the temperature on it.
scripts/robustness.py scores a label vocabulary of your own, and
scripts/fit_temperature.py fits a temperature on a few hundred of your own
decisions and says whether there were enough of them. Accuracy does not depend
on the temperature, so refitting changes only the number next to the answer, but
on unfamiliar wording that number is where this release is least trustworthy.
Withholding an answer
A calibrated probability is worth something only if the caller acts on it. Set a
confidence threshold, answer above it and send the rest to a human. The serving
API takes a threshold and returns the withheld decisions in a review list.
| Share answered | Confidence threshold | Accuracy on those | Errors per 1000 |
|---|---|---|---|
| 50% | 0.85 | 0.967 | 33 |
| 60% | 0.77 | 0.947 | 53 |
| 70% | 0.69 | 0.921 | 79 |
| 80% | 0.61 | 0.892 | 108 |
| 90% | 0.54 | 0.859 | 141 |
| 100% | 0.23 | 0.823 | 177 |
Limitations
Hard reasoning tasks with small training sets remain weak and are not fixed by
anything in this release: rwsd coreference, rcb three-way inference and
parus causal choice all train on fewer than a thousand rows each. For a 307M
encoder that is a size limit, not a data-wording limit. A larger backbone was
tried once, xlm-roberta-large at 560M, and lost to v1.
Those same tasks are where the two releases differ most in the comparable
table, and in both directions: v1.3 is 0.059 above v1.1 on rcb and 0.084 and
0.090 below it on rwsd and parus. Their validation slices hold 220, 204 and
100 rows, so one standard error is roughly 0.03 to 0.05 and none of those three
gaps is far outside it. Read them as scatter, not as a trend, and do not plan
around either direction.
The model answers only within the option set it is given. It cannot say that none of the options fit, so include an explicit "none of these" option when that outcome is possible.
Probabilities are calibrated in aggregate, not per task. A single temperature serves all thirteen tasks, so a task far from the mix average can be systematically overconfident or underconfident.
Every training recipe here was run once. The wording-robustness gap is large enough that seed noise is unlikely to account for it, but it has not been measured across seeds, and the small per-task differences in the comparable table certainly should not be read as reproducible.
Robustness to wording was measured systematically on the choice-from-options shape only, using the eighteen-way routing task. For yes/no there is a single renaming test, on toxicity, reported above. The ordinal-scale shape has no wording test at all, and it is the shape where renaming a level could also disturb the order, so nothing here should be assumed to carry over to it.
Speed on CPU
Apple M4, float32, median of 25 runs after 5 warm-up runs, nothing else running on the machine. Latency is the whole decision: tokenisation, the forward passes and the softmax.
| Candidates | Ru-Jev v1.3 | Laya | Ratio |
|---|---|---|---|
| 2 | 18.5 ms | 15.5 ms | 1.19 |
| 5 | 32.8 ms | 17.8 ms | 1.84 |
| 18 | 73.8 ms | 30.5 ms | 2.42 |
Ru-Jev scores each (context, option) pair in its own forward pass, so the cost of a decision grows linearly with the number of candidates, by roughly 4 ms per candidate on this machine. That is also why option order is not an input to it. Laya reads all options in one pass, so its cost is almost flat in the number of candidates and order is an input. Ru-Jev is slower in both settings and no speed advantage is claimed here.
Usage
import json, torch, torch.nn as nn
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModel, AutoTokenizer
REPO = "besktech/ru-jev"
class Scorer(nn.Module):
def __init__(self, repo):
super().__init__()
self.backbone = AutoModel.from_pretrained(repo)
self.head = nn.Linear(self.backbone.config.hidden_size, 1)
self.head.load_state_dict(
load_file(hf_hub_download(repo, "head.safetensors")))
def forward(self, ids, mask):
cls = self.backbone(input_ids=ids, attention_mask=mask).last_hidden_state[:, 0]
return self.head(cls).squeeze(-1)
calib = json.load(open(hf_hub_download(REPO, "calibration.json")))
T = calib["temperature"]
tok = AutoTokenizer.from_pretrained(REPO)
model = Scorer(REPO).eval()
context = "Приложение падает при входе после обновления."
options = ["техническая проблема", "оплата, счета и возвраты", "продажи"]
with torch.no_grad():
enc = tok([context] * len(options), options, truncation=True,
max_length=calib["max_len"], padding=True, return_tensors="pt")
probs = torch.softmax(model(enc["input_ids"], enc["attention_mask"]) / T, dim=0)
for option, p in zip(options, probs.tolist()):
print(f"{p:.3f} {option}")
Divide by the shipped temperature before the softmax. Without it the probabilities are overconfident and the threshold logic above does not hold.
Training data
Thirteen tasks from open Russian datasets, 143,991 training and 26,746
validation decisions. Russian SuperGLUE (terra, danetqa, russe, rwsd,
parus, rcb), RuCoLA, MERA (ruopenbookqa, ruworldtree), MTEB Russian
(RuReviewsClassification, GeoreviewClassification, amazon_massive_intent,
amazon_massive_scenario) and rudetoxifier_data. Each source keeps its own
licence; see DATASETS.md in the code repository.
Every fixed option set ships in several wordings. The builder verifies that all thirteen tasks are present and refuses to write the data if any is short. That check exists because an earlier run silently trained on eleven.
Citation
The two releases are compared in the preprint arXiv:2611.08439. The code, the per-row scores and the evaluation scripts behind every figure in it are archived at Zenodo under 10.5281/zenodo.23986846, which always resolves to the latest version. Cite 10.5281/zenodo.23986847 for the exact version the paper was written against.
@misc{beskorovainyi2026rujev,
title = {Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Words, Not on Abbreviations or Codes},
author = {Beskorovainyi, Vladimir},
year = {2026},
eprint = {2611.08439},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2611.08439}
}
@software{beskorovainyi2026rujevcode,
title = {Ru-Jev: open typed-decision model (v1.3) — code, per-row results and evaluation scripts},
author = {Beskorovainyi, Vladimir},
year = {2026},
version = {1.3.0},
publisher = {Zenodo},
doi = {10.5281/zenodo.23986846},
url = {https://doi.org/10.5281/zenodo.23986846}
}
License
MIT covers the code and the weights. The training data keeps the licences of
its sources, which are listed task by task in DATASETS.md in the code
repository.
Vladimir Beskorovainyi, Enterprise AI Architect & CTO, author of the Amazon bestseller "Enterprise AI: The KATA Way", founder of Besk Tech (besk.tech)