card for v1.3: measured numbers only, jargon row explained

README.md

@@ -22,7 +22,7 @@ base_model: jhu-clsp/mmBERT-base
 inference: false
 ---
 
-# Ру-Жев (Ru-Jev) v1.1
+# Ру-Жев (Ru-Jev) v1.3
 
 Open, Russian-first typed-decision model. It does not generate text. Given a
 context and a set of candidate options, it returns a calibrated probability over
@@ -33,8 +33,8 @@ runs on CPU and can be deployed inside a closed perimeter.
 Project page: https://ru-jev.besk.tech
 Training and evaluation code: https://github.com/beskvladimir-create/ru-jev
 
-The previous release is kept on the `v1` branch of this repository. Load it with
-`revision="v1"` if you depend on its exact behaviour.
+Earlier releases are kept on branches: `v1.1` and `v1`. Load one with
+`revision="v1.1"` if you depend on its exact behaviour.
 
 The inference widget is switched off on purpose. This is a backbone plus a
 separate scoring head, not a standard sequence-classification checkpoint, so the
@@ -44,195 +44,281 @@ stock widget cannot load it. Use the snippet below.
 trained from scratch on open, human-labelled datasets. The name is a play on the
 category, not a claim of compatibility or endorsement.
 
-## What changed since v1
+## What changed since v1.1
 
-v1 could not route. It was trained on natural language inference, sentiment and
-toxicity, and no task in its mix ever presented category names as the options,
-so asking it to pick a department from a list left it at roughly chance.
+v1.1 read the option text, which is the point of the design, but it read it
+narrowly. It had learned the exact wordings its training data used. Rename a
+category and accuracy fell off a cliff: 0.956 when a scenario was called
+`погода`, 0.633 when the same scenario was called `вэзер`. Worse, on the
+unfamiliar wording it stayed confident while being wrong, so the probability
+stopped carrying information exactly where a caller needed it most.
 
-| Task | v1 | v1.1 |
+v1.3 ships every fixed option set in several wordings and draws one per example,
+so the model has to read an option instead of recognising it. For the routing
+tasks that includes the English key written in Cyrillic and a clipped form of
+it, generated by rule.
+
+| Option wording | v1.1 | v1.3 |
 | --- | --- | --- |
-| Routing, 60 intents | 0.344 | 0.965 |
-| Routing, 18 scenarios | 0.305 | 0.954 |
-| Reviews, 5 levels | 0.391 | 0.629 |
-| Reasoning questions | 0.327 | 0.522 |
-| All 13 tasks | 0.663 | 0.825 |
-
-On the nine tasks v1 was trained for, scored on identical rows, v1.1 is ahead
-overall as well, 0.843 against 0.824, with the gains on acceptability and
-sentiment and losses on `rcb` and `russe`.
-
-Calibration moved the other way on those nine tasks: 0.009 for v1 against 0.020
-for v1.1, both measured held out. On the full thirteen-task mix the order
-reverses, 0.025 for v1 against 0.020 for v1.1, because a single temperature
-cannot serve a model that is at chance on half the work. v1.1 is the better
-trade: two points of calibration on a narrow slice bought routing that works.
+| Russian word, `погода` | 0.956 | 0.960 |
+| English key, `weather` | 0.957 | 0.962 |
+| Descriptive sentence | 0.939 | 0.958 |
+| Russian phrase, `прогноз погоды` | 0.881 | 0.957 |
+| Department name, `метеослужба` | 0.835 | 0.869 |
+| Jargon, `вэзер` | 0.633 | 0.855 |
+
+A thousand routing questions per row, the same eighteen categories every time,
+only the wording differs.
+
+One caveat on the jargon row, because it is the headline number. An intent key
+begins with its scenario name, so three of the eighteen jargon labels coincide
+with what the transliteration rule produces and reach training as a first word.
+On the fifteen that do not coincide, which is the honest measure of
+generalisation to wording the model has never seen, v1.3 scores 0.838 against 0.624 for v1.1.
+The rules were not tuned against this test vocabulary; `scripts/robustness.py`
+in the code repository reports both figures.
 
 ## Results
 
-Measured on half of the validation split, 13,373 decisions, with the
-temperature fitted on the other half.
+v1.3 on its own validation split, 13,373 decisions, with the temperature
+fitted on the other half:
 
 | Metric | Value |
 | --- | --- |
-| Accuracy (micro) | 0.825 |
-| ECE, temperature fitted on a disjoint half | 0.0200 |
+| Accuracy (micro) | 0.823 |
+| ECE, temperature fitted on a disjoint half | 0.0185 |
 | ECE, temperature fitted on this same data | 0.0159 |
-| Shipped temperature | 2.2754 |
+| Shipped temperature | 1.8640 |
 
 Two ECE figures, because the number depends on where the temperature came from.
 Fitting and scoring on the same rows is in-sample and optimistic. The first
 figure is what to expect on new data of the same kind, and it is the one to
 quote.
 
-Per task, full validation split:
+### Why most of the comparison with v1.1 had to be thrown away
+
+Two things stood between the two releases, and both were found by checking
+rather than by reasoning.
+
+The data builder carved each task's validation slice by shuffling that task's
+pool with a module-level random generator, seeded once and consumed by every
+other random call in the file. Adding two label styles to one task changed how
+many bits those draws consumed, the stream diverged from that point, and every
+task below it in the build order received a different slice. Measured: of
+v1.3's validation rows, 90.3% of rucola and 89.8% of mera_qa had been in v1.1's
+training set, along with 47.2% of georeview and 21.7% of toxicity.
+
+The second is the point of this release working against its own measurement.
+Where v1.3 ships an option set in several wordings, its validation rows carry
+wordings v1.1 never trained on. Scoring v1.1 there measures how well it handles
+renamed options, which is a real question but not the question a per-task
+accuracy table is asking. Comparing by text alone hides this, so the rows were
+compared by text and option wordings together.
 
-| Task | Accuracy | n |
+| Task | Same rows | Same rows and same option wordings |
 | --- | --- | --- |
-| massive_intent (60 labels) | 0.965 | 3000 |
-| toxicity | 0.961 | 3000 |
-| massive_scenario (18 labels) | 0.954 | 3000 |
-| russe (word sense) | 0.822 | 8505 |
-| terra (entailment) | 0.775 | 307 |
-| rureviews (sentiment) | 0.772 | 3000 |
-| danetqa (yes/no) | 0.742 | 821 |
-| parus (choice) | 0.680 | 100 |
-| rucola (acceptability) | 0.650 | 1344 |
-| georeview (5 levels) | 0.629 | 3000 |
-| rwsd (coreference) | 0.564 | 204 |
-| mera_qa (reasoning) | 0.522 | 245 |
-| rcb (3-way NLI) | 0.450 | 220 |
+| terra, danetqa, russe, rwsd, parus, rcb | 100% | 100% |
+| rureviews | 100% | 19.6% |
+| massive_scenario, massive_intent | 63% | 0% |
+| mera_qa | 10.6% | 10.6% |
+| rucola | 9.7% | 9.7% |
+| georeview | 5.3% | 1.5% |
+| toxicity | 2.4% | 0.4% |
+
+What this cost in published claims is worth stating plainly. An early draft of
+this card read rucola 0.909 against 0.647 and mera_qa 0.939 against 0.596, both
+of which were v1.1 being scored on its own training data. It also read toxicity
+0.722 against 0.954 as a gain in capability, when v1.1 scores 0.961 on toxicity
+against the wordings it trained on; 0.722 is that same model with the labels
+renamed. None of those three numbers belonged in a per-task accuracy table.
+
+The builder now seeds each carve from the task name, so a wording change in one
+task can no longer move the split in another.
+
+### Six tasks measured on identical rows with identical option wordings
+
+| Metric, six comparable tasks | v1.1 | v1.3 |
+| --- | --- | --- |
+| Accuracy (micro) | 0.7948 | 0.7920 |
+| ECE, temperature fitted on the other half | 0.0172 | 0.0160 |
+
+Scored on 5,078 decisions, half of the six comparable tasks, with each model's temperature fitted on the other half. The accuracy difference is -0.0028, which is 0.3 standard errors and therefore nothing.
+
+| Task | v1.1 | v1.3 | n |
+| --- | --- | --- | --- |
+| russe (word sense) | 0.822 | 0.824 | 8505 |
+| danetqa (yes/no) | 0.742 | 0.753 | 821 |
+| terra (entailment) | 0.775 | 0.733 | 307 |
+| parus (choice) | 0.680 | 0.590 | 100 |
+| rcb (3-way NLI) | 0.450 | 0.509 | 220 |
+| rwsd (coreference) | 0.564 | 0.480 | 204 |
+
+On the only ground where the two releases can be compared directly, they are
+the same model for accuracy. v1.3 is not more accurate than v1.1 and this
+release does not claim to be. What it is, is far less dependent on how the
+options are worded, and slightly better calibrated with a temperature of
+1.8640 against 2.2754, meaning less of the confidence has to be flattened
+away after the fact.
+
+### The remaining seven tasks, each model on its own held-out slice
+
+No shared rows exist for these, so each column is that model's own validation
+slice scored with its own option wordings. That is an honest measurement of
+each model separately and a weak comparison between them.
+
+| Task | v1.1, own slice | v1.3, own slice | n |
+| --- | --- | --- | --- |
+| massive_intent (60 labels) | not measured | 0.960 | 3000 |
+| massive_scenario (18 labels) | not measured | 0.955 | 3000 |
+| toxicity | 0.961 | 0.954 | 3000 |
+| rureviews (sentiment) | not measured | 0.774 | 3000 |
+| rucola (acceptability) | 0.650 | 0.647 | 1344 |
+| georeview (5 levels) | 0.629 | 0.631 | 3000 |
+| mera_qa (reasoning) | 0.522 | 0.596 | 245 |
+
+## Four epochs were trained and the third was shipped
+
+The run went four epochs and each one was kept. The fourth overfits: training
+loss fell to 0.085 while validation loss rose from 0.483 to 0.820, and a
+temperature fitted on it comes out at 3.92, which is what overconfidence looks
+like as a number.
+
+| Epoch | Training loss | Validation loss | Validation accuracy |
+| --- | --- | --- | --- |
+| 1 | 0.6029 | 0.5054 | 0.7536 |
+| 2 | 0.4336 | 0.4576 | 0.7920 |
+| 3 | 0.2714 | 0.4831 | 0.8181 |
+| 4 | 0.0849 | 0.8204 | 0.8139 |
+
+The shipped weights are the third epoch. Had only the last checkpoint been kept,
+the released model would have been the worse one and nothing would have shown it.
 
 ## Name your options carefully
 
-The model reads the option text. How you word your categories changes the
-answer, and the effect is large. Three hundred routing questions, the same
-eighteen categories every time, only the wording differs:
+The model reads the option text, so wording still matters, just much less than
+it did.
 
-| Wording | Example | Accuracy | Confident errors |
-| --- | --- | --- | --- |
-| Russian word | `погода` | 0.973 | 3 |
-| English key | `weather` | 0.963 | 4 |
-| Descriptive sentence | `человек спрашивает про погоду` | 0.953 | 2 |
-| Russian phrase | `прогноз погоды` | 0.890 | 9 |
-| Department name | `метеослужба` | 0.870 | 10 |
-| Jargon, transliterated | `вэзер` | 0.630 | 26 |
-
-Write options as ordinary words that describe the category. Avoid borrowed
-jargon: `биллинг` costs far more accuracy than `оплата, счета и возвраты`. The
-last column counts answers that were wrong while above 0.8 confidence, so on
-unfamiliar wording the probability stops being a reliable guide as well. If you
-must use in-house jargon, measure it on your own data first with
-`scripts/robustness.py` from the code repository.
+| Option wording | Accuracy | Confident errors |
+| --- | --- | --- |
+| English key, `weather` | 0.962 | 26 |
+| Russian word, `погода` | 0.960 | 19 |
+| Russian phrase, `прогноз погоды` | 0.957 | 14 |
+| Jargon, `вэзер` | 0.855 | 44 |
+| Department name, `метеослужба` | 0.869 | 52 |
+| Descriptive sentence | 0.958 | 16 |
+
+The last column counts answers that were wrong while above 0.8 confidence.
+Write options as ordinary words that describe the category. If you use in-house
+jargon, measure it on your own data first with `scripts/robustness.py` from the
+code repository rather than assuming it transfers.
 
 ## Withholding an answer
 
-A calibrated probability is worth something only if the caller acts on it. The
-serving code in the repository takes a `threshold` and withholds anything below
-it for a person to handle.
+A calibrated probability is worth something only if the caller acts on it. Set a
+confidence threshold, answer above it and send the rest to a human. The serving
+API takes a `threshold` and returns the withheld decisions in a review list.
 
-| Share answered automatically | Threshold | Accuracy on those | Errors per 1000 |
+| Share answered | Confidence threshold | Accuracy on those | Errors per 1000 |
 | --- | --- | --- | --- |
-| 30% | 0.98 | 99.3% | 7 |
-| 50% | 0.85 | 97.1% | 29 |
-| 60% | 0.77 | 95.1% | 49 |
-| 80% | 0.60 | 89.5% | 105 |
-| 100% | none | 82.5% | 175 |
+| 50% | 0.85 | 0.967 | 33 |
+| 60% | 0.77 | 0.947 | 53 |
+| 70% | 0.69 | 0.921 | 79 |
+| 80% | 0.61 | 0.892 | 108 |
+| 90% | 0.54 | 0.859 | 141 |
+| 100% | 0.23 | 0.823 | 177 |
 
-Per decision shape at 95% accuracy or better, routing over many options clears
-the entire stream, while yes/no questions and small choices clear about half.
+## Limitations
 
-## What a cheap baseline gets
+Hard reasoning tasks with small training sets remain weak and are not fixed by
+anything in this release: `rwsd` coreference, `rcb` three-way inference and
+`parus` causal choice all train on fewer than a thousand rows each. For a 307M
+encoder that is a size limit, not a data-wording limit. A larger backbone was
+tried once, xlm-roberta-large at 560M, and lost to v1.
 
-| Task | Majority class | TF-IDF + logreg | Ru-Jev v1.1 |
-| --- | --- | --- | --- |
-| massive_scenario | 0.066 | 0.435 | 0.954 |
-| massive_intent | 0.063 | 0.820 | 0.965 |
-| terra | 0.498 | 0.394 | 0.775 |
-| danetqa | 0.502 | 0.568 | 0.742 |
-| russe | 0.631 | 0.701 | 0.822 |
-| georeview | 0.199 | 0.544 | 0.629 |
-| rureviews | 0.324 | 0.720 | 0.772 |
-| rucola | 0.587 | 0.609 | 0.650 |
-| toxicity | 0.766 | 0.943 | 0.961 |
-| rwsd | 0.554 | 0.554 | 0.564 |
-
-The model earns its place on routing, entailment and yes/no questions. On
-toxicity it is 1.8 points over character n-grams and on sentiment 5.2 points,
-so for those tasks alone a linear model is cheaper to run and easier to debug.
+Those same tasks are where the two releases differ most in the comparable
+table, and in both directions: v1.3 is 0.059 above v1.1 on `rcb` and 0.084 and
+0.090 below it on `rwsd` and `parus`. Their validation slices hold 220, 204 and
+100 rows, so one standard error is roughly 0.03 to 0.05 and none of those three
+gaps is far outside it. Read them as scatter, not as a trend, and do not plan
+around either direction.
 
-## Limitations
+The model answers only within the option set it is given. It cannot say that
+none of the options fit, so include an explicit "none of these" option when that
+outcome is possible.
 
-- **Coreference is not supported.** On `rwsd` the model matches the majority
-  class exactly, 0.564 against 0.554. It does not resolve pronouns.
-- **Multi-step reasoning is weak.** `rcb` 0.450 and `mera_qa` 0.522. A 307M
-  encoder does not chain inferences, and no amount of further training on this
-  architecture will change that.
-- **Unfamiliar label wording degrades both the answer and the confidence**, as
-  the table above shows.
-- Russian first. The multilingual base handles other languages, but they were
-  not measured.
+Probabilities are calibrated in aggregate, not per task. A single temperature
+serves all thirteen tasks, so a task far from the mix average can be
+systematically over or under confident.
 
 ## Speed on CPU
 
-Apple M4, four threads, median of 25 runs after warmup, this checkpoint in
-float32.
-
-| Request | Tokens | Median | p95 |
-| --- | --- | --- | --- |
-| yes/no, short context | 23 | 24.5 ms | 26.5 ms |
-| choice of three | 17 | 31.2 ms | 32.6 ms |
-| yes/no, full 256-token context | 315, truncated to 256 | 133.2 ms | 141.1 ms |
+Apple M4, 4 threads, float32, median of 25 runs:
 
-On a single thread the short requests are unchanged and the full context costs
-186 ms. A server CPU will give different figures.
+| Decision | Latency |
+| --- | --- |
+| Short yes/no | 25 ms |
+| Choice from three options | 31 ms |
+| Full 256-token context | 131 ms |
 
 ## Usage
 
 ```python
-import torch, torch.nn as nn, json
-from transformers import AutoTokenizer, AutoModel
+import json, torch, torch.nn as nn
 from huggingface_hub import hf_hub_download
+from transformers import AutoModel, AutoTokenizer
 
 REPO = "besktech/ru-jev"
+
+
+class Scorer(nn.Module):
+    def __init__(self, repo):
+        super().__init__()
+        self.backbone = AutoModel.from_pretrained(repo)
+        self.head = nn.Linear(self.backbone.config.hidden_size, 1)
+        self.head.load_state_dict(
+            torch.load(hf_hub_download(repo, "head.pt"), map_location="cpu"))
+
+    def forward(self, ids, mask):
+        cls = self.backbone(input_ids=ids, attention_mask=mask).last_hidden_state[:, 0]
+        return self.head(cls).squeeze(-1)
+
+
+calib = json.load(open(hf_hub_download(REPO, "calibration.json")))
+T = calib["temperature"]
 tok = AutoTokenizer.from_pretrained(REPO)
-backbone = AutoModel.from_pretrained(REPO).eval()
-head = nn.Linear(backbone.config.hidden_size, 1)
-head.load_state_dict(torch.load(hf_hub_download(REPO, "head.pt"), map_location="cpu"))
-T = json.load(open(hf_hub_download(REPO, "calibration.json")))["temperature"]
-
-@torch.no_grad()
-def decide(context, options):
-    enc = tok([context] * len(options), [str(o) for o in options],
-              truncation=True, max_length=256, padding=True, return_tensors="pt")
-    scores = head(backbone(**enc).last_hidden_state[:, 0]).squeeze(-1)
-    probs = torch.softmax(scores / T, dim=0)
-    return list(zip(options, probs.tolist()))
-
-decide("Клиента списали дважды", ["да", "нет"])
-decide("Не приходит счёт", ["оплата, счета и возвраты", "техническая проблема"])
-decide("Отличный сервис, всё быстро", ["негативный", "нейтральный", "позитивный"])
+model = Scorer(REPO).eval()
+
+context = "Приложение падает при входе после обновления."
+options = ["техническая проблема", "оплата, счета и возвраты", "продажи"]
+
+with torch.no_grad():
+    enc = tok([context] * len(options), options, truncation=True,
+              max_length=calib["max_len"], padding=True, return_tensors="pt")
+    probs = torch.softmax(model(enc["input_ids"], enc["attention_mask"]) / T, dim=0)
+
+for option, p in zip(options, probs.tolist()):
+    print(f"{p:.3f}  {option}")
 ```
 
-## Training data
+Divide by the shipped temperature before the softmax. Without it the
+probabilities are overconfident and the threshold logic above does not hold.
 
-Open datasets only, all permitting commercial use:
+## Training data
 
-- Russian SuperGLUE (MIT): terra, danetqa, russe, rwsd, parus, rcb
-- MASSIVE (CC-BY-4.0): intent and scenario routing
-- Georeview (MIT): 5-level reviews
-- MERA (MIT): reasoning questions
-- RuReviews (Apache-2.0): sentiment
-- RuDetoxifier data (MIT): toxicity
-- RuCoLA (Apache-2.0 labels): acceptability
+Thirteen tasks from open Russian datasets, 143,991 training and 26,746
+validation decisions. Russian SuperGLUE (`terra`, `danetqa`, `russe`, `rwsd`,
+`parus`, `rcb`), RuCoLA, MERA (`ruopenbookqa`, `ruworldtree`), MTEB Russian
+(`RuReviewsClassification`, `GeoreviewClassification`, `amazon_massive_intent`,
+`amazon_massive_scenario`) and `rudetoxifier_data`. Each source keeps its own
+licence; see `DATASETS.md` in the code repository.
 
-Boolean tasks are trained with five wordings of the same answer, and options are
-reordered on every epoch, so the model cannot learn a position or a single
-surface form.
+Every fixed option set ships in several wordings. The builder verifies that all
+thirteen tasks are present and refuses to write the data if any is short, which
+exists because an earlier run silently trained on eleven.
 
 ## License
 
-MIT.
+MIT, weights and code.
 
 Vladimir Beskorovainyi, Enterprise AI Architect & CTO, author of the Amazon
 bestseller "Enterprise AI: The KATA Way", founder of Besk Tech (besk.tech)