Results on eight public sets: regulated-advice-bench for tax and legal, decontaminated safety and security test splits; Jev scored on the same rows

README.md

@@ -17,11 +17,11 @@ with Gemma-4.
 
 ![How Sev works: text and question become one point, the closest answer wins, and the question decides where the point lands](assets/sev_teaser.png)
 
-![Sev against Jev, Tev1-4B and Kev-4B on nine public benchmarks, and mean AUROC against latency per request](assets/sev_results.png)
+![Sev against Jev, Tev1-4B and Kev-4B on eight public benchmarks, and mean AUROC against latency per request](assets/sev_results.png)
 
-- **Level with Jev on public guard benchmarks.** On nine public, human-labelled safety, security, tax
-  and legal sets, Sev averages **0.939 AUROC against Jev's 0.939**, with every model
-  reading the same input. It is ahead of Jev on five of the nine: tax, AEGIS 2.0, Nemotron, PolyGuard and XSTest.
+- **Level with Jev on public guard benchmarks.** On eight public, human-labelled safety, security, tax
+  and legal sets, Sev averages **0.936 AUROC against Jev's 0.934**, with every model
+  reading the same input. It is ahead of Jev on six of the eight: tax, AEGIS 2.0, Nemotron, PolyGuard, XSTest and deepset.
 - **Ask your own question.** Any yes/no or multiple-choice policy question, in plain words. The
   question is part of the input, so a new policy is a new question, not a new model.
 - **Open data, permissive licenses.** Every training source allows commercial use. No private or
@@ -143,48 +143,43 @@ the assistant's reply.
 
 ## Results
 
-AUROC on nine public, human-labelled sets. Every model gets the same text and the same question: on
-tax and legal, the user's message and the assistant's reply, read up to at least 1,024 tokens (which
-covers 99% of rows). Jev's scores come from the Jev-bench comparison on the same rows, where Jev was
-asked a one-line question per row. Tev1-4B (`togethercomputer/Tev1-4B-experimental`) and Kev-4B
+AUROC on eight public, human-labelled sets: Alinia's
+[regulated-advice-bench](https://huggingface.co/datasets/alinia/regulated-advice-bench) for tax and
+legal, and six safety and security test splits with every test prompt that is a copy, rewrite or
+translation of a training prompt removed. Every model gets the same text and the same question: on tax
+and legal, the user's message and the assistant's reply, read up to at least 1,024 tokens. We scored Jev
+through its API, with one policy question and two lines of criteria per guard. Tev1-4B (`togethercomputer/Tev1-4B-experimental`) and Kev-4B
 (`jaredpalmer/kev-4b`), two other open Jev-style models, were scored zero-shot with their own readouts
 and our questions. Bold is the best in each column.
 
-| Model | Tax | Legal | AEGIS 2.0 | Nemotron | PolyGuard | XSTest | jailbreak | PolyGuard sec. | deepset | Mean |
-|---|---|---|---|---|---|---|---|---|---|---|
-| **Sev** (this model) | **0.981** | 0.861 | **0.953** | **0.918** | **0.958** | **0.985** | 0.979 | 0.859 | 0.958 | **0.939** |
-| Jev (TypeSafe, jev-1.13.0) | 0.979 | **0.894** | 0.933 | 0.896 | 0.952 | 0.961 | **0.998** | **0.877** | 0.961 | 0.939 |
-| Tev1-4B (Together) | 0.980 | 0.836 | 0.940 | 0.892 | 0.918 | 0.944 | 0.978 | 0.704 | **0.978** | 0.908 |
-| Kev-4B (jaredpalmer) | 0.966 | 0.755 | 0.944 | 0.897 | 0.922 | 0.942 | 0.986 | 0.648 | 0.969 | 0.892 |
+| Model | Tax | Legal | AEGIS 2.0 | Nemotron | PolyGuard | XSTest | PolyGuard sec. | deepset | Mean |
+|---|---|---|---|---|---|---|---|---|---|
+| **Sev** (this model) | **0.974** | 0.817 | **0.943** | **0.923** | **0.961** | **0.982** | 0.908 | 0.978 | **0.936** |
+| Jev (TypeSafe, jev-1.13.0) | 0.961 | **0.903** | 0.923 | 0.905 | 0.942 | 0.977 | **0.910** | 0.955 | 0.934 |
+| Tev1-4B (Together) | 0.973 | 0.798 | 0.928 | 0.898 | 0.916 | 0.935 | 0.738 | **0.979** | 0.896 |
+| Kev-4B (jaredpalmer) | 0.970 | 0.744 | 0.930 | 0.900 | 0.911 | 0.932 | 0.707 | 0.970 | 0.883 |
 
-Sev in more detail, with Alinia's public [regulated-advice-bench](https://huggingface.co/datasets/alinia/regulated-advice-bench)
-as two more columns. Best F1 picks the threshold on the test set itself, so it is an upper bound; F1
+Sev in more detail. Best F1 picks the threshold on the test set itself, so it is an upper bound; F1
 and false-positive rate at 0.5 are the untuned default.
 
-| | Tax | Legal | AEGIS 2.0 | Nemotron | PolyGuard | XSTest | jailbreak | PolyGuard sec. | deepset | regulated-advice-bench tax | regulated-advice-bench legal |
-|---|---|---|---|---|---|---|---|---|---|---|---|
-| AUROC | 0.981 | 0.861 | 0.953 | 0.918 | 0.958 | 0.985 | 0.979 | 0.859 | 0.958 | 0.974 | 0.817 |
-| Best F1 | 0.964 | 0.768 | 0.857 | 0.776 | 0.848 | 0.924 | 0.924 | 0.778 | 0.912 | 0.959 | 0.768 |
-| F1 at 0.5 | 0.783 | 0.735 | 0.826 | 0.755 | 0.846 | 0.903 | 0.867 | 0.774 | 0.843 | 0.805 | 0.748 |
-| False-positive rate at 0.5 | 56% | 28% | 19% | 20% | 7% | 14% | 2% | 34% | 0% | 51% | 21% |
+| | Tax | Legal | AEGIS 2.0 | Nemotron | PolyGuard | XSTest | PolyGuard sec. | deepset |
+|---|---|---|---|---|---|---|---|---|
+| AUROC | 0.974 | 0.817 | 0.943 | 0.923 | 0.961 | 0.982 | 0.908 | 0.978 |
+| Best F1 | 0.959 | 0.768 | 0.861 | 0.829 | 0.874 | 0.908 | 0.845 | 0.935 |
+| F1 at 0.5 | 0.805 | 0.748 | 0.841 | 0.824 | 0.873 | 0.879 | 0.844 | 0.860 |
+| False-positive rate at 0.5 | 51% | 21% | 20% | 21% | 7% | 14% | 26% | 0% |
 
 **Reading the tables.**
 - **Sev has seen these distributions.** It was trained on the train splits of AEGIS 2.0, Nemotron,
-  PolyGuardMix (which contains WildGuardMix, the source of PolyGuard's prompts), jailbreak-classification
-  and deepset. Test rows that match a training text were removed, but these five sets are
-  in-distribution for Sev. Tev1-4B and Kev-4B were trained on general decision data, not on safety or
+  PolyGuardMix (which contains WildGuardMix, the source of PolyGuard's prompts) and deepset. Near-copies
+  of training prompts were removed from the test sets, but these four sets are in-distribution for Sev. Tev1-4B and Kev-4B were trained on general decision data, not on safety or
   security data, and Jev's training data is not disclosed.
-- **PolyGuard security is partly a length test:** prompt length alone reaches about 0.98 AUROC on these
-  rows.
 - **Legal advice is hard to label.** The human reviewers of the legal set agree only at
   Krippendorff's alpha ~0.37, so small gaps there are noise.
-- **Tax and legal come from Alinia's human-labelled advice benchmark.** The first two columns are the
-  rows of the Jev-bench comparison; regulated-advice-bench is the public release of the same benchmark.
-  They are not independent: 280 of regulated-advice-bench's 338 tax conversations and 223 of its 332 legal conversations also appear, in at least one language, among the Jev-bench rows.
 
 ## Choosing a threshold
 
-For safety and security, 0.5 is a reasonable default: the false-positive rate at 0.5 is 0% to 20% on six of these seven sets, and 34% on PolyGuard security, whose compliant rows are harmful requests without a jailbreak wrapper.
+For safety and security, 0.5 is a reasonable default: the false-positive rate at 0.5 is 0% to 26% on these six sets.
 
 For tax and legal advice, compliant replies score high, so 0.5 flags many of them: 51% of
 compliant tax replies and 21% of compliant legal replies on regulated-advice-bench. A higher
@@ -207,14 +202,14 @@ If you have even a few hundred labelled examples, set it on them.
 ## Speed
 
 One A100-40GB, bf16. Latency is the p50 per request at batch 1; throughput is at batch 32. All three
-models were timed on the same 200 texts, drawn evenly from the nine sets, with tax and legal as user
+models were timed on the same 200 texts, drawn evenly from the evaluation sets, with tax and legal as user
 message + reply. A request is one text and one question; answer embeddings are cached.
 
 | Model | Architecture | p50 per request (ms) | Throughput, batch 32 (texts/s) | Mean AUROC |
 |---|---|---|---|---|
-| Sev | Gemma-4-E4B-it + LoRA, embedding readout | 90 | 17 | 0.939 |
-| Kev-4B | Qwen3.5-4B-Base + LoRA + pointer head | 107 | 21 | 0.892 |
-| Tev1-4B | Qwen3.5-4B, LoRA SFT, letter output | 123 | 7 | 0.908 |
+| Sev | Gemma-4-E4B-it + LoRA, embedding readout | 90 | 17 | 0.936 |
+| Kev-4B | Qwen3.5-4B-Base + LoRA + pointer head | 107 | 21 | 0.883 |
+| Tev1-4B | Qwen3.5-4B, LoRA SFT, letter output | 123 | 7 | 0.896 |
 
 ## Training
 
@@ -260,25 +255,23 @@ The model reads each one as user message + reply.
 
 | Set | Guard | Source | Rows | Positives |
 |---|---|---|---|---|
-| Tax advice (human labels) | tax | Alinia, not public (Jev-bench rows) | 2,871 | 1,467 |
-| Legal advice (human labels) | legal | Alinia, not public (Jev-bench rows) | 2,232 | 918 |
-| AEGIS 2.0 | safety | [nvidia/Aegis-AI-Content-Safety-Dataset-2.0](https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-2.0) (test prompts) | 1,303 | 494 |
-| Nemotron, 12 languages | safety | [nvidia/Nemotron-Safety-Guard-Dataset-v3](https://huggingface.co/datasets/nvidia/Nemotron-Safety-Guard-Dataset-v3) (test) | 12,485 | 3,938 |
-| PolyGuard, 17 languages | safety | [ToxicityPrompts/PolyGuardPrompts](https://huggingface.co/datasets/ToxicityPrompts/PolyGuardPrompts) (test) | 13,464 | 3,434 |
-| XSTest | safety (over-refusal) | [Paul/XSTest](https://huggingface.co/datasets/Paul/XSTest) | 425 | 175 |
-| jailbreak-classification | security | [jackhhao/jailbreak-classification](https://huggingface.co/datasets/jackhhao/jailbreak-classification) | 359 | 110 |
-| PolyGuard security | security | [ToxicityPrompts/PolyGuardPrompts](https://huggingface.co/datasets/ToxicityPrompts/PolyGuardPrompts) (adversarial flag) | 5,933 | 2,295 |
-| deepset prompt injections | security | [deepset/prompt-injections](https://huggingface.co/datasets/deepset/prompt-injections) (test) | 115 | 59 |
 | regulated-advice-bench, tax | tax | [alinia/regulated-advice-bench](https://huggingface.co/datasets/alinia/regulated-advice-bench) `tax` | 3,042 | 1,593 |
 | regulated-advice-bench, legal | legal | [alinia/regulated-advice-bench](https://huggingface.co/datasets/alinia/regulated-advice-bench) `legal` | 2,988 | 1,719 |
-
-Rows, labels and the harm-type scope (hate, sexual, violence, wrongdoing) of the first nine follow the
-Jev-bench suite configs, including their fixed exclusion lists, which remove the same items for every
-system.
+| AEGIS 2.0 | safety | [nvidia/Aegis-AI-Content-Safety-Dataset-2.0](https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-2.0) (test prompts) | 1,220 | 544 |
+| Nemotron, 12 languages | safety | [nvidia/Nemotron-Safety-Guard-Dataset-v3](https://huggingface.co/datasets/nvidia/Nemotron-Safety-Guard-Dataset-v3) (test) | 14,298 | 6,268 |
+| PolyGuard, 17 languages | safety | [ToxicityPrompts/PolyGuardPrompts](https://huggingface.co/datasets/ToxicityPrompts/PolyGuardPrompts) (test) | 24,051 | 8,571 |
+| XSTest | safety (over-refusal) | [Paul/XSTest](https://huggingface.co/datasets/Paul/XSTest) | 371 | 134 |
+| PolyGuard security | security | [ToxicityPrompts/PolyGuardPrompts](https://huggingface.co/datasets/ToxicityPrompts/PolyGuardPrompts) (adversarial flag) | 28,583 | 13,769 |
+| deepset prompt injections | security | [deepset/prompt-injections](https://huggingface.co/datasets/deepset/prompt-injections) (test) | 95 | 49 |
+
+The safety sets are scoped to four harm types (hate, sexual, violence, wrongdoing). Every safety and
+security set keeps its test split's labels, minus each test prompt whose nearest training prompt has
+cosine similarity of 0.95 or more under EmbeddingGemma-2, or is an exact copy. Every model is scored on
+the same rows.
 
 ## Limitations
 
-- **Legal advice is the weak spot** (0.861 AUROC against Jev's 0.894). It rests on synthetic
+- **Legal advice is the weak spot** (0.817 AUROC against Jev's 0.903). It rests on synthetic
   training data, and the human labels themselves are noisy.
 - **Thresholds.** See above: tax and legal need a higher threshold than 0.5.
 - **Long inputs.** The text is cut to its first 1,024 tokens. Training texts are single messages or one

results.json

@@ -1,123 +1,47 @@
 {
  "auroc_same_input": {
   "Sev": {
-   "alinia_public_tax": 0.9808323477375968,
-   "alinia_public_legal": 0.861346551135252,
-   "aegis": 0.9533174359307988,
-   "nemotron": 0.9182700258119252,
-   "polyguard": 0.9581884370185889,
-   "xstest": 0.9854628571428571,
-   "jailbreak": 0.9790069368382621,
-   "polyguard_sec": 0.8591660169045934,
-   "deepset": 0.9579297820823245
+   "rab_tax": 0.974301388450246,
+   "rab_legal": 0.8171577937399234,
+   "aegis": 0.9427835668290986,
+   "nemotron": 0.9234368306947225,
+   "polyguard": 0.9610062942854292,
+   "xstest": 0.9820202783550601,
+   "polyguard_sec": 0.9078515809218515,
+   "deepset": 0.9778172138420587
   },
   "Jev (TypeSafe, jev-1.13.0)": {
-   "alinia_public_tax": 0.979,
-   "alinia_public_legal": 0.894,
-   "aegis": 0.933,
-   "nemotron": 0.896,
-   "polyguard": 0.952,
-   "xstest": 0.961,
-   "jailbreak": 0.998,
-   "polyguard_sec": 0.877,
-   "deepset": 0.961
+   "rab_tax": 0.960554652276588,
+   "rab_legal": 0.9027120519700322,
+   "aegis": 0.9226037678384963,
+   "nemotron": 0.904750045895219,
+   "polyguard": 0.9419810266999138,
+   "xstest": 0.977029409912463,
+   "polyguard_sec": 0.9097988686458153,
+   "deepset": 0.9554125998225377
   },
   "Tev1-4B": {
-   "alinia_public_tax": 0.9804371384125985,
-   "alinia_public_legal": 0.8358104276718297,
-   "aegis": 0.9395464986513065,
-   "nemotron": 0.8921018117310652,
-   "polyguard": 0.9176947462795073,
-   "xstest": 0.9442628571428571,
-   "jailbreak": 0.9776925885359621,
-   "polyguard_sec": 0.7038263500379078,
-   "deepset": 0.9776029055690073
+   "rab_tax": 0.9727526007719245,
+   "rab_legal": 0.7977687377573507,
+   "aegis": 0.9276929059345632,
+   "nemotron": 0.8978858695177069,
+   "polyguard": 0.9160940029128933,
+   "xstest": 0.9351501983752124,
+   "polyguard_sec": 0.7384812236283134,
+   "deepset": 0.9791481810115351
   },
   "Kev-4B": {
-   "alinia_public_tax": 0.9664853753129145,
-   "alinia_public_legal": 0.755493876901344,
-   "aegis": 0.9438790829884449,
-   "nemotron": 0.896841311772749,
-   "polyguard": 0.9223887597545163,
-   "xstest": 0.9417142857142857,
-   "jailbreak": 0.986126323475721,
-   "polyguard_sec": 0.6481888705638019,
-   "deepset": 0.9694309927360775
+   "rab_tax": 0.9698135866153552,
+   "rab_legal": 0.7443278685217962,
+   "aegis": 0.9304788113470239,
+   "nemotron": 0.9002001607723431,
+   "polyguard": 0.9105472580907253,
+   "xstest": 0.9324264752188425,
+   "polyguard_sec": 0.706694782313543,
+   "deepset": 0.9702750665483585
   }
  },
  "sev": {
-  "alinia_public_tax": {
-   "auroc": 0.9808323477375968,
-   "best_f1": 0.9639734172787688,
-   "f1@0.5": 0.7833827893175074,
-   "fpr@0.5": 0.5612535612535613,
-   "n": 2871,
-   "positives": 1467
-  },
-  "alinia_public_legal": {
-   "auroc": 0.861346551135252,
-   "best_f1": 0.7675977653631285,
-   "f1@0.5": 0.7345526445872467,
-   "fpr@0.5": 0.2754946727549467,
-   "n": 2232,
-   "positives": 918
-  },
-  "aegis": {
-   "auroc": 0.9533174359307988,
-   "best_f1": 0.8568507157464212,
-   "f1@0.5": 0.8256549232158988,
-   "fpr@0.5": 0.19283065512978986,
-   "n": 1303,
-   "positives": 494
-  },
-  "nemotron": {
-   "auroc": 0.9182700258119252,
-   "best_f1": 0.7760338814150473,
-   "f1@0.5": 0.7547501372872049,
-   "fpr@0.5": 0.20252720252720252,
-   "n": 12485,
-   "positives": 3938
-  },
-  "polyguard": {
-   "auroc": 0.9581884370185889,
-   "best_f1": 0.8476485497043087,
-   "f1@0.5": 0.84625087596356,
-   "fpr@0.5": 0.06799601196410768,
-   "n": 13464,
-   "positives": 3434
-  },
-  "xstest": {
-   "auroc": 0.9854628571428571,
-   "best_f1": 0.9239130434782609,
-   "f1@0.5": 0.9028871391076115,
-   "fpr@0.5": 0.136,
-   "n": 425,
-   "positives": 175
-  },
-  "jailbreak": {
-   "auroc": 0.9790069368382621,
-   "best_f1": 0.9237668161434978,
-   "f1@0.5": 0.8669950738916257,
-   "fpr@0.5": 0.020080321285140562,
-   "n": 359,
-   "positives": 110
-  },
-  "polyguard_sec": {
-   "auroc": 0.8591660169045934,
-   "best_f1": 0.7775787965616044,
-   "f1@0.5": 0.7737556561085973,
-   "fpr@0.5": 0.33754810335349095,
-   "n": 5933,
-   "positives": 2295
-  },
-  "deepset": {
-   "auroc": 0.9579297820823245,
-   "best_f1": 0.9122807017543859,
-   "f1@0.5": 0.8431372549019608,
-   "fpr@0.5": 0.0,
-   "n": 115,
-   "positives": 59
-  },
   "rab_tax": {
    "auroc": 0.974301388450246,
    "best_f1": 0.9590377113133939,
@@ -133,6 +57,54 @@
    "fpr@0.5": 0.2127659574468085,
    "n": 2988,
    "positives": 1719
+  },
+  "aegis": {
+   "auroc": 0.9427835668290986,
+   "best_f1": 0.861111111111111,
+   "f1@0.5": 0.8407534246575342,
+   "fpr@0.5": 0.19674556213017752,
+   "n": 1220,
+   "positives": 544
+  },
+  "nemotron": {
+   "auroc": 0.9234368306947225,
+   "best_f1": 0.8286759554759865,
+   "f1@0.5": 0.8235468564650059,
+   "fpr@0.5": 0.20747198007471981,
+   "n": 14298,
+   "positives": 6268
+  },
+  "polyguard": {
+   "auroc": 0.9610062942854292,
+   "best_f1": 0.8741775836972343,
+   "f1@0.5": 0.8728405218004466,
+   "fpr@0.5": 0.06589147286821706,
+   "n": 24051,
+   "positives": 8571
+  },
+  "xstest": {
+   "auroc": 0.9820202783550601,
+   "best_f1": 0.9084507042253521,
+   "f1@0.5": 0.8791946308724832,
+   "fpr@0.5": 0.13924050632911392,
+   "n": 371,
+   "positives": 134
+  },
+  "polyguard_sec": {
+   "auroc": 0.9078515809218515,
+   "best_f1": 0.8446201789364339,
+   "f1@0.5": 0.8437890856428149,
+   "fpr@0.5": 0.2561765897124342,
+   "n": 28583,
+   "positives": 13769
+  },
+  "deepset": {
+   "auroc": 0.9778172138420587,
+   "best_f1": 0.9347826086956522,
+   "f1@0.5": 0.8604651162790697,
+   "fpr@0.5": 0.0,
+   "n": 95,
+   "positives": 49
   }
  },
  "thresholds": {
@@ -178,48 +150,6 @@
      "recall": 0.9409918392969241,
      "f1": 0.9264524103831892
     }
-   },
-   "alinia_public_tax": {
-    "0.5": {
-     "fpr": 0.5612535612535613,
-     "recall": 0.9897750511247444,
-     "f1": 0.7833827893175074
-    },
-    "0.8": {
-     "fpr": 0.48646723646723644,
-     "recall": 0.9822767552828903,
-     "f1": 0.8025619604566973
-    },
-    "0.9": {
-     "fpr": 0.4472934472934473,
-     "recall": 0.9802317655078391,
-     "f1": 0.8140390602887064
-    },
-    "0.95": {
-     "fpr": 0.40883190883190884,
-     "recall": 0.978186775732788,
-     "f1": 0.8256616800920599
-    },
-    "0.98": {
-     "fpr": 0.3425925925925926,
-     "recall": 0.9747784594410361,
-     "f1": 0.8466548253404381
-    },
-    "0.99": {
-     "fpr": 0.297008547008547,
-     "recall": 0.9713701431492843,
-     "f1": 0.8612873980054397
-    },
-    "0.995": {
-     "fpr": 0.24715099715099714,
-     "recall": 0.9693251533742331,
-     "f1": 0.8788627935723115
-    },
-    "0.999": {
-     "fpr": 0.12962962962962962,
-     "recall": 0.9591002044989775,
-     "f1": 0.9208115183246073
-    }
    }
   },
   "legal": {
@@ -264,65 +194,9 @@
      "recall": 0.5648632926119838,
      "f1": 0.7018431514275388
     }
-   },
-   "alinia_public_legal": {
-    "0.5": {
-     "fpr": 0.2754946727549467,
-     "recall": 0.809368191721133,
-     "f1": 0.7345526445872467
-    },
-    "0.8": {
-     "fpr": 0.2359208523592085,
-     "recall": 0.7973856209150327,
-     "f1": 0.746938775510204
-    },
-    "0.9": {
-     "fpr": 0.21232876712328766,
-     "recall": 0.7919389978213508,
-     "f1": 0.7557172557172557
-    },
-    "0.95": {
-     "fpr": 0.1917808219178082,
-     "recall": 0.7788671023965141,
-     "f1": 0.7586206896551724
-    },
-    "0.98": {
-     "fpr": 0.1750380517503805,
-     "recall": 0.7647058823529411,
-     "f1": 0.7589189189189189
-    },
-    "0.99": {
-     "fpr": 0.15449010654490106,
-     "recall": 0.7538126361655774,
-     "f1": 0.7633756205184776
-    },
-    "0.995": {
-     "fpr": 0.136986301369863,
-     "recall": 0.7407407407407407,
-     "f1": 0.7649043869516311
-    },
-    "0.999": {
-     "fpr": 0.09817351598173515,
-     "recall": 0.7004357298474946,
-     "f1": 0.7609467455621302
-    }
    }
   }
  },
- "benchmark_overlap": {
-  "tax": {
-   "rab_items": 338,
-   "items_sharing_a_reply": 280,
-   "shared_replies": 1299,
-   "jevbench_rows": 2871
-  },
-  "legal": {
-   "rab_items": 332,
-   "items_sharing_a_reply": 223,
-   "shared_replies": 1151,
-   "jevbench_rows": 2232
-  }
- },
  "latency_a100": {
   "Sev": {
    "p50_ms": 90.37737251492217,