Update model card
Browse files- QUICKSTART.md +2 -1
- README.md +40 -27
- SHA256SUMS +18 -17
- docs/BENCHMARKS.md +2 -2
- docs/assets/00-benchmark-card.png +2 -2
- docs/assets/00-benchmark-card.svg +183 -183
- docs/assets/05-gain-over-base.png +2 -2
- docs/assets/05-gain-over-base.svg +74 -74
- docs/assets/06-vs-jev.png +2 -2
- docs/assets/06-vs-jev.svg +94 -94
- docs/public-classification-suites.md +2 -0
- evidence/calibration-leaderboard-ablate-nosys.json +533 -209
- evidence/calibration-leaderboard-merged-native.json +184 -184
- evidence/calibration-per-type-served.json +537 -0
- evidence/prompt-wording-choice.json +81 -0
- evidence/served-nosys-easy-report.json +70 -70
- evidence/served-nosys-hard-report.json +228 -228
- evidence/served-nosys-original-report.json +131 -131
- release-manifest.json +16 -2
QUICKSTART.md
CHANGED
|
@@ -45,7 +45,8 @@ uv pip install --python .venv-native/bin/python -e './StandardOne-8B/server[nati
|
|
| 45 |
--host 0.0.0.0 --port 30120 --max-concurrency 1 \
|
| 46 |
--tokenizer-model mistralai/Ministral-3-8B-Instruct-2512-BF16 \
|
| 47 |
--tokenizer-revision f6fae9795746f63c9be8344932f01275f3c63734 \
|
| 48 |
-
--prompt-wording native --native-system-prompt none --default-temperature
|
|
|
|
| 49 |
```
|
| 50 |
|
| 51 |
6. Check health and models.
|
|
|
|
| 45 |
--host 0.0.0.0 --port 30120 --max-concurrency 1 \
|
| 46 |
--tokenizer-model mistralai/Ministral-3-8B-Instruct-2512-BF16 \
|
| 47 |
--tokenizer-revision f6fae9795746f63c9be8344932f01275f3c63734 \
|
| 48 |
+
--prompt-wording native --native-system-prompt none --default-temperature 0.85 \
|
| 49 |
+
--temperature-by-type choice=0.85,noul=0.85,score=0.70
|
| 50 |
```
|
| 51 |
|
| 52 |
6. Check health and models.
|
README.md
CHANGED
|
@@ -29,7 +29,6 @@ tags:
|
|
| 29 |
|
| 30 |
> **Updated weights (v2, 2026-09-26).** If you downloaded this model before, download it again or pin
|
| 31 |
> `revision="v2"`. Earlier versions stay available under the tags `v1` and `v1.1`.
|
| 32 |
-
> The benchmark figures and the serving temperature on this card are still those of v1.1 and are being updated for v2.
|
| 33 |
|
| 34 |
**Version:** v2
|
| 35 |
|
|
@@ -51,15 +50,14 @@ for the measurement conditions and limitations.
|
|
| 51 |
|
| 52 |
|
| 53 |
*The figure combines results from different measurement paths. See [Benchmarks](#benchmarks) for
|
| 54 |
-
served versus offline conditions; measured 24–
|
| 55 |
|
| 56 |
## At a glance
|
| 57 |
|
| 58 |
- Send a state and a bounded rubric to receive probabilities for the supplied labels:
|
| 59 |
`choice` selects among labeled options, `noul` is yes/no, and `score` uses an ordinal scale.
|
| 60 |
The endpoint scores the labels in one forward pass without decoding answer text.
|
| 61 |
-
- In the same-run **offline** comparison with its untuned base, 8B improves on four of six suites
|
| 62 |
-
ties on public easy, and declines by 8.11 percentage points on public hard. These are not
|
| 63 |
served-endpoint results.
|
| 64 |
- Probabilities are temperature-scaled and calibration-checked (hard-tier ECE, distribution
|
| 65 |
total-variation) — see Benchmarks below.
|
|
@@ -104,9 +102,24 @@ Adapter (`jev-adapter`, ships as `server/` in this repository):
|
|
| 104 |
--host 0.0.0.0 --port 30120 --max-concurrency 1 \
|
| 105 |
--tokenizer-model mistralai/Ministral-3-8B-Instruct-2512-BF16 \
|
| 106 |
--tokenizer-revision f6fae9795746f63c9be8344932f01275f3c63734 \
|
| 107 |
-
--prompt-wording native --native-system-prompt none --default-temperature
|
|
|
|
| 108 |
```
|
| 109 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 110 |
Try it:
|
| 111 |
|
| 112 |
```bash
|
|
@@ -132,7 +145,8 @@ Response shape (example values, default temperature applied):
|
|
| 132 |
"usage": {"input_tokens": 96, "output_tokens": 0},
|
| 133 |
"metadata": {
|
| 134 |
"confidence_method": "1 - normalized_entropy",
|
| 135 |
-
"temperature":
|
|
|
|
| 136 |
"evaluations": 1,
|
| 137 |
"adapter_elapsed_ms": 26.1
|
| 138 |
}
|
|
@@ -144,19 +158,19 @@ More (client command, 3B variant, request format): see [QUICKSTART.md](QUICKSTAR
|
|
| 144 |
## Benchmarks
|
| 145 |
|
| 146 |
**Served endpoint results (the release configuration).** Merged BF16 weights through SGLang 0.5.20
|
| 147 |
-
and `jev-adapter`, native wording, no system prompt, one option order,
|
| 148 |
measured on the same items through its hosted endpoint; its probabilities are raw, with no
|
| 149 |
temperature applied. These are our measurements, not official sealed-set JevBench scores.
|
| 150 |
|
| 151 |
| Suite | **Standard One 8B** | Jev 1.13 |
|
| 152 |
|---|---|---|
|
| 153 |
| JevBench public easy (48) | **100.00 %** | 100.00 % |
|
| 154 |
-
| JevBench public standard (72) | **
|
| 155 |
| JevBench public hard (111) | **54.95 %** | 72.07 % |
|
| 156 |
-
| judge proxy (600: routing + answer adequacy) | **
|
| 157 |
-
| realistic transfer set (600) | **90.
|
| 158 |
-
| stated-distribution probability (1,036) | **81.
|
| 159 |
-
| hard proxy (600) | **
|
| 160 |
|
| 161 |
**Offline comparison with the untuned base.** This separate `transformers` runner used native
|
| 162 |
wording, no system prompt, one option order, T=1. The base and tuned checkpoint were scored by the
|
|
@@ -164,17 +178,16 @@ same offline path; these numbers are indicative of the base-model change, not th
|
|
| 164 |
|
| 165 |
| Suite | Untuned base | **Standard One 8B** |
|
| 166 |
|---|---|---|
|
| 167 |
-
| JevBench public easy (48) | 100.00 % | **
|
| 168 |
| JevBench public standard (72) | 79.17 % | **97.22 %** |
|
| 169 |
-
| JevBench public hard (111) | 60.36 % | **
|
| 170 |
-
| judge proxy (600: routing + answer adequacy) | 79.33 % | **88.
|
| 171 |
| realistic transfer set (600) | 72.83 % | **90.00 %** |
|
| 172 |
-
| stated-distribution probability (1,036) | 34.85 % | **
|
| 173 |
|
| 174 |
-
Against the untuned base, four suites improve, public easy
|
| 175 |
-
|
| 176 |
-
|
| 177 |
-
(T=1.65) against Jev 1.13's 0.099 raw, and mean TV to the stated distributions is 0.145 at served T
|
| 178 |
against Jev's 0.192 raw; full calibration table: [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md#calibration).
|
| 179 |
|
| 180 |
**Speed** — raw serial latency on one H200 with SGLang 0.5.20, using a 242-decision profile averaging
|
|
@@ -191,7 +204,7 @@ for speed only; no accuracy comparison is implied.
|
|
| 191 |
|
| 192 |
Throughput on one H200 (hard+standard mix, 1,322 tokens/request): 29.3k tok/s at concurrency 1, rising to 39.5k tok/s at concurrency 64 (≈40 decisions/s at concurrency 8 on the 280-token profile above).
|
| 193 |
|
| 194 |
-
On the public classification and decision suites (400 cases/suite, seed 13, served endpoints): AG News 84.2 %, typed decisions
|
| 195 |
|
| 196 |
A JevBench v1.4.1 run has been requested; the sealed-set result is not yet available. Full report: [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).
|
| 197 |
|
|
@@ -200,7 +213,7 @@ A JevBench v1.4.1 run has been requested; the sealed-set result is not yet avail
|
|
| 200 |
- **Base model:** `mistralai/Ministral-3-8B-Instruct-2512-BF16`, revision `f6fae9795746f63c9be8344932f01275f3c63734` (Apache-2.0).
|
| 201 |
- **Adapter:** LoRA r=16, α=32, dropout 0, on `q_proj k_proj v_proj o_proj gate_proj up_proj down_proj` of the language-model projections only (vision tower and multimodal projector excluded), **44,564,480** trainable parameters, PEFT 0.21.0. Adapter file `adapter_model.safetensors`, 214,559,872 bytes, sha256 `53e41238cd55567cfbc75efdf61d354da39771573f56f74bb1440c9ffdd1bd4a`.
|
| 202 |
- **Merged BF16 checkpoint:** merging the adapter into the base changed **238 tensors** (293 unchanged), none outside the language-model projections, max absolute weight change **0.00201**.
|
| 203 |
-
- **Serving details:** native chat-template wording, no system prompt, fixed
|
| 204 |
|
| 205 |
| Path | Contents |
|
| 206 |
|---|---|
|
|
@@ -223,18 +236,18 @@ An exact-text overlap audit against the public JevBench tiers found 0 exact scen
|
|
| 223 |
## Limitations
|
| 224 |
|
| 225 |
- **Public hard tier:** the served 8B score is **54.95 %**, versus **72.07 %** for Jev 1.13. In the
|
| 226 |
-
separate offline base comparison, Standard One 8B scores **
|
| 227 |
**60.36 %**.
|
| 228 |
-
- Served probabilities are temperature-scaled by
|
| 229 |
materially different question distribution, re-fitting that temperature is advisable rather than
|
| 230 |
-
assuming
|
| 231 |
- At most 26 options per question (one uppercase letter per option, `A`–`Z`).
|
| 232 |
- The sealed JevBench set has not been measured for this model.
|
| 233 |
-
- Served and offline probabilities can differ on identical prompts (mean total-variation ≈0.
|
| 234 |
hard tier); served numbers are treated as authoritative.
|
| 235 |
- Korean is a small share of multilingual training relative to the other seven languages.
|
| 236 |
- The card reports text benchmarks; it does not establish decision accuracy on image inputs.
|
| 237 |
-
- Ten-way support triage (
|
| 238 |
for those.
|
| 239 |
|
| 240 |
## Licence
|
|
|
|
| 29 |
|
| 30 |
> **Updated weights (v2, 2026-09-26).** If you downloaded this model before, download it again or pin
|
| 31 |
> `revision="v2"`. Earlier versions stay available under the tags `v1` and `v1.1`.
|
|
|
|
| 32 |
|
| 33 |
**Version:** v2
|
| 34 |
|
|
|
|
| 50 |
|
| 51 |
|
| 52 |
*The figure combines results from different measurement paths. See [Benchmarks](#benchmarks) for
|
| 53 |
+
served versus offline conditions; measured 24–26 September 2026.*
|
| 54 |
|
| 55 |
## At a glance
|
| 56 |
|
| 57 |
- Send a state and a bounded rubric to receive probabilities for the supplied labels:
|
| 58 |
`choice` selects among labeled options, `noul` is yes/no, and `score` uses an ordinal scale.
|
| 59 |
The endpoint scores the labels in one forward pass without decoding answer text.
|
| 60 |
+
- In the same-run **offline** comparison with its untuned base, 8B improves on four of six suites and declines on public easy (2.08 percentage points) and public hard (4.50 percentage points). These are not
|
|
|
|
| 61 |
served-endpoint results.
|
| 62 |
- Probabilities are temperature-scaled and calibration-checked (hard-tier ECE, distribution
|
| 63 |
total-variation) — see Benchmarks below.
|
|
|
|
| 102 |
--host 0.0.0.0 --port 30120 --max-concurrency 1 \
|
| 103 |
--tokenizer-model mistralai/Ministral-3-8B-Instruct-2512-BF16 \
|
| 104 |
--tokenizer-revision f6fae9795746f63c9be8344932f01275f3c63734 \
|
| 105 |
+
--prompt-wording native --native-system-prompt none --default-temperature 0.85 \
|
| 106 |
+
--temperature-by-type choice=0.85,noul=0.85,score=0.70
|
| 107 |
```
|
| 108 |
|
| 109 |
+
### Prompt wording
|
| 110 |
+
|
| 111 |
+
`jev-adapter` can phrase a request in two ways. Both were measured on the same served endpoint (no system prompt) and each has its
|
| 112 |
+
own fitted temperatures; the recommended default is `native` unless `served` scores at least 1.0 percentage point higher on
|
| 113 |
+
the suites below and an offline check agrees.
|
| 114 |
+
|
| 115 |
+
| `--prompt-wording` | What the prompt looks like | Temperatures (default; choice / noul / score) | Mean accuracy, 10 suites |
|
| 116 |
+
|---|---|---|---:|
|
| 117 |
+
| `native` (recommended default) | `State:` / `Question:` / `Options:` headers, options as `A. name: description` | 0.85; 0.85 / 0.85 / 0.70 | 76.6 % |
|
| 118 |
+
| `served` | `Context:` / `Question:` / `Options:` headers, options as `A: name: description` | 0.80; 0.80 / 0.90 / 0.95 | 76.4 % |
|
| 119 |
+
|
| 120 |
+
The 10 suites: judge proxy, hard proxy, stated-distribution probability, realistic transfer set, MuSiQue (multiple choice), SQuAD 2.0 unanswerable questions, ContractNLI, PAWS-X (English), a held-out hard decision set and a consistency set. None of them is a JevBench tier, and no JevBench item was used to choose the wording or the temperatures.
|
| 121 |
+
To use `served`, pass `--prompt-wording served --default-temperature 0.80 --temperature-by-type choice=0.80,noul=0.90,score=0.95`.
|
| 122 |
+
|
| 123 |
Try it:
|
| 124 |
|
| 125 |
```bash
|
|
|
|
| 145 |
"usage": {"input_tokens": 96, "output_tokens": 0},
|
| 146 |
"metadata": {
|
| 147 |
"confidence_method": "1 - normalized_entropy",
|
| 148 |
+
"temperature": 0.85,
|
| 149 |
+
"temperature_by_type": {"choice": 0.85, "noul": 0.85, "score": 0.70},
|
| 150 |
"evaluations": 1,
|
| 151 |
"adapter_elapsed_ms": 26.1
|
| 152 |
}
|
|
|
|
| 158 |
## Benchmarks
|
| 159 |
|
| 160 |
**Served endpoint results (the release configuration).** Merged BF16 weights through SGLang 0.5.20
|
| 161 |
+
and `jev-adapter`, native wording, no system prompt, one option order, per-answer-type temperatures (choice 0.85, noul 0.85, score 0.70). The wording and the temperatures were chosen on non-JevBench data. Jev 1.13 was
|
| 162 |
measured on the same items through its hosted endpoint; its probabilities are raw, with no
|
| 163 |
temperature applied. These are our measurements, not official sealed-set JevBench scores.
|
| 164 |
|
| 165 |
| Suite | **Standard One 8B** | Jev 1.13 |
|
| 166 |
|---|---|---|
|
| 167 |
| JevBench public easy (48) | **100.00 %** | 100.00 % |
|
| 168 |
+
| JevBench public standard (72) | **93.06 %** | 98.61 % |
|
| 169 |
| JevBench public hard (111) | **54.95 %** | 72.07 % |
|
| 170 |
+
| judge proxy (600: routing + answer adequacy) | **89.33 %** | 90.50 % |
|
| 171 |
+
| realistic transfer set (600) | **90.83 %** | 86.67 % |
|
| 172 |
+
| stated-distribution probability (1,036) | **81.18 %** | 72.97 % |
|
| 173 |
+
| hard proxy (600) | **53.00 %** | 54.83 % |
|
| 174 |
|
| 175 |
**Offline comparison with the untuned base.** This separate `transformers` runner used native
|
| 176 |
wording, no system prompt, one option order, T=1. The base and tuned checkpoint were scored by the
|
|
|
|
| 178 |
|
| 179 |
| Suite | Untuned base | **Standard One 8B** |
|
| 180 |
|---|---|---|
|
| 181 |
+
| JevBench public easy (48) | 100.00 % | **97.92 %** |
|
| 182 |
| JevBench public standard (72) | 79.17 % | **97.22 %** |
|
| 183 |
+
| JevBench public hard (111) | 60.36 % | **55.86 %** |
|
| 184 |
+
| judge proxy (600: routing + answer adequacy) | 79.33 % | **88.33 %** |
|
| 185 |
| realistic transfer set (600) | 72.83 % | **90.00 %** |
|
| 186 |
+
| stated-distribution probability (1,036) | 34.85 % | **82.63 %** |
|
| 187 |
|
| 188 |
+
Against the untuned base, four suites improve, and public easy falls by 2.08 percentage points and public hard falls by 4.50 percentage points on this offline run. Served and offline probabilities differ even on identical
|
| 189 |
+
prompts, so use the served table for expected endpoint behavior. Hard-tier ECE at the served temperatures is
|
| 190 |
+
0.184 against Jev 1.13's 0.099 raw, and mean TV to the stated distributions is 0.108 at served T
|
|
|
|
| 191 |
against Jev's 0.192 raw; full calibration table: [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md#calibration).
|
| 192 |
|
| 193 |
**Speed** — raw serial latency on one H200 with SGLang 0.5.20, using a 242-decision profile averaging
|
|
|
|
| 204 |
|
| 205 |
Throughput on one H200 (hard+standard mix, 1,322 tokens/request): 29.3k tok/s at concurrency 1, rising to 39.5k tok/s at concurrency 64 (≈40 decisions/s at concurrency 8 on the 280-token profile above).
|
| 206 |
|
| 207 |
+
On the public classification and decision suites (400 cases/suite, seed 13, served endpoints): AG News 84.2 %, typed decisions 71.1 %, MASSIVE intent mean 86.1 %, email spam 91.8 %, phishing 85.2 %. Full table, per-language and per-workflow breakdown: [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md#public-classification-and-decision-suites) and [`docs/public-classification-suites.md`](docs/public-classification-suites.md).
|
| 208 |
|
| 209 |
A JevBench v1.4.1 run has been requested; the sealed-set result is not yet available. Full report: [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).
|
| 210 |
|
|
|
|
| 213 |
- **Base model:** `mistralai/Ministral-3-8B-Instruct-2512-BF16`, revision `f6fae9795746f63c9be8344932f01275f3c63734` (Apache-2.0).
|
| 214 |
- **Adapter:** LoRA r=16, α=32, dropout 0, on `q_proj k_proj v_proj o_proj gate_proj up_proj down_proj` of the language-model projections only (vision tower and multimodal projector excluded), **44,564,480** trainable parameters, PEFT 0.21.0. Adapter file `adapter_model.safetensors`, 214,559,872 bytes, sha256 `53e41238cd55567cfbc75efdf61d354da39771573f56f74bb1440c9ffdd1bd4a`.
|
| 215 |
- **Merged BF16 checkpoint:** merging the adapter into the base changed **238 tensors** (293 unchanged), none outside the language-model projections, max absolute weight change **0.00201**.
|
| 216 |
+
- **Serving details:** native chat-template wording, no system prompt, fixed per-answer-type temperatures (choice 0.85, noul 0.85, score 0.70; fitted on held-out and public-train calibration data, no JevBench item); served model name `standard-one-8b` behind stock SGLang 0.5.20 via `jev-adapter` (`POST /v1/systemone`); single caller-supplied option order, no rotation ensemble; 8,192-token context.
|
| 217 |
|
| 218 |
| Path | Contents |
|
| 219 |
|---|---|
|
|
|
|
| 236 |
## Limitations
|
| 237 |
|
| 238 |
- **Public hard tier:** the served 8B score is **54.95 %**, versus **72.07 %** for Jev 1.13. In the
|
| 239 |
+
separate offline base comparison, Standard One 8B scores **55.86 %**, below the untuned base's
|
| 240 |
**60.36 %**.
|
| 241 |
+
- Served probabilities are temperature-scaled by one value per answer type; if you apply this model to a
|
| 242 |
materially different question distribution, re-fitting that temperature is advisable rather than
|
| 243 |
+
assuming these values transfer.
|
| 244 |
- At most 26 options per question (one uppercase letter per option, `A`–`Z`).
|
| 245 |
- The sealed JevBench set has not been measured for this model.
|
| 246 |
+
- Served and offline probabilities can differ on identical prompts (mean total-variation ≈0.06 on the
|
| 247 |
hard tier); served numbers are treated as authoritative.
|
| 248 |
- Korean is a small share of multilingual training relative to the other seven languages.
|
| 249 |
- The card reports text benchmarks; it does not establish decision accuracy on image inputs.
|
| 250 |
+
- Ten-way support triage (36 %) and RAG passage relevance (59 %) are weak zero-shot; fine-tune
|
| 251 |
for those.
|
| 252 |
|
| 253 |
## Licence
|
SHA256SUMS
CHANGED
|
@@ -1,29 +1,31 @@
|
|
| 1 |
1495e1e757ef4d0925a2350563cf5754bb23c51701a8ec4fb3c5cdcbedae6747 LICENSE
|
| 2 |
0fae9e8380c795cd6d6fa9f1733c408df7b27bf009d947ebb15adc96ad40dfab MERGE_REPORT.json
|
| 3 |
20486cf6b2e42440b0aea913ea475c818664f34e76e4501651d6cb21987a508b NOTICE
|
| 4 |
-
|
| 5 |
-
|
| 6 |
66e0fda8c8ab269ccd010dca6196931d475a22c3f2eebe5b6b3a2ec33a354d0e SYSTEM_PROMPT.txt
|
| 7 |
74eeb55fd3341286ec3fd44e902b7120721acc81cd394e96b431f85e93a1ea56 chat_template.jinja
|
| 8 |
111827aa7ae6d936f48afb2f0776f51b00add45c857c708af2b893499f76d882 config.json
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
7313f2b0645ee401d082572986dcee0c08d3daed1ae79fa650cfa685570ad7e2 docs/assets/02-latency-vs-qwen.png
|
| 13 |
0b48a0dd4b4c9ac7c1842836940348edc2e1ba697184943f0c84c3e3d3c7f2d0 docs/assets/02-latency-vs-qwen.svg
|
| 14 |
5f430f8841e393548f080833524840adb93deef1c6f75dca93d0571262a2ac68 docs/assets/04-throughput.png
|
| 15 |
dd888be78c0e32894cf04a8e3fbf8757e7218873668fa4443ba47c7c1c48b6dc docs/assets/04-throughput.svg
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
|
|
|
|
|
|
| 25 |
d47f43be5bd7767b80ed27a96a9a85ece1495b37fb9365e575b1df6b32fbc509 evidence/served-nosys-latency-idle-gpu.json
|
| 26 |
-
|
| 27 |
e0923390059f84a9180b00e5501778acc45ea9856cd7f2fd68208b360927c677 generation_config.json
|
| 28 |
5b29a2e3e86366c0b2e21c5038a16fa1f2bed8a2ca69e5e775608126570c89d4 model-00001-of-00004.safetensors
|
| 29 |
65c9000a4aafa0cb0978540cfdbb6017e39f14a208a8d0523d375656f5cf0343 model-00002-of-00004.safetensors
|
|
@@ -32,8 +34,7 @@ fe537b22d9b005873c25b0f8ca0548e24f6d3a4fb611db11becc6ba86c849a03 model-00003-of
|
|
| 32 |
71e022361d37d84c2b6224acbddca2c74bedcc7addc523cfc6a57420d4b22ceb model.safetensors.index.json
|
| 33 |
81b8377f36c5b3d60900b333214d85160a0500b84576e969092c6fd214f69538 params.json
|
| 34 |
ece2373c2ae391bce18785a1810543bc6173a1f28b6767cffab2c35dbea5f002 processor_config.json
|
| 35 |
-
|
| 36 |
-
eef2a88a000d2ec294379755770c147327cb79d3d7278c711899bd36797d7a02 server/.gitignore
|
| 37 |
1495e1e757ef4d0925a2350563cf5754bb23c51701a8ec4fb3c5cdcbedae6747 server/LICENSE
|
| 38 |
0f1f023b916d38d7c12af3a71e09f5e82b819a7e156e4992118bb160ee63ede1 server/NOTICE
|
| 39 |
e58911cf75448ce8951377d46eeb531158c46db0586faedd22cccdfd69193013 server/README.md
|
|
|
|
| 1 |
1495e1e757ef4d0925a2350563cf5754bb23c51701a8ec4fb3c5cdcbedae6747 LICENSE
|
| 2 |
0fae9e8380c795cd6d6fa9f1733c408df7b27bf009d947ebb15adc96ad40dfab MERGE_REPORT.json
|
| 3 |
20486cf6b2e42440b0aea913ea475c818664f34e76e4501651d6cb21987a508b NOTICE
|
| 4 |
+
0ffad36ca79c67234dd1bfbb9e24d1d236beaf8d2297c3c2052b9a1d4df30f20 QUICKSTART.md
|
| 5 |
+
e38cdd0ef9869ed4730f96623af226bdbedc657351962b23c450ba5daa81c7c9 README.md
|
| 6 |
66e0fda8c8ab269ccd010dca6196931d475a22c3f2eebe5b6b3a2ec33a354d0e SYSTEM_PROMPT.txt
|
| 7 |
74eeb55fd3341286ec3fd44e902b7120721acc81cd394e96b431f85e93a1ea56 chat_template.jinja
|
| 8 |
111827aa7ae6d936f48afb2f0776f51b00add45c857c708af2b893499f76d882 config.json
|
| 9 |
+
04cfde9163128e5fbd0f224750c906b062a38a5f2a4fca1a22658a6251190cf0 docs/BENCHMARKS.md
|
| 10 |
+
e7e9f42a23bf9bf1cbe71a6fb6195be5b2f8f57a9acd968ccb3b12f3227c8569 docs/assets/00-benchmark-card.png
|
| 11 |
+
606768546085a23defaef5a18820afa2feead163a843dd19ba37d1b2809b9741 docs/assets/00-benchmark-card.svg
|
| 12 |
7313f2b0645ee401d082572986dcee0c08d3daed1ae79fa650cfa685570ad7e2 docs/assets/02-latency-vs-qwen.png
|
| 13 |
0b48a0dd4b4c9ac7c1842836940348edc2e1ba697184943f0c84c3e3d3c7f2d0 docs/assets/02-latency-vs-qwen.svg
|
| 14 |
5f430f8841e393548f080833524840adb93deef1c6f75dca93d0571262a2ac68 docs/assets/04-throughput.png
|
| 15 |
dd888be78c0e32894cf04a8e3fbf8757e7218873668fa4443ba47c7c1c48b6dc docs/assets/04-throughput.svg
|
| 16 |
+
95a53a9907990ac5fcbfe1b4aaa8f09f268636802a1375f4d8d0034c82dcf998 docs/assets/05-gain-over-base.png
|
| 17 |
+
22396bd8908b697d912bd1ec54a510eff2b2b1884cd00b5e1bedd81db450d2fa docs/assets/05-gain-over-base.svg
|
| 18 |
+
c43fb809f0b6bff34530fe2970fbd59817ae56e93813da7f45af32a8b9413987 docs/assets/06-vs-jev.png
|
| 19 |
+
23fe4cfc32b756068dd3967339bae947391a18f1c4c0de5bad5c0dceeaff3463 docs/assets/06-vs-jev.svg
|
| 20 |
+
df15177d9f8022598ac7af696600afcee14dd531875c519c2e2db74f0fc2498c docs/public-classification-suites.md
|
| 21 |
+
7866009cccf468a71c56163d0ec27eee5185e5d8dc4c1a495f91445229eb4c95 evidence/calibration-leaderboard-ablate-nosys.json
|
| 22 |
+
e3b7eb0fff1504035ab7ddbe01af976db48236872e6b77fc7bcdc766a60fd000 evidence/calibration-leaderboard-merged-native.json
|
| 23 |
+
a717a16ab338fb5a3aefe0822a5421732cb2f68d965691fdb8138ded0b4d70b3 evidence/calibration-per-type-served.json
|
| 24 |
+
88e86d3ae6f8272ea479e9c55bb83c96c4512aebd43e9c4cf5ff137b730ad9c6 evidence/prompt-wording-choice.json
|
| 25 |
+
a33036e10498a440481258cbbc8611b531498d6239115a9278c3e6cafe10f6ad evidence/served-nosys-easy-report.json
|
| 26 |
+
8ea7f7db1900e0d3e3963c44fb55c996b0bd91744f98645fe57a87eb71adb5e8 evidence/served-nosys-hard-report.json
|
| 27 |
d47f43be5bd7767b80ed27a96a9a85ece1495b37fb9365e575b1df6b32fbc509 evidence/served-nosys-latency-idle-gpu.json
|
| 28 |
+
262f5c7f85fcc5b8a9ccfaa7485d8b9f11a70f3d5a2ff60868ff9ccd1e887b43 evidence/served-nosys-original-report.json
|
| 29 |
e0923390059f84a9180b00e5501778acc45ea9856cd7f2fd68208b360927c677 generation_config.json
|
| 30 |
5b29a2e3e86366c0b2e21c5038a16fa1f2bed8a2ca69e5e775608126570c89d4 model-00001-of-00004.safetensors
|
| 31 |
65c9000a4aafa0cb0978540cfdbb6017e39f14a208a8d0523d375656f5cf0343 model-00002-of-00004.safetensors
|
|
|
|
| 34 |
71e022361d37d84c2b6224acbddca2c74bedcc7addc523cfc6a57420d4b22ceb model.safetensors.index.json
|
| 35 |
81b8377f36c5b3d60900b333214d85160a0500b84576e969092c6fd214f69538 params.json
|
| 36 |
ece2373c2ae391bce18785a1810543bc6173a1f28b6767cffab2c35dbea5f002 processor_config.json
|
| 37 |
+
12068f5b55e15718a228e32c5a29d482749216a557f415faa2345c858ad79094 release-manifest.json
|
|
|
|
| 38 |
1495e1e757ef4d0925a2350563cf5754bb23c51701a8ec4fb3c5cdcbedae6747 server/LICENSE
|
| 39 |
0f1f023b916d38d7c12af3a71e09f5e82b819a7e156e4992118bb160ee63ede1 server/NOTICE
|
| 40 |
e58911cf75448ce8951377d46eeb531158c46db0586faedd22cccdfd69193013 server/README.md
|
docs/BENCHMARKS.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
# Standard One — full benchmark report
|
| 2 |
|
| 3 |
-
> **Note.** The
|
| 4 |
-
> overlap-audit sections
|
| 5 |
|
| 6 |
This report covers both released systems, **Standard One 8B** (`StandardThinking/StandardOne-8B` / `-LoRA`) and
|
| 7 |
**Standard One 3B** (`StandardThinking/StandardOne-3B` / `-LoRA`). It is our own measurement, not an official JevBench
|
|
|
|
| 1 |
# Standard One — full benchmark report
|
| 2 |
|
| 3 |
+
> **Note.** The v2 figures for both sizes are on their model cards (Standard One 8B and Standard One 3B). The benchmark figures in
|
| 4 |
+
> this report are still those of v1.1; the training-data provenance and overlap-audit sections already describe the v2 weights.
|
| 5 |
|
| 6 |
This report covers both released systems, **Standard One 8B** (`StandardThinking/StandardOne-8B` / `-LoRA`) and
|
| 7 |
**Standard One 3B** (`StandardThinking/StandardOne-3B` / `-LoRA`). It is our own measurement, not an official JevBench
|
docs/assets/00-benchmark-card.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
docs/assets/00-benchmark-card.svg
CHANGED
|
|
docs/assets/05-gain-over-base.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
docs/assets/05-gain-over-base.svg
CHANGED
|
|
docs/assets/06-vs-jev.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
docs/assets/06-vs-jev.svg
CHANGED
|
|
docs/public-classification-suites.md
CHANGED
|
@@ -1,5 +1,7 @@
|
|
| 1 |
# Public classification and decision suites
|
| 2 |
|
|
|
|
|
|
|
| 3 |
Measured 25 September 2026 through the served endpoints (merged BF16, SGLang 0.5.20, jev-adapter, native wording, no system prompt, single option order). Each suite: 400 cases sampled with seed 13 from the public test split, one request per case; ECE (10-bin, top label) uses the served temperature (8B T=1.65, 3B T=1.55). Accuracy and macro-F1 are argmax metrics and do not depend on the temperature. banking77 (77 labels) is not runnable: the endpoint accepts at most 26 options per question.
|
| 4 |
|
| 5 |
## Public datasets
|
|
|
|
| 1 |
# Public classification and decision suites
|
| 2 |
|
| 3 |
+
> **Note.** The Standard One 8B v2 figures for these suites are on its model card; the tables below are still those of v1.1.
|
| 4 |
+
|
| 5 |
Measured 25 September 2026 through the served endpoints (merged BF16, SGLang 0.5.20, jev-adapter, native wording, no system prompt, single option order). Each suite: 400 cases sampled with seed 13 from the public test split, one request per case; ECE (10-bin, top label) uses the served temperature (8B T=1.65, 3B T=1.55). Accuracy and macro-F1 are argmax metrics and do not depend on the temperature. banking77 (77 labels) is not runnable: the endpoint accepts at most 26 options per question.
|
| 6 |
|
| 7 |
## Public datasets
|
evidence/calibration-leaderboard-ablate-nosys.json
CHANGED
|
@@ -1,237 +1,399 @@
|
|
| 1 |
{
|
| 2 |
"arm_fitted": "original",
|
| 3 |
"inputs": [
|
| 4 |
-
"runs/ablate-
|
| 5 |
],
|
| 6 |
"fit_suites": [
|
| 7 |
"distprob-calib-v2",
|
| 8 |
"internal-eval-hardproxy-calib-v1"
|
| 9 |
],
|
| 10 |
-
"temperature":
|
| 11 |
"objective": "leaderboard",
|
| 12 |
-
"axis_proxy_at_T1":
|
| 13 |
-
"axis_proxy_at_T":
|
| 14 |
"at_T1": {
|
| 15 |
"jevbench-easy": {
|
| 16 |
"original": {
|
| 17 |
"questions": 48,
|
| 18 |
-
"accuracy":
|
| 19 |
-
"ece_10bin": 0.
|
| 20 |
-
"calibration_axis_ece_part":
|
| 21 |
},
|
| 22 |
"rotation": {
|
| 23 |
"questions": 48,
|
| 24 |
-
"accuracy":
|
| 25 |
-
"ece_10bin": 0.
|
| 26 |
-
"calibration_axis_ece_part":
|
| 27 |
}
|
| 28 |
},
|
| 29 |
"jevbench-original": {
|
| 30 |
"original": {
|
| 31 |
"questions": 72,
|
| 32 |
"accuracy": 0.9722222222222222,
|
| 33 |
-
"ece_10bin": 0.
|
| 34 |
-
"calibration_axis_ece_part":
|
| 35 |
},
|
| 36 |
"rotation": {
|
| 37 |
"questions": 72,
|
| 38 |
"accuracy": 0.9722222222222222,
|
| 39 |
-
"ece_10bin": 0.
|
| 40 |
-
"calibration_axis_ece_part":
|
| 41 |
}
|
| 42 |
},
|
| 43 |
"jevbench-hard": {
|
| 44 |
"original": {
|
| 45 |
"questions": 111,
|
| 46 |
-
"accuracy": 0.
|
| 47 |
-
"ece_10bin": 0.
|
| 48 |
-
"calibration_axis_ece_part":
|
| 49 |
},
|
| 50 |
"rotation": {
|
| 51 |
"questions": 111,
|
| 52 |
-
"accuracy": 0.
|
| 53 |
-
"ece_10bin": 0.
|
| 54 |
-
"calibration_axis_ece_part":
|
| 55 |
}
|
| 56 |
},
|
| 57 |
"judge-proxy-v1": {
|
| 58 |
"original": {
|
| 59 |
"questions": 600,
|
| 60 |
-
"accuracy": 0.
|
| 61 |
-
"ece_10bin": 0.
|
| 62 |
-
"calibration_axis_ece_part":
|
| 63 |
},
|
| 64 |
"rotation": {
|
| 65 |
"questions": 600,
|
| 66 |
-
"accuracy": 0.
|
| 67 |
-
"ece_10bin": 0.
|
| 68 |
-
"calibration_axis_ece_part":
|
| 69 |
}
|
| 70 |
},
|
| 71 |
"internal-eval-hardproxy-v1": {
|
| 72 |
"original": {
|
| 73 |
"questions": 600,
|
| 74 |
-
"accuracy": 0.
|
| 75 |
-
"ece_10bin": 0.
|
| 76 |
-
"calibration_axis_ece_part":
|
| 77 |
},
|
| 78 |
"rotation": {
|
| 79 |
"questions": 600,
|
| 80 |
-
"accuracy": 0.
|
| 81 |
-
"ece_10bin": 0.
|
| 82 |
-
"calibration_axis_ece_part":
|
| 83 |
}
|
| 84 |
},
|
| 85 |
"internal-eval-hardproxy-v2": {
|
| 86 |
"original": {
|
| 87 |
"questions": 900,
|
| 88 |
-
"accuracy": 0.
|
| 89 |
-
"ece_10bin": 0.
|
| 90 |
-
"calibration_axis_ece_part":
|
| 91 |
"probability_items": 80,
|
| 92 |
-
"mean_tv_distance": 0.
|
| 93 |
-
"fidelity_part":
|
| 94 |
-
"calibration_axis_estimate":
|
| 95 |
},
|
| 96 |
"rotation": {
|
| 97 |
"questions": 900,
|
| 98 |
-
"accuracy": 0.
|
| 99 |
-
"ece_10bin": 0.
|
| 100 |
-
"calibration_axis_ece_part":
|
| 101 |
"probability_items": 80,
|
| 102 |
-
"mean_tv_distance": 0.
|
| 103 |
-
"fidelity_part":
|
| 104 |
-
"calibration_axis_estimate":
|
| 105 |
}
|
| 106 |
},
|
| 107 |
"internal-eval-hardproxy-v3": {
|
| 108 |
"original": {
|
| 109 |
"questions": 960,
|
| 110 |
-
"accuracy": 0.
|
| 111 |
-
"ece_10bin": 0.
|
| 112 |
-
"calibration_axis_ece_part":
|
| 113 |
"probability_items": 120,
|
| 114 |
-
"mean_tv_distance": 0.
|
| 115 |
-
"fidelity_part":
|
| 116 |
-
"calibration_axis_estimate":
|
| 117 |
},
|
| 118 |
"rotation": {
|
| 119 |
"questions": 960,
|
| 120 |
-
"accuracy": 0.
|
| 121 |
-
"ece_10bin": 0.
|
| 122 |
-
"calibration_axis_ece_part":
|
| 123 |
"probability_items": 120,
|
| 124 |
-
"mean_tv_distance": 0.
|
| 125 |
-
"fidelity_part":
|
| 126 |
-
"calibration_axis_estimate":
|
| 127 |
}
|
| 128 |
},
|
| 129 |
"internal-eval-hardproxy-calib-v1": {
|
| 130 |
"original": {
|
| 131 |
"questions": 600,
|
| 132 |
-
"accuracy": 0.
|
| 133 |
-
"ece_10bin": 0.
|
| 134 |
-
"calibration_axis_ece_part":
|
| 135 |
},
|
| 136 |
"rotation": {
|
| 137 |
"questions": 600,
|
| 138 |
-
"accuracy": 0.
|
| 139 |
-
"ece_10bin": 0.
|
| 140 |
-
"calibration_axis_ece_part":
|
| 141 |
}
|
| 142 |
},
|
| 143 |
"distprob-calib-v2": {
|
| 144 |
"original": {
|
| 145 |
"questions": 916,
|
| 146 |
-
"accuracy": 0.
|
| 147 |
-
"ece_10bin": 0.
|
| 148 |
-
"calibration_axis_ece_part":
|
| 149 |
"probability_items": 916,
|
| 150 |
-
"mean_tv_distance": 0.
|
| 151 |
-
"fidelity_part":
|
| 152 |
-
"calibration_axis_estimate":
|
| 153 |
},
|
| 154 |
"rotation": {
|
| 155 |
"questions": 916,
|
| 156 |
-
"accuracy": 0.
|
| 157 |
-
"ece_10bin": 0.
|
| 158 |
-
"calibration_axis_ece_part":
|
| 159 |
"probability_items": 916,
|
| 160 |
-
"mean_tv_distance": 0.
|
| 161 |
-
"fidelity_part":
|
| 162 |
-
"calibration_axis_estimate":
|
| 163 |
}
|
| 164 |
},
|
| 165 |
"distprob-val-v2": {
|
| 166 |
"original": {
|
| 167 |
"questions": 1036,
|
| 168 |
-
"accuracy": 0.
|
| 169 |
-
"ece_10bin": 0.
|
| 170 |
-
"calibration_axis_ece_part":
|
| 171 |
"probability_items": 1036,
|
| 172 |
-
"mean_tv_distance": 0.
|
| 173 |
-
"fidelity_part": 89.
|
| 174 |
-
"calibration_axis_estimate":
|
| 175 |
},
|
| 176 |
"rotation": {
|
| 177 |
"questions": 1036,
|
| 178 |
-
"accuracy": 0.
|
| 179 |
-
"ece_10bin": 0.
|
| 180 |
-
"calibration_axis_ece_part":
|
| 181 |
"probability_items": 1036,
|
| 182 |
-
"mean_tv_distance": 0.
|
| 183 |
-
"fidelity_part": 89.
|
| 184 |
-
"calibration_axis_estimate":
|
| 185 |
}
|
| 186 |
},
|
| 187 |
"internal-eval-v2-realistic": {
|
| 188 |
"original": {
|
| 189 |
"questions": 600,
|
| 190 |
"accuracy": 0.9,
|
| 191 |
-
"ece_10bin": 0.
|
| 192 |
-
"calibration_axis_ece_part":
|
| 193 |
"probability_items": 120,
|
| 194 |
-
"mean_tv_distance": 0.
|
| 195 |
-
"fidelity_part":
|
| 196 |
-
"calibration_axis_estimate":
|
| 197 |
},
|
| 198 |
"rotation": {
|
| 199 |
"questions": 600,
|
| 200 |
"accuracy": 0.9,
|
| 201 |
-
"ece_10bin": 0.
|
| 202 |
-
"calibration_axis_ece_part":
|
| 203 |
"probability_items": 120,
|
| 204 |
-
"mean_tv_distance": 0.
|
| 205 |
-
"fidelity_part":
|
| 206 |
-
"calibration_axis_estimate":
|
| 207 |
}
|
| 208 |
},
|
| 209 |
"games-harness-eval-v1": {
|
| 210 |
"original": {
|
| 211 |
"questions": 360,
|
| 212 |
"accuracy": 0.9638888888888889,
|
| 213 |
-
"ece_10bin": 0.
|
| 214 |
-
"calibration_axis_ece_part":
|
| 215 |
},
|
| 216 |
"rotation": {
|
| 217 |
"questions": 360,
|
| 218 |
"accuracy": 0.9638888888888889,
|
| 219 |
-
"ece_10bin": 0.
|
| 220 |
-
"calibration_axis_ece_part":
|
| 221 |
}
|
| 222 |
},
|
| 223 |
"tetris-drought-eval-v1": {
|
| 224 |
"original": {
|
| 225 |
"questions": 120,
|
| 226 |
-
"accuracy": 0.
|
| 227 |
-
"ece_10bin": 0.
|
| 228 |
-
"calibration_axis_ece_part":
|
| 229 |
},
|
| 230 |
"rotation": {
|
| 231 |
"questions": 120,
|
| 232 |
-
"accuracy": 0.
|
| 233 |
-
"ece_10bin": 0.
|
| 234 |
-
"calibration_axis_ece_part":
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 235 |
}
|
| 236 |
}
|
| 237 |
},
|
|
@@ -239,224 +401,386 @@
|
|
| 239 |
"jevbench-easy": {
|
| 240 |
"original": {
|
| 241 |
"questions": 48,
|
| 242 |
-
"accuracy":
|
| 243 |
-
"ece_10bin": 0.
|
| 244 |
-
"calibration_axis_ece_part":
|
| 245 |
},
|
| 246 |
"rotation": {
|
| 247 |
"questions": 48,
|
| 248 |
-
"accuracy":
|
| 249 |
-
"ece_10bin": 0.
|
| 250 |
-
"calibration_axis_ece_part":
|
| 251 |
}
|
| 252 |
},
|
| 253 |
"jevbench-original": {
|
| 254 |
"original": {
|
| 255 |
"questions": 72,
|
| 256 |
"accuracy": 0.9722222222222222,
|
| 257 |
-
"ece_10bin": 0.
|
| 258 |
-
"calibration_axis_ece_part":
|
| 259 |
},
|
| 260 |
"rotation": {
|
| 261 |
"questions": 72,
|
| 262 |
"accuracy": 0.9722222222222222,
|
| 263 |
-
"ece_10bin": 0.
|
| 264 |
-
"calibration_axis_ece_part":
|
| 265 |
}
|
| 266 |
},
|
| 267 |
"jevbench-hard": {
|
| 268 |
"original": {
|
| 269 |
"questions": 111,
|
| 270 |
-
"accuracy": 0.
|
| 271 |
-
"ece_10bin": 0.
|
| 272 |
-
"calibration_axis_ece_part":
|
| 273 |
},
|
| 274 |
"rotation": {
|
| 275 |
"questions": 111,
|
| 276 |
-
"accuracy": 0.
|
| 277 |
-
"ece_10bin": 0.
|
| 278 |
-
"calibration_axis_ece_part":
|
| 279 |
}
|
| 280 |
},
|
| 281 |
"judge-proxy-v1": {
|
| 282 |
"original": {
|
| 283 |
"questions": 600,
|
| 284 |
-
"accuracy": 0.
|
| 285 |
-
"ece_10bin": 0.
|
| 286 |
-
"calibration_axis_ece_part":
|
| 287 |
},
|
| 288 |
"rotation": {
|
| 289 |
"questions": 600,
|
| 290 |
-
"accuracy": 0.
|
| 291 |
-
"ece_10bin": 0.
|
| 292 |
-
"calibration_axis_ece_part":
|
| 293 |
}
|
| 294 |
},
|
| 295 |
"internal-eval-hardproxy-v1": {
|
| 296 |
"original": {
|
| 297 |
"questions": 600,
|
| 298 |
-
"accuracy": 0.
|
| 299 |
-
"ece_10bin": 0.
|
| 300 |
-
"calibration_axis_ece_part":
|
| 301 |
},
|
| 302 |
"rotation": {
|
| 303 |
"questions": 600,
|
| 304 |
-
"accuracy": 0.
|
| 305 |
-
"ece_10bin": 0.
|
| 306 |
-
"calibration_axis_ece_part":
|
| 307 |
}
|
| 308 |
},
|
| 309 |
"internal-eval-hardproxy-v2": {
|
| 310 |
"original": {
|
| 311 |
"questions": 900,
|
| 312 |
-
"accuracy": 0.
|
| 313 |
-
"ece_10bin": 0.
|
| 314 |
-
"calibration_axis_ece_part":
|
| 315 |
"probability_items": 80,
|
| 316 |
-
"mean_tv_distance": 0.
|
| 317 |
-
"fidelity_part":
|
| 318 |
-
"calibration_axis_estimate":
|
| 319 |
},
|
| 320 |
"rotation": {
|
| 321 |
"questions": 900,
|
| 322 |
-
"accuracy": 0.
|
| 323 |
-
"ece_10bin": 0.
|
| 324 |
-
"calibration_axis_ece_part":
|
| 325 |
"probability_items": 80,
|
| 326 |
-
"mean_tv_distance": 0.
|
| 327 |
-
"fidelity_part":
|
| 328 |
-
"calibration_axis_estimate":
|
| 329 |
}
|
| 330 |
},
|
| 331 |
"internal-eval-hardproxy-v3": {
|
| 332 |
"original": {
|
| 333 |
"questions": 960,
|
| 334 |
-
"accuracy": 0.
|
| 335 |
-
"ece_10bin": 0.
|
| 336 |
-
"calibration_axis_ece_part":
|
| 337 |
"probability_items": 120,
|
| 338 |
-
"mean_tv_distance": 0.
|
| 339 |
-
"fidelity_part":
|
| 340 |
-
"calibration_axis_estimate":
|
| 341 |
},
|
| 342 |
"rotation": {
|
| 343 |
"questions": 960,
|
| 344 |
-
"accuracy": 0.
|
| 345 |
-
"ece_10bin": 0.
|
| 346 |
-
"calibration_axis_ece_part":
|
| 347 |
"probability_items": 120,
|
| 348 |
-
"mean_tv_distance": 0.
|
| 349 |
-
"fidelity_part":
|
| 350 |
-
"calibration_axis_estimate":
|
| 351 |
}
|
| 352 |
},
|
| 353 |
"internal-eval-hardproxy-calib-v1": {
|
| 354 |
"original": {
|
| 355 |
"questions": 600,
|
| 356 |
-
"accuracy": 0.
|
| 357 |
-
"ece_10bin": 0.
|
| 358 |
-
"calibration_axis_ece_part":
|
| 359 |
},
|
| 360 |
"rotation": {
|
| 361 |
"questions": 600,
|
| 362 |
-
"accuracy": 0.
|
| 363 |
-
"ece_10bin": 0.
|
| 364 |
-
"calibration_axis_ece_part":
|
| 365 |
}
|
| 366 |
},
|
| 367 |
"distprob-calib-v2": {
|
| 368 |
"original": {
|
| 369 |
"questions": 916,
|
| 370 |
-
"accuracy": 0.
|
| 371 |
-
"ece_10bin": 0.
|
| 372 |
-
"calibration_axis_ece_part":
|
| 373 |
"probability_items": 916,
|
| 374 |
-
"mean_tv_distance": 0.
|
| 375 |
-
"fidelity_part":
|
| 376 |
-
"calibration_axis_estimate":
|
| 377 |
},
|
| 378 |
"rotation": {
|
| 379 |
"questions": 916,
|
| 380 |
-
"accuracy": 0.
|
| 381 |
-
"ece_10bin": 0.
|
| 382 |
-
"calibration_axis_ece_part":
|
| 383 |
"probability_items": 916,
|
| 384 |
-
"mean_tv_distance": 0.
|
| 385 |
-
"fidelity_part":
|
| 386 |
-
"calibration_axis_estimate":
|
| 387 |
}
|
| 388 |
},
|
| 389 |
"distprob-val-v2": {
|
| 390 |
"original": {
|
| 391 |
"questions": 1036,
|
| 392 |
-
"accuracy": 0.
|
| 393 |
-
"ece_10bin": 0.
|
| 394 |
-
"calibration_axis_ece_part":
|
| 395 |
"probability_items": 1036,
|
| 396 |
-
"mean_tv_distance": 0.
|
| 397 |
-
"fidelity_part":
|
| 398 |
-
"calibration_axis_estimate":
|
| 399 |
},
|
| 400 |
"rotation": {
|
| 401 |
"questions": 1036,
|
| 402 |
-
"accuracy": 0.
|
| 403 |
-
"ece_10bin": 0.
|
| 404 |
-
"calibration_axis_ece_part":
|
| 405 |
"probability_items": 1036,
|
| 406 |
-
"mean_tv_distance": 0.
|
| 407 |
-
"fidelity_part":
|
| 408 |
-
"calibration_axis_estimate":
|
| 409 |
}
|
| 410 |
},
|
| 411 |
"internal-eval-v2-realistic": {
|
| 412 |
"original": {
|
| 413 |
"questions": 600,
|
| 414 |
"accuracy": 0.9,
|
| 415 |
-
"ece_10bin": 0.
|
| 416 |
-
"calibration_axis_ece_part":
|
| 417 |
"probability_items": 120,
|
| 418 |
-
"mean_tv_distance": 0.
|
| 419 |
-
"fidelity_part":
|
| 420 |
-
"calibration_axis_estimate":
|
| 421 |
},
|
| 422 |
"rotation": {
|
| 423 |
"questions": 600,
|
| 424 |
"accuracy": 0.9,
|
| 425 |
-
"ece_10bin": 0.
|
| 426 |
-
"calibration_axis_ece_part":
|
| 427 |
"probability_items": 120,
|
| 428 |
-
"mean_tv_distance": 0.
|
| 429 |
-
"fidelity_part":
|
| 430 |
-
"calibration_axis_estimate":
|
| 431 |
}
|
| 432 |
},
|
| 433 |
"games-harness-eval-v1": {
|
| 434 |
"original": {
|
| 435 |
"questions": 360,
|
| 436 |
"accuracy": 0.9638888888888889,
|
| 437 |
-
"ece_10bin": 0.
|
| 438 |
-
"calibration_axis_ece_part": 95.
|
| 439 |
},
|
| 440 |
"rotation": {
|
| 441 |
"questions": 360,
|
| 442 |
"accuracy": 0.9638888888888889,
|
| 443 |
-
"ece_10bin": 0.
|
| 444 |
-
"calibration_axis_ece_part": 95.
|
| 445 |
}
|
| 446 |
},
|
| 447 |
"tetris-drought-eval-v1": {
|
| 448 |
"original": {
|
| 449 |
"questions": 120,
|
| 450 |
-
"accuracy": 0.
|
| 451 |
-
"ece_10bin": 0.
|
| 452 |
-
"calibration_axis_ece_part":
|
| 453 |
},
|
| 454 |
"rotation": {
|
| 455 |
"questions": 120,
|
| 456 |
-
"accuracy": 0.
|
| 457 |
-
"ece_10bin": 0.
|
| 458 |
-
"calibration_axis_ece_part":
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 459 |
}
|
| 460 |
}
|
| 461 |
}
|
| 462 |
-
}
|
|
|
|
| 1 |
{
|
| 2 |
"arm_fitted": "original",
|
| 3 |
"inputs": [
|
| 4 |
+
"runs/ablate-r5-8b-nosys/predictions.jsonl"
|
| 5 |
],
|
| 6 |
"fit_suites": [
|
| 7 |
"distprob-calib-v2",
|
| 8 |
"internal-eval-hardproxy-calib-v1"
|
| 9 |
],
|
| 10 |
+
"temperature": 0.8500000000000001,
|
| 11 |
"objective": "leaderboard",
|
| 12 |
+
"axis_proxy_at_T1": 88.07514148307334,
|
| 13 |
+
"axis_proxy_at_T": 89.39493420296604,
|
| 14 |
"at_T1": {
|
| 15 |
"jevbench-easy": {
|
| 16 |
"original": {
|
| 17 |
"questions": 48,
|
| 18 |
+
"accuracy": 0.9791666666666666,
|
| 19 |
+
"ece_10bin": 0.05492118312761862,
|
| 20 |
+
"calibration_axis_ece_part": 89.01576337447628
|
| 21 |
},
|
| 22 |
"rotation": {
|
| 23 |
"questions": 48,
|
| 24 |
+
"accuracy": 0.9791666666666666,
|
| 25 |
+
"ece_10bin": 0.05492118312761862,
|
| 26 |
+
"calibration_axis_ece_part": 89.01576337447628
|
| 27 |
}
|
| 28 |
},
|
| 29 |
"jevbench-original": {
|
| 30 |
"original": {
|
| 31 |
"questions": 72,
|
| 32 |
"accuracy": 0.9722222222222222,
|
| 33 |
+
"ece_10bin": 0.09139810704127882,
|
| 34 |
+
"calibration_axis_ece_part": 81.72037859174424
|
| 35 |
},
|
| 36 |
"rotation": {
|
| 37 |
"questions": 72,
|
| 38 |
"accuracy": 0.9722222222222222,
|
| 39 |
+
"ece_10bin": 0.09139810704127882,
|
| 40 |
+
"calibration_axis_ece_part": 81.72037859174424
|
| 41 |
}
|
| 42 |
},
|
| 43 |
"jevbench-hard": {
|
| 44 |
"original": {
|
| 45 |
"questions": 111,
|
| 46 |
+
"accuracy": 0.5585585585585585,
|
| 47 |
+
"ece_10bin": 0.14352073043752922,
|
| 48 |
+
"calibration_axis_ece_part": 71.29585391249415
|
| 49 |
},
|
| 50 |
"rotation": {
|
| 51 |
"questions": 111,
|
| 52 |
+
"accuracy": 0.5585585585585585,
|
| 53 |
+
"ece_10bin": 0.14352073043752922,
|
| 54 |
+
"calibration_axis_ece_part": 71.29585391249415
|
| 55 |
}
|
| 56 |
},
|
| 57 |
"judge-proxy-v1": {
|
| 58 |
"original": {
|
| 59 |
"questions": 600,
|
| 60 |
+
"accuracy": 0.8833333333333333,
|
| 61 |
+
"ece_10bin": 0.037457916102559934,
|
| 62 |
+
"calibration_axis_ece_part": 92.50841677948802
|
| 63 |
},
|
| 64 |
"rotation": {
|
| 65 |
"questions": 600,
|
| 66 |
+
"accuracy": 0.8833333333333333,
|
| 67 |
+
"ece_10bin": 0.037457916102559934,
|
| 68 |
+
"calibration_axis_ece_part": 92.50841677948802
|
| 69 |
}
|
| 70 |
},
|
| 71 |
"internal-eval-hardproxy-v1": {
|
| 72 |
"original": {
|
| 73 |
"questions": 600,
|
| 74 |
+
"accuracy": 0.5266666666666666,
|
| 75 |
+
"ece_10bin": 0.041925456824368594,
|
| 76 |
+
"calibration_axis_ece_part": 91.61490863512628
|
| 77 |
},
|
| 78 |
"rotation": {
|
| 79 |
"questions": 600,
|
| 80 |
+
"accuracy": 0.5266666666666666,
|
| 81 |
+
"ece_10bin": 0.041925456824368594,
|
| 82 |
+
"calibration_axis_ece_part": 91.61490863512628
|
| 83 |
}
|
| 84 |
},
|
| 85 |
"internal-eval-hardproxy-v2": {
|
| 86 |
"original": {
|
| 87 |
"questions": 900,
|
| 88 |
+
"accuracy": 0.8433333333333334,
|
| 89 |
+
"ece_10bin": 0.08948587834522262,
|
| 90 |
+
"calibration_axis_ece_part": 82.10282433095549,
|
| 91 |
"probability_items": 80,
|
| 92 |
+
"mean_tv_distance": 0.1229657172506469,
|
| 93 |
+
"fidelity_part": 87.70342827493532,
|
| 94 |
+
"calibration_axis_estimate": 84.9031263029454
|
| 95 |
},
|
| 96 |
"rotation": {
|
| 97 |
"questions": 900,
|
| 98 |
+
"accuracy": 0.8433333333333334,
|
| 99 |
+
"ece_10bin": 0.08948587834522262,
|
| 100 |
+
"calibration_axis_ece_part": 82.10282433095549,
|
| 101 |
"probability_items": 80,
|
| 102 |
+
"mean_tv_distance": 0.1229657172506469,
|
| 103 |
+
"fidelity_part": 87.70342827493532,
|
| 104 |
+
"calibration_axis_estimate": 84.9031263029454
|
| 105 |
}
|
| 106 |
},
|
| 107 |
"internal-eval-hardproxy-v3": {
|
| 108 |
"original": {
|
| 109 |
"questions": 960,
|
| 110 |
+
"accuracy": 0.7552083333333334,
|
| 111 |
+
"ece_10bin": 0.08735852250592542,
|
| 112 |
+
"calibration_axis_ece_part": 82.52829549881491,
|
| 113 |
"probability_items": 120,
|
| 114 |
+
"mean_tv_distance": 0.2901798287985715,
|
| 115 |
+
"fidelity_part": 70.98201712014284,
|
| 116 |
+
"calibration_axis_estimate": 76.75515630947888
|
| 117 |
},
|
| 118 |
"rotation": {
|
| 119 |
"questions": 960,
|
| 120 |
+
"accuracy": 0.7552083333333334,
|
| 121 |
+
"ece_10bin": 0.08735852250592542,
|
| 122 |
+
"calibration_axis_ece_part": 82.52829549881491,
|
| 123 |
"probability_items": 120,
|
| 124 |
+
"mean_tv_distance": 0.2901798287985715,
|
| 125 |
+
"fidelity_part": 70.98201712014284,
|
| 126 |
+
"calibration_axis_estimate": 76.75515630947888
|
| 127 |
}
|
| 128 |
},
|
| 129 |
"internal-eval-hardproxy-calib-v1": {
|
| 130 |
"original": {
|
| 131 |
"questions": 600,
|
| 132 |
+
"accuracy": 0.5583333333333333,
|
| 133 |
+
"ece_10bin": 0.043326916120365315,
|
| 134 |
+
"calibration_axis_ece_part": 91.33461677592693
|
| 135 |
},
|
| 136 |
"rotation": {
|
| 137 |
"questions": 600,
|
| 138 |
+
"accuracy": 0.5583333333333333,
|
| 139 |
+
"ece_10bin": 0.043326916120365315,
|
| 140 |
+
"calibration_axis_ece_part": 91.33461677592693
|
| 141 |
}
|
| 142 |
},
|
| 143 |
"distprob-calib-v2": {
|
| 144 |
"original": {
|
| 145 |
"questions": 916,
|
| 146 |
+
"accuracy": 0.8067685589519651,
|
| 147 |
+
"ece_10bin": 0.21569683672585738,
|
| 148 |
+
"calibration_axis_ece_part": 56.86063265482852,
|
| 149 |
"probability_items": 916,
|
| 150 |
+
"mean_tv_distance": 0.15184333809780262,
|
| 151 |
+
"fidelity_part": 84.81566619021974,
|
| 152 |
+
"calibration_axis_estimate": 70.83814942252414
|
| 153 |
},
|
| 154 |
"rotation": {
|
| 155 |
"questions": 916,
|
| 156 |
+
"accuracy": 0.8067685589519651,
|
| 157 |
+
"ece_10bin": 0.21569683672585738,
|
| 158 |
+
"calibration_axis_ece_part": 56.86063265482852,
|
| 159 |
"probability_items": 916,
|
| 160 |
+
"mean_tv_distance": 0.15184333809780262,
|
| 161 |
+
"fidelity_part": 84.81566619021974,
|
| 162 |
+
"calibration_axis_estimate": 70.83814942252414
|
| 163 |
}
|
| 164 |
},
|
| 165 |
"distprob-val-v2": {
|
| 166 |
"original": {
|
| 167 |
"questions": 1036,
|
| 168 |
+
"accuracy": 0.8262548262548263,
|
| 169 |
+
"ece_10bin": 0.24138813803021536,
|
| 170 |
+
"calibration_axis_ece_part": 51.72237239395693,
|
| 171 |
"probability_items": 1036,
|
| 172 |
+
"mean_tv_distance": 0.10756285377961519,
|
| 173 |
+
"fidelity_part": 89.24371462203848,
|
| 174 |
+
"calibration_axis_estimate": 70.4830435079977
|
| 175 |
},
|
| 176 |
"rotation": {
|
| 177 |
"questions": 1036,
|
| 178 |
+
"accuracy": 0.8262548262548263,
|
| 179 |
+
"ece_10bin": 0.24138813803021536,
|
| 180 |
+
"calibration_axis_ece_part": 51.72237239395693,
|
| 181 |
"probability_items": 1036,
|
| 182 |
+
"mean_tv_distance": 0.10756285377961519,
|
| 183 |
+
"fidelity_part": 89.24371462203848,
|
| 184 |
+
"calibration_axis_estimate": 70.4830435079977
|
| 185 |
}
|
| 186 |
},
|
| 187 |
"internal-eval-v2-realistic": {
|
| 188 |
"original": {
|
| 189 |
"questions": 600,
|
| 190 |
"accuracy": 0.9,
|
| 191 |
+
"ece_10bin": 0.08695022105009712,
|
| 192 |
+
"calibration_axis_ece_part": 82.60995578998059,
|
| 193 |
"probability_items": 120,
|
| 194 |
+
"mean_tv_distance": 0.11081345146552232,
|
| 195 |
+
"fidelity_part": 88.91865485344776,
|
| 196 |
+
"calibration_axis_estimate": 85.76430532171418
|
| 197 |
},
|
| 198 |
"rotation": {
|
| 199 |
"questions": 600,
|
| 200 |
"accuracy": 0.9,
|
| 201 |
+
"ece_10bin": 0.08695022105009712,
|
| 202 |
+
"calibration_axis_ece_part": 82.60995578998059,
|
| 203 |
"probability_items": 120,
|
| 204 |
+
"mean_tv_distance": 0.11081345146552232,
|
| 205 |
+
"fidelity_part": 88.91865485344776,
|
| 206 |
+
"calibration_axis_estimate": 85.76430532171418
|
| 207 |
}
|
| 208 |
},
|
| 209 |
"games-harness-eval-v1": {
|
| 210 |
"original": {
|
| 211 |
"questions": 360,
|
| 212 |
"accuracy": 0.9638888888888889,
|
| 213 |
+
"ece_10bin": 0.044962278747440826,
|
| 214 |
+
"calibration_axis_ece_part": 91.00754425051183
|
| 215 |
},
|
| 216 |
"rotation": {
|
| 217 |
"questions": 360,
|
| 218 |
"accuracy": 0.9638888888888889,
|
| 219 |
+
"ece_10bin": 0.044962278747440826,
|
| 220 |
+
"calibration_axis_ece_part": 91.00754425051183
|
| 221 |
}
|
| 222 |
},
|
| 223 |
"tetris-drought-eval-v1": {
|
| 224 |
"original": {
|
| 225 |
"questions": 120,
|
| 226 |
+
"accuracy": 0.6833333333333333,
|
| 227 |
+
"ece_10bin": 0.09665632869540049,
|
| 228 |
+
"calibration_axis_ece_part": 80.6687342609199
|
| 229 |
},
|
| 230 |
"rotation": {
|
| 231 |
"questions": 120,
|
| 232 |
+
"accuracy": 0.6833333333333333,
|
| 233 |
+
"ece_10bin": 0.09665632869540049,
|
| 234 |
+
"calibration_axis_ece_part": 80.6687342609199
|
| 235 |
+
}
|
| 236 |
+
},
|
| 237 |
+
"musique-mc": {
|
| 238 |
+
"original": {
|
| 239 |
+
"questions": 500,
|
| 240 |
+
"accuracy": 0.792,
|
| 241 |
+
"ece_10bin": 0.1905555418030256,
|
| 242 |
+
"calibration_axis_ece_part": 61.888891639394885
|
| 243 |
+
},
|
| 244 |
+
"rotation": {
|
| 245 |
+
"questions": 500,
|
| 246 |
+
"accuracy": 0.792,
|
| 247 |
+
"ece_10bin": 0.1905555418030256,
|
| 248 |
+
"calibration_axis_ece_part": 61.888891639394885
|
| 249 |
+
}
|
| 250 |
+
},
|
| 251 |
+
"squad2-abstain": {
|
| 252 |
+
"original": {
|
| 253 |
+
"questions": 500,
|
| 254 |
+
"accuracy": 0.838,
|
| 255 |
+
"ece_10bin": 0.09906661484285945,
|
| 256 |
+
"calibration_axis_ece_part": 80.1866770314281
|
| 257 |
+
},
|
| 258 |
+
"rotation": {
|
| 259 |
+
"questions": 500,
|
| 260 |
+
"accuracy": 0.838,
|
| 261 |
+
"ece_10bin": 0.09906661484285945,
|
| 262 |
+
"calibration_axis_ece_part": 80.1866770314281
|
| 263 |
+
}
|
| 264 |
+
},
|
| 265 |
+
"contractnli": {
|
| 266 |
+
"original": {
|
| 267 |
+
"questions": 500,
|
| 268 |
+
"accuracy": 0.738,
|
| 269 |
+
"ece_10bin": 0.052794876942178645,
|
| 270 |
+
"calibration_axis_ece_part": 89.44102461156427
|
| 271 |
+
},
|
| 272 |
+
"rotation": {
|
| 273 |
+
"questions": 500,
|
| 274 |
+
"accuracy": 0.738,
|
| 275 |
+
"ece_10bin": 0.052794876942178645,
|
| 276 |
+
"calibration_axis_ece_part": 89.44102461156427
|
| 277 |
+
}
|
| 278 |
+
},
|
| 279 |
+
"paws-x-en": {
|
| 280 |
+
"original": {
|
| 281 |
+
"questions": 500,
|
| 282 |
+
"accuracy": 0.774,
|
| 283 |
+
"ece_10bin": 0.05806977200000005,
|
| 284 |
+
"calibration_axis_ece_part": 88.38604559999999
|
| 285 |
+
},
|
| 286 |
+
"rotation": {
|
| 287 |
+
"questions": 500,
|
| 288 |
+
"accuracy": 0.774,
|
| 289 |
+
"ece_10bin": 0.05806977200000005,
|
| 290 |
+
"calibration_axis_ece_part": 88.38604559999999
|
| 291 |
+
}
|
| 292 |
+
},
|
| 293 |
+
"calib-public-v1": {
|
| 294 |
+
"original": {
|
| 295 |
+
"questions": 1200,
|
| 296 |
+
"accuracy": 0.7883333333333333,
|
| 297 |
+
"ece_10bin": 0.08406420036767286,
|
| 298 |
+
"calibration_axis_ece_part": 83.18715992646543
|
| 299 |
+
},
|
| 300 |
+
"rotation": {
|
| 301 |
+
"questions": 1200,
|
| 302 |
+
"accuracy": 0.7883333333333333,
|
| 303 |
+
"ece_10bin": 0.08406420036767286,
|
| 304 |
+
"calibration_axis_ece_part": 83.18715992646543
|
| 305 |
+
}
|
| 306 |
+
},
|
| 307 |
+
"shortcut-diagnostic-eval-v1": {
|
| 308 |
+
"original": {
|
| 309 |
+
"questions": 601,
|
| 310 |
+
"accuracy": 0.8386023294509152,
|
| 311 |
+
"ece_10bin": 0.0688669227644725,
|
| 312 |
+
"calibration_axis_ece_part": 86.2266154471055
|
| 313 |
+
},
|
| 314 |
+
"rotation": {
|
| 315 |
+
"questions": 601,
|
| 316 |
+
"accuracy": 0.8386023294509152,
|
| 317 |
+
"ece_10bin": 0.0688669227644725,
|
| 318 |
+
"calibration_axis_ece_part": 86.2266154471055
|
| 319 |
+
}
|
| 320 |
+
},
|
| 321 |
+
"consistency-eval-v1": {
|
| 322 |
+
"original": {
|
| 323 |
+
"questions": 1200,
|
| 324 |
+
"accuracy": 0.7766666666666666,
|
| 325 |
+
"ece_10bin": 0.0725575099165813,
|
| 326 |
+
"calibration_axis_ece_part": 85.48849801668375
|
| 327 |
+
},
|
| 328 |
+
"rotation": {
|
| 329 |
+
"questions": 1200,
|
| 330 |
+
"accuracy": 0.7766666666666666,
|
| 331 |
+
"ece_10bin": 0.0725575099165813,
|
| 332 |
+
"calibration_axis_ece_part": 85.48849801668375
|
| 333 |
+
}
|
| 334 |
+
},
|
| 335 |
+
"hard-eval-v4": {
|
| 336 |
+
"original": {
|
| 337 |
+
"questions": 800,
|
| 338 |
+
"accuracy": 0.61,
|
| 339 |
+
"ece_10bin": 0.06506552576475072,
|
| 340 |
+
"calibration_axis_ece_part": 86.98689484704985
|
| 341 |
+
},
|
| 342 |
+
"rotation": {
|
| 343 |
+
"questions": 800,
|
| 344 |
+
"accuracy": 0.61,
|
| 345 |
+
"ece_10bin": 0.06506552576475072,
|
| 346 |
+
"calibration_axis_ece_part": 86.98689484704985
|
| 347 |
+
}
|
| 348 |
+
},
|
| 349 |
+
"multihop-v4-eval": {
|
| 350 |
+
"original": {
|
| 351 |
+
"questions": 597,
|
| 352 |
+
"accuracy": 0.7169179229480737,
|
| 353 |
+
"ece_10bin": 0.10006403003213532,
|
| 354 |
+
"calibration_axis_ece_part": 79.98719399357293
|
| 355 |
+
},
|
| 356 |
+
"rotation": {
|
| 357 |
+
"questions": 597,
|
| 358 |
+
"accuracy": 0.7169179229480737,
|
| 359 |
+
"ece_10bin": 0.10006403003213532,
|
| 360 |
+
"calibration_axis_ece_part": 79.98719399357293
|
| 361 |
+
}
|
| 362 |
+
},
|
| 363 |
+
"ood-paraphrase-eval-v1": {
|
| 364 |
+
"original": {
|
| 365 |
+
"questions": 586,
|
| 366 |
+
"accuracy": 0.7866894197952219,
|
| 367 |
+
"ece_10bin": 0.09275922147430427,
|
| 368 |
+
"calibration_axis_ece_part": 81.44815570513914,
|
| 369 |
+
"probability_items": 70,
|
| 370 |
+
"mean_tv_distance": 0.27008534467469203,
|
| 371 |
+
"fidelity_part": 72.9914655325308,
|
| 372 |
+
"calibration_axis_estimate": 77.21981061883497
|
| 373 |
+
},
|
| 374 |
+
"rotation": {
|
| 375 |
+
"questions": 586,
|
| 376 |
+
"accuracy": 0.7866894197952219,
|
| 377 |
+
"ece_10bin": 0.09275922147430427,
|
| 378 |
+
"calibration_axis_ece_part": 81.44815570513914,
|
| 379 |
+
"probability_items": 70,
|
| 380 |
+
"mean_tv_distance": 0.27008534467469203,
|
| 381 |
+
"fidelity_part": 72.9914655325308,
|
| 382 |
+
"calibration_axis_estimate": 77.21981061883497
|
| 383 |
+
}
|
| 384 |
+
},
|
| 385 |
+
"gen-v5-eval": {
|
| 386 |
+
"original": {
|
| 387 |
+
"questions": 738,
|
| 388 |
+
"accuracy": 0.8008130081300813,
|
| 389 |
+
"ece_10bin": 0.06862755202134709,
|
| 390 |
+
"calibration_axis_ece_part": 86.27448959573059
|
| 391 |
+
},
|
| 392 |
+
"rotation": {
|
| 393 |
+
"questions": 738,
|
| 394 |
+
"accuracy": 0.8008130081300813,
|
| 395 |
+
"ece_10bin": 0.06862755202134709,
|
| 396 |
+
"calibration_axis_ece_part": 86.27448959573059
|
| 397 |
}
|
| 398 |
}
|
| 399 |
},
|
|
|
|
| 401 |
"jevbench-easy": {
|
| 402 |
"original": {
|
| 403 |
"questions": 48,
|
| 404 |
+
"accuracy": 0.9791666666666666,
|
| 405 |
+
"ece_10bin": 0.03524161987063875,
|
| 406 |
+
"calibration_axis_ece_part": 92.95167602587226
|
| 407 |
},
|
| 408 |
"rotation": {
|
| 409 |
"questions": 48,
|
| 410 |
+
"accuracy": 0.9791666666666666,
|
| 411 |
+
"ece_10bin": 0.03524161987063875,
|
| 412 |
+
"calibration_axis_ece_part": 92.95167602587226
|
| 413 |
}
|
| 414 |
},
|
| 415 |
"jevbench-original": {
|
| 416 |
"original": {
|
| 417 |
"questions": 72,
|
| 418 |
"accuracy": 0.9722222222222222,
|
| 419 |
+
"ece_10bin": 0.05747967311316008,
|
| 420 |
+
"calibration_axis_ece_part": 88.50406537736798
|
| 421 |
},
|
| 422 |
"rotation": {
|
| 423 |
"questions": 72,
|
| 424 |
"accuracy": 0.9722222222222222,
|
| 425 |
+
"ece_10bin": 0.05747967311316008,
|
| 426 |
+
"calibration_axis_ece_part": 88.50406537736798
|
| 427 |
}
|
| 428 |
},
|
| 429 |
"jevbench-hard": {
|
| 430 |
"original": {
|
| 431 |
"questions": 111,
|
| 432 |
+
"accuracy": 0.5585585585585585,
|
| 433 |
+
"ece_10bin": 0.17667966122573142,
|
| 434 |
+
"calibration_axis_ece_part": 64.66406775485372
|
| 435 |
},
|
| 436 |
"rotation": {
|
| 437 |
"questions": 111,
|
| 438 |
+
"accuracy": 0.5585585585585585,
|
| 439 |
+
"ece_10bin": 0.17667966122573142,
|
| 440 |
+
"calibration_axis_ece_part": 64.66406775485372
|
| 441 |
}
|
| 442 |
},
|
| 443 |
"judge-proxy-v1": {
|
| 444 |
"original": {
|
| 445 |
"questions": 600,
|
| 446 |
+
"accuracy": 0.8833333333333333,
|
| 447 |
+
"ece_10bin": 0.054559563806900344,
|
| 448 |
+
"calibration_axis_ece_part": 89.08808723861993
|
| 449 |
},
|
| 450 |
"rotation": {
|
| 451 |
"questions": 600,
|
| 452 |
+
"accuracy": 0.8833333333333333,
|
| 453 |
+
"ece_10bin": 0.054559563806900344,
|
| 454 |
+
"calibration_axis_ece_part": 89.08808723861993
|
| 455 |
}
|
| 456 |
},
|
| 457 |
"internal-eval-hardproxy-v1": {
|
| 458 |
"original": {
|
| 459 |
"questions": 600,
|
| 460 |
+
"accuracy": 0.5266666666666666,
|
| 461 |
+
"ece_10bin": 0.07147612518045485,
|
| 462 |
+
"calibration_axis_ece_part": 85.70477496390903
|
| 463 |
},
|
| 464 |
"rotation": {
|
| 465 |
"questions": 600,
|
| 466 |
+
"accuracy": 0.5266666666666666,
|
| 467 |
+
"ece_10bin": 0.07147612518045485,
|
| 468 |
+
"calibration_axis_ece_part": 85.70477496390903
|
| 469 |
}
|
| 470 |
},
|
| 471 |
"internal-eval-hardproxy-v2": {
|
| 472 |
"original": {
|
| 473 |
"questions": 900,
|
| 474 |
+
"accuracy": 0.8433333333333334,
|
| 475 |
+
"ece_10bin": 0.053991315352306155,
|
| 476 |
+
"calibration_axis_ece_part": 89.20173692953877,
|
| 477 |
"probability_items": 80,
|
| 478 |
+
"mean_tv_distance": 0.11599699197747633,
|
| 479 |
+
"fidelity_part": 88.40030080225236,
|
| 480 |
+
"calibration_axis_estimate": 88.80101886589557
|
| 481 |
},
|
| 482 |
"rotation": {
|
| 483 |
"questions": 900,
|
| 484 |
+
"accuracy": 0.8433333333333334,
|
| 485 |
+
"ece_10bin": 0.053991315352306155,
|
| 486 |
+
"calibration_axis_ece_part": 89.20173692953877,
|
| 487 |
"probability_items": 80,
|
| 488 |
+
"mean_tv_distance": 0.11599699197747633,
|
| 489 |
+
"fidelity_part": 88.40030080225236,
|
| 490 |
+
"calibration_axis_estimate": 88.80101886589557
|
| 491 |
}
|
| 492 |
},
|
| 493 |
"internal-eval-hardproxy-v3": {
|
| 494 |
"original": {
|
| 495 |
"questions": 960,
|
| 496 |
+
"accuracy": 0.7552083333333334,
|
| 497 |
+
"ece_10bin": 0.07767264665837811,
|
| 498 |
+
"calibration_axis_ece_part": 84.46547066832437,
|
| 499 |
"probability_items": 120,
|
| 500 |
+
"mean_tv_distance": 0.3203937627499176,
|
| 501 |
+
"fidelity_part": 67.96062372500825,
|
| 502 |
+
"calibration_axis_estimate": 76.21304719666631
|
| 503 |
},
|
| 504 |
"rotation": {
|
| 505 |
"questions": 960,
|
| 506 |
+
"accuracy": 0.7552083333333334,
|
| 507 |
+
"ece_10bin": 0.07767264665837811,
|
| 508 |
+
"calibration_axis_ece_part": 84.46547066832437,
|
| 509 |
"probability_items": 120,
|
| 510 |
+
"mean_tv_distance": 0.3203937627499176,
|
| 511 |
+
"fidelity_part": 67.96062372500825,
|
| 512 |
+
"calibration_axis_estimate": 76.21304719666631
|
| 513 |
}
|
| 514 |
},
|
| 515 |
"internal-eval-hardproxy-calib-v1": {
|
| 516 |
"original": {
|
| 517 |
"questions": 600,
|
| 518 |
+
"accuracy": 0.5583333333333333,
|
| 519 |
+
"ece_10bin": 0.03356776673780446,
|
| 520 |
+
"calibration_axis_ece_part": 93.28644665243911
|
| 521 |
},
|
| 522 |
"rotation": {
|
| 523 |
"questions": 600,
|
| 524 |
+
"accuracy": 0.5583333333333333,
|
| 525 |
+
"ece_10bin": 0.03356776673780446,
|
| 526 |
+
"calibration_axis_ece_part": 93.28644665243911
|
| 527 |
}
|
| 528 |
},
|
| 529 |
"distprob-calib-v2": {
|
| 530 |
"original": {
|
| 531 |
"questions": 916,
|
| 532 |
+
"accuracy": 0.8067685589519651,
|
| 533 |
+
"ece_10bin": 0.18683086851036443,
|
| 534 |
+
"calibration_axis_ece_part": 62.63382629792711,
|
| 535 |
"probability_items": 916,
|
| 536 |
+
"mean_tv_distance": 0.1449657824650703,
|
| 537 |
+
"fidelity_part": 85.50342175349297,
|
| 538 |
+
"calibration_axis_estimate": 74.06862402571004
|
| 539 |
},
|
| 540 |
"rotation": {
|
| 541 |
"questions": 916,
|
| 542 |
+
"accuracy": 0.8067685589519651,
|
| 543 |
+
"ece_10bin": 0.18683086851036443,
|
| 544 |
+
"calibration_axis_ece_part": 62.63382629792711,
|
| 545 |
"probability_items": 916,
|
| 546 |
+
"mean_tv_distance": 0.1449657824650703,
|
| 547 |
+
"fidelity_part": 85.50342175349297,
|
| 548 |
+
"calibration_axis_estimate": 74.06862402571004
|
| 549 |
}
|
| 550 |
},
|
| 551 |
"distprob-val-v2": {
|
| 552 |
"original": {
|
| 553 |
"questions": 1036,
|
| 554 |
+
"accuracy": 0.8262548262548263,
|
| 555 |
+
"ece_10bin": 0.21362855032331066,
|
| 556 |
+
"calibration_axis_ece_part": 57.274289935337876,
|
| 557 |
"probability_items": 1036,
|
| 558 |
+
"mean_tv_distance": 0.10162736557917454,
|
| 559 |
+
"fidelity_part": 89.83726344208254,
|
| 560 |
+
"calibration_axis_estimate": 73.55577668871021
|
| 561 |
},
|
| 562 |
"rotation": {
|
| 563 |
"questions": 1036,
|
| 564 |
+
"accuracy": 0.8262548262548263,
|
| 565 |
+
"ece_10bin": 0.21362855032331066,
|
| 566 |
+
"calibration_axis_ece_part": 57.274289935337876,
|
| 567 |
"probability_items": 1036,
|
| 568 |
+
"mean_tv_distance": 0.10162736557917454,
|
| 569 |
+
"fidelity_part": 89.83726344208254,
|
| 570 |
+
"calibration_axis_estimate": 73.55577668871021
|
| 571 |
}
|
| 572 |
},
|
| 573 |
"internal-eval-v2-realistic": {
|
| 574 |
"original": {
|
| 575 |
"questions": 600,
|
| 576 |
"accuracy": 0.9,
|
| 577 |
+
"ece_10bin": 0.0665036226816713,
|
| 578 |
+
"calibration_axis_ece_part": 86.69927546366574,
|
| 579 |
"probability_items": 120,
|
| 580 |
+
"mean_tv_distance": 0.10493778102223708,
|
| 581 |
+
"fidelity_part": 89.5062218977763,
|
| 582 |
+
"calibration_axis_estimate": 88.10274868072102
|
| 583 |
},
|
| 584 |
"rotation": {
|
| 585 |
"questions": 600,
|
| 586 |
"accuracy": 0.9,
|
| 587 |
+
"ece_10bin": 0.0665036226816713,
|
| 588 |
+
"calibration_axis_ece_part": 86.69927546366574,
|
| 589 |
"probability_items": 120,
|
| 590 |
+
"mean_tv_distance": 0.10493778102223708,
|
| 591 |
+
"fidelity_part": 89.5062218977763,
|
| 592 |
+
"calibration_axis_estimate": 88.10274868072102
|
| 593 |
}
|
| 594 |
},
|
| 595 |
"games-harness-eval-v1": {
|
| 596 |
"original": {
|
| 597 |
"questions": 360,
|
| 598 |
"accuracy": 0.9638888888888889,
|
| 599 |
+
"ece_10bin": 0.02184635319154184,
|
| 600 |
+
"calibration_axis_ece_part": 95.63072936169164
|
| 601 |
},
|
| 602 |
"rotation": {
|
| 603 |
"questions": 360,
|
| 604 |
"accuracy": 0.9638888888888889,
|
| 605 |
+
"ece_10bin": 0.02184635319154184,
|
| 606 |
+
"calibration_axis_ece_part": 95.63072936169164
|
| 607 |
}
|
| 608 |
},
|
| 609 |
"tetris-drought-eval-v1": {
|
| 610 |
"original": {
|
| 611 |
"questions": 120,
|
| 612 |
+
"accuracy": 0.6833333333333333,
|
| 613 |
+
"ece_10bin": 0.11049695827247336,
|
| 614 |
+
"calibration_axis_ece_part": 77.90060834550533
|
| 615 |
},
|
| 616 |
"rotation": {
|
| 617 |
"questions": 120,
|
| 618 |
+
"accuracy": 0.6833333333333333,
|
| 619 |
+
"ece_10bin": 0.11049695827247336,
|
| 620 |
+
"calibration_axis_ece_part": 77.90060834550533
|
| 621 |
+
}
|
| 622 |
+
},
|
| 623 |
+
"musique-mc": {
|
| 624 |
+
"original": {
|
| 625 |
+
"questions": 500,
|
| 626 |
+
"accuracy": 0.792,
|
| 627 |
+
"ece_10bin": 0.13486604240531933,
|
| 628 |
+
"calibration_axis_ece_part": 73.02679151893614
|
| 629 |
+
},
|
| 630 |
+
"rotation": {
|
| 631 |
+
"questions": 500,
|
| 632 |
+
"accuracy": 0.792,
|
| 633 |
+
"ece_10bin": 0.13486604240531933,
|
| 634 |
+
"calibration_axis_ece_part": 73.02679151893614
|
| 635 |
+
}
|
| 636 |
+
},
|
| 637 |
+
"squad2-abstain": {
|
| 638 |
+
"original": {
|
| 639 |
+
"questions": 500,
|
| 640 |
+
"accuracy": 0.838,
|
| 641 |
+
"ece_10bin": 0.05573655874873827,
|
| 642 |
+
"calibration_axis_ece_part": 88.85268825025234
|
| 643 |
+
},
|
| 644 |
+
"rotation": {
|
| 645 |
+
"questions": 500,
|
| 646 |
+
"accuracy": 0.838,
|
| 647 |
+
"ece_10bin": 0.05573655874873827,
|
| 648 |
+
"calibration_axis_ece_part": 88.85268825025234
|
| 649 |
+
}
|
| 650 |
+
},
|
| 651 |
+
"contractnli": {
|
| 652 |
+
"original": {
|
| 653 |
+
"questions": 500,
|
| 654 |
+
"accuracy": 0.738,
|
| 655 |
+
"ece_10bin": 0.02914397113023941,
|
| 656 |
+
"calibration_axis_ece_part": 94.17120577395211
|
| 657 |
+
},
|
| 658 |
+
"rotation": {
|
| 659 |
+
"questions": 500,
|
| 660 |
+
"accuracy": 0.738,
|
| 661 |
+
"ece_10bin": 0.02914397113023941,
|
| 662 |
+
"calibration_axis_ece_part": 94.17120577395211
|
| 663 |
+
}
|
| 664 |
+
},
|
| 665 |
+
"paws-x-en": {
|
| 666 |
+
"original": {
|
| 667 |
+
"questions": 500,
|
| 668 |
+
"accuracy": 0.774,
|
| 669 |
+
"ece_10bin": 0.052597046548974354,
|
| 670 |
+
"calibration_axis_ece_part": 89.48059069020513
|
| 671 |
+
},
|
| 672 |
+
"rotation": {
|
| 673 |
+
"questions": 500,
|
| 674 |
+
"accuracy": 0.774,
|
| 675 |
+
"ece_10bin": 0.052597046548974354,
|
| 676 |
+
"calibration_axis_ece_part": 89.48059069020513
|
| 677 |
+
}
|
| 678 |
+
},
|
| 679 |
+
"calib-public-v1": {
|
| 680 |
+
"original": {
|
| 681 |
+
"questions": 1200,
|
| 682 |
+
"accuracy": 0.7883333333333333,
|
| 683 |
+
"ece_10bin": 0.04692776335400818,
|
| 684 |
+
"calibration_axis_ece_part": 90.61444732919837
|
| 685 |
+
},
|
| 686 |
+
"rotation": {
|
| 687 |
+
"questions": 1200,
|
| 688 |
+
"accuracy": 0.7883333333333333,
|
| 689 |
+
"ece_10bin": 0.04692776335400818,
|
| 690 |
+
"calibration_axis_ece_part": 90.61444732919837
|
| 691 |
+
}
|
| 692 |
+
},
|
| 693 |
+
"shortcut-diagnostic-eval-v1": {
|
| 694 |
+
"original": {
|
| 695 |
+
"questions": 601,
|
| 696 |
+
"accuracy": 0.8386023294509152,
|
| 697 |
+
"ece_10bin": 0.04566385758624237,
|
| 698 |
+
"calibration_axis_ece_part": 90.86722848275153
|
| 699 |
+
},
|
| 700 |
+
"rotation": {
|
| 701 |
+
"questions": 601,
|
| 702 |
+
"accuracy": 0.8386023294509152,
|
| 703 |
+
"ece_10bin": 0.04566385758624237,
|
| 704 |
+
"calibration_axis_ece_part": 90.86722848275153
|
| 705 |
+
}
|
| 706 |
+
},
|
| 707 |
+
"consistency-eval-v1": {
|
| 708 |
+
"original": {
|
| 709 |
+
"questions": 1200,
|
| 710 |
+
"accuracy": 0.7766666666666666,
|
| 711 |
+
"ece_10bin": 0.03654737270461749,
|
| 712 |
+
"calibration_axis_ece_part": 92.6905254590765
|
| 713 |
+
},
|
| 714 |
+
"rotation": {
|
| 715 |
+
"questions": 1200,
|
| 716 |
+
"accuracy": 0.7766666666666666,
|
| 717 |
+
"ece_10bin": 0.03654737270461749,
|
| 718 |
+
"calibration_axis_ece_part": 92.6905254590765
|
| 719 |
+
}
|
| 720 |
+
},
|
| 721 |
+
"hard-eval-v4": {
|
| 722 |
+
"original": {
|
| 723 |
+
"questions": 800,
|
| 724 |
+
"accuracy": 0.61,
|
| 725 |
+
"ece_10bin": 0.08232350161574703,
|
| 726 |
+
"calibration_axis_ece_part": 83.5352996768506
|
| 727 |
+
},
|
| 728 |
+
"rotation": {
|
| 729 |
+
"questions": 800,
|
| 730 |
+
"accuracy": 0.61,
|
| 731 |
+
"ece_10bin": 0.08232350161574703,
|
| 732 |
+
"calibration_axis_ece_part": 83.5352996768506
|
| 733 |
+
}
|
| 734 |
+
},
|
| 735 |
+
"multihop-v4-eval": {
|
| 736 |
+
"original": {
|
| 737 |
+
"questions": 597,
|
| 738 |
+
"accuracy": 0.7169179229480737,
|
| 739 |
+
"ece_10bin": 0.0693297434671131,
|
| 740 |
+
"calibration_axis_ece_part": 86.13405130657739
|
| 741 |
+
},
|
| 742 |
+
"rotation": {
|
| 743 |
+
"questions": 597,
|
| 744 |
+
"accuracy": 0.7169179229480737,
|
| 745 |
+
"ece_10bin": 0.0693297434671131,
|
| 746 |
+
"calibration_axis_ece_part": 86.13405130657739
|
| 747 |
+
}
|
| 748 |
+
},
|
| 749 |
+
"ood-paraphrase-eval-v1": {
|
| 750 |
+
"original": {
|
| 751 |
+
"questions": 586,
|
| 752 |
+
"accuracy": 0.7866894197952219,
|
| 753 |
+
"ece_10bin": 0.0577406986619374,
|
| 754 |
+
"calibration_axis_ece_part": 88.45186026761252,
|
| 755 |
+
"probability_items": 70,
|
| 756 |
+
"mean_tv_distance": 0.2945346301884306,
|
| 757 |
+
"fidelity_part": 70.54653698115693,
|
| 758 |
+
"calibration_axis_estimate": 79.49919862438472
|
| 759 |
+
},
|
| 760 |
+
"rotation": {
|
| 761 |
+
"questions": 586,
|
| 762 |
+
"accuracy": 0.7866894197952219,
|
| 763 |
+
"ece_10bin": 0.0577406986619374,
|
| 764 |
+
"calibration_axis_ece_part": 88.45186026761252,
|
| 765 |
+
"probability_items": 70,
|
| 766 |
+
"mean_tv_distance": 0.2945346301884306,
|
| 767 |
+
"fidelity_part": 70.54653698115693,
|
| 768 |
+
"calibration_axis_estimate": 79.49919862438472
|
| 769 |
+
}
|
| 770 |
+
},
|
| 771 |
+
"gen-v5-eval": {
|
| 772 |
+
"original": {
|
| 773 |
+
"questions": 738,
|
| 774 |
+
"accuracy": 0.8008130081300813,
|
| 775 |
+
"ece_10bin": 0.03632140751073094,
|
| 776 |
+
"calibration_axis_ece_part": 92.73571849785381
|
| 777 |
+
},
|
| 778 |
+
"rotation": {
|
| 779 |
+
"questions": 738,
|
| 780 |
+
"accuracy": 0.8008130081300813,
|
| 781 |
+
"ece_10bin": 0.03632140751073094,
|
| 782 |
+
"calibration_axis_ece_part": 92.73571849785381
|
| 783 |
}
|
| 784 |
}
|
| 785 |
}
|
| 786 |
+
}
|
evidence/calibration-leaderboard-merged-native.json
CHANGED
|
@@ -1,207 +1,207 @@
|
|
| 1 |
{
|
| 2 |
"arm_fitted": "original",
|
| 3 |
"inputs": [
|
| 4 |
-
"runs/merged-
|
| 5 |
],
|
| 6 |
"fit_suites": [
|
| 7 |
"distprob-calib-v2",
|
| 8 |
"internal-eval-hardproxy-calib-v1"
|
| 9 |
],
|
| 10 |
-
"temperature":
|
| 11 |
"objective": "leaderboard",
|
| 12 |
-
"axis_proxy_at_T1":
|
| 13 |
-
"axis_proxy_at_T":
|
| 14 |
"at_T1": {
|
| 15 |
"jevbench-easy": {
|
| 16 |
"original": {
|
| 17 |
"questions": 48,
|
| 18 |
-
"accuracy":
|
| 19 |
-
"ece_10bin": 0.
|
| 20 |
-
"calibration_axis_ece_part":
|
| 21 |
},
|
| 22 |
"rotation": {
|
| 23 |
"questions": 48,
|
| 24 |
-
"accuracy":
|
| 25 |
-
"ece_10bin": 0.
|
| 26 |
-
"calibration_axis_ece_part":
|
| 27 |
}
|
| 28 |
},
|
| 29 |
"jevbench-original": {
|
| 30 |
"original": {
|
| 31 |
"questions": 72,
|
| 32 |
-
"accuracy": 0.
|
| 33 |
-
"ece_10bin": 0.
|
| 34 |
-
"calibration_axis_ece_part":
|
| 35 |
},
|
| 36 |
"rotation": {
|
| 37 |
"questions": 72,
|
| 38 |
-
"accuracy": 0.
|
| 39 |
-
"ece_10bin": 0.
|
| 40 |
-
"calibration_axis_ece_part":
|
| 41 |
}
|
| 42 |
},
|
| 43 |
"jevbench-hard": {
|
| 44 |
"original": {
|
| 45 |
"questions": 111,
|
| 46 |
-
"accuracy": 0.
|
| 47 |
-
"ece_10bin": 0.
|
| 48 |
-
"calibration_axis_ece_part":
|
| 49 |
},
|
| 50 |
"rotation": {
|
| 51 |
"questions": 111,
|
| 52 |
-
"accuracy": 0.
|
| 53 |
-
"ece_10bin": 0.
|
| 54 |
-
"calibration_axis_ece_part":
|
| 55 |
}
|
| 56 |
},
|
| 57 |
"judge-proxy-v1": {
|
| 58 |
"original": {
|
| 59 |
"questions": 600,
|
| 60 |
-
"accuracy": 0.
|
| 61 |
-
"ece_10bin": 0.
|
| 62 |
-
"calibration_axis_ece_part":
|
| 63 |
},
|
| 64 |
"rotation": {
|
| 65 |
"questions": 600,
|
| 66 |
-
"accuracy": 0.
|
| 67 |
-
"ece_10bin": 0.
|
| 68 |
-
"calibration_axis_ece_part":
|
| 69 |
}
|
| 70 |
},
|
| 71 |
"internal-eval-v2-realistic": {
|
| 72 |
"original": {
|
| 73 |
"questions": 600,
|
| 74 |
-
"accuracy": 0.
|
| 75 |
-
"ece_10bin": 0.
|
| 76 |
-
"calibration_axis_ece_part":
|
| 77 |
"probability_items": 120,
|
| 78 |
-
"mean_tv_distance": 0.
|
| 79 |
-
"fidelity_part":
|
| 80 |
-
"calibration_axis_estimate":
|
| 81 |
},
|
| 82 |
"rotation": {
|
| 83 |
"questions": 600,
|
| 84 |
-
"accuracy": 0.
|
| 85 |
-
"ece_10bin": 0.
|
| 86 |
-
"calibration_axis_ece_part":
|
| 87 |
"probability_items": 120,
|
| 88 |
-
"mean_tv_distance": 0.
|
| 89 |
-
"fidelity_part":
|
| 90 |
-
"calibration_axis_estimate":
|
| 91 |
}
|
| 92 |
},
|
| 93 |
"distprob-val-v2": {
|
| 94 |
"original": {
|
| 95 |
"questions": 1036,
|
| 96 |
-
"accuracy": 0.
|
| 97 |
-
"ece_10bin": 0.
|
| 98 |
-
"calibration_axis_ece_part":
|
| 99 |
"probability_items": 1036,
|
| 100 |
-
"mean_tv_distance": 0.
|
| 101 |
-
"fidelity_part": 89.
|
| 102 |
-
"calibration_axis_estimate":
|
| 103 |
},
|
| 104 |
"rotation": {
|
| 105 |
"questions": 1036,
|
| 106 |
-
"accuracy": 0.
|
| 107 |
-
"ece_10bin": 0.
|
| 108 |
-
"calibration_axis_ece_part":
|
| 109 |
"probability_items": 1036,
|
| 110 |
-
"mean_tv_distance": 0.
|
| 111 |
-
"fidelity_part": 89.
|
| 112 |
-
"calibration_axis_estimate":
|
| 113 |
}
|
| 114 |
},
|
| 115 |
"internal-eval-v1": {
|
| 116 |
"original": {
|
| 117 |
"questions": 600,
|
| 118 |
-
"accuracy": 0.
|
| 119 |
-
"ece_10bin": 0.
|
| 120 |
-
"calibration_axis_ece_part":
|
| 121 |
},
|
| 122 |
"rotation": {
|
| 123 |
"questions": 600,
|
| 124 |
-
"accuracy": 0.
|
| 125 |
-
"ece_10bin": 0.
|
| 126 |
-
"calibration_axis_ece_part":
|
| 127 |
}
|
| 128 |
},
|
| 129 |
"internal-eval-hardproxy-v1": {
|
| 130 |
"original": {
|
| 131 |
"questions": 600,
|
| 132 |
-
"accuracy": 0.
|
| 133 |
-
"ece_10bin": 0.
|
| 134 |
-
"calibration_axis_ece_part":
|
| 135 |
},
|
| 136 |
"rotation": {
|
| 137 |
"questions": 600,
|
| 138 |
-
"accuracy": 0.
|
| 139 |
-
"ece_10bin": 0.
|
| 140 |
-
"calibration_axis_ece_part":
|
| 141 |
}
|
| 142 |
},
|
| 143 |
"games-harness-eval-v1": {
|
| 144 |
"original": {
|
| 145 |
"questions": 360,
|
| 146 |
-
"accuracy": 0.
|
| 147 |
-
"ece_10bin": 0.
|
| 148 |
-
"calibration_axis_ece_part":
|
| 149 |
},
|
| 150 |
"rotation": {
|
| 151 |
"questions": 360,
|
| 152 |
-
"accuracy": 0.
|
| 153 |
-
"ece_10bin": 0.
|
| 154 |
-
"calibration_axis_ece_part":
|
| 155 |
}
|
| 156 |
},
|
| 157 |
"tetris-drought-eval-v1": {
|
| 158 |
"original": {
|
| 159 |
"questions": 120,
|
| 160 |
-
"accuracy": 0.
|
| 161 |
-
"ece_10bin": 0.
|
| 162 |
-
"calibration_axis_ece_part":
|
| 163 |
},
|
| 164 |
"rotation": {
|
| 165 |
"questions": 120,
|
| 166 |
-
"accuracy": 0.
|
| 167 |
-
"ece_10bin": 0.
|
| 168 |
-
"calibration_axis_ece_part":
|
| 169 |
}
|
| 170 |
},
|
| 171 |
"internal-eval-hardproxy-calib-v1": {
|
| 172 |
"original": {
|
| 173 |
"questions": 600,
|
| 174 |
-
"accuracy": 0.
|
| 175 |
-
"ece_10bin": 0.
|
| 176 |
-
"calibration_axis_ece_part":
|
| 177 |
},
|
| 178 |
"rotation": {
|
| 179 |
"questions": 600,
|
| 180 |
-
"accuracy": 0.
|
| 181 |
-
"ece_10bin": 0.
|
| 182 |
-
"calibration_axis_ece_part":
|
| 183 |
}
|
| 184 |
},
|
| 185 |
"distprob-calib-v2": {
|
| 186 |
"original": {
|
| 187 |
"questions": 916,
|
| 188 |
-
"accuracy": 0.
|
| 189 |
-
"ece_10bin": 0.
|
| 190 |
-
"calibration_axis_ece_part":
|
| 191 |
"probability_items": 916,
|
| 192 |
-
"mean_tv_distance": 0.
|
| 193 |
-
"fidelity_part": 85.
|
| 194 |
-
"calibration_axis_estimate":
|
| 195 |
},
|
| 196 |
"rotation": {
|
| 197 |
"questions": 916,
|
| 198 |
-
"accuracy": 0.
|
| 199 |
-
"ece_10bin": 0.
|
| 200 |
-
"calibration_axis_ece_part":
|
| 201 |
"probability_items": 916,
|
| 202 |
-
"mean_tv_distance": 0.
|
| 203 |
-
"fidelity_part": 85.
|
| 204 |
-
"calibration_axis_estimate":
|
| 205 |
}
|
| 206 |
}
|
| 207 |
},
|
|
@@ -209,193 +209,193 @@
|
|
| 209 |
"jevbench-easy": {
|
| 210 |
"original": {
|
| 211 |
"questions": 48,
|
| 212 |
-
"accuracy":
|
| 213 |
-
"ece_10bin": 0.
|
| 214 |
-
"calibration_axis_ece_part":
|
| 215 |
},
|
| 216 |
"rotation": {
|
| 217 |
"questions": 48,
|
| 218 |
-
"accuracy":
|
| 219 |
-
"ece_10bin": 0.
|
| 220 |
-
"calibration_axis_ece_part":
|
| 221 |
}
|
| 222 |
},
|
| 223 |
"jevbench-original": {
|
| 224 |
"original": {
|
| 225 |
"questions": 72,
|
| 226 |
-
"accuracy": 0.
|
| 227 |
-
"ece_10bin": 0.
|
| 228 |
-
"calibration_axis_ece_part":
|
| 229 |
},
|
| 230 |
"rotation": {
|
| 231 |
"questions": 72,
|
| 232 |
-
"accuracy": 0.
|
| 233 |
-
"ece_10bin": 0.
|
| 234 |
-
"calibration_axis_ece_part":
|
| 235 |
}
|
| 236 |
},
|
| 237 |
"jevbench-hard": {
|
| 238 |
"original": {
|
| 239 |
"questions": 111,
|
| 240 |
-
"accuracy": 0.
|
| 241 |
-
"ece_10bin": 0.
|
| 242 |
-
"calibration_axis_ece_part":
|
| 243 |
},
|
| 244 |
"rotation": {
|
| 245 |
"questions": 111,
|
| 246 |
-
"accuracy": 0.
|
| 247 |
-
"ece_10bin": 0.
|
| 248 |
-
"calibration_axis_ece_part":
|
| 249 |
}
|
| 250 |
},
|
| 251 |
"judge-proxy-v1": {
|
| 252 |
"original": {
|
| 253 |
"questions": 600,
|
| 254 |
-
"accuracy": 0.
|
| 255 |
-
"ece_10bin": 0.
|
| 256 |
-
"calibration_axis_ece_part": 92.
|
| 257 |
},
|
| 258 |
"rotation": {
|
| 259 |
"questions": 600,
|
| 260 |
-
"accuracy": 0.
|
| 261 |
-
"ece_10bin": 0.
|
| 262 |
-
"calibration_axis_ece_part": 92.
|
| 263 |
}
|
| 264 |
},
|
| 265 |
"internal-eval-v2-realistic": {
|
| 266 |
"original": {
|
| 267 |
"questions": 600,
|
| 268 |
-
"accuracy": 0.
|
| 269 |
-
"ece_10bin": 0.
|
| 270 |
-
"calibration_axis_ece_part":
|
| 271 |
"probability_items": 120,
|
| 272 |
-
"mean_tv_distance": 0.
|
| 273 |
-
"fidelity_part": 88.
|
| 274 |
-
"calibration_axis_estimate":
|
| 275 |
},
|
| 276 |
"rotation": {
|
| 277 |
"questions": 600,
|
| 278 |
-
"accuracy": 0.
|
| 279 |
-
"ece_10bin": 0.
|
| 280 |
-
"calibration_axis_ece_part":
|
| 281 |
"probability_items": 120,
|
| 282 |
-
"mean_tv_distance": 0.
|
| 283 |
-
"fidelity_part": 88.
|
| 284 |
-
"calibration_axis_estimate":
|
| 285 |
}
|
| 286 |
},
|
| 287 |
"distprob-val-v2": {
|
| 288 |
"original": {
|
| 289 |
"questions": 1036,
|
| 290 |
-
"accuracy": 0.
|
| 291 |
-
"ece_10bin": 0.
|
| 292 |
-
"calibration_axis_ece_part":
|
| 293 |
"probability_items": 1036,
|
| 294 |
-
"mean_tv_distance": 0.
|
| 295 |
-
"fidelity_part":
|
| 296 |
-
"calibration_axis_estimate":
|
| 297 |
},
|
| 298 |
"rotation": {
|
| 299 |
"questions": 1036,
|
| 300 |
-
"accuracy": 0.
|
| 301 |
-
"ece_10bin": 0.
|
| 302 |
-
"calibration_axis_ece_part":
|
| 303 |
"probability_items": 1036,
|
| 304 |
-
"mean_tv_distance": 0.
|
| 305 |
-
"fidelity_part":
|
| 306 |
-
"calibration_axis_estimate":
|
| 307 |
}
|
| 308 |
},
|
| 309 |
"internal-eval-v1": {
|
| 310 |
"original": {
|
| 311 |
"questions": 600,
|
| 312 |
-
"accuracy": 0.
|
| 313 |
-
"ece_10bin": 0.
|
| 314 |
-
"calibration_axis_ece_part":
|
| 315 |
},
|
| 316 |
"rotation": {
|
| 317 |
"questions": 600,
|
| 318 |
-
"accuracy": 0.
|
| 319 |
-
"ece_10bin": 0.
|
| 320 |
-
"calibration_axis_ece_part":
|
| 321 |
}
|
| 322 |
},
|
| 323 |
"internal-eval-hardproxy-v1": {
|
| 324 |
"original": {
|
| 325 |
"questions": 600,
|
| 326 |
-
"accuracy": 0.
|
| 327 |
-
"ece_10bin": 0.
|
| 328 |
-
"calibration_axis_ece_part":
|
| 329 |
},
|
| 330 |
"rotation": {
|
| 331 |
"questions": 600,
|
| 332 |
-
"accuracy": 0.
|
| 333 |
-
"ece_10bin": 0.
|
| 334 |
-
"calibration_axis_ece_part":
|
| 335 |
}
|
| 336 |
},
|
| 337 |
"games-harness-eval-v1": {
|
| 338 |
"original": {
|
| 339 |
"questions": 360,
|
| 340 |
-
"accuracy": 0.
|
| 341 |
-
"ece_10bin": 0.
|
| 342 |
-
"calibration_axis_ece_part": 92.
|
| 343 |
},
|
| 344 |
"rotation": {
|
| 345 |
"questions": 360,
|
| 346 |
-
"accuracy": 0.
|
| 347 |
-
"ece_10bin": 0.
|
| 348 |
-
"calibration_axis_ece_part": 92.
|
| 349 |
}
|
| 350 |
},
|
| 351 |
"tetris-drought-eval-v1": {
|
| 352 |
"original": {
|
| 353 |
"questions": 120,
|
| 354 |
-
"accuracy": 0.
|
| 355 |
-
"ece_10bin": 0.
|
| 356 |
-
"calibration_axis_ece_part":
|
| 357 |
},
|
| 358 |
"rotation": {
|
| 359 |
"questions": 120,
|
| 360 |
-
"accuracy": 0.
|
| 361 |
-
"ece_10bin": 0.
|
| 362 |
-
"calibration_axis_ece_part":
|
| 363 |
}
|
| 364 |
},
|
| 365 |
"internal-eval-hardproxy-calib-v1": {
|
| 366 |
"original": {
|
| 367 |
"questions": 600,
|
| 368 |
-
"accuracy": 0.
|
| 369 |
-
"ece_10bin": 0.
|
| 370 |
-
"calibration_axis_ece_part":
|
| 371 |
},
|
| 372 |
"rotation": {
|
| 373 |
"questions": 600,
|
| 374 |
-
"accuracy": 0.
|
| 375 |
-
"ece_10bin": 0.
|
| 376 |
-
"calibration_axis_ece_part":
|
| 377 |
}
|
| 378 |
},
|
| 379 |
"distprob-calib-v2": {
|
| 380 |
"original": {
|
| 381 |
"questions": 916,
|
| 382 |
-
"accuracy": 0.
|
| 383 |
-
"ece_10bin": 0.
|
| 384 |
-
"calibration_axis_ece_part":
|
| 385 |
"probability_items": 916,
|
| 386 |
-
"mean_tv_distance": 0.
|
| 387 |
-
"fidelity_part":
|
| 388 |
-
"calibration_axis_estimate":
|
| 389 |
},
|
| 390 |
"rotation": {
|
| 391 |
"questions": 916,
|
| 392 |
-
"accuracy": 0.
|
| 393 |
-
"ece_10bin": 0.
|
| 394 |
-
"calibration_axis_ece_part":
|
| 395 |
"probability_items": 916,
|
| 396 |
-
"mean_tv_distance": 0.
|
| 397 |
-
"fidelity_part":
|
| 398 |
-
"calibration_axis_estimate":
|
| 399 |
}
|
| 400 |
}
|
| 401 |
}
|
|
|
|
| 1 |
{
|
| 2 |
"arm_fitted": "original",
|
| 3 |
"inputs": [
|
| 4 |
+
"runs/merged-v2r5-8b-native/predictions.jsonl"
|
| 5 |
],
|
| 6 |
"fit_suites": [
|
| 7 |
"distprob-calib-v2",
|
| 8 |
"internal-eval-hardproxy-calib-v1"
|
| 9 |
],
|
| 10 |
+
"temperature": 0.9,
|
| 11 |
"objective": "leaderboard",
|
| 12 |
+
"axis_proxy_at_T1": 89.39094405825404,
|
| 13 |
+
"axis_proxy_at_T": 90.02942353534496,
|
| 14 |
"at_T1": {
|
| 15 |
"jevbench-easy": {
|
| 16 |
"original": {
|
| 17 |
"questions": 48,
|
| 18 |
+
"accuracy": 0.9791666666666666,
|
| 19 |
+
"ece_10bin": 0.06084584077400829,
|
| 20 |
+
"calibration_axis_ece_part": 87.83083184519835
|
| 21 |
},
|
| 22 |
"rotation": {
|
| 23 |
"questions": 48,
|
| 24 |
+
"accuracy": 0.9791666666666666,
|
| 25 |
+
"ece_10bin": 0.06084584077400829,
|
| 26 |
+
"calibration_axis_ece_part": 87.83083184519835
|
| 27 |
}
|
| 28 |
},
|
| 29 |
"jevbench-original": {
|
| 30 |
"original": {
|
| 31 |
"questions": 72,
|
| 32 |
+
"accuracy": 0.9722222222222222,
|
| 33 |
+
"ece_10bin": 0.09698746242927771,
|
| 34 |
+
"calibration_axis_ece_part": 80.60250751414446
|
| 35 |
},
|
| 36 |
"rotation": {
|
| 37 |
"questions": 72,
|
| 38 |
+
"accuracy": 0.9722222222222222,
|
| 39 |
+
"ece_10bin": 0.09698746242927771,
|
| 40 |
+
"calibration_axis_ece_part": 80.60250751414446
|
| 41 |
}
|
| 42 |
},
|
| 43 |
"jevbench-hard": {
|
| 44 |
"original": {
|
| 45 |
"questions": 111,
|
| 46 |
+
"accuracy": 0.5405405405405406,
|
| 47 |
+
"ece_10bin": 0.1718513293706684,
|
| 48 |
+
"calibration_axis_ece_part": 65.62973412586632
|
| 49 |
},
|
| 50 |
"rotation": {
|
| 51 |
"questions": 111,
|
| 52 |
+
"accuracy": 0.5405405405405406,
|
| 53 |
+
"ece_10bin": 0.1718513293706684,
|
| 54 |
+
"calibration_axis_ece_part": 65.62973412586632
|
| 55 |
}
|
| 56 |
},
|
| 57 |
"judge-proxy-v1": {
|
| 58 |
"original": {
|
| 59 |
"questions": 600,
|
| 60 |
+
"accuracy": 0.8983333333333333,
|
| 61 |
+
"ece_10bin": 0.032301507962339375,
|
| 62 |
+
"calibration_axis_ece_part": 93.53969840753213
|
| 63 |
},
|
| 64 |
"rotation": {
|
| 65 |
"questions": 600,
|
| 66 |
+
"accuracy": 0.8983333333333333,
|
| 67 |
+
"ece_10bin": 0.032301507962339375,
|
| 68 |
+
"calibration_axis_ece_part": 93.53969840753213
|
| 69 |
}
|
| 70 |
},
|
| 71 |
"internal-eval-v2-realistic": {
|
| 72 |
"original": {
|
| 73 |
"questions": 600,
|
| 74 |
+
"accuracy": 0.9,
|
| 75 |
+
"ece_10bin": 0.09031304679451015,
|
| 76 |
+
"calibration_axis_ece_part": 81.93739064109798,
|
| 77 |
"probability_items": 120,
|
| 78 |
+
"mean_tv_distance": 0.116874445744356,
|
| 79 |
+
"fidelity_part": 88.3125554255644,
|
| 80 |
+
"calibration_axis_estimate": 85.12497303333119
|
| 81 |
},
|
| 82 |
"rotation": {
|
| 83 |
"questions": 600,
|
| 84 |
+
"accuracy": 0.9,
|
| 85 |
+
"ece_10bin": 0.09031304679451015,
|
| 86 |
+
"calibration_axis_ece_part": 81.93739064109798,
|
| 87 |
"probability_items": 120,
|
| 88 |
+
"mean_tv_distance": 0.116874445744356,
|
| 89 |
+
"fidelity_part": 88.3125554255644,
|
| 90 |
+
"calibration_axis_estimate": 85.12497303333119
|
| 91 |
}
|
| 92 |
},
|
| 93 |
"distprob-val-v2": {
|
| 94 |
"original": {
|
| 95 |
"questions": 1036,
|
| 96 |
+
"accuracy": 0.8262548262548263,
|
| 97 |
+
"ece_10bin": 0.2358515207860438,
|
| 98 |
+
"calibration_axis_ece_part": 52.829695842791246,
|
| 99 |
"probability_items": 1036,
|
| 100 |
+
"mean_tv_distance": 0.1035510813851473,
|
| 101 |
+
"fidelity_part": 89.64489186148526,
|
| 102 |
+
"calibration_axis_estimate": 71.23729385213825
|
| 103 |
},
|
| 104 |
"rotation": {
|
| 105 |
"questions": 1036,
|
| 106 |
+
"accuracy": 0.8262548262548263,
|
| 107 |
+
"ece_10bin": 0.2358515207860438,
|
| 108 |
+
"calibration_axis_ece_part": 52.829695842791246,
|
| 109 |
"probability_items": 1036,
|
| 110 |
+
"mean_tv_distance": 0.1035510813851473,
|
| 111 |
+
"fidelity_part": 89.64489186148526,
|
| 112 |
+
"calibration_axis_estimate": 71.23729385213825
|
| 113 |
}
|
| 114 |
},
|
| 115 |
"internal-eval-v1": {
|
| 116 |
"original": {
|
| 117 |
"questions": 600,
|
| 118 |
+
"accuracy": 0.77,
|
| 119 |
+
"ece_10bin": 0.08272740577314419,
|
| 120 |
+
"calibration_axis_ece_part": 83.45451884537115
|
| 121 |
},
|
| 122 |
"rotation": {
|
| 123 |
"questions": 600,
|
| 124 |
+
"accuracy": 0.77,
|
| 125 |
+
"ece_10bin": 0.08272740577314419,
|
| 126 |
+
"calibration_axis_ece_part": 83.45451884537115
|
| 127 |
}
|
| 128 |
},
|
| 129 |
"internal-eval-hardproxy-v1": {
|
| 130 |
"original": {
|
| 131 |
"questions": 600,
|
| 132 |
+
"accuracy": 0.545,
|
| 133 |
+
"ece_10bin": 0.03828401064102784,
|
| 134 |
+
"calibration_axis_ece_part": 92.34319787179443
|
| 135 |
},
|
| 136 |
"rotation": {
|
| 137 |
"questions": 600,
|
| 138 |
+
"accuracy": 0.545,
|
| 139 |
+
"ece_10bin": 0.03828401064102784,
|
| 140 |
+
"calibration_axis_ece_part": 92.34319787179443
|
| 141 |
}
|
| 142 |
},
|
| 143 |
"games-harness-eval-v1": {
|
| 144 |
"original": {
|
| 145 |
"questions": 360,
|
| 146 |
+
"accuracy": 0.975,
|
| 147 |
+
"ece_10bin": 0.04861687139484637,
|
| 148 |
+
"calibration_axis_ece_part": 90.27662572103073
|
| 149 |
},
|
| 150 |
"rotation": {
|
| 151 |
"questions": 360,
|
| 152 |
+
"accuracy": 0.975,
|
| 153 |
+
"ece_10bin": 0.04861687139484637,
|
| 154 |
+
"calibration_axis_ece_part": 90.27662572103073
|
| 155 |
}
|
| 156 |
},
|
| 157 |
"tetris-drought-eval-v1": {
|
| 158 |
"original": {
|
| 159 |
"questions": 120,
|
| 160 |
+
"accuracy": 0.6666666666666666,
|
| 161 |
+
"ece_10bin": 0.1224225426051953,
|
| 162 |
+
"calibration_axis_ece_part": 75.51549147896094
|
| 163 |
},
|
| 164 |
"rotation": {
|
| 165 |
"questions": 120,
|
| 166 |
+
"accuracy": 0.6666666666666666,
|
| 167 |
+
"ece_10bin": 0.1224225426051953,
|
| 168 |
+
"calibration_axis_ece_part": 75.51549147896094
|
| 169 |
}
|
| 170 |
},
|
| 171 |
"internal-eval-hardproxy-calib-v1": {
|
| 172 |
"original": {
|
| 173 |
"questions": 600,
|
| 174 |
+
"accuracy": 0.55,
|
| 175 |
+
"ece_10bin": 0.03251703333599121,
|
| 176 |
+
"calibration_axis_ece_part": 93.49659333280175
|
| 177 |
},
|
| 178 |
"rotation": {
|
| 179 |
"questions": 600,
|
| 180 |
+
"accuracy": 0.55,
|
| 181 |
+
"ece_10bin": 0.03251703333599121,
|
| 182 |
+
"calibration_axis_ece_part": 93.49659333280175
|
| 183 |
}
|
| 184 |
},
|
| 185 |
"distprob-calib-v2": {
|
| 186 |
"original": {
|
| 187 |
"questions": 916,
|
| 188 |
+
"accuracy": 0.8056768558951966,
|
| 189 |
+
"ece_10bin": 0.20882153550285798,
|
| 190 |
+
"calibration_axis_ece_part": 58.2356928994284,
|
| 191 |
"probability_items": 916,
|
| 192 |
+
"mean_tv_distance": 0.14714705216293703,
|
| 193 |
+
"fidelity_part": 85.2852947837063,
|
| 194 |
+
"calibration_axis_estimate": 71.76049384156735
|
| 195 |
},
|
| 196 |
"rotation": {
|
| 197 |
"questions": 916,
|
| 198 |
+
"accuracy": 0.8056768558951966,
|
| 199 |
+
"ece_10bin": 0.20882153550285798,
|
| 200 |
+
"calibration_axis_ece_part": 58.2356928994284,
|
| 201 |
"probability_items": 916,
|
| 202 |
+
"mean_tv_distance": 0.14714705216293703,
|
| 203 |
+
"fidelity_part": 85.2852947837063,
|
| 204 |
+
"calibration_axis_estimate": 71.76049384156735
|
| 205 |
}
|
| 206 |
}
|
| 207 |
},
|
|
|
|
| 209 |
"jevbench-easy": {
|
| 210 |
"original": {
|
| 211 |
"questions": 48,
|
| 212 |
+
"accuracy": 0.9791666666666666,
|
| 213 |
+
"ece_10bin": 0.045379196836452335,
|
| 214 |
+
"calibration_axis_ece_part": 90.92416063270954
|
| 215 |
},
|
| 216 |
"rotation": {
|
| 217 |
"questions": 48,
|
| 218 |
+
"accuracy": 0.9791666666666666,
|
| 219 |
+
"ece_10bin": 0.045379196836452335,
|
| 220 |
+
"calibration_axis_ece_part": 90.92416063270954
|
| 221 |
}
|
| 222 |
},
|
| 223 |
"jevbench-original": {
|
| 224 |
"original": {
|
| 225 |
"questions": 72,
|
| 226 |
+
"accuracy": 0.9722222222222222,
|
| 227 |
+
"ece_10bin": 0.07893674689684578,
|
| 228 |
+
"calibration_axis_ece_part": 84.21265062063085
|
| 229 |
},
|
| 230 |
"rotation": {
|
| 231 |
"questions": 72,
|
| 232 |
+
"accuracy": 0.9722222222222222,
|
| 233 |
+
"ece_10bin": 0.07893674689684578,
|
| 234 |
+
"calibration_axis_ece_part": 84.21265062063085
|
| 235 |
}
|
| 236 |
},
|
| 237 |
"jevbench-hard": {
|
| 238 |
"original": {
|
| 239 |
"questions": 111,
|
| 240 |
+
"accuracy": 0.5405405405405406,
|
| 241 |
+
"ece_10bin": 0.18339181130178053,
|
| 242 |
+
"calibration_axis_ece_part": 63.3216377396439
|
| 243 |
},
|
| 244 |
"rotation": {
|
| 245 |
"questions": 111,
|
| 246 |
+
"accuracy": 0.5405405405405406,
|
| 247 |
+
"ece_10bin": 0.18339181130178053,
|
| 248 |
+
"calibration_axis_ece_part": 63.3216377396439
|
| 249 |
}
|
| 250 |
},
|
| 251 |
"judge-proxy-v1": {
|
| 252 |
"original": {
|
| 253 |
"questions": 600,
|
| 254 |
+
"accuracy": 0.8983333333333333,
|
| 255 |
+
"ece_10bin": 0.03979876869189099,
|
| 256 |
+
"calibration_axis_ece_part": 92.0402462616218
|
| 257 |
},
|
| 258 |
"rotation": {
|
| 259 |
"questions": 600,
|
| 260 |
+
"accuracy": 0.8983333333333333,
|
| 261 |
+
"ece_10bin": 0.03979876869189099,
|
| 262 |
+
"calibration_axis_ece_part": 92.0402462616218
|
| 263 |
}
|
| 264 |
},
|
| 265 |
"internal-eval-v2-realistic": {
|
| 266 |
"original": {
|
| 267 |
"questions": 600,
|
| 268 |
+
"accuracy": 0.9,
|
| 269 |
+
"ece_10bin": 0.0736220706476437,
|
| 270 |
+
"calibration_axis_ece_part": 85.27558587047126,
|
| 271 |
"probability_items": 120,
|
| 272 |
+
"mean_tv_distance": 0.11236686629256984,
|
| 273 |
+
"fidelity_part": 88.76331337074302,
|
| 274 |
+
"calibration_axis_estimate": 87.01944962060713
|
| 275 |
},
|
| 276 |
"rotation": {
|
| 277 |
"questions": 600,
|
| 278 |
+
"accuracy": 0.9,
|
| 279 |
+
"ece_10bin": 0.0736220706476437,
|
| 280 |
+
"calibration_axis_ece_part": 85.27558587047126,
|
| 281 |
"probability_items": 120,
|
| 282 |
+
"mean_tv_distance": 0.11236686629256984,
|
| 283 |
+
"fidelity_part": 88.76331337074302,
|
| 284 |
+
"calibration_axis_estimate": 87.01944962060713
|
| 285 |
}
|
| 286 |
},
|
| 287 |
"distprob-val-v2": {
|
| 288 |
"original": {
|
| 289 |
"questions": 1036,
|
| 290 |
+
"accuracy": 0.8262548262548263,
|
| 291 |
+
"ece_10bin": 0.21796425974031988,
|
| 292 |
+
"calibration_axis_ece_part": 56.407148051936026,
|
| 293 |
"probability_items": 1036,
|
| 294 |
+
"mean_tv_distance": 0.09967736998468674,
|
| 295 |
+
"fidelity_part": 90.03226300153133,
|
| 296 |
+
"calibration_axis_estimate": 73.21970552673368
|
| 297 |
},
|
| 298 |
"rotation": {
|
| 299 |
"questions": 1036,
|
| 300 |
+
"accuracy": 0.8262548262548263,
|
| 301 |
+
"ece_10bin": 0.21796425974031988,
|
| 302 |
+
"calibration_axis_ece_part": 56.407148051936026,
|
| 303 |
"probability_items": 1036,
|
| 304 |
+
"mean_tv_distance": 0.09967736998468674,
|
| 305 |
+
"fidelity_part": 90.03226300153133,
|
| 306 |
+
"calibration_axis_estimate": 73.21970552673368
|
| 307 |
}
|
| 308 |
},
|
| 309 |
"internal-eval-v1": {
|
| 310 |
"original": {
|
| 311 |
"questions": 600,
|
| 312 |
+
"accuracy": 0.77,
|
| 313 |
+
"ece_10bin": 0.10198733183329939,
|
| 314 |
+
"calibration_axis_ece_part": 79.60253363334012
|
| 315 |
},
|
| 316 |
"rotation": {
|
| 317 |
"questions": 600,
|
| 318 |
+
"accuracy": 0.77,
|
| 319 |
+
"ece_10bin": 0.10198733183329939,
|
| 320 |
+
"calibration_axis_ece_part": 79.60253363334012
|
| 321 |
}
|
| 322 |
},
|
| 323 |
"internal-eval-hardproxy-v1": {
|
| 324 |
"original": {
|
| 325 |
"questions": 600,
|
| 326 |
+
"accuracy": 0.545,
|
| 327 |
+
"ece_10bin": 0.04782078898436224,
|
| 328 |
+
"calibration_axis_ece_part": 90.43584220312755
|
| 329 |
},
|
| 330 |
"rotation": {
|
| 331 |
"questions": 600,
|
| 332 |
+
"accuracy": 0.545,
|
| 333 |
+
"ece_10bin": 0.04782078898436224,
|
| 334 |
+
"calibration_axis_ece_part": 90.43584220312755
|
| 335 |
}
|
| 336 |
},
|
| 337 |
"games-harness-eval-v1": {
|
| 338 |
"original": {
|
| 339 |
"questions": 360,
|
| 340 |
+
"accuracy": 0.975,
|
| 341 |
+
"ece_10bin": 0.03682651060774796,
|
| 342 |
+
"calibration_axis_ece_part": 92.63469787845041
|
| 343 |
},
|
| 344 |
"rotation": {
|
| 345 |
"questions": 360,
|
| 346 |
+
"accuracy": 0.975,
|
| 347 |
+
"ece_10bin": 0.03682651060774796,
|
| 348 |
+
"calibration_axis_ece_part": 92.63469787845041
|
| 349 |
}
|
| 350 |
},
|
| 351 |
"tetris-drought-eval-v1": {
|
| 352 |
"original": {
|
| 353 |
"questions": 120,
|
| 354 |
+
"accuracy": 0.6666666666666666,
|
| 355 |
+
"ece_10bin": 0.1250689728260685,
|
| 356 |
+
"calibration_axis_ece_part": 74.9862054347863
|
| 357 |
},
|
| 358 |
"rotation": {
|
| 359 |
"questions": 120,
|
| 360 |
+
"accuracy": 0.6666666666666666,
|
| 361 |
+
"ece_10bin": 0.1250689728260685,
|
| 362 |
+
"calibration_axis_ece_part": 74.9862054347863
|
| 363 |
}
|
| 364 |
},
|
| 365 |
"internal-eval-hardproxy-calib-v1": {
|
| 366 |
"original": {
|
| 367 |
"questions": 600,
|
| 368 |
+
"accuracy": 0.55,
|
| 369 |
+
"ece_10bin": 0.02832898176475044,
|
| 370 |
+
"calibration_axis_ece_part": 94.3342036470499
|
| 371 |
},
|
| 372 |
"rotation": {
|
| 373 |
"questions": 600,
|
| 374 |
+
"accuracy": 0.55,
|
| 375 |
+
"ece_10bin": 0.02832898176475044,
|
| 376 |
+
"calibration_axis_ece_part": 94.3342036470499
|
| 377 |
}
|
| 378 |
},
|
| 379 |
"distprob-calib-v2": {
|
| 380 |
"original": {
|
| 381 |
"questions": 916,
|
| 382 |
+
"accuracy": 0.8056768558951966,
|
| 383 |
+
"ece_10bin": 0.1901457810150631,
|
| 384 |
+
"calibration_axis_ece_part": 61.97084379698738,
|
| 385 |
"probability_items": 916,
|
| 386 |
+
"mean_tv_distance": 0.14275356576359963,
|
| 387 |
+
"fidelity_part": 85.72464342364003,
|
| 388 |
+
"calibration_axis_estimate": 73.8477436103137
|
| 389 |
},
|
| 390 |
"rotation": {
|
| 391 |
"questions": 916,
|
| 392 |
+
"accuracy": 0.8056768558951966,
|
| 393 |
+
"ece_10bin": 0.1901457810150631,
|
| 394 |
+
"calibration_axis_ece_part": 61.97084379698738,
|
| 395 |
"probability_items": 916,
|
| 396 |
+
"mean_tv_distance": 0.14275356576359963,
|
| 397 |
+
"fidelity_part": 85.72464342364003,
|
| 398 |
+
"calibration_axis_estimate": 73.8477436103137
|
| 399 |
}
|
| 400 |
}
|
| 401 |
}
|
evidence/calibration-per-type-served.json
ADDED
|
@@ -0,0 +1,537 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"arm_fitted": "original",
|
| 3 |
+
"inputs": [
|
| 4 |
+
"runs/served-v2r5-8b-native/predictions.jsonl"
|
| 5 |
+
],
|
| 6 |
+
"fit_suites": [
|
| 7 |
+
"calib-public-v1",
|
| 8 |
+
"distprob-calib-v2",
|
| 9 |
+
"internal-eval-hardproxy-calib-v1"
|
| 10 |
+
],
|
| 11 |
+
"temperature": {
|
| 12 |
+
"*": 0.8500000000000001,
|
| 13 |
+
"choice": 0.8500000000000001,
|
| 14 |
+
"noul": 0.8500000000000001,
|
| 15 |
+
"score": 0.7
|
| 16 |
+
},
|
| 17 |
+
"per_type": {
|
| 18 |
+
"*": {
|
| 19 |
+
"temperature": 0.8500000000000001,
|
| 20 |
+
"axis_proxy_at_T": 90.31624912014627
|
| 21 |
+
},
|
| 22 |
+
"choice": {
|
| 23 |
+
"fit_items": 1648,
|
| 24 |
+
"temperature": 0.8500000000000001
|
| 25 |
+
},
|
| 26 |
+
"noul": {
|
| 27 |
+
"fit_items": 782,
|
| 28 |
+
"temperature": 0.8500000000000001
|
| 29 |
+
},
|
| 30 |
+
"score": {
|
| 31 |
+
"fit_items": 286,
|
| 32 |
+
"temperature": 0.7
|
| 33 |
+
},
|
| 34 |
+
"combined_axis_proxy": 90.4802401453947
|
| 35 |
+
},
|
| 36 |
+
"objective": "leaderboard",
|
| 37 |
+
"at_T1": {
|
| 38 |
+
"jevbench-easy": {
|
| 39 |
+
"original": {
|
| 40 |
+
"questions": 48,
|
| 41 |
+
"accuracy": 1.0,
|
| 42 |
+
"ece_10bin": 0.05131341842374401,
|
| 43 |
+
"calibration_axis_ece_part": 89.73731631525119
|
| 44 |
+
},
|
| 45 |
+
"rotation": {
|
| 46 |
+
"questions": 48,
|
| 47 |
+
"accuracy": 1.0,
|
| 48 |
+
"ece_10bin": 0.05131341842374401,
|
| 49 |
+
"calibration_axis_ece_part": 89.73731631525119
|
| 50 |
+
}
|
| 51 |
+
},
|
| 52 |
+
"jevbench-original": {
|
| 53 |
+
"original": {
|
| 54 |
+
"questions": 72,
|
| 55 |
+
"accuracy": 0.9305555555555556,
|
| 56 |
+
"ece_10bin": 0.08352987068619123,
|
| 57 |
+
"calibration_axis_ece_part": 83.29402586276176
|
| 58 |
+
},
|
| 59 |
+
"rotation": {
|
| 60 |
+
"questions": 72,
|
| 61 |
+
"accuracy": 0.9305555555555556,
|
| 62 |
+
"ece_10bin": 0.08352987068619123,
|
| 63 |
+
"calibration_axis_ece_part": 83.29402586276176
|
| 64 |
+
}
|
| 65 |
+
},
|
| 66 |
+
"jevbench-hard": {
|
| 67 |
+
"original": {
|
| 68 |
+
"questions": 111,
|
| 69 |
+
"accuracy": 0.5495495495495496,
|
| 70 |
+
"ece_10bin": 0.155305409757105,
|
| 71 |
+
"calibration_axis_ece_part": 68.938918048579
|
| 72 |
+
},
|
| 73 |
+
"rotation": {
|
| 74 |
+
"questions": 111,
|
| 75 |
+
"accuracy": 0.5495495495495496,
|
| 76 |
+
"ece_10bin": 0.155305409757105,
|
| 77 |
+
"calibration_axis_ece_part": 68.938918048579
|
| 78 |
+
}
|
| 79 |
+
},
|
| 80 |
+
"judge-proxy-v1": {
|
| 81 |
+
"original": {
|
| 82 |
+
"questions": 600,
|
| 83 |
+
"accuracy": 0.8933333333333333,
|
| 84 |
+
"ece_10bin": 0.03321891827960434,
|
| 85 |
+
"calibration_axis_ece_part": 93.35621634407913
|
| 86 |
+
},
|
| 87 |
+
"rotation": {
|
| 88 |
+
"questions": 600,
|
| 89 |
+
"accuracy": 0.8933333333333333,
|
| 90 |
+
"ece_10bin": 0.03321891827960434,
|
| 91 |
+
"calibration_axis_ece_part": 93.35621634407913
|
| 92 |
+
}
|
| 93 |
+
},
|
| 94 |
+
"internal-eval-hardproxy-v1": {
|
| 95 |
+
"original": {
|
| 96 |
+
"questions": 600,
|
| 97 |
+
"accuracy": 0.53,
|
| 98 |
+
"ece_10bin": 0.04558131777549149,
|
| 99 |
+
"calibration_axis_ece_part": 90.88373644490171
|
| 100 |
+
},
|
| 101 |
+
"rotation": {
|
| 102 |
+
"questions": 600,
|
| 103 |
+
"accuracy": 0.53,
|
| 104 |
+
"ece_10bin": 0.04558131777549149,
|
| 105 |
+
"calibration_axis_ece_part": 90.88373644490171
|
| 106 |
+
}
|
| 107 |
+
},
|
| 108 |
+
"internal-eval-hardproxy-calib-v1": {
|
| 109 |
+
"original": {
|
| 110 |
+
"questions": 600,
|
| 111 |
+
"accuracy": 0.5516666666666666,
|
| 112 |
+
"ece_10bin": 0.0463225213333465,
|
| 113 |
+
"calibration_axis_ece_part": 90.7354957333307
|
| 114 |
+
},
|
| 115 |
+
"rotation": {
|
| 116 |
+
"questions": 600,
|
| 117 |
+
"accuracy": 0.5516666666666666,
|
| 118 |
+
"ece_10bin": 0.0463225213333465,
|
| 119 |
+
"calibration_axis_ece_part": 90.7354957333307
|
| 120 |
+
}
|
| 121 |
+
},
|
| 122 |
+
"distprob-calib-v2": {
|
| 123 |
+
"original": {
|
| 124 |
+
"questions": 916,
|
| 125 |
+
"accuracy": 0.8078602620087336,
|
| 126 |
+
"ece_10bin": 0.2152625258729562,
|
| 127 |
+
"calibration_axis_ece_part": 56.94749482540876,
|
| 128 |
+
"probability_items": 916,
|
| 129 |
+
"mean_tv_distance": 0.15316257835395553,
|
| 130 |
+
"fidelity_part": 84.68374216460445,
|
| 131 |
+
"calibration_axis_estimate": 70.8156184950066
|
| 132 |
+
},
|
| 133 |
+
"rotation": {
|
| 134 |
+
"questions": 916,
|
| 135 |
+
"accuracy": 0.8078602620087336,
|
| 136 |
+
"ece_10bin": 0.2152625258729562,
|
| 137 |
+
"calibration_axis_ece_part": 56.94749482540876,
|
| 138 |
+
"probability_items": 916,
|
| 139 |
+
"mean_tv_distance": 0.15316257835395553,
|
| 140 |
+
"fidelity_part": 84.68374216460445,
|
| 141 |
+
"calibration_axis_estimate": 70.8156184950066
|
| 142 |
+
}
|
| 143 |
+
},
|
| 144 |
+
"distprob-val-v2": {
|
| 145 |
+
"original": {
|
| 146 |
+
"questions": 1036,
|
| 147 |
+
"accuracy": 0.8117760617760618,
|
| 148 |
+
"ece_10bin": 0.22716643540800202,
|
| 149 |
+
"calibration_axis_ece_part": 54.566712918399595,
|
| 150 |
+
"probability_items": 1036,
|
| 151 |
+
"mean_tv_distance": 0.11045876364449934,
|
| 152 |
+
"fidelity_part": 88.95412363555006,
|
| 153 |
+
"calibration_axis_estimate": 71.76041827697483
|
| 154 |
+
},
|
| 155 |
+
"rotation": {
|
| 156 |
+
"questions": 1036,
|
| 157 |
+
"accuracy": 0.8117760617760618,
|
| 158 |
+
"ece_10bin": 0.22716643540800202,
|
| 159 |
+
"calibration_axis_ece_part": 54.566712918399595,
|
| 160 |
+
"probability_items": 1036,
|
| 161 |
+
"mean_tv_distance": 0.11045876364449934,
|
| 162 |
+
"fidelity_part": 88.95412363555006,
|
| 163 |
+
"calibration_axis_estimate": 71.76041827697483
|
| 164 |
+
}
|
| 165 |
+
},
|
| 166 |
+
"internal-eval-v2-realistic": {
|
| 167 |
+
"original": {
|
| 168 |
+
"questions": 600,
|
| 169 |
+
"accuracy": 0.9083333333333333,
|
| 170 |
+
"ece_10bin": 0.10108927189999369,
|
| 171 |
+
"calibration_axis_ece_part": 79.78214562000126,
|
| 172 |
+
"probability_items": 120,
|
| 173 |
+
"mean_tv_distance": 0.11869095735250425,
|
| 174 |
+
"fidelity_part": 88.13090426474957,
|
| 175 |
+
"calibration_axis_estimate": 83.95652494237541
|
| 176 |
+
},
|
| 177 |
+
"rotation": {
|
| 178 |
+
"questions": 600,
|
| 179 |
+
"accuracy": 0.9083333333333333,
|
| 180 |
+
"ece_10bin": 0.10108927189999369,
|
| 181 |
+
"calibration_axis_ece_part": 79.78214562000126,
|
| 182 |
+
"probability_items": 120,
|
| 183 |
+
"mean_tv_distance": 0.11869095735250425,
|
| 184 |
+
"fidelity_part": 88.13090426474957,
|
| 185 |
+
"calibration_axis_estimate": 83.95652494237541
|
| 186 |
+
}
|
| 187 |
+
},
|
| 188 |
+
"musique-mc": {
|
| 189 |
+
"original": {
|
| 190 |
+
"questions": 500,
|
| 191 |
+
"accuracy": 0.776,
|
| 192 |
+
"ece_10bin": 0.1762487833910149,
|
| 193 |
+
"calibration_axis_ece_part": 64.75024332179702
|
| 194 |
+
},
|
| 195 |
+
"rotation": {
|
| 196 |
+
"questions": 500,
|
| 197 |
+
"accuracy": 0.776,
|
| 198 |
+
"ece_10bin": 0.1762487833910149,
|
| 199 |
+
"calibration_axis_ece_part": 64.75024332179702
|
| 200 |
+
}
|
| 201 |
+
},
|
| 202 |
+
"squad2-abstain": {
|
| 203 |
+
"original": {
|
| 204 |
+
"questions": 500,
|
| 205 |
+
"accuracy": 0.838,
|
| 206 |
+
"ece_10bin": 0.09252912817743919,
|
| 207 |
+
"calibration_axis_ece_part": 81.49417436451216
|
| 208 |
+
},
|
| 209 |
+
"rotation": {
|
| 210 |
+
"questions": 500,
|
| 211 |
+
"accuracy": 0.838,
|
| 212 |
+
"ece_10bin": 0.09252912817743919,
|
| 213 |
+
"calibration_axis_ece_part": 81.49417436451216
|
| 214 |
+
}
|
| 215 |
+
},
|
| 216 |
+
"contractnli": {
|
| 217 |
+
"original": {
|
| 218 |
+
"questions": 500,
|
| 219 |
+
"accuracy": 0.734,
|
| 220 |
+
"ece_10bin": 0.05028456533752318,
|
| 221 |
+
"calibration_axis_ece_part": 89.94308693249536
|
| 222 |
+
},
|
| 223 |
+
"rotation": {
|
| 224 |
+
"questions": 500,
|
| 225 |
+
"accuracy": 0.734,
|
| 226 |
+
"ece_10bin": 0.05028456533752318,
|
| 227 |
+
"calibration_axis_ece_part": 89.94308693249536
|
| 228 |
+
}
|
| 229 |
+
},
|
| 230 |
+
"paws-x-en": {
|
| 231 |
+
"original": {
|
| 232 |
+
"questions": 500,
|
| 233 |
+
"accuracy": 0.776,
|
| 234 |
+
"ece_10bin": 0.05967536000000001,
|
| 235 |
+
"calibration_axis_ece_part": 88.064928
|
| 236 |
+
},
|
| 237 |
+
"rotation": {
|
| 238 |
+
"questions": 500,
|
| 239 |
+
"accuracy": 0.776,
|
| 240 |
+
"ece_10bin": 0.05967536000000001,
|
| 241 |
+
"calibration_axis_ece_part": 88.064928
|
| 242 |
+
}
|
| 243 |
+
},
|
| 244 |
+
"hard-eval-v4": {
|
| 245 |
+
"original": {
|
| 246 |
+
"questions": 800,
|
| 247 |
+
"accuracy": 0.6125,
|
| 248 |
+
"ece_10bin": 0.06262397630161659,
|
| 249 |
+
"calibration_axis_ece_part": 87.47520473967668
|
| 250 |
+
},
|
| 251 |
+
"rotation": {
|
| 252 |
+
"questions": 800,
|
| 253 |
+
"accuracy": 0.6125,
|
| 254 |
+
"ece_10bin": 0.06262397630161659,
|
| 255 |
+
"calibration_axis_ece_part": 87.47520473967668
|
| 256 |
+
}
|
| 257 |
+
},
|
| 258 |
+
"consistency-eval-v1": {
|
| 259 |
+
"original": {
|
| 260 |
+
"questions": 1200,
|
| 261 |
+
"accuracy": 0.7791666666666667,
|
| 262 |
+
"ece_10bin": 0.07439833133746974,
|
| 263 |
+
"calibration_axis_ece_part": 85.12033373250605
|
| 264 |
+
},
|
| 265 |
+
"rotation": {
|
| 266 |
+
"questions": 1200,
|
| 267 |
+
"accuracy": 0.7791666666666667,
|
| 268 |
+
"ece_10bin": 0.07439833133746974,
|
| 269 |
+
"calibration_axis_ece_part": 85.12033373250605
|
| 270 |
+
}
|
| 271 |
+
},
|
| 272 |
+
"calib-public-v1": {
|
| 273 |
+
"original": {
|
| 274 |
+
"questions": 1200,
|
| 275 |
+
"accuracy": 0.7891666666666667,
|
| 276 |
+
"ece_10bin": 0.08250858745875968,
|
| 277 |
+
"calibration_axis_ece_part": 83.49828250824807
|
| 278 |
+
},
|
| 279 |
+
"rotation": {
|
| 280 |
+
"questions": 1200,
|
| 281 |
+
"accuracy": 0.7891666666666667,
|
| 282 |
+
"ece_10bin": 0.08250858745875968,
|
| 283 |
+
"calibration_axis_ece_part": 83.49828250824807
|
| 284 |
+
}
|
| 285 |
+
}
|
| 286 |
+
},
|
| 287 |
+
"at_T": {
|
| 288 |
+
"jevbench-easy": {
|
| 289 |
+
"original": {
|
| 290 |
+
"questions": 48,
|
| 291 |
+
"accuracy": 1.0,
|
| 292 |
+
"ece_10bin": 0.031363848289634476,
|
| 293 |
+
"calibration_axis_ece_part": 93.72723034207311
|
| 294 |
+
},
|
| 295 |
+
"rotation": {
|
| 296 |
+
"questions": 48,
|
| 297 |
+
"accuracy": 1.0,
|
| 298 |
+
"ece_10bin": 0.031363848289634476,
|
| 299 |
+
"calibration_axis_ece_part": 93.72723034207311
|
| 300 |
+
}
|
| 301 |
+
},
|
| 302 |
+
"jevbench-original": {
|
| 303 |
+
"original": {
|
| 304 |
+
"questions": 72,
|
| 305 |
+
"accuracy": 0.9305555555555556,
|
| 306 |
+
"ece_10bin": 0.05261988353339139,
|
| 307 |
+
"calibration_axis_ece_part": 89.47602329332172
|
| 308 |
+
},
|
| 309 |
+
"rotation": {
|
| 310 |
+
"questions": 72,
|
| 311 |
+
"accuracy": 0.9305555555555556,
|
| 312 |
+
"ece_10bin": 0.05261988353339139,
|
| 313 |
+
"calibration_axis_ece_part": 89.47602329332172
|
| 314 |
+
}
|
| 315 |
+
},
|
| 316 |
+
"jevbench-hard": {
|
| 317 |
+
"original": {
|
| 318 |
+
"questions": 111,
|
| 319 |
+
"accuracy": 0.5495495495495496,
|
| 320 |
+
"ece_10bin": 0.18375048168380848,
|
| 321 |
+
"calibration_axis_ece_part": 63.24990366323831
|
| 322 |
+
},
|
| 323 |
+
"rotation": {
|
| 324 |
+
"questions": 111,
|
| 325 |
+
"accuracy": 0.5495495495495496,
|
| 326 |
+
"ece_10bin": 0.18375048168380848,
|
| 327 |
+
"calibration_axis_ece_part": 63.24990366323831
|
| 328 |
+
}
|
| 329 |
+
},
|
| 330 |
+
"judge-proxy-v1": {
|
| 331 |
+
"original": {
|
| 332 |
+
"questions": 600,
|
| 333 |
+
"accuracy": 0.8933333333333333,
|
| 334 |
+
"ece_10bin": 0.047426294346464816,
|
| 335 |
+
"calibration_axis_ece_part": 90.51474113070704
|
| 336 |
+
},
|
| 337 |
+
"rotation": {
|
| 338 |
+
"questions": 600,
|
| 339 |
+
"accuracy": 0.8933333333333333,
|
| 340 |
+
"ece_10bin": 0.047426294346464816,
|
| 341 |
+
"calibration_axis_ece_part": 90.51474113070704
|
| 342 |
+
}
|
| 343 |
+
},
|
| 344 |
+
"internal-eval-hardproxy-v1": {
|
| 345 |
+
"original": {
|
| 346 |
+
"questions": 600,
|
| 347 |
+
"accuracy": 0.53,
|
| 348 |
+
"ece_10bin": 0.06331511952187803,
|
| 349 |
+
"calibration_axis_ece_part": 87.3369760956244
|
| 350 |
+
},
|
| 351 |
+
"rotation": {
|
| 352 |
+
"questions": 600,
|
| 353 |
+
"accuracy": 0.53,
|
| 354 |
+
"ece_10bin": 0.06331511952187803,
|
| 355 |
+
"calibration_axis_ece_part": 87.3369760956244
|
| 356 |
+
}
|
| 357 |
+
},
|
| 358 |
+
"internal-eval-hardproxy-calib-v1": {
|
| 359 |
+
"original": {
|
| 360 |
+
"questions": 600,
|
| 361 |
+
"accuracy": 0.5516666666666666,
|
| 362 |
+
"ece_10bin": 0.046857953345990605,
|
| 363 |
+
"calibration_axis_ece_part": 90.62840933080187
|
| 364 |
+
},
|
| 365 |
+
"rotation": {
|
| 366 |
+
"questions": 600,
|
| 367 |
+
"accuracy": 0.5516666666666666,
|
| 368 |
+
"ece_10bin": 0.046857953345990605,
|
| 369 |
+
"calibration_axis_ece_part": 90.62840933080187
|
| 370 |
+
}
|
| 371 |
+
},
|
| 372 |
+
"distprob-calib-v2": {
|
| 373 |
+
"original": {
|
| 374 |
+
"questions": 916,
|
| 375 |
+
"accuracy": 0.8078602620087336,
|
| 376 |
+
"ece_10bin": 0.1764814098828335,
|
| 377 |
+
"calibration_axis_ece_part": 64.7037180234333,
|
| 378 |
+
"probability_items": 916,
|
| 379 |
+
"mean_tv_distance": 0.1486326330918305,
|
| 380 |
+
"fidelity_part": 85.13673669081695,
|
| 381 |
+
"calibration_axis_estimate": 74.92022735712513
|
| 382 |
+
},
|
| 383 |
+
"rotation": {
|
| 384 |
+
"questions": 916,
|
| 385 |
+
"accuracy": 0.8078602620087336,
|
| 386 |
+
"ece_10bin": 0.1764814098828335,
|
| 387 |
+
"calibration_axis_ece_part": 64.7037180234333,
|
| 388 |
+
"probability_items": 916,
|
| 389 |
+
"mean_tv_distance": 0.1486326330918305,
|
| 390 |
+
"fidelity_part": 85.13673669081695,
|
| 391 |
+
"calibration_axis_estimate": 74.92022735712513
|
| 392 |
+
}
|
| 393 |
+
},
|
| 394 |
+
"distprob-val-v2": {
|
| 395 |
+
"original": {
|
| 396 |
+
"questions": 1036,
|
| 397 |
+
"accuracy": 0.8117760617760618,
|
| 398 |
+
"ece_10bin": 0.18923197859731913,
|
| 399 |
+
"calibration_axis_ece_part": 62.15360428053618,
|
| 400 |
+
"probability_items": 1036,
|
| 401 |
+
"mean_tv_distance": 0.10813236971372132,
|
| 402 |
+
"fidelity_part": 89.18676302862787,
|
| 403 |
+
"calibration_axis_estimate": 75.67018365458202
|
| 404 |
+
},
|
| 405 |
+
"rotation": {
|
| 406 |
+
"questions": 1036,
|
| 407 |
+
"accuracy": 0.8117760617760618,
|
| 408 |
+
"ece_10bin": 0.18923197859731913,
|
| 409 |
+
"calibration_axis_ece_part": 62.15360428053618,
|
| 410 |
+
"probability_items": 1036,
|
| 411 |
+
"mean_tv_distance": 0.10813236971372132,
|
| 412 |
+
"fidelity_part": 89.18676302862787,
|
| 413 |
+
"calibration_axis_estimate": 75.67018365458202
|
| 414 |
+
}
|
| 415 |
+
},
|
| 416 |
+
"internal-eval-v2-realistic": {
|
| 417 |
+
"original": {
|
| 418 |
+
"questions": 600,
|
| 419 |
+
"accuracy": 0.9083333333333333,
|
| 420 |
+
"ece_10bin": 0.0930138502276407,
|
| 421 |
+
"calibration_axis_ece_part": 81.39722995447187,
|
| 422 |
+
"probability_items": 120,
|
| 423 |
+
"mean_tv_distance": 0.11052496279849079,
|
| 424 |
+
"fidelity_part": 88.94750372015092,
|
| 425 |
+
"calibration_axis_estimate": 85.17236683731139
|
| 426 |
+
},
|
| 427 |
+
"rotation": {
|
| 428 |
+
"questions": 600,
|
| 429 |
+
"accuracy": 0.9083333333333333,
|
| 430 |
+
"ece_10bin": 0.0930138502276407,
|
| 431 |
+
"calibration_axis_ece_part": 81.39722995447187,
|
| 432 |
+
"probability_items": 120,
|
| 433 |
+
"mean_tv_distance": 0.11052496279849079,
|
| 434 |
+
"fidelity_part": 88.94750372015092,
|
| 435 |
+
"calibration_axis_estimate": 85.17236683731139
|
| 436 |
+
}
|
| 437 |
+
},
|
| 438 |
+
"musique-mc": {
|
| 439 |
+
"original": {
|
| 440 |
+
"questions": 500,
|
| 441 |
+
"accuracy": 0.776,
|
| 442 |
+
"ece_10bin": 0.12098928569367604,
|
| 443 |
+
"calibration_axis_ece_part": 75.80214286126478
|
| 444 |
+
},
|
| 445 |
+
"rotation": {
|
| 446 |
+
"questions": 500,
|
| 447 |
+
"accuracy": 0.776,
|
| 448 |
+
"ece_10bin": 0.12098928569367604,
|
| 449 |
+
"calibration_axis_ece_part": 75.80214286126478
|
| 450 |
+
}
|
| 451 |
+
},
|
| 452 |
+
"squad2-abstain": {
|
| 453 |
+
"original": {
|
| 454 |
+
"questions": 500,
|
| 455 |
+
"accuracy": 0.838,
|
| 456 |
+
"ece_10bin": 0.06032627228820835,
|
| 457 |
+
"calibration_axis_ece_part": 87.93474554235833
|
| 458 |
+
},
|
| 459 |
+
"rotation": {
|
| 460 |
+
"questions": 500,
|
| 461 |
+
"accuracy": 0.838,
|
| 462 |
+
"ece_10bin": 0.06032627228820835,
|
| 463 |
+
"calibration_axis_ece_part": 87.93474554235833
|
| 464 |
+
}
|
| 465 |
+
},
|
| 466 |
+
"contractnli": {
|
| 467 |
+
"original": {
|
| 468 |
+
"questions": 500,
|
| 469 |
+
"accuracy": 0.734,
|
| 470 |
+
"ece_10bin": 0.02339666792395878,
|
| 471 |
+
"calibration_axis_ece_part": 95.32066641520825
|
| 472 |
+
},
|
| 473 |
+
"rotation": {
|
| 474 |
+
"questions": 500,
|
| 475 |
+
"accuracy": 0.734,
|
| 476 |
+
"ece_10bin": 0.02339666792395878,
|
| 477 |
+
"calibration_axis_ece_part": 95.32066641520825
|
| 478 |
+
}
|
| 479 |
+
},
|
| 480 |
+
"paws-x-en": {
|
| 481 |
+
"original": {
|
| 482 |
+
"questions": 500,
|
| 483 |
+
"accuracy": 0.776,
|
| 484 |
+
"ece_10bin": 0.05241691245501016,
|
| 485 |
+
"calibration_axis_ece_part": 89.51661750899797
|
| 486 |
+
},
|
| 487 |
+
"rotation": {
|
| 488 |
+
"questions": 500,
|
| 489 |
+
"accuracy": 0.776,
|
| 490 |
+
"ece_10bin": 0.05241691245501016,
|
| 491 |
+
"calibration_axis_ece_part": 89.51661750899797
|
| 492 |
+
}
|
| 493 |
+
},
|
| 494 |
+
"hard-eval-v4": {
|
| 495 |
+
"original": {
|
| 496 |
+
"questions": 800,
|
| 497 |
+
"accuracy": 0.6125,
|
| 498 |
+
"ece_10bin": 0.09082334730679965,
|
| 499 |
+
"calibration_axis_ece_part": 81.83533053864008
|
| 500 |
+
},
|
| 501 |
+
"rotation": {
|
| 502 |
+
"questions": 800,
|
| 503 |
+
"accuracy": 0.6125,
|
| 504 |
+
"ece_10bin": 0.09082334730679965,
|
| 505 |
+
"calibration_axis_ece_part": 81.83533053864008
|
| 506 |
+
}
|
| 507 |
+
},
|
| 508 |
+
"consistency-eval-v1": {
|
| 509 |
+
"original": {
|
| 510 |
+
"questions": 1200,
|
| 511 |
+
"accuracy": 0.7791666666666667,
|
| 512 |
+
"ece_10bin": 0.034963130144631066,
|
| 513 |
+
"calibration_axis_ece_part": 93.00737397107379
|
| 514 |
+
},
|
| 515 |
+
"rotation": {
|
| 516 |
+
"questions": 1200,
|
| 517 |
+
"accuracy": 0.7791666666666667,
|
| 518 |
+
"ece_10bin": 0.034963130144631066,
|
| 519 |
+
"calibration_axis_ece_part": 93.00737397107379
|
| 520 |
+
}
|
| 521 |
+
},
|
| 522 |
+
"calib-public-v1": {
|
| 523 |
+
"original": {
|
| 524 |
+
"questions": 1200,
|
| 525 |
+
"accuracy": 0.7891666666666667,
|
| 526 |
+
"ece_10bin": 0.04602219448214145,
|
| 527 |
+
"calibration_axis_ece_part": 90.79556110357171
|
| 528 |
+
},
|
| 529 |
+
"rotation": {
|
| 530 |
+
"questions": 1200,
|
| 531 |
+
"accuracy": 0.7891666666666667,
|
| 532 |
+
"ece_10bin": 0.04602219448214145,
|
| 533 |
+
"calibration_axis_ece_part": 90.79556110357171
|
| 534 |
+
}
|
| 535 |
+
}
|
| 536 |
+
}
|
| 537 |
+
}
|
evidence/prompt-wording-choice.json
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"method": "decision on the unweighted mean accuracy over choice_suites (served predictions): native unless served leads by >= 1.0 pp and the offline cross-check agrees; report_only suites play no part",
|
| 3 |
+
"choice_suites": [
|
| 4 |
+
"judge-proxy-v1",
|
| 5 |
+
"internal-eval-hardproxy-v1",
|
| 6 |
+
"distprob-val-v2",
|
| 7 |
+
"internal-eval-v2-realistic",
|
| 8 |
+
"musique-mc",
|
| 9 |
+
"squad2-abstain",
|
| 10 |
+
"contractnli",
|
| 11 |
+
"paws-x-en",
|
| 12 |
+
"hard-eval-v4",
|
| 13 |
+
"consistency-eval-v1"
|
| 14 |
+
],
|
| 15 |
+
"report_only_suites": [
|
| 16 |
+
"jevbench-easy",
|
| 17 |
+
"jevbench-original",
|
| 18 |
+
"jevbench-hard"
|
| 19 |
+
],
|
| 20 |
+
"size": "8B",
|
| 21 |
+
"basis": "served predictions (SGLang 0.5.20 + jev-adapter, no system prompt, T=1, one option order)",
|
| 22 |
+
"arms": {
|
| 23 |
+
"native": {
|
| 24 |
+
"wording": "native (jev-adapter --prompt-wording native --native-system-prompt none)",
|
| 25 |
+
"served_choice_suites": {
|
| 26 |
+
"judge-proxy-v1": 89.33,
|
| 27 |
+
"internal-eval-hardproxy-v1": 53.0,
|
| 28 |
+
"distprob-val-v2": 81.18,
|
| 29 |
+
"internal-eval-v2-realistic": 90.83,
|
| 30 |
+
"musique-mc": 77.6,
|
| 31 |
+
"squad2-abstain": 83.8,
|
| 32 |
+
"contractnli": 73.4,
|
| 33 |
+
"paws-x-en": 77.6,
|
| 34 |
+
"hard-eval-v4": 61.25,
|
| 35 |
+
"consistency-eval-v1": 77.92
|
| 36 |
+
},
|
| 37 |
+
"served_mean": 76.591,
|
| 38 |
+
"report_only_jevbench_served": {
|
| 39 |
+
"jevbench-easy": 100.0,
|
| 40 |
+
"jevbench-original": 93.06,
|
| 41 |
+
"jevbench-hard": 54.95
|
| 42 |
+
},
|
| 43 |
+
"per_type_T": {
|
| 44 |
+
"choice": 0.85,
|
| 45 |
+
"noul": 0.85,
|
| 46 |
+
"score": 0.7
|
| 47 |
+
}
|
| 48 |
+
},
|
| 49 |
+
"served": {
|
| 50 |
+
"wording": "served (jev-adapter --prompt-wording served)",
|
| 51 |
+
"served_choice_suites": {
|
| 52 |
+
"judge-proxy-v1": 90.33,
|
| 53 |
+
"internal-eval-hardproxy-v1": 51.67,
|
| 54 |
+
"distprob-val-v2": 81.85,
|
| 55 |
+
"internal-eval-v2-realistic": 90.0,
|
| 56 |
+
"musique-mc": 77.2,
|
| 57 |
+
"squad2-abstain": 83.2,
|
| 58 |
+
"contractnli": 72.2,
|
| 59 |
+
"paws-x-en": 77.2,
|
| 60 |
+
"hard-eval-v4": 62.62,
|
| 61 |
+
"consistency-eval-v1": 77.83
|
| 62 |
+
},
|
| 63 |
+
"served_mean": 76.41,
|
| 64 |
+
"report_only_jevbench_served": {
|
| 65 |
+
"jevbench-easy": 100.0,
|
| 66 |
+
"jevbench-original": 98.61,
|
| 67 |
+
"jevbench-hard": 56.76
|
| 68 |
+
},
|
| 69 |
+
"per_type_T": {
|
| 70 |
+
"choice": 0.8,
|
| 71 |
+
"noul": 0.9,
|
| 72 |
+
"score": 0.95
|
| 73 |
+
}
|
| 74 |
+
}
|
| 75 |
+
},
|
| 76 |
+
"chosen": "native",
|
| 77 |
+
"chosen_wording": "native (jev-adapter --prompt-wording native --native-system-prompt none)",
|
| 78 |
+
"margin_pp": 0.181,
|
| 79 |
+
"rule": "served only if its served mean leads native by >= 1.0 pp and the offline check agrees; otherwise native",
|
| 80 |
+
"served_lead_pp": -0.181
|
| 81 |
+
}
|
evidence/served-nosys-easy-report.json
CHANGED
|
@@ -8,22 +8,22 @@
|
|
| 8 |
"served_models": [
|
| 9 |
"decision-model"
|
| 10 |
],
|
| 11 |
-
"elapsed_measurement_s": 2.
|
| 12 |
-
"requests_per_second":
|
| 13 |
-
"questions_per_second":
|
| 14 |
"latency_ms": {
|
| 15 |
"n": 48,
|
| 16 |
-
"mean":
|
| 17 |
-
"p50": 42.
|
| 18 |
-
"p95":
|
| 19 |
-
"p99":
|
| 20 |
},
|
| 21 |
"adapter_ms": {
|
| 22 |
"n": 48,
|
| 23 |
-
"mean":
|
| 24 |
-
"p50": 40.
|
| 25 |
-
"p95":
|
| 26 |
-
"p99":
|
| 27 |
},
|
| 28 |
"input_tokens_per_request": {
|
| 29 |
"n": 48,
|
|
@@ -37,13 +37,13 @@
|
|
| 37 |
"clean": {
|
| 38 |
"n": 48,
|
| 39 |
"acc": 1.0,
|
| 40 |
-
"nll": 0.
|
| 41 |
-
"brier":
|
| 42 |
-
"ece": 0.
|
| 43 |
-
"mean_conf": 0.
|
| 44 |
-
"confidence_bias": -0.
|
| 45 |
"confident_error_rate": 0.0,
|
| 46 |
-
"coverage_at_0_9":
|
| 47 |
"accuracy_at_0_9": 1.0,
|
| 48 |
"coverage_at_5pct_error": 1.0,
|
| 49 |
"coverage_at_1pct_error": 1.0
|
|
@@ -52,13 +52,13 @@
|
|
| 52 |
"extraction": {
|
| 53 |
"n": 12,
|
| 54 |
"acc": 1.0,
|
| 55 |
-
"nll": 0.
|
| 56 |
-
"brier":
|
| 57 |
-
"ece": 0.
|
| 58 |
-
"mean_conf": 0.
|
| 59 |
-
"confidence_bias": -0.
|
| 60 |
"confident_error_rate": 0.0,
|
| 61 |
-
"coverage_at_0_9":
|
| 62 |
"accuracy_at_0_9": 1.0,
|
| 63 |
"coverage_at_5pct_error": 1.0,
|
| 64 |
"coverage_at_1pct_error": 1.0
|
|
@@ -66,11 +66,11 @@
|
|
| 66 |
"fact": {
|
| 67 |
"n": 12,
|
| 68 |
"acc": 1.0,
|
| 69 |
-
"nll": 0.
|
| 70 |
-
"brier":
|
| 71 |
-
"ece": 0.
|
| 72 |
-
"mean_conf": 0.
|
| 73 |
-
"confidence_bias": -0.
|
| 74 |
"confident_error_rate": 0.0,
|
| 75 |
"coverage_at_0_9": 1.0,
|
| 76 |
"accuracy_at_0_9": 1.0,
|
|
@@ -80,11 +80,11 @@
|
|
| 80 |
"intent": {
|
| 81 |
"n": 12,
|
| 82 |
"acc": 1.0,
|
| 83 |
-
"nll":
|
| 84 |
-
"brier":
|
| 85 |
-
"ece":
|
| 86 |
-
"mean_conf": 0.
|
| 87 |
-
"confidence_bias": -
|
| 88 |
"confident_error_rate": 0.0,
|
| 89 |
"coverage_at_0_9": 1.0,
|
| 90 |
"accuracy_at_0_9": 1.0,
|
|
@@ -94,11 +94,11 @@
|
|
| 94 |
"tool_selection": {
|
| 95 |
"n": 12,
|
| 96 |
"acc": 1.0,
|
| 97 |
-
"nll":
|
| 98 |
-
"brier":
|
| 99 |
-
"ece":
|
| 100 |
-
"mean_conf": 0.
|
| 101 |
-
"confidence_bias": -
|
| 102 |
"confident_error_rate": 0.0,
|
| 103 |
"coverage_at_0_9": 1.0,
|
| 104 |
"accuracy_at_0_9": 1.0,
|
|
@@ -110,13 +110,13 @@
|
|
| 110 |
"extraction": {
|
| 111 |
"n": 12,
|
| 112 |
"acc": 1.0,
|
| 113 |
-
"nll": 0.
|
| 114 |
-
"brier":
|
| 115 |
-
"ece": 0.
|
| 116 |
-
"mean_conf": 0.
|
| 117 |
-
"confidence_bias": -0.
|
| 118 |
"confident_error_rate": 0.0,
|
| 119 |
-
"coverage_at_0_9":
|
| 120 |
"accuracy_at_0_9": 1.0,
|
| 121 |
"coverage_at_5pct_error": 1.0,
|
| 122 |
"coverage_at_1pct_error": 1.0
|
|
@@ -124,11 +124,11 @@
|
|
| 124 |
"fact": {
|
| 125 |
"n": 12,
|
| 126 |
"acc": 1.0,
|
| 127 |
-
"nll": 0.
|
| 128 |
-
"brier":
|
| 129 |
-
"ece": 0.
|
| 130 |
-
"mean_conf": 0.
|
| 131 |
-
"confidence_bias": -0.
|
| 132 |
"confident_error_rate": 0.0,
|
| 133 |
"coverage_at_0_9": 1.0,
|
| 134 |
"accuracy_at_0_9": 1.0,
|
|
@@ -138,11 +138,11 @@
|
|
| 138 |
"intent": {
|
| 139 |
"n": 12,
|
| 140 |
"acc": 1.0,
|
| 141 |
-
"nll":
|
| 142 |
-
"brier":
|
| 143 |
-
"ece":
|
| 144 |
-
"mean_conf": 0.
|
| 145 |
-
"confidence_bias": -
|
| 146 |
"confident_error_rate": 0.0,
|
| 147 |
"coverage_at_0_9": 1.0,
|
| 148 |
"accuracy_at_0_9": 1.0,
|
|
@@ -152,11 +152,11 @@
|
|
| 152 |
"tool_selection": {
|
| 153 |
"n": 12,
|
| 154 |
"acc": 1.0,
|
| 155 |
-
"nll":
|
| 156 |
-
"brier":
|
| 157 |
-
"ece":
|
| 158 |
-
"mean_conf": 0.
|
| 159 |
-
"confidence_bias": -
|
| 160 |
"confident_error_rate": 0.0,
|
| 161 |
"coverage_at_0_9": 1.0,
|
| 162 |
"accuracy_at_0_9": 1.0,
|
|
@@ -168,20 +168,20 @@
|
|
| 168 |
"clean": {
|
| 169 |
"n": 48,
|
| 170 |
"acc": 1.0,
|
| 171 |
-
"nll": 0.
|
| 172 |
-
"brier":
|
| 173 |
-
"ece": 0.
|
| 174 |
-
"mean_conf": 0.
|
| 175 |
-
"confidence_bias": -0.
|
| 176 |
"confident_error_rate": 0.0,
|
| 177 |
-
"coverage_at_0_9":
|
| 178 |
"accuracy_at_0_9": 1.0,
|
| 179 |
"coverage_at_5pct_error": 1.0,
|
| 180 |
"coverage_at_1pct_error": 1.0
|
| 181 |
}
|
| 182 |
},
|
| 183 |
"macro_task_acc": 1.0,
|
| 184 |
-
"objective": -0.
|
| 185 |
"paired_flip": {
|
| 186 |
"pairs": 0,
|
| 187 |
"incomplete_pairs": 0,
|
|
@@ -211,10 +211,10 @@
|
|
| 211 |
},
|
| 212 |
"latency_ms": {
|
| 213 |
"n": 48,
|
| 214 |
-
"mean":
|
| 215 |
-
"p50": 42.
|
| 216 |
-
"p95":
|
| 217 |
-
"p99":
|
| 218 |
}
|
| 219 |
}
|
| 220 |
},
|
|
|
|
| 8 |
"served_models": [
|
| 9 |
"decision-model"
|
| 10 |
],
|
| 11 |
+
"elapsed_measurement_s": 2.2737864300142974,
|
| 12 |
+
"requests_per_second": 21.11016204793613,
|
| 13 |
+
"questions_per_second": 21.11016204793613,
|
| 14 |
"latency_ms": {
|
| 15 |
"n": 48,
|
| 16 |
+
"mean": 47.24399989330171,
|
| 17 |
+
"p50": 42.06029849592596,
|
| 18 |
+
"p95": 82.6151950052008,
|
| 19 |
+
"p99": 90.90929455996958
|
| 20 |
},
|
| 21 |
"adapter_ms": {
|
| 22 |
"n": 48,
|
| 23 |
+
"mean": 45.70769293059129,
|
| 24 |
+
"p50": 40.59192197746597,
|
| 25 |
+
"p95": 81.13628662831616,
|
| 26 |
+
"p99": 89.3814675655449
|
| 27 |
},
|
| 28 |
"input_tokens_per_request": {
|
| 29 |
"n": 48,
|
|
|
|
| 37 |
"clean": {
|
| 38 |
"n": 48,
|
| 39 |
"acc": 1.0,
|
| 40 |
+
"nll": 0.05602756001861146,
|
| 41 |
+
"brier": 0.010436963591683486,
|
| 42 |
+
"ece": 0.05139835018305874,
|
| 43 |
+
"mean_conf": 0.9486016498169413,
|
| 44 |
+
"confidence_bias": -0.051398350183058716,
|
| 45 |
"confident_error_rate": 0.0,
|
| 46 |
+
"coverage_at_0_9": 0.7708333333333334,
|
| 47 |
"accuracy_at_0_9": 1.0,
|
| 48 |
"coverage_at_5pct_error": 1.0,
|
| 49 |
"coverage_at_1pct_error": 1.0
|
|
|
|
| 52 |
"extraction": {
|
| 53 |
"n": 12,
|
| 54 |
"acc": 1.0,
|
| 55 |
+
"nll": 0.1631958391839688,
|
| 56 |
+
"brier": 0.03874857951547743,
|
| 57 |
+
"ece": 0.1456061756692298,
|
| 58 |
+
"mean_conf": 0.8543938243307702,
|
| 59 |
+
"confidence_bias": -0.14560617566922984,
|
| 60 |
"confident_error_rate": 0.0,
|
| 61 |
+
"coverage_at_0_9": 0.08333333333333333,
|
| 62 |
"accuracy_at_0_9": 1.0,
|
| 63 |
"coverage_at_5pct_error": 1.0,
|
| 64 |
"coverage_at_1pct_error": 1.0
|
|
|
|
| 66 |
"fact": {
|
| 67 |
"n": 12,
|
| 68 |
"acc": 1.0,
|
| 69 |
+
"nll": 0.02283396445572382,
|
| 70 |
+
"brier": 0.001844080081996955,
|
| 71 |
+
"ece": 0.022352008246930843,
|
| 72 |
+
"mean_conf": 0.9776479917530692,
|
| 73 |
+
"confidence_bias": -0.022352008246930843,
|
| 74 |
"confident_error_rate": 0.0,
|
| 75 |
"coverage_at_0_9": 1.0,
|
| 76 |
"accuracy_at_0_9": 1.0,
|
|
|
|
| 80 |
"intent": {
|
| 81 |
"n": 12,
|
| 82 |
"acc": 1.0,
|
| 83 |
+
"nll": 0.022857902830035415,
|
| 84 |
+
"brier": 0.0008136618646276942,
|
| 85 |
+
"ece": 0.02254590948808477,
|
| 86 |
+
"mean_conf": 0.9774540905119152,
|
| 87 |
+
"confidence_bias": -0.02254590948808477,
|
| 88 |
"confident_error_rate": 0.0,
|
| 89 |
"coverage_at_0_9": 1.0,
|
| 90 |
"accuracy_at_0_9": 1.0,
|
|
|
|
| 94 |
"tool_selection": {
|
| 95 |
"n": 12,
|
| 96 |
"acc": 1.0,
|
| 97 |
+
"nll": 0.015222533604717798,
|
| 98 |
+
"brier": 0.0003415329046318565,
|
| 99 |
+
"ece": 0.015089307327989299,
|
| 100 |
+
"mean_conf": 0.9849106926720107,
|
| 101 |
+
"confidence_bias": -0.015089307327989299,
|
| 102 |
"confident_error_rate": 0.0,
|
| 103 |
"coverage_at_0_9": 1.0,
|
| 104 |
"accuracy_at_0_9": 1.0,
|
|
|
|
| 110 |
"extraction": {
|
| 111 |
"n": 12,
|
| 112 |
"acc": 1.0,
|
| 113 |
+
"nll": 0.1631958391839688,
|
| 114 |
+
"brier": 0.03874857951547743,
|
| 115 |
+
"ece": 0.1456061756692298,
|
| 116 |
+
"mean_conf": 0.8543938243307702,
|
| 117 |
+
"confidence_bias": -0.14560617566922984,
|
| 118 |
"confident_error_rate": 0.0,
|
| 119 |
+
"coverage_at_0_9": 0.08333333333333333,
|
| 120 |
"accuracy_at_0_9": 1.0,
|
| 121 |
"coverage_at_5pct_error": 1.0,
|
| 122 |
"coverage_at_1pct_error": 1.0
|
|
|
|
| 124 |
"fact": {
|
| 125 |
"n": 12,
|
| 126 |
"acc": 1.0,
|
| 127 |
+
"nll": 0.02283396445572382,
|
| 128 |
+
"brier": 0.001844080081996955,
|
| 129 |
+
"ece": 0.022352008246930843,
|
| 130 |
+
"mean_conf": 0.9776479917530692,
|
| 131 |
+
"confidence_bias": -0.022352008246930843,
|
| 132 |
"confident_error_rate": 0.0,
|
| 133 |
"coverage_at_0_9": 1.0,
|
| 134 |
"accuracy_at_0_9": 1.0,
|
|
|
|
| 138 |
"intent": {
|
| 139 |
"n": 12,
|
| 140 |
"acc": 1.0,
|
| 141 |
+
"nll": 0.022857902830035415,
|
| 142 |
+
"brier": 0.0008136618646276942,
|
| 143 |
+
"ece": 0.02254590948808477,
|
| 144 |
+
"mean_conf": 0.9774540905119152,
|
| 145 |
+
"confidence_bias": -0.02254590948808477,
|
| 146 |
"confident_error_rate": 0.0,
|
| 147 |
"coverage_at_0_9": 1.0,
|
| 148 |
"accuracy_at_0_9": 1.0,
|
|
|
|
| 152 |
"tool_selection": {
|
| 153 |
"n": 12,
|
| 154 |
"acc": 1.0,
|
| 155 |
+
"nll": 0.015222533604717798,
|
| 156 |
+
"brier": 0.0003415329046318565,
|
| 157 |
+
"ece": 0.015089307327989299,
|
| 158 |
+
"mean_conf": 0.9849106926720107,
|
| 159 |
+
"confidence_bias": -0.015089307327989299,
|
| 160 |
"confident_error_rate": 0.0,
|
| 161 |
"coverage_at_0_9": 1.0,
|
| 162 |
"accuracy_at_0_9": 1.0,
|
|
|
|
| 168 |
"clean": {
|
| 169 |
"n": 48,
|
| 170 |
"acc": 1.0,
|
| 171 |
+
"nll": 0.05602756001861146,
|
| 172 |
+
"brier": 0.010436963591683486,
|
| 173 |
+
"ece": 0.05139835018305874,
|
| 174 |
+
"mean_conf": 0.9486016498169413,
|
| 175 |
+
"confidence_bias": -0.051398350183058716,
|
| 176 |
"confident_error_rate": 0.0,
|
| 177 |
+
"coverage_at_0_9": 0.7708333333333334,
|
| 178 |
"accuracy_at_0_9": 1.0,
|
| 179 |
"coverage_at_5pct_error": 1.0,
|
| 180 |
"coverage_at_1pct_error": 1.0
|
| 181 |
}
|
| 182 |
},
|
| 183 |
"macro_task_acc": 1.0,
|
| 184 |
+
"objective": -0.05602756001861146,
|
| 185 |
"paired_flip": {
|
| 186 |
"pairs": 0,
|
| 187 |
"incomplete_pairs": 0,
|
|
|
|
| 211 |
},
|
| 212 |
"latency_ms": {
|
| 213 |
"n": 48,
|
| 214 |
+
"mean": 47.24399989330171,
|
| 215 |
+
"p50": 42.06029849592596,
|
| 216 |
+
"p95": 82.6151950052008,
|
| 217 |
+
"p99": 90.90929455996958
|
| 218 |
}
|
| 219 |
}
|
| 220 |
},
|
evidence/served-nosys-hard-report.json
CHANGED
|
@@ -8,22 +8,22 @@
|
|
| 8 |
"served_models": [
|
| 9 |
"decision-model"
|
| 10 |
],
|
| 11 |
-
"elapsed_measurement_s":
|
| 12 |
-
"requests_per_second":
|
| 13 |
-
"questions_per_second":
|
| 14 |
"latency_ms": {
|
| 15 |
"n": 111,
|
| 16 |
-
"mean":
|
| 17 |
-
"p50":
|
| 18 |
-
"p95":
|
| 19 |
-
"p99":
|
| 20 |
},
|
| 21 |
"adapter_ms": {
|
| 22 |
"n": 111,
|
| 23 |
-
"mean":
|
| 24 |
-
"p50":
|
| 25 |
-
"p95":
|
| 26 |
-
"p99":
|
| 27 |
},
|
| 28 |
"input_tokens_per_request": {
|
| 29 |
"n": 111,
|
|
@@ -36,119 +36,119 @@
|
|
| 36 |
"jevbench-hard/public": {
|
| 37 |
"clean": {
|
| 38 |
"n": 111,
|
| 39 |
-
"acc": 0.
|
| 40 |
-
"nll":
|
| 41 |
-
"brier": 0.
|
| 42 |
-
"ece": 0.
|
| 43 |
-
"mean_conf": 0.
|
| 44 |
-
"confidence_bias": 0.
|
| 45 |
-
"confident_error_rate": 0.
|
| 46 |
-
"coverage_at_0_9": 0.
|
| 47 |
-
"accuracy_at_0_9": 0.
|
| 48 |
-
"coverage_at_5pct_error": 0.
|
| 49 |
-
"coverage_at_1pct_error": 0.
|
| 50 |
-
"score_mae": 0.
|
| 51 |
-
"ranked_probability_score": 0.
|
| 52 |
},
|
| 53 |
"tasks": {
|
| 54 |
"adversarial": {
|
| 55 |
"n": 6,
|
| 56 |
-
"acc":
|
| 57 |
-
"nll": 0.
|
| 58 |
-
"brier": 0.
|
| 59 |
-
"ece": 0.
|
| 60 |
-
"mean_conf": 0.
|
| 61 |
-
"confidence_bias": -0.
|
| 62 |
"confident_error_rate": 0.0,
|
| 63 |
-
"coverage_at_0_9": 0.
|
| 64 |
"accuracy_at_0_9": 1.0,
|
| 65 |
-
"coverage_at_5pct_error":
|
| 66 |
-
"coverage_at_1pct_error":
|
| 67 |
-
"score_mae": 0.
|
| 68 |
-
"ranked_probability_score": 0.
|
| 69 |
},
|
| 70 |
"ambiguous": {
|
| 71 |
"n": 7,
|
| 72 |
-
"acc": 0.
|
| 73 |
-
"nll": 1.
|
| 74 |
-
"brier": 0.
|
| 75 |
-
"ece": 0.
|
| 76 |
-
"mean_conf": 0.
|
| 77 |
-
"confidence_bias": 0.
|
| 78 |
-
"confident_error_rate": 0.
|
| 79 |
-
"coverage_at_0_9": 0.
|
| 80 |
-
"accuracy_at_0_9":
|
| 81 |
"coverage_at_5pct_error": 0.14285714285714285,
|
| 82 |
"coverage_at_1pct_error": 0.14285714285714285
|
| 83 |
},
|
| 84 |
"judge_hard": {
|
| 85 |
"n": 17,
|
| 86 |
"acc": 0.7647058823529411,
|
| 87 |
-
"nll": 0.
|
| 88 |
-
"brier": 0.
|
| 89 |
-
"ece": 0.
|
| 90 |
-
"mean_conf": 0.
|
| 91 |
-
"confidence_bias": 0.
|
| 92 |
-
"confident_error_rate": 0.
|
| 93 |
-
"coverage_at_0_9": 0.
|
| 94 |
-
"accuracy_at_0_9":
|
| 95 |
"coverage_at_5pct_error": 0.5294117647058824,
|
| 96 |
"coverage_at_1pct_error": 0.5294117647058824
|
| 97 |
},
|
| 98 |
"long_policy": {
|
| 99 |
"n": 19,
|
| 100 |
"acc": 0.47368421052631576,
|
| 101 |
-
"nll": 1.
|
| 102 |
-
"brier": 0.
|
| 103 |
-
"ece": 0.
|
| 104 |
-
"mean_conf": 0.
|
| 105 |
-
"confidence_bias": 0.
|
| 106 |
-
"confident_error_rate": 0.
|
| 107 |
-
"coverage_at_0_9": 0.
|
| 108 |
-
"accuracy_at_0_9":
|
| 109 |
-
"coverage_at_5pct_error": 0.
|
| 110 |
-
"coverage_at_1pct_error": 0.
|
| 111 |
-
"score_mae": 0.
|
| 112 |
-
"ranked_probability_score": 0.
|
| 113 |
},
|
| 114 |
"multi_hop": {
|
| 115 |
"n": 18,
|
| 116 |
-
"acc": 0.
|
| 117 |
-
"nll": 1.
|
| 118 |
-
"brier": 0.
|
| 119 |
-
"ece": 0.
|
| 120 |
-
"mean_conf": 0.
|
| 121 |
-
"confidence_bias": 0.
|
| 122 |
-
"confident_error_rate": 0.
|
| 123 |
-
"coverage_at_0_9": 0.
|
| 124 |
-
"accuracy_at_0_9":
|
| 125 |
-
"coverage_at_5pct_error": 0.
|
| 126 |
-
"coverage_at_1pct_error": 0.
|
| 127 |
-
"score_mae":
|
| 128 |
-
"ranked_probability_score": 0.
|
| 129 |
},
|
| 130 |
"probability": {
|
| 131 |
"n": 10,
|
| 132 |
-
"acc": 0.
|
| 133 |
-
"nll": 0.
|
| 134 |
-
"brier": 0.
|
| 135 |
-
"ece": 0.
|
| 136 |
-
"mean_conf": 0.
|
| 137 |
-
"confidence_bias": 0.
|
| 138 |
"confident_error_rate": 0.0,
|
| 139 |
-
"coverage_at_0_9": 0.
|
| 140 |
-
"accuracy_at_0_9":
|
| 141 |
-
"coverage_at_5pct_error": 0.
|
| 142 |
-
"coverage_at_1pct_error": 0.
|
| 143 |
},
|
| 144 |
"routing_hard": {
|
| 145 |
"n": 5,
|
| 146 |
"acc": 1.0,
|
| 147 |
-
"nll": 0.
|
| 148 |
-
"brier":
|
| 149 |
-
"ece": 0.
|
| 150 |
-
"mean_conf": 0.
|
| 151 |
-
"confidence_bias": -0.
|
| 152 |
"confident_error_rate": 0.0,
|
| 153 |
"coverage_at_0_9": 1.0,
|
| 154 |
"accuracy_at_0_9": 1.0,
|
|
@@ -158,149 +158,149 @@
|
|
| 158 |
"temporal_numeric": {
|
| 159 |
"n": 15,
|
| 160 |
"acc": 0.2,
|
| 161 |
-
"nll":
|
| 162 |
-
"brier":
|
| 163 |
-
"ece": 0.
|
| 164 |
-
"mean_conf": 0.
|
| 165 |
-
"confidence_bias": 0.
|
| 166 |
"confident_error_rate": 0.06666666666666667,
|
| 167 |
"coverage_at_0_9": 0.06666666666666667,
|
| 168 |
"accuracy_at_0_9": 0.0,
|
| 169 |
"coverage_at_5pct_error": 0.0,
|
| 170 |
"coverage_at_1pct_error": 0.0,
|
| 171 |
-
"score_mae": 0.
|
| 172 |
-
"ranked_probability_score": 0.
|
| 173 |
},
|
| 174 |
"tradeoff": {
|
| 175 |
"n": 6,
|
| 176 |
"acc": 0.3333333333333333,
|
| 177 |
-
"nll":
|
| 178 |
-
"brier": 1.
|
| 179 |
-
"ece": 0.
|
| 180 |
-
"mean_conf": 0.
|
| 181 |
-
"confidence_bias": 0.
|
| 182 |
-
"confident_error_rate": 0.
|
| 183 |
-
"coverage_at_0_9": 0.
|
| 184 |
-
"accuracy_at_0_9":
|
| 185 |
"coverage_at_5pct_error": 0.0,
|
| 186 |
"coverage_at_1pct_error": 0.0
|
| 187 |
},
|
| 188 |
"trap": {
|
| 189 |
"n": 8,
|
| 190 |
"acc": 1.0,
|
| 191 |
-
"nll": 0.
|
| 192 |
-
"brier": 0.
|
| 193 |
-
"ece": 0.
|
| 194 |
-
"mean_conf": 0.
|
| 195 |
-
"confidence_bias": -0.
|
| 196 |
"confident_error_rate": 0.0,
|
| 197 |
-
"coverage_at_0_9":
|
| 198 |
"accuracy_at_0_9": 1.0,
|
| 199 |
"coverage_at_5pct_error": 1.0,
|
| 200 |
"coverage_at_1pct_error": 1.0,
|
| 201 |
-
"score_mae": 0.
|
| 202 |
-
"ranked_probability_score": 0.
|
| 203 |
}
|
| 204 |
},
|
| 205 |
"sources": {
|
| 206 |
"adversarial": {
|
| 207 |
"n": 6,
|
| 208 |
-
"acc":
|
| 209 |
-
"nll": 0.
|
| 210 |
-
"brier": 0.
|
| 211 |
-
"ece": 0.
|
| 212 |
-
"mean_conf": 0.
|
| 213 |
-
"confidence_bias": -0.
|
| 214 |
"confident_error_rate": 0.0,
|
| 215 |
-
"coverage_at_0_9": 0.
|
| 216 |
"accuracy_at_0_9": 1.0,
|
| 217 |
-
"coverage_at_5pct_error":
|
| 218 |
-
"coverage_at_1pct_error":
|
| 219 |
-
"score_mae": 0.
|
| 220 |
-
"ranked_probability_score": 0.
|
| 221 |
},
|
| 222 |
"ambiguous": {
|
| 223 |
"n": 7,
|
| 224 |
-
"acc": 0.
|
| 225 |
-
"nll": 1.
|
| 226 |
-
"brier": 0.
|
| 227 |
-
"ece": 0.
|
| 228 |
-
"mean_conf": 0.
|
| 229 |
-
"confidence_bias": 0.
|
| 230 |
-
"confident_error_rate": 0.
|
| 231 |
-
"coverage_at_0_9": 0.
|
| 232 |
-
"accuracy_at_0_9":
|
| 233 |
"coverage_at_5pct_error": 0.14285714285714285,
|
| 234 |
"coverage_at_1pct_error": 0.14285714285714285
|
| 235 |
},
|
| 236 |
"judge_hard": {
|
| 237 |
"n": 17,
|
| 238 |
"acc": 0.7647058823529411,
|
| 239 |
-
"nll": 0.
|
| 240 |
-
"brier": 0.
|
| 241 |
-
"ece": 0.
|
| 242 |
-
"mean_conf": 0.
|
| 243 |
-
"confidence_bias": 0.
|
| 244 |
-
"confident_error_rate": 0.
|
| 245 |
-
"coverage_at_0_9": 0.
|
| 246 |
-
"accuracy_at_0_9":
|
| 247 |
"coverage_at_5pct_error": 0.5294117647058824,
|
| 248 |
"coverage_at_1pct_error": 0.5294117647058824
|
| 249 |
},
|
| 250 |
"long_policy": {
|
| 251 |
"n": 19,
|
| 252 |
"acc": 0.47368421052631576,
|
| 253 |
-
"nll": 1.
|
| 254 |
-
"brier": 0.
|
| 255 |
-
"ece": 0.
|
| 256 |
-
"mean_conf": 0.
|
| 257 |
-
"confidence_bias": 0.
|
| 258 |
-
"confident_error_rate": 0.
|
| 259 |
-
"coverage_at_0_9": 0.
|
| 260 |
-
"accuracy_at_0_9":
|
| 261 |
-
"coverage_at_5pct_error": 0.
|
| 262 |
-
"coverage_at_1pct_error": 0.
|
| 263 |
-
"score_mae": 0.
|
| 264 |
-
"ranked_probability_score": 0.
|
| 265 |
},
|
| 266 |
"multi_hop": {
|
| 267 |
"n": 18,
|
| 268 |
-
"acc": 0.
|
| 269 |
-
"nll": 1.
|
| 270 |
-
"brier": 0.
|
| 271 |
-
"ece": 0.
|
| 272 |
-
"mean_conf": 0.
|
| 273 |
-
"confidence_bias": 0.
|
| 274 |
-
"confident_error_rate": 0.
|
| 275 |
-
"coverage_at_0_9": 0.
|
| 276 |
-
"accuracy_at_0_9":
|
| 277 |
-
"coverage_at_5pct_error": 0.
|
| 278 |
-
"coverage_at_1pct_error": 0.
|
| 279 |
-
"score_mae":
|
| 280 |
-
"ranked_probability_score": 0.
|
| 281 |
},
|
| 282 |
"probability": {
|
| 283 |
"n": 10,
|
| 284 |
-
"acc": 0.
|
| 285 |
-
"nll": 0.
|
| 286 |
-
"brier": 0.
|
| 287 |
-
"ece": 0.
|
| 288 |
-
"mean_conf": 0.
|
| 289 |
-
"confidence_bias": 0.
|
| 290 |
"confident_error_rate": 0.0,
|
| 291 |
-
"coverage_at_0_9": 0.
|
| 292 |
-
"accuracy_at_0_9":
|
| 293 |
-
"coverage_at_5pct_error": 0.
|
| 294 |
-
"coverage_at_1pct_error": 0.
|
| 295 |
},
|
| 296 |
"routing_hard": {
|
| 297 |
"n": 5,
|
| 298 |
"acc": 1.0,
|
| 299 |
-
"nll": 0.
|
| 300 |
-
"brier":
|
| 301 |
-
"ece": 0.
|
| 302 |
-
"mean_conf": 0.
|
| 303 |
-
"confidence_bias": -0.
|
| 304 |
"confident_error_rate": 0.0,
|
| 305 |
"coverage_at_0_9": 1.0,
|
| 306 |
"accuracy_at_0_9": 1.0,
|
|
@@ -310,70 +310,70 @@
|
|
| 310 |
"temporal_numeric": {
|
| 311 |
"n": 15,
|
| 312 |
"acc": 0.2,
|
| 313 |
-
"nll":
|
| 314 |
-
"brier":
|
| 315 |
-
"ece": 0.
|
| 316 |
-
"mean_conf": 0.
|
| 317 |
-
"confidence_bias": 0.
|
| 318 |
"confident_error_rate": 0.06666666666666667,
|
| 319 |
"coverage_at_0_9": 0.06666666666666667,
|
| 320 |
"accuracy_at_0_9": 0.0,
|
| 321 |
"coverage_at_5pct_error": 0.0,
|
| 322 |
"coverage_at_1pct_error": 0.0,
|
| 323 |
-
"score_mae": 0.
|
| 324 |
-
"ranked_probability_score": 0.
|
| 325 |
},
|
| 326 |
"tradeoff": {
|
| 327 |
"n": 6,
|
| 328 |
"acc": 0.3333333333333333,
|
| 329 |
-
"nll":
|
| 330 |
-
"brier": 1.
|
| 331 |
-
"ece": 0.
|
| 332 |
-
"mean_conf": 0.
|
| 333 |
-
"confidence_bias": 0.
|
| 334 |
-
"confident_error_rate": 0.
|
| 335 |
-
"coverage_at_0_9": 0.
|
| 336 |
-
"accuracy_at_0_9":
|
| 337 |
"coverage_at_5pct_error": 0.0,
|
| 338 |
"coverage_at_1pct_error": 0.0
|
| 339 |
},
|
| 340 |
"trap": {
|
| 341 |
"n": 8,
|
| 342 |
"acc": 1.0,
|
| 343 |
-
"nll": 0.
|
| 344 |
-
"brier": 0.
|
| 345 |
-
"ece": 0.
|
| 346 |
-
"mean_conf": 0.
|
| 347 |
-
"confidence_bias": -0.
|
| 348 |
"confident_error_rate": 0.0,
|
| 349 |
-
"coverage_at_0_9":
|
| 350 |
"accuracy_at_0_9": 1.0,
|
| 351 |
"coverage_at_5pct_error": 1.0,
|
| 352 |
"coverage_at_1pct_error": 1.0,
|
| 353 |
-
"score_mae": 0.
|
| 354 |
-
"ranked_probability_score": 0.
|
| 355 |
}
|
| 356 |
},
|
| 357 |
"variants": {
|
| 358 |
"clean": {
|
| 359 |
"n": 111,
|
| 360 |
-
"acc": 0.
|
| 361 |
-
"nll":
|
| 362 |
-
"brier": 0.
|
| 363 |
-
"ece": 0.
|
| 364 |
-
"mean_conf": 0.
|
| 365 |
-
"confidence_bias": 0.
|
| 366 |
-
"confident_error_rate": 0.
|
| 367 |
-
"coverage_at_0_9": 0.
|
| 368 |
-
"accuracy_at_0_9": 0.
|
| 369 |
-
"coverage_at_5pct_error": 0.
|
| 370 |
-
"coverage_at_1pct_error": 0.
|
| 371 |
-
"score_mae": 0.
|
| 372 |
-
"ranked_probability_score": 0.
|
| 373 |
}
|
| 374 |
},
|
| 375 |
-
"macro_task_acc": 0.
|
| 376 |
-
"objective": -
|
| 377 |
"paired_flip": {
|
| 378 |
"pairs": 0,
|
| 379 |
"incomplete_pairs": 0,
|
|
@@ -385,9 +385,9 @@
|
|
| 385 |
},
|
| 386 |
"reference_distribution": {
|
| 387 |
"n": 10,
|
| 388 |
-
"squared_l2": 0.
|
| 389 |
-
"total_variation": 0.
|
| 390 |
-
"kl_reference_to_model": 0.
|
| 391 |
"note": "Exact reference-distribution fidelity, separate from hard-label Brier/ECE."
|
| 392 |
},
|
| 393 |
"unknowable": null,
|
|
@@ -409,10 +409,10 @@
|
|
| 409 |
},
|
| 410 |
"latency_ms": {
|
| 411 |
"n": 111,
|
| 412 |
-
"mean":
|
| 413 |
-
"p50":
|
| 414 |
-
"p95":
|
| 415 |
-
"p99":
|
| 416 |
}
|
| 417 |
}
|
| 418 |
},
|
|
|
|
| 8 |
"served_models": [
|
| 9 |
"decision-model"
|
| 10 |
],
|
| 11 |
+
"elapsed_measurement_s": 11.638510651013348,
|
| 12 |
+
"requests_per_second": 9.537302781119626,
|
| 13 |
+
"questions_per_second": 9.537302781119626,
|
| 14 |
"latency_ms": {
|
| 15 |
"n": 111,
|
| 16 |
+
"mean": 104.71516278381502,
|
| 17 |
+
"p50": 79.18975199572742,
|
| 18 |
+
"p95": 259.6404929936398,
|
| 19 |
+
"p99": 314.9512562784367
|
| 20 |
},
|
| 21 |
"adapter_ms": {
|
| 22 |
"n": 111,
|
| 23 |
+
"mean": 103.09326519082124,
|
| 24 |
+
"p50": 77.53870100714266,
|
| 25 |
+
"p95": 257.935585512314,
|
| 26 |
+
"p99": 313.29736551269895
|
| 27 |
},
|
| 28 |
"input_tokens_per_request": {
|
| 29 |
"n": 111,
|
|
|
|
| 36 |
"jevbench-hard/public": {
|
| 37 |
"clean": {
|
| 38 |
"n": 111,
|
| 39 |
+
"acc": 0.5675675675675675,
|
| 40 |
+
"nll": 0.8978270366878996,
|
| 41 |
+
"brier": 0.5290505153484923,
|
| 42 |
+
"ece": 0.1370716552864825,
|
| 43 |
+
"mean_conf": 0.6984911875031684,
|
| 44 |
+
"confidence_bias": 0.13092361993560087,
|
| 45 |
+
"confident_error_rate": 0.009009009009009009,
|
| 46 |
+
"coverage_at_0_9": 0.23423423423423423,
|
| 47 |
+
"accuracy_at_0_9": 0.9615384615384616,
|
| 48 |
+
"coverage_at_5pct_error": 0.23423423423423423,
|
| 49 |
+
"coverage_at_1pct_error": 0.2072072072072072,
|
| 50 |
+
"score_mae": 0.6886011145197463,
|
| 51 |
+
"ranked_probability_score": 0.1312675912608615
|
| 52 |
},
|
| 53 |
"tasks": {
|
| 54 |
"adversarial": {
|
| 55 |
"n": 6,
|
| 56 |
+
"acc": 0.8333333333333334,
|
| 57 |
+
"nll": 0.30267000472066447,
|
| 58 |
+
"brier": 0.1846073015854883,
|
| 59 |
+
"ece": 0.21392364620757084,
|
| 60 |
+
"mean_conf": 0.8207123290002708,
|
| 61 |
+
"confidence_bias": -0.012621004333062569,
|
| 62 |
"confident_error_rate": 0.0,
|
| 63 |
+
"coverage_at_0_9": 0.5,
|
| 64 |
"accuracy_at_0_9": 1.0,
|
| 65 |
+
"coverage_at_5pct_error": 0.6666666666666666,
|
| 66 |
+
"coverage_at_1pct_error": 0.6666666666666666,
|
| 67 |
+
"score_mae": 0.3511274106868094,
|
| 68 |
+
"ranked_probability_score": 0.04695474627196716
|
| 69 |
},
|
| 70 |
"ambiguous": {
|
| 71 |
"n": 7,
|
| 72 |
+
"acc": 0.5714285714285714,
|
| 73 |
+
"nll": 1.166961099131194,
|
| 74 |
+
"brier": 0.7148034003369997,
|
| 75 |
+
"ece": 0.32083780443031323,
|
| 76 |
+
"mean_conf": 0.6696371954169369,
|
| 77 |
+
"confidence_bias": 0.0982086239883655,
|
| 78 |
+
"confident_error_rate": 0.0,
|
| 79 |
+
"coverage_at_0_9": 0.14285714285714285,
|
| 80 |
+
"accuracy_at_0_9": 1.0,
|
| 81 |
"coverage_at_5pct_error": 0.14285714285714285,
|
| 82 |
"coverage_at_1pct_error": 0.14285714285714285
|
| 83 |
},
|
| 84 |
"judge_hard": {
|
| 85 |
"n": 17,
|
| 86 |
"acc": 0.7647058823529411,
|
| 87 |
+
"nll": 0.4486737076244502,
|
| 88 |
+
"brier": 0.3105305830264557,
|
| 89 |
+
"ece": 0.163894819606467,
|
| 90 |
+
"mean_conf": 0.8579047132123342,
|
| 91 |
+
"confidence_bias": 0.09319883085939307,
|
| 92 |
+
"confident_error_rate": 0.0,
|
| 93 |
+
"coverage_at_0_9": 0.47058823529411764,
|
| 94 |
+
"accuracy_at_0_9": 1.0,
|
| 95 |
"coverage_at_5pct_error": 0.5294117647058824,
|
| 96 |
"coverage_at_1pct_error": 0.5294117647058824
|
| 97 |
},
|
| 98 |
"long_policy": {
|
| 99 |
"n": 19,
|
| 100 |
"acc": 0.47368421052631576,
|
| 101 |
+
"nll": 1.1202630393770878,
|
| 102 |
+
"brier": 0.6462411617272702,
|
| 103 |
+
"ece": 0.21854545548305468,
|
| 104 |
+
"mean_conf": 0.6851194875501027,
|
| 105 |
+
"confidence_bias": 0.2114352770237869,
|
| 106 |
+
"confident_error_rate": 0.0,
|
| 107 |
+
"coverage_at_0_9": 0.05263157894736842,
|
| 108 |
+
"accuracy_at_0_9": 1.0,
|
| 109 |
+
"coverage_at_5pct_error": 0.10526315789473684,
|
| 110 |
+
"coverage_at_1pct_error": 0.10526315789473684,
|
| 111 |
+
"score_mae": 0.9935835242669617,
|
| 112 |
+
"ranked_probability_score": 0.19958679277138075
|
| 113 |
},
|
| 114 |
"multi_hop": {
|
| 115 |
"n": 18,
|
| 116 |
+
"acc": 0.5,
|
| 117 |
+
"nll": 1.0199473339651695,
|
| 118 |
+
"brier": 0.5715218070950877,
|
| 119 |
+
"ece": 0.1377783888676486,
|
| 120 |
+
"mean_conf": 0.5964976311551695,
|
| 121 |
+
"confidence_bias": 0.09649763115516952,
|
| 122 |
+
"confident_error_rate": 0.0,
|
| 123 |
+
"coverage_at_0_9": 0.05555555555555555,
|
| 124 |
+
"accuracy_at_0_9": 1.0,
|
| 125 |
+
"coverage_at_5pct_error": 0.2222222222222222,
|
| 126 |
+
"coverage_at_1pct_error": 0.2222222222222222,
|
| 127 |
+
"score_mae": 0.9931654294749996,
|
| 128 |
+
"ranked_probability_score": 0.19713385339410178
|
| 129 |
},
|
| 130 |
"probability": {
|
| 131 |
"n": 10,
|
| 132 |
+
"acc": 0.5,
|
| 133 |
+
"nll": 0.7440881305202426,
|
| 134 |
+
"brier": 0.48740179466256217,
|
| 135 |
+
"ece": 0.17888073597680856,
|
| 136 |
+
"mean_conf": 0.5791129289438599,
|
| 137 |
+
"confidence_bias": 0.0791129289438599,
|
| 138 |
"confident_error_rate": 0.0,
|
| 139 |
+
"coverage_at_0_9": 0.0,
|
| 140 |
+
"accuracy_at_0_9": null,
|
| 141 |
+
"coverage_at_5pct_error": 0.2,
|
| 142 |
+
"coverage_at_1pct_error": 0.2
|
| 143 |
},
|
| 144 |
"routing_hard": {
|
| 145 |
"n": 5,
|
| 146 |
"acc": 1.0,
|
| 147 |
+
"nll": 0.05150349119576013,
|
| 148 |
+
"brier": 0.0035313144795324223,
|
| 149 |
+
"ece": 0.050124770239333594,
|
| 150 |
+
"mean_conf": 0.9498752297606664,
|
| 151 |
+
"confidence_bias": -0.050124770239333594,
|
| 152 |
"confident_error_rate": 0.0,
|
| 153 |
"coverage_at_0_9": 1.0,
|
| 154 |
"accuracy_at_0_9": 1.0,
|
|
|
|
| 158 |
"temporal_numeric": {
|
| 159 |
"n": 15,
|
| 160 |
"acc": 0.2,
|
| 161 |
+
"nll": 1.5813456095311673,
|
| 162 |
+
"brier": 0.8848137973448025,
|
| 163 |
+
"ece": 0.295391954903108,
|
| 164 |
+
"mean_conf": 0.495391954903108,
|
| 165 |
+
"confidence_bias": 0.295391954903108,
|
| 166 |
"confident_error_rate": 0.06666666666666667,
|
| 167 |
"coverage_at_0_9": 0.06666666666666667,
|
| 168 |
"accuracy_at_0_9": 0.0,
|
| 169 |
"coverage_at_5pct_error": 0.0,
|
| 170 |
"coverage_at_1pct_error": 0.0,
|
| 171 |
+
"score_mae": 0.38261118788247117,
|
| 172 |
+
"ranked_probability_score": 0.11832173048417309
|
| 173 |
},
|
| 174 |
"tradeoff": {
|
| 175 |
"n": 6,
|
| 176 |
"acc": 0.3333333333333333,
|
| 177 |
+
"nll": 1.6991903500800731,
|
| 178 |
+
"brier": 1.0732609261958943,
|
| 179 |
+
"ece": 0.5969757753736276,
|
| 180 |
+
"mean_conf": 0.7234564640731044,
|
| 181 |
+
"confidence_bias": 0.39012313073977106,
|
| 182 |
+
"confident_error_rate": 0.0,
|
| 183 |
+
"coverage_at_0_9": 0.0,
|
| 184 |
+
"accuracy_at_0_9": null,
|
| 185 |
"coverage_at_5pct_error": 0.0,
|
| 186 |
"coverage_at_1pct_error": 0.0
|
| 187 |
},
|
| 188 |
"trap": {
|
| 189 |
"n": 8,
|
| 190 |
"acc": 1.0,
|
| 191 |
+
"nll": 0.09860319454149577,
|
| 192 |
+
"brier": 0.020612255432572606,
|
| 193 |
+
"ece": 0.09124297801879305,
|
| 194 |
+
"mean_conf": 0.908757021981207,
|
| 195 |
+
"confidence_bias": -0.091242978018793,
|
| 196 |
"confident_error_rate": 0.0,
|
| 197 |
+
"coverage_at_0_9": 0.75,
|
| 198 |
"accuracy_at_0_9": 1.0,
|
| 199 |
"coverage_at_5pct_error": 1.0,
|
| 200 |
"coverage_at_1pct_error": 1.0,
|
| 201 |
+
"score_mae": 0.417535610540274,
|
| 202 |
+
"ranked_probability_score": 0.02602163187216543
|
| 203 |
}
|
| 204 |
},
|
| 205 |
"sources": {
|
| 206 |
"adversarial": {
|
| 207 |
"n": 6,
|
| 208 |
+
"acc": 0.8333333333333334,
|
| 209 |
+
"nll": 0.30267000472066447,
|
| 210 |
+
"brier": 0.1846073015854883,
|
| 211 |
+
"ece": 0.21392364620757084,
|
| 212 |
+
"mean_conf": 0.8207123290002708,
|
| 213 |
+
"confidence_bias": -0.012621004333062569,
|
| 214 |
"confident_error_rate": 0.0,
|
| 215 |
+
"coverage_at_0_9": 0.5,
|
| 216 |
"accuracy_at_0_9": 1.0,
|
| 217 |
+
"coverage_at_5pct_error": 0.6666666666666666,
|
| 218 |
+
"coverage_at_1pct_error": 0.6666666666666666,
|
| 219 |
+
"score_mae": 0.3511274106868094,
|
| 220 |
+
"ranked_probability_score": 0.04695474627196716
|
| 221 |
},
|
| 222 |
"ambiguous": {
|
| 223 |
"n": 7,
|
| 224 |
+
"acc": 0.5714285714285714,
|
| 225 |
+
"nll": 1.166961099131194,
|
| 226 |
+
"brier": 0.7148034003369997,
|
| 227 |
+
"ece": 0.32083780443031323,
|
| 228 |
+
"mean_conf": 0.6696371954169369,
|
| 229 |
+
"confidence_bias": 0.0982086239883655,
|
| 230 |
+
"confident_error_rate": 0.0,
|
| 231 |
+
"coverage_at_0_9": 0.14285714285714285,
|
| 232 |
+
"accuracy_at_0_9": 1.0,
|
| 233 |
"coverage_at_5pct_error": 0.14285714285714285,
|
| 234 |
"coverage_at_1pct_error": 0.14285714285714285
|
| 235 |
},
|
| 236 |
"judge_hard": {
|
| 237 |
"n": 17,
|
| 238 |
"acc": 0.7647058823529411,
|
| 239 |
+
"nll": 0.4486737076244502,
|
| 240 |
+
"brier": 0.3105305830264557,
|
| 241 |
+
"ece": 0.163894819606467,
|
| 242 |
+
"mean_conf": 0.8579047132123342,
|
| 243 |
+
"confidence_bias": 0.09319883085939307,
|
| 244 |
+
"confident_error_rate": 0.0,
|
| 245 |
+
"coverage_at_0_9": 0.47058823529411764,
|
| 246 |
+
"accuracy_at_0_9": 1.0,
|
| 247 |
"coverage_at_5pct_error": 0.5294117647058824,
|
| 248 |
"coverage_at_1pct_error": 0.5294117647058824
|
| 249 |
},
|
| 250 |
"long_policy": {
|
| 251 |
"n": 19,
|
| 252 |
"acc": 0.47368421052631576,
|
| 253 |
+
"nll": 1.1202630393770878,
|
| 254 |
+
"brier": 0.6462411617272702,
|
| 255 |
+
"ece": 0.21854545548305468,
|
| 256 |
+
"mean_conf": 0.6851194875501027,
|
| 257 |
+
"confidence_bias": 0.2114352770237869,
|
| 258 |
+
"confident_error_rate": 0.0,
|
| 259 |
+
"coverage_at_0_9": 0.05263157894736842,
|
| 260 |
+
"accuracy_at_0_9": 1.0,
|
| 261 |
+
"coverage_at_5pct_error": 0.10526315789473684,
|
| 262 |
+
"coverage_at_1pct_error": 0.10526315789473684,
|
| 263 |
+
"score_mae": 0.9935835242669617,
|
| 264 |
+
"ranked_probability_score": 0.19958679277138075
|
| 265 |
},
|
| 266 |
"multi_hop": {
|
| 267 |
"n": 18,
|
| 268 |
+
"acc": 0.5,
|
| 269 |
+
"nll": 1.0199473339651695,
|
| 270 |
+
"brier": 0.5715218070950877,
|
| 271 |
+
"ece": 0.1377783888676486,
|
| 272 |
+
"mean_conf": 0.5964976311551695,
|
| 273 |
+
"confidence_bias": 0.09649763115516952,
|
| 274 |
+
"confident_error_rate": 0.0,
|
| 275 |
+
"coverage_at_0_9": 0.05555555555555555,
|
| 276 |
+
"accuracy_at_0_9": 1.0,
|
| 277 |
+
"coverage_at_5pct_error": 0.2222222222222222,
|
| 278 |
+
"coverage_at_1pct_error": 0.2222222222222222,
|
| 279 |
+
"score_mae": 0.9931654294749996,
|
| 280 |
+
"ranked_probability_score": 0.19713385339410178
|
| 281 |
},
|
| 282 |
"probability": {
|
| 283 |
"n": 10,
|
| 284 |
+
"acc": 0.5,
|
| 285 |
+
"nll": 0.7440881305202426,
|
| 286 |
+
"brier": 0.48740179466256217,
|
| 287 |
+
"ece": 0.17888073597680856,
|
| 288 |
+
"mean_conf": 0.5791129289438599,
|
| 289 |
+
"confidence_bias": 0.0791129289438599,
|
| 290 |
"confident_error_rate": 0.0,
|
| 291 |
+
"coverage_at_0_9": 0.0,
|
| 292 |
+
"accuracy_at_0_9": null,
|
| 293 |
+
"coverage_at_5pct_error": 0.2,
|
| 294 |
+
"coverage_at_1pct_error": 0.2
|
| 295 |
},
|
| 296 |
"routing_hard": {
|
| 297 |
"n": 5,
|
| 298 |
"acc": 1.0,
|
| 299 |
+
"nll": 0.05150349119576013,
|
| 300 |
+
"brier": 0.0035313144795324223,
|
| 301 |
+
"ece": 0.050124770239333594,
|
| 302 |
+
"mean_conf": 0.9498752297606664,
|
| 303 |
+
"confidence_bias": -0.050124770239333594,
|
| 304 |
"confident_error_rate": 0.0,
|
| 305 |
"coverage_at_0_9": 1.0,
|
| 306 |
"accuracy_at_0_9": 1.0,
|
|
|
|
| 310 |
"temporal_numeric": {
|
| 311 |
"n": 15,
|
| 312 |
"acc": 0.2,
|
| 313 |
+
"nll": 1.5813456095311673,
|
| 314 |
+
"brier": 0.8848137973448025,
|
| 315 |
+
"ece": 0.295391954903108,
|
| 316 |
+
"mean_conf": 0.495391954903108,
|
| 317 |
+
"confidence_bias": 0.295391954903108,
|
| 318 |
"confident_error_rate": 0.06666666666666667,
|
| 319 |
"coverage_at_0_9": 0.06666666666666667,
|
| 320 |
"accuracy_at_0_9": 0.0,
|
| 321 |
"coverage_at_5pct_error": 0.0,
|
| 322 |
"coverage_at_1pct_error": 0.0,
|
| 323 |
+
"score_mae": 0.38261118788247117,
|
| 324 |
+
"ranked_probability_score": 0.11832173048417309
|
| 325 |
},
|
| 326 |
"tradeoff": {
|
| 327 |
"n": 6,
|
| 328 |
"acc": 0.3333333333333333,
|
| 329 |
+
"nll": 1.6991903500800731,
|
| 330 |
+
"brier": 1.0732609261958943,
|
| 331 |
+
"ece": 0.5969757753736276,
|
| 332 |
+
"mean_conf": 0.7234564640731044,
|
| 333 |
+
"confidence_bias": 0.39012313073977106,
|
| 334 |
+
"confident_error_rate": 0.0,
|
| 335 |
+
"coverage_at_0_9": 0.0,
|
| 336 |
+
"accuracy_at_0_9": null,
|
| 337 |
"coverage_at_5pct_error": 0.0,
|
| 338 |
"coverage_at_1pct_error": 0.0
|
| 339 |
},
|
| 340 |
"trap": {
|
| 341 |
"n": 8,
|
| 342 |
"acc": 1.0,
|
| 343 |
+
"nll": 0.09860319454149577,
|
| 344 |
+
"brier": 0.020612255432572606,
|
| 345 |
+
"ece": 0.09124297801879305,
|
| 346 |
+
"mean_conf": 0.908757021981207,
|
| 347 |
+
"confidence_bias": -0.091242978018793,
|
| 348 |
"confident_error_rate": 0.0,
|
| 349 |
+
"coverage_at_0_9": 0.75,
|
| 350 |
"accuracy_at_0_9": 1.0,
|
| 351 |
"coverage_at_5pct_error": 1.0,
|
| 352 |
"coverage_at_1pct_error": 1.0,
|
| 353 |
+
"score_mae": 0.417535610540274,
|
| 354 |
+
"ranked_probability_score": 0.02602163187216543
|
| 355 |
}
|
| 356 |
},
|
| 357 |
"variants": {
|
| 358 |
"clean": {
|
| 359 |
"n": 111,
|
| 360 |
+
"acc": 0.5675675675675675,
|
| 361 |
+
"nll": 0.8978270366878996,
|
| 362 |
+
"brier": 0.5290505153484923,
|
| 363 |
+
"ece": 0.1370716552864825,
|
| 364 |
+
"mean_conf": 0.6984911875031684,
|
| 365 |
+
"confidence_bias": 0.13092361993560087,
|
| 366 |
+
"confident_error_rate": 0.009009009009009009,
|
| 367 |
+
"coverage_at_0_9": 0.23423423423423423,
|
| 368 |
+
"accuracy_at_0_9": 0.9615384615384616,
|
| 369 |
+
"coverage_at_5pct_error": 0.23423423423423423,
|
| 370 |
+
"coverage_at_1pct_error": 0.2072072072072072,
|
| 371 |
+
"score_mae": 0.6886011145197463,
|
| 372 |
+
"ranked_probability_score": 0.1312675912608615
|
| 373 |
}
|
| 374 |
},
|
| 375 |
+
"macro_task_acc": 0.6176485330974495,
|
| 376 |
+
"objective": -0.8233245960687304,
|
| 377 |
"paired_flip": {
|
| 378 |
"pairs": 0,
|
| 379 |
"incomplete_pairs": 0,
|
|
|
|
| 385 |
},
|
| 386 |
"reference_distribution": {
|
| 387 |
"n": 10,
|
| 388 |
+
"squared_l2": 0.09922415881974203,
|
| 389 |
+
"total_variation": 0.2065482283787275,
|
| 390 |
+
"kl_reference_to_model": 0.12609642211581057,
|
| 391 |
"note": "Exact reference-distribution fidelity, separate from hard-label Brier/ECE."
|
| 392 |
},
|
| 393 |
"unknowable": null,
|
|
|
|
| 409 |
},
|
| 410 |
"latency_ms": {
|
| 411 |
"n": 111,
|
| 412 |
+
"mean": 104.71516278381502,
|
| 413 |
+
"p50": 79.18975199572742,
|
| 414 |
+
"p95": 259.6404929936398,
|
| 415 |
+
"p99": 314.9512562784367
|
| 416 |
}
|
| 417 |
}
|
| 418 |
},
|
evidence/served-nosys-original-report.json
CHANGED
|
@@ -8,22 +8,22 @@
|
|
| 8 |
"served_models": [
|
| 9 |
"decision-model"
|
| 10 |
],
|
| 11 |
-
"elapsed_measurement_s":
|
| 12 |
-
"requests_per_second":
|
| 13 |
-
"questions_per_second":
|
| 14 |
"latency_ms": {
|
| 15 |
"n": 72,
|
| 16 |
-
"mean":
|
| 17 |
-
"p50":
|
| 18 |
-
"p95":
|
| 19 |
-
"p99":
|
| 20 |
},
|
| 21 |
"adapter_ms": {
|
| 22 |
"n": 72,
|
| 23 |
-
"mean":
|
| 24 |
-
"p50": 39.
|
| 25 |
-
"p95":
|
| 26 |
-
"p99":
|
| 27 |
},
|
| 28 |
"input_tokens_per_request": {
|
| 29 |
"n": 72,
|
|
@@ -36,45 +36,45 @@
|
|
| 36 |
"jevbench-original/public": {
|
| 37 |
"clean": {
|
| 38 |
"n": 72,
|
| 39 |
-
"acc": 0.
|
| 40 |
-
"nll": 0.
|
| 41 |
-
"brier": 0.
|
| 42 |
-
"ece": 0.
|
| 43 |
-
"mean_conf": 0.
|
| 44 |
-
"confidence_bias": 0.
|
| 45 |
-
"confident_error_rate": 0.
|
| 46 |
-
"coverage_at_0_9": 0.
|
| 47 |
-
"accuracy_at_0_9":
|
| 48 |
"coverage_at_5pct_error": 0.9444444444444444,
|
| 49 |
-
"coverage_at_1pct_error": 0.
|
| 50 |
-
"score_mae": 0.
|
| 51 |
-
"ranked_probability_score":
|
| 52 |
},
|
| 53 |
"tasks": {
|
| 54 |
"adequacy": {
|
| 55 |
"n": 12,
|
| 56 |
"acc": 0.8333333333333334,
|
| 57 |
-
"nll": 0.
|
| 58 |
-
"brier": 0.
|
| 59 |
-
"ece": 0.
|
| 60 |
-
"mean_conf": 0.
|
| 61 |
-
"confidence_bias": 0.
|
| 62 |
-
"confident_error_rate": 0.
|
| 63 |
-
"coverage_at_0_9": 0.
|
| 64 |
-
"accuracy_at_0_9":
|
| 65 |
"coverage_at_5pct_error": 0.6666666666666666,
|
| 66 |
"coverage_at_1pct_error": 0.6666666666666666
|
| 67 |
},
|
| 68 |
"extraction": {
|
| 69 |
"n": 12,
|
| 70 |
"acc": 1.0,
|
| 71 |
-
"nll": 0.
|
| 72 |
-
"brier": 0.
|
| 73 |
-
"ece": 0.
|
| 74 |
-
"mean_conf": 0.
|
| 75 |
-
"confidence_bias": -0.
|
| 76 |
"confident_error_rate": 0.0,
|
| 77 |
-
"coverage_at_0_9":
|
| 78 |
"accuracy_at_0_9": 1.0,
|
| 79 |
"coverage_at_5pct_error": 1.0,
|
| 80 |
"coverage_at_1pct_error": 1.0
|
|
@@ -82,87 +82,87 @@
|
|
| 82 |
"intent": {
|
| 83 |
"n": 12,
|
| 84 |
"acc": 0.8333333333333334,
|
| 85 |
-
"nll": 0.
|
| 86 |
-
"brier": 0.
|
| 87 |
-
"ece": 0.
|
| 88 |
-
"mean_conf": 0.
|
| 89 |
-
"confidence_bias": 0.
|
| 90 |
"confident_error_rate": 0.0,
|
| 91 |
-
"coverage_at_0_9": 0.
|
| 92 |
"accuracy_at_0_9": 1.0,
|
| 93 |
-
"coverage_at_5pct_error": 0.
|
| 94 |
-
"coverage_at_1pct_error": 0.
|
| 95 |
},
|
| 96 |
"ordinal": {
|
| 97 |
"n": 12,
|
| 98 |
"acc": 1.0,
|
| 99 |
-
"nll": 0.
|
| 100 |
-
"brier":
|
| 101 |
-
"ece": 0.
|
| 102 |
-
"mean_conf": 0.
|
| 103 |
-
"confidence_bias": -0.
|
| 104 |
"confident_error_rate": 0.0,
|
| 105 |
-
"coverage_at_0_9":
|
| 106 |
"accuracy_at_0_9": 1.0,
|
| 107 |
"coverage_at_5pct_error": 1.0,
|
| 108 |
"coverage_at_1pct_error": 1.0,
|
| 109 |
-
"score_mae": 0.
|
| 110 |
-
"ranked_probability_score":
|
| 111 |
},
|
| 112 |
"policy": {
|
| 113 |
"n": 12,
|
| 114 |
"acc": 1.0,
|
| 115 |
-
"nll": 0.
|
| 116 |
-
"brier": 0.
|
| 117 |
-
"ece": 0.
|
| 118 |
-
"mean_conf": 0.
|
| 119 |
-
"confidence_bias": -0.
|
| 120 |
"confident_error_rate": 0.0,
|
| 121 |
-
"coverage_at_0_9":
|
| 122 |
"accuracy_at_0_9": 1.0,
|
| 123 |
"coverage_at_5pct_error": 1.0,
|
| 124 |
"coverage_at_1pct_error": 1.0
|
| 125 |
},
|
| 126 |
"routing": {
|
| 127 |
"n": 12,
|
| 128 |
-
"acc":
|
| 129 |
-
"nll": 0.
|
| 130 |
-
"brier": 0.
|
| 131 |
-
"ece": 0.
|
| 132 |
-
"mean_conf": 0.
|
| 133 |
-
"confidence_bias": -0.
|
| 134 |
"confident_error_rate": 0.0,
|
| 135 |
-
"coverage_at_0_9": 0.
|
| 136 |
"accuracy_at_0_9": 1.0,
|
| 137 |
-
"coverage_at_5pct_error":
|
| 138 |
-
"coverage_at_1pct_error":
|
| 139 |
}
|
| 140 |
},
|
| 141 |
"sources": {
|
| 142 |
"adequacy": {
|
| 143 |
"n": 12,
|
| 144 |
"acc": 0.8333333333333334,
|
| 145 |
-
"nll": 0.
|
| 146 |
-
"brier": 0.
|
| 147 |
-
"ece": 0.
|
| 148 |
-
"mean_conf": 0.
|
| 149 |
-
"confidence_bias": 0.
|
| 150 |
-
"confident_error_rate": 0.
|
| 151 |
-
"coverage_at_0_9": 0.
|
| 152 |
-
"accuracy_at_0_9":
|
| 153 |
"coverage_at_5pct_error": 0.6666666666666666,
|
| 154 |
"coverage_at_1pct_error": 0.6666666666666666
|
| 155 |
},
|
| 156 |
"extraction": {
|
| 157 |
"n": 12,
|
| 158 |
"acc": 1.0,
|
| 159 |
-
"nll": 0.
|
| 160 |
-
"brier": 0.
|
| 161 |
-
"ece": 0.
|
| 162 |
-
"mean_conf": 0.
|
| 163 |
-
"confidence_bias": -0.
|
| 164 |
"confident_error_rate": 0.0,
|
| 165 |
-
"coverage_at_0_9":
|
| 166 |
"accuracy_at_0_9": 1.0,
|
| 167 |
"coverage_at_5pct_error": 1.0,
|
| 168 |
"coverage_at_1pct_error": 1.0
|
|
@@ -170,82 +170,82 @@
|
|
| 170 |
"intent": {
|
| 171 |
"n": 12,
|
| 172 |
"acc": 0.8333333333333334,
|
| 173 |
-
"nll": 0.
|
| 174 |
-
"brier": 0.
|
| 175 |
-
"ece": 0.
|
| 176 |
-
"mean_conf": 0.
|
| 177 |
-
"confidence_bias": 0.
|
| 178 |
"confident_error_rate": 0.0,
|
| 179 |
-
"coverage_at_0_9": 0.
|
| 180 |
"accuracy_at_0_9": 1.0,
|
| 181 |
-
"coverage_at_5pct_error": 0.
|
| 182 |
-
"coverage_at_1pct_error": 0.
|
| 183 |
},
|
| 184 |
"ordinal": {
|
| 185 |
"n": 12,
|
| 186 |
"acc": 1.0,
|
| 187 |
-
"nll": 0.
|
| 188 |
-
"brier":
|
| 189 |
-
"ece": 0.
|
| 190 |
-
"mean_conf": 0.
|
| 191 |
-
"confidence_bias": -0.
|
| 192 |
"confident_error_rate": 0.0,
|
| 193 |
-
"coverage_at_0_9":
|
| 194 |
"accuracy_at_0_9": 1.0,
|
| 195 |
"coverage_at_5pct_error": 1.0,
|
| 196 |
"coverage_at_1pct_error": 1.0,
|
| 197 |
-
"score_mae": 0.
|
| 198 |
-
"ranked_probability_score":
|
| 199 |
},
|
| 200 |
"policy": {
|
| 201 |
"n": 12,
|
| 202 |
"acc": 1.0,
|
| 203 |
-
"nll": 0.
|
| 204 |
-
"brier": 0.
|
| 205 |
-
"ece": 0.
|
| 206 |
-
"mean_conf": 0.
|
| 207 |
-
"confidence_bias": -0.
|
| 208 |
"confident_error_rate": 0.0,
|
| 209 |
-
"coverage_at_0_9":
|
| 210 |
"accuracy_at_0_9": 1.0,
|
| 211 |
"coverage_at_5pct_error": 1.0,
|
| 212 |
"coverage_at_1pct_error": 1.0
|
| 213 |
},
|
| 214 |
"routing": {
|
| 215 |
"n": 12,
|
| 216 |
-
"acc":
|
| 217 |
-
"nll": 0.
|
| 218 |
-
"brier": 0.
|
| 219 |
-
"ece": 0.
|
| 220 |
-
"mean_conf": 0.
|
| 221 |
-
"confidence_bias": -0.
|
| 222 |
"confident_error_rate": 0.0,
|
| 223 |
-
"coverage_at_0_9": 0.
|
| 224 |
"accuracy_at_0_9": 1.0,
|
| 225 |
-
"coverage_at_5pct_error":
|
| 226 |
-
"coverage_at_1pct_error":
|
| 227 |
}
|
| 228 |
},
|
| 229 |
"variants": {
|
| 230 |
"clean": {
|
| 231 |
"n": 72,
|
| 232 |
-
"acc": 0.
|
| 233 |
-
"nll": 0.
|
| 234 |
-
"brier": 0.
|
| 235 |
-
"ece": 0.
|
| 236 |
-
"mean_conf": 0.
|
| 237 |
-
"confidence_bias": 0.
|
| 238 |
-
"confident_error_rate": 0.
|
| 239 |
-
"coverage_at_0_9": 0.
|
| 240 |
-
"accuracy_at_0_9":
|
| 241 |
"coverage_at_5pct_error": 0.9444444444444444,
|
| 242 |
-
"coverage_at_1pct_error": 0.
|
| 243 |
-
"score_mae": 0.
|
| 244 |
-
"ranked_probability_score":
|
| 245 |
}
|
| 246 |
},
|
| 247 |
-
"macro_task_acc": 0.
|
| 248 |
-
"objective": -0.
|
| 249 |
"paired_flip": {
|
| 250 |
"pairs": 0,
|
| 251 |
"incomplete_pairs": 0,
|
|
@@ -275,10 +275,10 @@
|
|
| 275 |
},
|
| 276 |
"latency_ms": {
|
| 277 |
"n": 72,
|
| 278 |
-
"mean":
|
| 279 |
-
"p50":
|
| 280 |
-
"p95":
|
| 281 |
-
"p99":
|
| 282 |
}
|
| 283 |
}
|
| 284 |
},
|
|
|
|
| 8 |
"served_models": [
|
| 9 |
"decision-model"
|
| 10 |
],
|
| 11 |
+
"elapsed_measurement_s": 3.404803980025463,
|
| 12 |
+
"requests_per_second": 21.146591822141122,
|
| 13 |
+
"questions_per_second": 21.146591822141122,
|
| 14 |
"latency_ms": {
|
| 15 |
"n": 72,
|
| 16 |
+
"mean": 47.179252678890606,
|
| 17 |
+
"p50": 41.166106500895694,
|
| 18 |
+
"p95": 90.7176292530494,
|
| 19 |
+
"p99": 103.64775064750576
|
| 20 |
},
|
| 21 |
"adapter_ms": {
|
| 22 |
"n": 72,
|
| 23 |
+
"mean": 45.77490902900334,
|
| 24 |
+
"p50": 39.72933301702142,
|
| 25 |
+
"p95": 89.21472410147545,
|
| 26 |
+
"p99": 101.97920961014464
|
| 27 |
},
|
| 28 |
"input_tokens_per_request": {
|
| 29 |
"n": 72,
|
|
|
|
| 36 |
"jevbench-original/public": {
|
| 37 |
"clean": {
|
| 38 |
"n": 72,
|
| 39 |
+
"acc": 0.9305555555555556,
|
| 40 |
+
"nll": 0.21445778495560835,
|
| 41 |
+
"brier": 0.10677210008799252,
|
| 42 |
+
"ece": 0.07532617080884235,
|
| 43 |
+
"mean_conf": 0.867553070835708,
|
| 44 |
+
"confidence_bias": -0.0630024847198476,
|
| 45 |
+
"confident_error_rate": 0.0,
|
| 46 |
+
"coverage_at_0_9": 0.5694444444444444,
|
| 47 |
+
"accuracy_at_0_9": 1.0,
|
| 48 |
"coverage_at_5pct_error": 0.9444444444444444,
|
| 49 |
+
"coverage_at_1pct_error": 0.6944444444444444,
|
| 50 |
+
"score_mae": 0.06320981328072943,
|
| 51 |
+
"ranked_probability_score": 0.0022976210425868964
|
| 52 |
},
|
| 53 |
"tasks": {
|
| 54 |
"adequacy": {
|
| 55 |
"n": 12,
|
| 56 |
"acc": 0.8333333333333334,
|
| 57 |
+
"nll": 0.39601716170144025,
|
| 58 |
+
"brier": 0.265688298438643,
|
| 59 |
+
"ece": 0.21775140100679702,
|
| 60 |
+
"mean_conf": 0.8963503037291343,
|
| 61 |
+
"confidence_bias": 0.0630169703958009,
|
| 62 |
+
"confident_error_rate": 0.0,
|
| 63 |
+
"coverage_at_0_9": 0.6666666666666666,
|
| 64 |
+
"accuracy_at_0_9": 1.0,
|
| 65 |
"coverage_at_5pct_error": 0.6666666666666666,
|
| 66 |
"coverage_at_1pct_error": 0.6666666666666666
|
| 67 |
},
|
| 68 |
"extraction": {
|
| 69 |
"n": 12,
|
| 70 |
"acc": 1.0,
|
| 71 |
+
"nll": 0.21387533479827292,
|
| 72 |
+
"brier": 0.0743310288444152,
|
| 73 |
+
"ece": 0.1816862006180468,
|
| 74 |
+
"mean_conf": 0.818313799381953,
|
| 75 |
+
"confidence_bias": -0.18168620061804697,
|
| 76 |
"confident_error_rate": 0.0,
|
| 77 |
+
"coverage_at_0_9": 0.3333333333333333,
|
| 78 |
"accuracy_at_0_9": 1.0,
|
| 79 |
"coverage_at_5pct_error": 1.0,
|
| 80 |
"coverage_at_1pct_error": 1.0
|
|
|
|
| 82 |
"intent": {
|
| 83 |
"n": 12,
|
| 84 |
"acc": 0.8333333333333334,
|
| 85 |
+
"nll": 0.32148899172819584,
|
| 86 |
+
"brier": 0.18188962676594325,
|
| 87 |
+
"ece": 0.1363148434412054,
|
| 88 |
+
"mean_conf": 0.7923110617523501,
|
| 89 |
+
"confidence_bias": -0.04102227158098326,
|
| 90 |
"confident_error_rate": 0.0,
|
| 91 |
+
"coverage_at_0_9": 0.3333333333333333,
|
| 92 |
"accuracy_at_0_9": 1.0,
|
| 93 |
+
"coverage_at_5pct_error": 0.8333333333333334,
|
| 94 |
+
"coverage_at_1pct_error": 0.8333333333333334
|
| 95 |
},
|
| 96 |
"ordinal": {
|
| 97 |
"n": 12,
|
| 98 |
"acc": 1.0,
|
| 99 |
+
"nll": 0.08771804229600423,
|
| 100 |
+
"brier": 0.01411749127368562,
|
| 101 |
+
"ece": 0.08218311475062134,
|
| 102 |
+
"mean_conf": 0.9178168852493788,
|
| 103 |
+
"confidence_bias": -0.08218311475062123,
|
| 104 |
"confident_error_rate": 0.0,
|
| 105 |
+
"coverage_at_0_9": 0.6666666666666666,
|
| 106 |
"accuracy_at_0_9": 1.0,
|
| 107 |
"coverage_at_5pct_error": 1.0,
|
| 108 |
"coverage_at_1pct_error": 1.0,
|
| 109 |
+
"score_mae": 0.06320981328072943,
|
| 110 |
+
"ranked_probability_score": 0.0022976210425868964
|
| 111 |
},
|
| 112 |
"policy": {
|
| 113 |
"n": 12,
|
| 114 |
"acc": 1.0,
|
| 115 |
+
"nll": 0.05347539889458897,
|
| 116 |
+
"brier": 0.009156676638993574,
|
| 117 |
+
"ece": 0.05095000022134209,
|
| 118 |
+
"mean_conf": 0.949049999778658,
|
| 119 |
+
"confidence_bias": -0.050950000221341996,
|
| 120 |
"confident_error_rate": 0.0,
|
| 121 |
+
"coverage_at_0_9": 0.9166666666666666,
|
| 122 |
"accuracy_at_0_9": 1.0,
|
| 123 |
"coverage_at_5pct_error": 1.0,
|
| 124 |
"coverage_at_1pct_error": 1.0
|
| 125 |
},
|
| 126 |
"routing": {
|
| 127 |
"n": 12,
|
| 128 |
+
"acc": 0.9166666666666666,
|
| 129 |
+
"nll": 0.2141717803151479,
|
| 130 |
+
"brier": 0.09544947856627449,
|
| 131 |
+
"ece": 0.1591324080778615,
|
| 132 |
+
"mean_conf": 0.831476375122774,
|
| 133 |
+
"confidence_bias": -0.08519029154389268,
|
| 134 |
"confident_error_rate": 0.0,
|
| 135 |
+
"coverage_at_0_9": 0.5,
|
| 136 |
"accuracy_at_0_9": 1.0,
|
| 137 |
+
"coverage_at_5pct_error": 0.9166666666666666,
|
| 138 |
+
"coverage_at_1pct_error": 0.9166666666666666
|
| 139 |
}
|
| 140 |
},
|
| 141 |
"sources": {
|
| 142 |
"adequacy": {
|
| 143 |
"n": 12,
|
| 144 |
"acc": 0.8333333333333334,
|
| 145 |
+
"nll": 0.39601716170144025,
|
| 146 |
+
"brier": 0.265688298438643,
|
| 147 |
+
"ece": 0.21775140100679702,
|
| 148 |
+
"mean_conf": 0.8963503037291343,
|
| 149 |
+
"confidence_bias": 0.0630169703958009,
|
| 150 |
+
"confident_error_rate": 0.0,
|
| 151 |
+
"coverage_at_0_9": 0.6666666666666666,
|
| 152 |
+
"accuracy_at_0_9": 1.0,
|
| 153 |
"coverage_at_5pct_error": 0.6666666666666666,
|
| 154 |
"coverage_at_1pct_error": 0.6666666666666666
|
| 155 |
},
|
| 156 |
"extraction": {
|
| 157 |
"n": 12,
|
| 158 |
"acc": 1.0,
|
| 159 |
+
"nll": 0.21387533479827292,
|
| 160 |
+
"brier": 0.0743310288444152,
|
| 161 |
+
"ece": 0.1816862006180468,
|
| 162 |
+
"mean_conf": 0.818313799381953,
|
| 163 |
+
"confidence_bias": -0.18168620061804697,
|
| 164 |
"confident_error_rate": 0.0,
|
| 165 |
+
"coverage_at_0_9": 0.3333333333333333,
|
| 166 |
"accuracy_at_0_9": 1.0,
|
| 167 |
"coverage_at_5pct_error": 1.0,
|
| 168 |
"coverage_at_1pct_error": 1.0
|
|
|
|
| 170 |
"intent": {
|
| 171 |
"n": 12,
|
| 172 |
"acc": 0.8333333333333334,
|
| 173 |
+
"nll": 0.32148899172819584,
|
| 174 |
+
"brier": 0.18188962676594325,
|
| 175 |
+
"ece": 0.1363148434412054,
|
| 176 |
+
"mean_conf": 0.7923110617523501,
|
| 177 |
+
"confidence_bias": -0.04102227158098326,
|
| 178 |
"confident_error_rate": 0.0,
|
| 179 |
+
"coverage_at_0_9": 0.3333333333333333,
|
| 180 |
"accuracy_at_0_9": 1.0,
|
| 181 |
+
"coverage_at_5pct_error": 0.8333333333333334,
|
| 182 |
+
"coverage_at_1pct_error": 0.8333333333333334
|
| 183 |
},
|
| 184 |
"ordinal": {
|
| 185 |
"n": 12,
|
| 186 |
"acc": 1.0,
|
| 187 |
+
"nll": 0.08771804229600423,
|
| 188 |
+
"brier": 0.01411749127368562,
|
| 189 |
+
"ece": 0.08218311475062134,
|
| 190 |
+
"mean_conf": 0.9178168852493788,
|
| 191 |
+
"confidence_bias": -0.08218311475062123,
|
| 192 |
"confident_error_rate": 0.0,
|
| 193 |
+
"coverage_at_0_9": 0.6666666666666666,
|
| 194 |
"accuracy_at_0_9": 1.0,
|
| 195 |
"coverage_at_5pct_error": 1.0,
|
| 196 |
"coverage_at_1pct_error": 1.0,
|
| 197 |
+
"score_mae": 0.06320981328072943,
|
| 198 |
+
"ranked_probability_score": 0.0022976210425868964
|
| 199 |
},
|
| 200 |
"policy": {
|
| 201 |
"n": 12,
|
| 202 |
"acc": 1.0,
|
| 203 |
+
"nll": 0.05347539889458897,
|
| 204 |
+
"brier": 0.009156676638993574,
|
| 205 |
+
"ece": 0.05095000022134209,
|
| 206 |
+
"mean_conf": 0.949049999778658,
|
| 207 |
+
"confidence_bias": -0.050950000221341996,
|
| 208 |
"confident_error_rate": 0.0,
|
| 209 |
+
"coverage_at_0_9": 0.9166666666666666,
|
| 210 |
"accuracy_at_0_9": 1.0,
|
| 211 |
"coverage_at_5pct_error": 1.0,
|
| 212 |
"coverage_at_1pct_error": 1.0
|
| 213 |
},
|
| 214 |
"routing": {
|
| 215 |
"n": 12,
|
| 216 |
+
"acc": 0.9166666666666666,
|
| 217 |
+
"nll": 0.2141717803151479,
|
| 218 |
+
"brier": 0.09544947856627449,
|
| 219 |
+
"ece": 0.1591324080778615,
|
| 220 |
+
"mean_conf": 0.831476375122774,
|
| 221 |
+
"confidence_bias": -0.08519029154389268,
|
| 222 |
"confident_error_rate": 0.0,
|
| 223 |
+
"coverage_at_0_9": 0.5,
|
| 224 |
"accuracy_at_0_9": 1.0,
|
| 225 |
+
"coverage_at_5pct_error": 0.9166666666666666,
|
| 226 |
+
"coverage_at_1pct_error": 0.9166666666666666
|
| 227 |
}
|
| 228 |
},
|
| 229 |
"variants": {
|
| 230 |
"clean": {
|
| 231 |
"n": 72,
|
| 232 |
+
"acc": 0.9305555555555556,
|
| 233 |
+
"nll": 0.21445778495560835,
|
| 234 |
+
"brier": 0.10677210008799252,
|
| 235 |
+
"ece": 0.07532617080884235,
|
| 236 |
+
"mean_conf": 0.867553070835708,
|
| 237 |
+
"confidence_bias": -0.0630024847198476,
|
| 238 |
+
"confident_error_rate": 0.0,
|
| 239 |
+
"coverage_at_0_9": 0.5694444444444444,
|
| 240 |
+
"accuracy_at_0_9": 1.0,
|
| 241 |
"coverage_at_5pct_error": 0.9444444444444444,
|
| 242 |
+
"coverage_at_1pct_error": 0.6944444444444444,
|
| 243 |
+
"score_mae": 0.06320981328072943,
|
| 244 |
+
"ranked_probability_score": 0.0022976210425868964
|
| 245 |
}
|
| 246 |
},
|
| 247 |
+
"macro_task_acc": 0.9305555555555555,
|
| 248 |
+
"objective": -0.21445778495560833,
|
| 249 |
"paired_flip": {
|
| 250 |
"pairs": 0,
|
| 251 |
"incomplete_pairs": 0,
|
|
|
|
| 275 |
},
|
| 276 |
"latency_ms": {
|
| 277 |
"n": 72,
|
| 278 |
+
"mean": 47.179252678890606,
|
| 279 |
+
"p50": 41.166106500895694,
|
| 280 |
+
"p95": 90.7176292530494,
|
| 281 |
+
"p99": 103.64775064750576
|
| 282 |
}
|
| 283 |
}
|
| 284 |
},
|
release-manifest.json
CHANGED
|
@@ -18,9 +18,23 @@
|
|
| 18 |
"serving": {
|
| 19 |
"wording": "native",
|
| 20 |
"system_prompt": null,
|
| 21 |
-
"temperature": 1.65,
|
| 22 |
"served_model_name": "standard-one-8b",
|
| 23 |
"engine": "SGLang 0.5.20",
|
| 24 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
}
|
| 26 |
}
|
|
|
|
| 18 |
"serving": {
|
| 19 |
"wording": "native",
|
| 20 |
"system_prompt": null,
|
|
|
|
| 21 |
"served_model_name": "standard-one-8b",
|
| 22 |
"engine": "SGLang 0.5.20",
|
| 23 |
+
"temperature_by_type": {
|
| 24 |
+
"choice": 0.85,
|
| 25 |
+
"noul": 0.85,
|
| 26 |
+
"score": 0.7
|
| 27 |
+
},
|
| 28 |
+
"default_temperature": 0.85,
|
| 29 |
+
"selection": "wording and temperatures chosen on non-JevBench data (evidence/prompt-wording-choice.json, evidence/calibration-per-type-served.json)",
|
| 30 |
+
"alternative_wording": {
|
| 31 |
+
"wording": "served",
|
| 32 |
+
"default_temperature": 0.8,
|
| 33 |
+
"temperature_by_type": {
|
| 34 |
+
"choice": 0.8,
|
| 35 |
+
"noul": 0.9,
|
| 36 |
+
"score": 0.95
|
| 37 |
+
}
|
| 38 |
+
}
|
| 39 |
}
|
| 40 |
}
|