Add --revision HF Hub support in serve_systemone.py and refine Table 2 notes
Browse files
README.md
CHANGED
|
@@ -35,7 +35,7 @@ pipeline_tag: question-answering
|
|
| 35 |
|
| 36 |
| Public Benchmark | Dataset & Split | Score | Calibration & Validity | Latency / Cost (L4 GPU) |
|
| 37 |
| :--- | :--- | :---: | :---: | :---: |
|
| 38 |
-
| **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions)** | Official `test` (`400` cases · `2,000` decisions) | **`75.40%` Accuracy** (
|
| 39 |
| **[Benchmark Heaven / `jevbench`](https://github.com/fstandhartinger/jevbench)** | Official Public Suite (`231` items via `/v1/systemone`) | **`100.0%` Easy (`48/48`)** · `62.50%` Original (`45/72`) · `37.84%` Hard (`42/111`) | **`100.0%` Strict Schema Validity** (`Easy Brier 0.0207`, `ECE 0.0605`) | **`64.8 ms` `p50`** · **`$0.00667 / 1k`** (`encoder_classifier_<=0.6B`) |
|
| 40 |
| **[Stanford `rajpurkar/squad_v2`](https://huggingface.co/datasets/rajpurkar/squad_v2)** | Zero-overlap `validation` (`N = 500`: `250` HasAns + `250` NoAns) | **`87.00%` Exact Match** · **`89.36%` F1** | `HasAns F1 91.52%` · `NoAns EM 87.20%` | `10.30 ms` `p50` · `$0.0017 / 1k` |
|
| 41 |
| **[`NullspanBench v3.2`](https://hemanth.github.io/nullspan/bench.html)** | `N = 500` (`336` Open + `164` Sealed: SQuAD2, `96` Contracts, `8` Adversarial) | **`98.2` Capability** · **`98.0%` Intelligence** (`95.7` Hybrid Composite) | **`98.5` Calibration** (`AUROC 0.9933`, `ECE 0.0142`; Int8 ONNX `ECE 0.0127`) | `10.30 ms` CUDA `p50` (`0.040 ms` Hybrid) · `$0.0003–$0.0017 / 1k` |
|
|
|
|
| 35 |
|
| 36 |
| Public Benchmark | Dataset & Split | Score | Calibration & Validity | Latency / Cost (L4 GPU) |
|
| 37 |
| :--- | :--- | :---: | :---: | :---: |
|
| 38 |
+
| **[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions)** | Official `test` (`400` cases · `2,000` decisions) | **`75.40%` Accuracy** (Table 2 fitted cross-encoder; `#2` vs Hub `.eval_results` widget) | `KL 0.1515` · `Brier 0.0770` · `ECE 0.0944` | `78.0 ms/dec` (`390.0 ms/case`) · `0` output tokens |
|
| 39 |
| **[Benchmark Heaven / `jevbench`](https://github.com/fstandhartinger/jevbench)** | Official Public Suite (`231` items via `/v1/systemone`) | **`100.0%` Easy (`48/48`)** · `62.50%` Original (`45/72`) · `37.84%` Hard (`42/111`) | **`100.0%` Strict Schema Validity** (`Easy Brier 0.0207`, `ECE 0.0605`) | **`64.8 ms` `p50`** · **`$0.00667 / 1k`** (`encoder_classifier_<=0.6B`) |
|
| 40 |
| **[Stanford `rajpurkar/squad_v2`](https://huggingface.co/datasets/rajpurkar/squad_v2)** | Zero-overlap `validation` (`N = 500`: `250` HasAns + `250` NoAns) | **`87.00%` Exact Match** · **`89.36%` F1** | `HasAns F1 91.52%` · `NoAns EM 87.20%` | `10.30 ms` `p50` · `$0.0017 / 1k` |
|
| 41 |
| **[`NullspanBench v3.2`](https://hemanth.github.io/nullspan/bench.html)** | `N = 500` (`336` Open + `164` Sealed: SQuAD2, `96` Contracts, `8` Adversarial) | **`98.2` Capability** · **`98.0%` Intelligence** (`95.7` Hybrid Composite) | **`98.5` Calibration** (`AUROC 0.9933`, `ECE 0.0142`; Int8 ONNX `ECE 0.0127`) | `10.30 ms` CUDA `p50` (`0.040 ms` Hybrid) · `$0.0003–$0.0017 / 1k` |
|