Buckets:
| # RL Agent evaluation | |
| Metrics after calibration. Zero-shot = task families held out of training entirely. | |
| ## In-task | |
| | task family | questions | accuracy | ECE | NLL | | |
| |---|---|---|---|---| | |
| | conversation outcomes | 3600 | 0.482 | 0.019 | 0.693 | | |
| | email triage | 2691 | 0.732 | 0.017 | 0.595 | | |
| | emotion and tone | 1825 | 0.906 | 0.018 | 0.238 | | |
| | inference and fact checking | 3022 | 0.883 | 0.054 | 0.340 | | |
| | instruction-following tasks | 600 | 0.878 | 0.046 | 0.302 | | |
| | intent and routing | 1475 | 0.991 | 0.009 | 0.181 | | |
| | moderation and safety | 2708 | 0.967 | 0.061 | 0.153 | | |
| | reading comprehension | 770 | 0.847 | 0.083 | 0.409 | | |
| | response quality scoring | 3146 | 0.581 | 0.023 | 1.009 | | |
| | robustness checks | 744 | 0.851 | 0.108 | 1.058 | | |
| | search relevance | 733 | 0.628 | 0.066 | 0.728 | | |
| | sentiment and rating | 961 | 0.442 | 0.438 | 3.545 | | |
| | topic classification | 749 | 0.939 | 0.029 | 0.196 | | |
| Overall: accuracy 0.753, ECE 0.030, Brier 0.308, accuracy at 50% coverage 0.947 | |
| ## Zero-shot | |
| | task family | questions | accuracy | ECE | NLL | | |
| |---|---|---|---|---| | |
| | emotion and tone | 600 | 0.583 | 0.318 | 1.976 | | |
| | instruction-following tasks | 600 | 0.863 | 0.045 | 0.319 | | |
| | moderation and safety | 600 | 0.797 | 0.171 | 1.415 | | |
| | sentiment and rating | 600 | 0.362 | 0.291 | 1.798 | | |
| Overall: accuracy 0.651, ECE 0.204, Brier 0.532, accuracy at 50% coverage 0.818 | |
| ## Latency | |
| ``` | |
| { | |
| "1_questions": { | |
| "p50_ms": 38.4, | |
| "p95_ms": 42.1 | |
| }, | |
| "10_questions": { | |
| "p50_ms": 156.0, | |
| "p95_ms": 158.4 | |
| }, | |
| "50_questions": { | |
| "p50_ms": 721.4, | |
| "p95_ms": 733.0 | |
| } | |
| } | |
| ``` | |
Xet Storage Details
- Size:
- 1.61 kB
- Xet hash:
- 09eb9fa7b5064dd32a9fce76d55908653bd1f3419e0167c7ebd14fd80fa9ea04
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.