NullspanBench — Decision & Extraction Model Leaderboard
Following the Benchmark Heaven (JevBench) evaluation protocol: comparing 19 systems across Intelligence (I), Probability Calibration (C), Speed (S), and Cost (K) over 500 zero-leakage observations (S+P) spanning Open structured extraction (n=336) and Sealed Stanford SQuAD v2.0 (n=60), Multi-Numeric Legal/Commercial Contracts (n=96), and Adversarial prompt-injection traps (n=8).
Pool Filter:
Official External Public Benchmarks · NVIDIA L4 GPU · Zero Test Contamination
Typed Decisions · JevBench · SQuAD v2.0
| External Leaderboard | Evaluated Split | Headline Score | Calibration & Schema Validity | Latency & Cost (L4 GPU) | Public Rank / Notes |
|---|---|---|---|---|---|
| LocalLLaMA/typed-decisions | Official test (400 cases · 2,000 decisions) |
79.00% Accuracy (noul 84.2%, choice 76.8%, score 76.8%) | KL 0.0760 (#2) · Brier 0.0420 (#1 tied) · ECE 0.1665 | 9.56 ms/dec (47.8 ms/case) · 0 output tokens | Specialist (general + train) · #1 Hub widget · 96.3% of Bayes ceiling |
| Benchmark Heaven / fstandhartinger/jevbench | Public Suite (231 items via /v1/systemone) |
100.0% Easy (48/48) · 68.06% Original · 36.04% Hard | 100.0% Strict Valid · ECE 0.0755 · v1.3 Geo 46.27 | 38.82 ms p50 · $0.00604 / 1k | 100% fact & tool_selection · 91.7% intent & ordinal |
| Stanford rajpurkar/squad_v2 | Zero-Overlap validation (500 items) |
87.80% Exact Match · 90.90% F1 | HasAns F1 90.61% · NoAns EM 91.20% | 10.30 ms p50 · $0.0017 / 1k | NullspanForExtraction 345.3M + qa_null_head |
1. NullspanBench Capability Score · Mean of Intelligence (I) & Calibration (C)
95% Bootstrap CIs (B=1,000) · S+P Pool
2. Pareto Frontier: Capability Against Cost & Speed
Upper-Right is Best (More Capable & Cheaper / Faster)
Capability vs Cost ($ / 1,000 Decisions · Log Scale)
Encoders generate 0 output tokens; autoregressive API decoders pay per generated JSON token.
Capability vs Speed (p50 Latency ms / Decision · Log Scale)
Bubble size follows the official Composite Score; dashed line traces the Pareto-optimal frontier.
3. Composite Score Leaderboard · Four Axes (0–100) Weighted Harmonic Mean
19 Ranked Systems
| # ▲ | System ↕ | Architecture / Runtime ↕ | Composite ↕ | Cap. ↕ | Intel. (I) ↕ | Calib. (C) ↕ | Speed (S) ↕ | Cost (K) ↕ | $/1k dec. ↕ | p50 ms ↕ | Null ECE ↕ | Open (S) ↕ | Sealed (P) ↕ |
|---|
4. Compare Two Systems · Multi-Axis Radar Diagnostics
Four Axes · Request Types · Difficulty Tiers · Subject Domains
The Four Score Axes (0–100)
Competence per Request Type (Open / Sealed)
Competence per Difficulty Tier (Open & Sealed)
Competence by Subject Domain / Use Case
5. Competence Heatmap by Request Type, Difficulty Tier & Subject Domain
Greener = Stronger Within Column · Exact Observation Counts (n=500)
| System | Span (Open) | Span (Sealed) | Noul / p_null (Open) | Noul / p_null (Sealed) | Choice (Bool/Enum) | Math & Invariants | Score (Quantiles) | Tier: Easy | Tier: Standard | Tier: Judge | Tier: Hard | SRE Incident | Cloud Invoice | SQuAD v2.0 QA | Contracts (96) | Adversarial |
|---|