Thanks, Dipankar, both points are fair to raise.

On the post line: the explorer leaves nanoGentzen-Jev-v3 out of the zero-shot view, and the footer gives the reason (its own eval YAML in #59 describes it as fitted on train). The post followed that view without saying so; I should have. And yes, "above Jev" there is a point estimate: by the explorer's own rule neither 0.731 nor 0.743 separates from 0.727.

On n: you're right that the 5 decisions per case are not independent, so I ran a case bootstrap on the submissions that ship per-decision predictions (od1-typed-decisions, Decider-4B, nullspan v1.3.0, Qwen3.8-27B from #60), using the scorer's own accuracy rule. Design effects came out at 1.0–1.4, so n_eff is about 1,450–2,000, not 400. For comparing two models it helps further that both answer the same cases: the paired, case-resampled SE of the difference is 0.64–0.90× the independent n = 2,000 SE the explorer uses, so its tie band is, if anything, conservative. Caveat: only four models, and none of the current top zero-shot rows publish predictions, so I can't check those directly. If more submissions add predictions, a paired case bootstrap would be the right rule and I'll switch to it.

Thanks for re-running it. Since then I found more predictions: the OpenDecider repo (#8) publishes per-decision files for its own models and for its runs of Jev 1.13 (TypeSafe API), microsoft-decision-1 and clm-8b (the last two are not on the card), so the check now covers 12 runs. Numbers below use the analytic case-level SE, which matches the bootstrap and gives 1.35 for od1 as you found.

On the top: deff does grow with accuracy (Spearman 0.82 over the 12), from 1.02–1.18 below 0.67 to 1.28–1.40 above 0.78. The highest is Decider-4B at 0.827 with 1.40, n_eff about 1,430. So it rises, but in this sample it levels off near 1.4.

Paired against Jev (OpenDecider's whole-case run, 0.7305 under the dataset scorer's argmax rule, 0.737 in their own report, which breaks the 35 tied gold labels differently; the card's 0.727 is a separate run), the paired SE is 0.72–0.93x the independent one, median 0.79. d3-mini's +0.016 stays inside the band even at 0.72x.

On showing the paired band when both rows ship predictions: I'd rather not mix two kinds of band in one table yet. Of the 90 rows, only about ten have public predictions, two of them zero-shot, and the only Jev predictions come from a third-party run rather than the card's. The cleaner fix is upstream: score_with_jobs.py writes only the summary to results.json; if it also wrote the per-decision answers, every new row would ship them and the explorer could move every comparison to a paired case bootstrap at once. @sandeep85 @codelion , would that be welcome as a PR?

Script, sources and output: https://huggingface.co/spaces/tardellirs/typed-decisions-explorer/tree/main/analysis

Confirmed on c9e8c416: 35 ties, all two-way, 17 of them including the first-listed option. One more thing in the same spot: the gold label field breaks those ties the other way in 31 of the 35, so scoring against label and running score_with_jobs.py give different numbers for the same predictions. Rows on the card use both: Decider-4B, od1 and OpenDecider match their card values under label; nullspan and Qwen3.8-27B (#60) match under the scorer's argmax. Across the 12 runs the two rules differ by 0.001 to 0.007.

On Jev, the 0.006 to 0.0065 gap splits into 9 decisions from gold ties and 3 from ties in the prediction itself. Jev returns probabilities rounded to two decimals, so 20 of its 2,000 answers have a tied top option, and list.index breaks those by option order too (OpenDecider-medium has 104).

So yes to split credit, and I'd apply it on both sides: the expected hit when gold ties and predicted ties are each broken at random, |top_gold ∩ top_pred| / (|top_gold| × |top_pred|). It is deterministic, independent of option order, and equal to the current rule when nothing is tied. For Jev's whole-case run it gives 0.7328, between the scorer's 0.7305 and label's 0.7350.

Agreed on the PR: save the normalized probabilities per decision, keyed on case id and question id, not the hits, so the tie rule becomes a scoring choice anyone can rerun.

Per-rule table for all 12 runs: https://huggingface.co/spaces/tardellirs/typed-decisions-explorer/tree/main/analysis

Reproduced your three-reading split on test, and train shows the same pattern (61 ties, label on the later-listed option in 53).

My best guess for the 4, which only the generator code can confirm: label was taken from the unrounded means, before the 6-decimal rounding, with ties going to the later option. Two things point that way. The gold values are means of three 2-decimal samples (p x 3 sits on the 0.01 grid for 98.5% of the 28,400 probabilities). And the noul ties are all exactly 0.5, with label "true" in 9 of 10 on test and 21 of 22 on train, which is what a ">= 0.5 is true" rule gives. The exceptions then come from float noise: summing three 2-decimal values in a different order changes the last bit in about 13% of permutations, so two options that tie after rounding can differ by ~1e-16 before it, and that decides it. customer_service_000066, the one noul tie labelled "false", is one of your 4.

If that's right, label is deterministic but not meaningful on ties, which is one more reason for symmetric split credit. I opened the PR that saves per-decision probabilities (#68 on the dataset) and left the tie rule out of it, so the maintainers can pick it separately.