Typed Decision Leaderboard
๐ฏ
34
Answer verifiers on one identical test set
Proto_AGI PRO
mayafree
AI & ML interests
None yet
Recent Activity
updated a Space about 11 hours ago
mayafree/typed-decision-leaderboard upvoted an article 1 day ago
Your model already knows it's wrong. Asking costs 0.06 seconds and zero tokens. liked a model 1 day ago
FINAL-Bench/ZTC-Judge-9BOrganizations
updated a
Space about 11 hours ago
upvoted an article 1 day ago
Article
Your model already knows it's wrong. Asking costs 0.06 seconds and zero tokens.
FINAL-Bench
โข โข 11 liked 2
models 1 day ago
reacted to SeaWolf-AI's post with ๐ 1 day ago
Post
2430
Ask a language model how confident it is and you get an AUC of 0.5000. Exactly a coin flip. We measured it across 2,018 items.
FINAL-Bench/gate-tetris
https://huggingface.co/blog/FINAL-Bench/ztc
Collection: https://huggingface.co/collections/FINAL-Bench/ztc-models-jev-ecosystems
Zero-Token Confidence (ZTC) reads it. One forward pass over the model's hidden state returns a calibrated probability that the answer is correct. Zero generated tokens.
It sits at the top of the shared board. Same 2,018 items, same harness for every entry: ZTC on Darwin-397B 0.7394, JEV 0.7335, ZTC-Judge-27B 0.7255, a surface baseline that reads only answer length and formatting 0.7036, Lynx 8B 0.5157, the model's own self-reported confidence 0.5000, HHEM 0.4852. First and third place both emit nothing at all.
The number worth staring at is 0.7036. That is a baseline reading no content whatsoever, just how long the answer is and how it is formatted. Any verifier scoring below it is not reading content either.
On speed, one gate call costs 0.0615 seconds, measured on four B200s across 2,000 items. Generating a single candidate answer takes 1.631 seconds, so the gate is 26 times cheaper than the work it guards. A verifier that generates competes with your agent for the same budget. A verifier that only reads can be attached to every action instead of a sampled few.
We built it so you can watch it decide. Three lanes receive the same stream of proposed actions and the same time budget. One has no gate and must execute everything. One uses a text-reading verifier. One uses ZTC. Right action plus one, wrong action minus one, hold zero. Over 400 matches: no gate minus 3.9, text verifier plus 13.0, ZTC plus 29.1, with ZTC taking 98 percent of matches. Gating lifts executed accuracy from 49 percent to 65 percent.
FINAL-Bench/gate-tetris
https://huggingface.co/blog/FINAL-Bench/ztc
Collection: https://huggingface.co/collections/FINAL-Bench/ztc-models-jev-ecosystems
Zero-Token Confidence (ZTC) reads it. One forward pass over the model's hidden state returns a calibrated probability that the answer is correct. Zero generated tokens.
It sits at the top of the shared board. Same 2,018 items, same harness for every entry: ZTC on Darwin-397B 0.7394, JEV 0.7335, ZTC-Judge-27B 0.7255, a surface baseline that reads only answer length and formatting 0.7036, Lynx 8B 0.5157, the model's own self-reported confidence 0.5000, HHEM 0.4852. First and third place both emit nothing at all.
The number worth staring at is 0.7036. That is a baseline reading no content whatsoever, just how long the answer is and how it is formatted. Any verifier scoring below it is not reading content either.
On speed, one gate call costs 0.0615 seconds, measured on four B200s across 2,000 items. Generating a single candidate answer takes 1.631 seconds, so the gate is 26 times cheaper than the work it guards. A verifier that generates competes with your agent for the same budget. A verifier that only reads can be attached to every action instead of a sampled few.
We built it so you can watch it decide. Three lanes receive the same stream of proposed actions and the same time budget. One has no gate and must execute everything. One uses a text-reading verifier. One uses ZTC. Right action plus one, wrong action minus one, hold zero. Over 400 matches: no gate minus 3.9, text verifier plus 13.0, ZTC plus 29.1, with ZTC taking 98 percent of matches. Gating lifts executed accuracy from 49 percent to 65 percent.
reacted to their post with ๐ค๐๐ 2 days ago
Post
3361
JEV Ecosystems โ every answer-verification vendor publishes a benchmark, and every one of them wins it. So we ran 13 of them on one test set: 2,018 items, identical labels, same grading code.
๐ฏ Leaderboard
mayafree/typed-decision-leaderboard
๐ Full write-up (method, mechanism, limits)
https://huggingface.co/blog/mayafree/jve-ecosystems
๐งช Try it โ ZTC, JEV and Laya on the same input, side by side
mayafree/verifier-playground
Three results
1๏ธโฃ Only three systems clear 0.70 โ ZTC (397B) 0.7364 ยท JEV 0.7350 ยท ZTC (27B) 0.7282. First and second differ by 0.0014, so no rank is assigned.
2๏ธโฃ A baseline that reads nothing but answer length and formatting scores 0.7036. Eight of the thirteen fall below it. A leaderboard without that line is flattering its entrants.
3๏ธโฃ Bigger does not win. On scientific reasoning, 27B 0.7410 beats 397B 0.6287 โ a model fourteen times larger scoring 0.11 lower.
And AUC is not the number you deploy on.
Same 20% retry budget, wired into an agent loop, against a 74.83% no-gate baseline:
ZTC +1.34 pp ยท JEV โ0.07 pp ยท random โ0.25 pp.
The mechanism is the interesting part. Re-answering is double-edged: 38% of wrong answers get fixed, and 30% of right answers get broken. So a gate is paid for by precision, not recall. Of the 403 items JEV routed for a retry, 216 were already correct.
0.0014 AUC apart; 1.4 points of end-to-end agent accuracy apart.
Scores, labels and grading code are published in full. Four public reproductions that would not run from their released artefacts are listed too, with the failure and a link, and no score.
Don't take the table's word for it โ paste your own case into the playground and watch all three answer at once. Want a system added? Open a discussion on the Space.
๐ฏ Leaderboard
mayafree/typed-decision-leaderboard
๐ Full write-up (method, mechanism, limits)
https://huggingface.co/blog/mayafree/jve-ecosystems
๐งช Try it โ ZTC, JEV and Laya on the same input, side by side
mayafree/verifier-playground
Three results
1๏ธโฃ Only three systems clear 0.70 โ ZTC (397B) 0.7364 ยท JEV 0.7350 ยท ZTC (27B) 0.7282. First and second differ by 0.0014, so no rank is assigned.
2๏ธโฃ A baseline that reads nothing but answer length and formatting scores 0.7036. Eight of the thirteen fall below it. A leaderboard without that line is flattering its entrants.
3๏ธโฃ Bigger does not win. On scientific reasoning, 27B 0.7410 beats 397B 0.6287 โ a model fourteen times larger scoring 0.11 lower.
And AUC is not the number you deploy on.
Same 20% retry budget, wired into an agent loop, against a 74.83% no-gate baseline:
ZTC +1.34 pp ยท JEV โ0.07 pp ยท random โ0.25 pp.
The mechanism is the interesting part. Re-answering is double-edged: 38% of wrong answers get fixed, and 30% of right answers get broken. So a gate is paid for by precision, not recall. Of the 403 items JEV routed for a retry, 216 were already correct.
0.0014 AUC apart; 1.4 points of end-to-end agent accuracy apart.
Scores, labels and grading code are published in full. Four public reproductions that would not run from their released artefacts are listed too, with the failure and a link, and no score.
Don't take the table's word for it โ paste your own case into the playground and watch all three answer at once. Want a system added? Open a discussion on the Space.
liked a
Space 2 days ago
liked a
model 2 days ago
upvoted a collection 2 days ago
liked a
Space 2 days ago
reacted to their post with ๐ฅ 2 days ago
Post
3361
JEV Ecosystems โ every answer-verification vendor publishes a benchmark, and every one of them wins it. So we ran 13 of them on one test set: 2,018 items, identical labels, same grading code.
๐ฏ Leaderboard
mayafree/typed-decision-leaderboard
๐ Full write-up (method, mechanism, limits)
https://huggingface.co/blog/mayafree/jve-ecosystems
๐งช Try it โ ZTC, JEV and Laya on the same input, side by side
mayafree/verifier-playground
Three results
1๏ธโฃ Only three systems clear 0.70 โ ZTC (397B) 0.7364 ยท JEV 0.7350 ยท ZTC (27B) 0.7282. First and second differ by 0.0014, so no rank is assigned.
2๏ธโฃ A baseline that reads nothing but answer length and formatting scores 0.7036. Eight of the thirteen fall below it. A leaderboard without that line is flattering its entrants.
3๏ธโฃ Bigger does not win. On scientific reasoning, 27B 0.7410 beats 397B 0.6287 โ a model fourteen times larger scoring 0.11 lower.
And AUC is not the number you deploy on.
Same 20% retry budget, wired into an agent loop, against a 74.83% no-gate baseline:
ZTC +1.34 pp ยท JEV โ0.07 pp ยท random โ0.25 pp.
The mechanism is the interesting part. Re-answering is double-edged: 38% of wrong answers get fixed, and 30% of right answers get broken. So a gate is paid for by precision, not recall. Of the 403 items JEV routed for a retry, 216 were already correct.
0.0014 AUC apart; 1.4 points of end-to-end agent accuracy apart.
Scores, labels and grading code are published in full. Four public reproductions that would not run from their released artefacts are listed too, with the failure and a link, and no score.
Don't take the table's word for it โ paste your own case into the playground and watch all three answer at once. Want a system added? Open a discussion on the Space.
๐ฏ Leaderboard
mayafree/typed-decision-leaderboard
๐ Full write-up (method, mechanism, limits)
https://huggingface.co/blog/mayafree/jve-ecosystems
๐งช Try it โ ZTC, JEV and Laya on the same input, side by side
mayafree/verifier-playground
Three results
1๏ธโฃ Only three systems clear 0.70 โ ZTC (397B) 0.7364 ยท JEV 0.7350 ยท ZTC (27B) 0.7282. First and second differ by 0.0014, so no rank is assigned.
2๏ธโฃ A baseline that reads nothing but answer length and formatting scores 0.7036. Eight of the thirteen fall below it. A leaderboard without that line is flattering its entrants.
3๏ธโฃ Bigger does not win. On scientific reasoning, 27B 0.7410 beats 397B 0.6287 โ a model fourteen times larger scoring 0.11 lower.
And AUC is not the number you deploy on.
Same 20% retry budget, wired into an agent loop, against a 74.83% no-gate baseline:
ZTC +1.34 pp ยท JEV โ0.07 pp ยท random โ0.25 pp.
The mechanism is the interesting part. Re-answering is double-edged: 38% of wrong answers get fixed, and 30% of right answers get broken. So a gate is paid for by precision, not recall. Of the 403 items JEV routed for a retry, 216 were already correct.
0.0014 AUC apart; 1.4 points of end-to-end agent accuracy apart.
Scores, labels and grading code are published in full. Four public reproductions that would not run from their released artefacts are listed too, with the failure and a link, and no score.
Don't take the table's word for it โ paste your own case into the playground and watch all three answer at once. Want a system added? Open a discussion on the Space.
posted an update 2 days ago
Post
3361
JEV Ecosystems โ every answer-verification vendor publishes a benchmark, and every one of them wins it. So we ran 13 of them on one test set: 2,018 items, identical labels, same grading code.
๐ฏ Leaderboard
mayafree/typed-decision-leaderboard
๐ Full write-up (method, mechanism, limits)
https://huggingface.co/blog/mayafree/jve-ecosystems
๐งช Try it โ ZTC, JEV and Laya on the same input, side by side
mayafree/verifier-playground
Three results
1๏ธโฃ Only three systems clear 0.70 โ ZTC (397B) 0.7364 ยท JEV 0.7350 ยท ZTC (27B) 0.7282. First and second differ by 0.0014, so no rank is assigned.
2๏ธโฃ A baseline that reads nothing but answer length and formatting scores 0.7036. Eight of the thirteen fall below it. A leaderboard without that line is flattering its entrants.
3๏ธโฃ Bigger does not win. On scientific reasoning, 27B 0.7410 beats 397B 0.6287 โ a model fourteen times larger scoring 0.11 lower.
And AUC is not the number you deploy on.
Same 20% retry budget, wired into an agent loop, against a 74.83% no-gate baseline:
ZTC +1.34 pp ยท JEV โ0.07 pp ยท random โ0.25 pp.
The mechanism is the interesting part. Re-answering is double-edged: 38% of wrong answers get fixed, and 30% of right answers get broken. So a gate is paid for by precision, not recall. Of the 403 items JEV routed for a retry, 216 were already correct.
0.0014 AUC apart; 1.4 points of end-to-end agent accuracy apart.
Scores, labels and grading code are published in full. Four public reproductions that would not run from their released artefacts are listed too, with the failure and a link, and no score.
Don't take the table's word for it โ paste your own case into the playground and watch all three answer at once. Want a system added? Open a discussion on the Space.
๐ฏ Leaderboard
mayafree/typed-decision-leaderboard
๐ Full write-up (method, mechanism, limits)
https://huggingface.co/blog/mayafree/jve-ecosystems
๐งช Try it โ ZTC, JEV and Laya on the same input, side by side
mayafree/verifier-playground
Three results
1๏ธโฃ Only three systems clear 0.70 โ ZTC (397B) 0.7364 ยท JEV 0.7350 ยท ZTC (27B) 0.7282. First and second differ by 0.0014, so no rank is assigned.
2๏ธโฃ A baseline that reads nothing but answer length and formatting scores 0.7036. Eight of the thirteen fall below it. A leaderboard without that line is flattering its entrants.
3๏ธโฃ Bigger does not win. On scientific reasoning, 27B 0.7410 beats 397B 0.6287 โ a model fourteen times larger scoring 0.11 lower.
And AUC is not the number you deploy on.
Same 20% retry budget, wired into an agent loop, against a 74.83% no-gate baseline:
ZTC +1.34 pp ยท JEV โ0.07 pp ยท random โ0.25 pp.
The mechanism is the interesting part. Re-answering is double-edged: 38% of wrong answers get fixed, and 30% of right answers get broken. So a gate is paid for by precision, not recall. Of the 403 items JEV routed for a retry, 216 were already correct.
0.0014 AUC apart; 1.4 points of end-to-end agent accuracy apart.
Scores, labels and grading code are published in full. Four public reproductions that would not run from their released artefacts are listed too, with the failure and a link, and no score.
Don't take the table's word for it โ paste your own case into the playground and watch all three answer at once. Want a system added? Open a discussion on the Space.
liked a
Space 2 days ago
updated a
Space 2 days ago
published a
Space 2 days ago
upvoted an article 2 days ago
Article
Inside the JEV Ecosystem: 13 Answer Verifiers on One Test Set
mayafree
โข โข 11 published an article 2 days ago
Article
Inside the JEV Ecosystem: 13 Answer Verifiers on One Test Set
mayafree
โข โข 11 published a
Space 2 days ago