ZTC Models: JEV ecosystems
Collection
Answer verification from one forward pass, zero generated tokens. Measured against JEV and 10 others on 2,018 identical items. • 7 items • Updated • 16
ZTC ecosystem
Zero-Token Confidence (ZTC) reads a model's own internal state to judge whether an answer is right — generating zero tokens. Darwin-397B-ZTC ships a self-readout probe; ZTC-Judge-27B judges other models' answers; the arcade shows the gate deciding execute or hold in real time.
ZTC-Judge-27B
Ask a model how confident it is and you get a coin flip (AUC 0.500). This reads the same forward pass and gets 0.7282 on domains it has never seen — above the surface baseline in all five.
ZTC-Judge-27B takes a question and an answer written by any model and scores whether that answer can be trusted — without generating a single token.
ZTC — Zero-Token Confidence Judge — it evaluates someone else's answer, not its own
How it works
[question + answer] → one forward pass through the 27B model
→ final-layer hidden state at the last position (5,120-d)
→ one dot product with the probe
→ score
Generated tokens: 0. No access to the answering model's weights, logits or log-probabilities is required — the text of the answer is the only input. Because nothing is decoded, latency is one forward pass and batching translates directly into throughput.
Usage
from ztc_judge import ZTCJudge
judge = ZTCJudge.from_pretrained("FINAL-Bench/ZTC-Judge-27B")
judge.score(
"Which defensive compound does the insect release when it meets a predator?",
"C. Allomone",
domain="scientific_reasoning",
)
# {'score': -0.61, 'verdict': 'review', 'domain_auc': 0.7410, 'generated_tokens': 0}
Batched scoring (recommended — there is no decode step, so batch size is throughput):
judge.score_batch([(q1, a1), (q2, a2), ...], domain="scientific_reasoning")
The score is an unbounded real number; higher means more likely correct. It is a ranking signal, not a calibrated probability. Pick a threshold from your own review budget (see below).
Evaluation
Two independent measurements are reported. They use different test sets and different protocols, and are not interchangeable — read the protocol line before quoting a number.
A. Independent leaderboard — 2,018 items, leave-one-domain-out
The Typed Decision Leaderboard scores answer verifiers from several vendors on one identical item set with identical labels: https://huggingface.co/spaces/mayafree/typed-decision-leaderboard
| System | AUC |
|---|---|
| Darwin-397B-ZTC | 0.7364 |
| JEV (TypeSafe AI) | 0.7350 |
| ZTC-Judge-27B | 0.7282 |
| GPT-5.2 asked directly | 0.7148 |
| open-jev 4B | 0.6844 |
| Answer length and formatting only | 0.6223 |
| Patronus Lynx 8B | 0.5179 |
| The answering model's own stated confidence | 0.5000 |
Per domain, and against the surface baseline in the same domain:
| Domain | Baseline | ZTC-Judge-27B | Items | Wrong |
|---|---|---|---|---|
| Professional exams (law · math · biology) | 0.7138 | 0.8462 | 400 | 80 |
| Scientific reasoning | 0.7272 | 0.7410 | 198 | 20 |
| Biology & medicine | 0.5908 | 0.7154 | 917 | 169 |
| Disaster & safety procedures | 0.5949 | 0.6961 | 225 | 109 |
| General multi-step reasoning | 0.5420 | 0.6172 | 278 | 130 |
| Size-weighted mean | 0.6223 | 0.7282 | 2,018 | 508 |
Above the surface baseline in all five domains. That baseline — a model reading nothing but answer length, digit count and formatting — is the bar that matters; a verifier below it is detecting shape, not correctness.
Protocol. Every figure comes from a domain the probe never saw. Hyper-parameters are selected inside the training domains only. Scores are computed per domain and then size-weighted; pooling all items into one AUC inflates the result, because score scales differ between domains.
Self-readout. Given only the question, this model answers on its own and the same forward pass tells whether it was right: 0.7322 (3 domains, 1,595 items). Tools that see only text from outside a model cannot do this at all.
B. Repository-internal measurement — 400 items, random folds
| Judge | Qwen3.5-27B, revision fc05daec18b0 (weights unmodified) |
| Items | 400 professional-exam items, out-of-fold |
| Result | 0.7999 |
| Permutation null | z = 25.68 |
This number is higher than the leaderboard figure because the protocol is easier: one domain, random folds instead of held-out domains. Quote 0.7282 when comparing against other systems.
Two probes ship with this model
| File | Size | Produces |
|---|---|---|
ztc_probe.npz |
59 KB | linear readout — one dot product |
ztc_curve_probe.npz |
5.3 MB | the leaderboard figure, 0.7282 — 256 anchors, RBF kernel |
usage_curve.py |
— | runnable example for both |
Both read the same input: the final-layer hidden state at the last position, from a single forward pass with zero generated tokens. The curved probe is the one to use when the number matters; the linear probe is there for deployments where 59 KB and a single dot product are worth the difference.
Review budget
If you can only re-check part of your traffic, sort by score ascending and review the lowest.
| Reviewed | Errors caught | vs. random |
|---|---|---|
| 5% | 14.0% | 2.8× |
| 10% | 28.0% | 2.8× |
| 15% | 38.4% | 2.6× |
| 20% | 47.2% | 2.4× |
| 30% | 62.0% | 2.1× |
| 40% | 72.6% | 1.8× |
Intended use
- Triage of LLM output at scale — route low-scoring answers to a human, a larger model, or a retrieval step, and let the rest through.
- Regression monitoring — track the score distribution of a deployed system over time.
- Cheap pre-filter in front of an expensive verifier — costs one forward pass, no decoding.
Out of scope
- Not a grounding checker. It does not take a source document and decide whether the answer follows from it. Tools built for that task target a different axis.
- Not a safety, toxicity or policy classifier.
- Not a calibrated probability. Use it to rank and threshold, not as a likelihood.
- Not a substitute for human review in high-stakes decisions.
Limitations
- Domain coverage. Scores are only meaningful for the five domains above. When
domain_aucreturnsNone, the domain was never measured and the score carries no published guarantee. - Weakest domain is 0.6172 (general multi-step reasoning). General multi-step reasoning is the hardest case and is listed first in this section on purpose.
- Revision lock. The probe is fitted to one specific revision of the base model. Running it on a different Qwen3.5-27B revision produces no error and silently wrong scores. This repository ships the matching weights so that failure mode cannot occur.
- It reports the verifier's judgement, which is not the same thing as the answering model's own confidence — that quantity measures 0.500 here.
- Long answers are read within the model's context window; answers exceeding it are truncated before scoring.
Lineage
| Base model | Qwen/Qwen3.5-27B, revision fc05daec18b0, Apache-2.0 |
| Modification to base weights | none — the probe is a separate 59 KB file |
| Added by FINAL-Bench | probe, inference code, evaluation protocol and tables |
What this repository contains
| Included | Base weights · tokenizer · probe · inference code · evaluation configuration |
| Not included | Training corpus · hidden-state matrices · fitting pipeline |
The probe is the product; the material it was fitted from is not distributed.
Independent comparison
This model is listed on the Typed Decision Leaderboard, which measures answer verifiers from several vendors on one identical item set with identical labels, and publishes scores, labels and grading code:
https://huggingface.co/spaces/mayafree/typed-decision-leaderboard
License
The base model, Qwen3.5-27B, is Apache-2.0 and redistributable. The probe, the inference code and the evaluation tables are assets of FINAL-Bench / VIDRAFT.
Citation
@misc{ztc_judge_27b_2026,
title = {ZTC-Judge-27B: zero-token answer verification from a single forward pass},
author = {FINAL-Bench},
year = {2026},
url = {https://huggingface.co/FINAL-Bench/ZTC-Judge-27B}
}
- Downloads last month
- 6