ZTC-Judge-4B

Answer verification from a single forward pass, with zero generated tokens — at 4B.

ZTC-Judge-4B takes a question and an answer written by any model and scores whether that answer can be trusted. It is the bottom rung of a four-point size ladder measured under one identical protocol, and it is published so the shape of that ladder can be checked rather than asserted.

ZTC — Zero-Token Confidence · Judge — it evaluates someone else's answer, not its own

---

Read this before deploying

This model is not the strongest member of the family, and the card says so with numbers.

Model	Leaderboard AUC
Darwin-397B-ZTC	0.7364
ZTC-Judge-27B	0.7282
ZTC-Judge-9B	0.6506
ZTC-Judge-4B	0.6360
Answer length and formatting only	0.6223

The ladder does not decline smoothly — it steps. Between 9B and 27B the score moves 0.078, while between 4B and 9B it moves 0.015. Whatever carries verification quality is largely absent below 27B on this axis.

Where this model is worth deploying is one specific place, and it is a real one:

Domain	Surface baseline	4B	Margin
Professional exams (law · math · biology)	0.7138	0.7787	+0.0649
Scientific reasoning	0.7272	0.5576	🔴 -0.1696
Biology & medicine	0.5908	0.6223	+0.0315
Disaster & safety procedures	0.5949	0.5842	🔴 -0.0107
General multi-step reasoning	0.5420	0.5738	+0.0318
Size-weighted mean	0.6223	0.6360	+0.0137

🔴 Do not use this model for disaster and safety content. In that domain it does not clear the surface baseline, which means it is reading answer shape rather than correctness there.

✅ Professional-exam style content is where it earns its size. It runs on a laptop, on CPU, and inside networks that never reach the internet — places a hosted API cannot go.

How it works

[question + answer]  →  one forward pass
                     →  final-layer hidden state at the last position (2560-d)
                     →  probe
                     →  score

Generated tokens: 0. No access to the answering model's weights or logits is required; the text of the answer is the only input. Latency is one forward pass, and batching converts directly into throughput.

Usage

import json
import numpy as np, torch
from huggingface_hub import hf_hub_download, snapshot_download
from transformers import AutoModel, AutoTokenizer

REPO = "FINAL-Bench/ZTC-Judge-4B"
cfg  = json.load(open(hf_hub_download(REPO, "ztc_config.json"), encoding="utf-8"))
path = snapshot_download(REPO)

tok   = AutoTokenizer.from_pretrained(path)
model = AutoModel.from_pretrained(path, dtype=torch.bfloat16, low_cpu_mem_usage=True).eval()

def hidden(question, answer):
    b = tok([cfg["template"] % (question, answer)], return_tensors="pt",
            truncation=True, max_length=cfg["max_length"])
    dev = next(model.parameters()).device
    with torch.no_grad():
        h = model(input_ids=b["input_ids"].to(dev),
                  attention_mask=b["attention_mask"].to(dev)).last_hidden_state
    return h[0, int(b["attention_mask"].sum()) - 1].float().numpy().astype(np.float64)

# linear probe — one dot product
p = np.load(hf_hub_download(REPO, "ztc_probe.npz"))
v = hidden("Which defensive chemical does an insect release?", "C. Allomone")
print(float(((v - p["mu"]) / p["sd"]) @ p["w"]))

The score is an unbounded real number; higher means more likely correct. It is a ranking signal, not a calibrated probability — choose a threshold from your own review budget.

Two probes ship with this model

File	Produces
ztc_probe.npz	linear readout — one dot product
ztc_curve_probe.npz	the reported figure 0.6360 — 256 anchors, RBF kernel

Both read the same input. The curved probe is the one to use when the number matters.

Protocol

Items	2,018 · 508 incorrect · 5 domains · answers written by 4 different models
Metric	AUC — how well wrong answers sort to the bottom. Threshold-free. 0.5 = coin flip
Selection	Leave-one-domain-out. Every figure comes from a domain the probe never saw; hyper-parameters are chosen inside the training domains only
Aggregation	Per domain, then size-weighted. Pooling all items into one AUC inflates the result

The same protocol, item set and grading code are applied to every rung of the ladder and to the other systems on the independent leaderboard: https://huggingface.co/spaces/mayafree/typed-decision-leaderboard

Out of scope

• Not a grounding checker. It does not take a source document and decide whether the answer follows from it.
• Not a safety, toxicity or policy classifier.
• Not a calibrated probability. Use it to rank and threshold.
• Not a general-purpose verifier at this size. See the domain table above.

Limitations

• Domain coverage. Scores are meaningful only for the five domains listed. Outside them nothing has been measured and no guarantee is published.
• Below the surface baseline on disaster and safety. Stated in the table rather than omitted.
• Revision lock. The probe is fitted to one specific revision of the base model. Running it on a different revision produces no error and silently wrong scores; this repository ships the matching weights so that failure mode cannot occur.
• It reports the verifier's judgement, which is not the answering model's own confidence — that quantity measures 0.5000 on this set.

Lineage

Base model	Qwen/Qwen3.5-4B, Apache-2.0
Modification to base weights	none — the probes are separate files
Added by FINAL-Bench	probes, inference code, evaluation protocol and tables

What this repository contains

Included	Base weights · tokenizer · linear probe · curved probe · configuration
Not included	Training corpus · hidden-state matrices · fitting pipeline

The rest of the ladder

ZTC-Judge-27B · Darwin-397B-ZTC · ZTC-Judge-9B · ZTC-Judge-4B

License

The base model is Apache-2.0 and redistributable. The probes, the inference code and the evaluation tables are assets of FINAL-Bench / VIDRAFT.