AbsoluteZero NLI 9B

AbsoluteZero NLI 9B

Observable NLI with think traces

Provide a text (premise) and a statement (hypothesis), and it returns calibrated probabilities that the text entails, contradicts, or is neutral to the statement. One model answers in two ways:

  • Fast mode: a single forward pass and a classifier head (similar to cross-encoders).
  • Think mode: the model writes a short reasoning trace, then the scoring head reads it and gives the verdict. Use it on the hard cases, such as multi-step logic, quantifiers and negation, long contracts and policies. Traces are short (~60 tokens on average on ANLI, ~250 on first-order-logic problems).

Benefits of observation

Typical zero-shot encoder models cannot provide information that elucidates why they chose their classification. Think traces enable users to inspect and determine how to augment hypotheses to fix or improve classification. This could be done in conjunction with automation for grouping classification failures and using an LLM to test changes to the hypothesis against a holdout set.

Examples

Use Text → statement Result (fast mode)
Zero-shot classification / tagging review → "This review is positive." 93.6% (zs_meta, 18 tasks)
RAG grounding retrieved passages → generated answer 87.0% (RAGTruth)
Guardrails / Moderation message → policy statement 93.0% (CrossingGuard)
Clinical extraction clinical note → "The patient is unemployed." 98.5% (SDOH-NLI)
Contract review contract → "The agreement permits sharing data with third parties." 93.5% (ContractNLI)

9.0B parameters · documents up to 32k tokens · #1 on the OpenNLI Skill Index (64.5, 11-18-26, fast mode) · Apache-2.0

from sentence_transformers import CrossEncoder

model = CrossEncoder("dleemiller/AbsoluteZero-nli-9b", trust_remote_code=True)
text = "My card was charged twice for the same order."
labels = ["billing", "shipping", "account access"]
probs = model.predict([(text, f"This text is about {l}.") for l in labels], apply_softmax=True)
print({l: round(float(p[0]), 3) for l, p in zip(labels, probs)})  # p[0] = P(entailment)
# {'billing': 0.956, 'shipping': 0.001, 'account access': 0.004}

More in Think Mode, Recommended Applications and the 45-dataset evaluation.


Table of Contents


NLI Evaluation Results

Comparison on the OpenNLI leaderboard (v0.2.11), fast mode.

Model Params Skill Index Accuracy mnli mnli_mismatched snli anli_r1 anli_r2 anli_r3 wanli lingnli CAL Throughput (pairs/s)
dleemiller/AbsoluteZero-nli-9b 9.0B 64.5 0.8508 0.9276 0.9279 0.9380 0.8460 0.7360 0.6825 0.7744 0.8902 6.2 84
dleemiller/AbsoluteZero-nli-4b 4.2B 61.8 0.8372 0.9211 0.9227 0.9366 0.8250 0.7060 0.6517 0.7718 0.8829 7.2 159
dleemiller/AbsoluteZero-nli-2b 1.9B 59.6 0.8262 0.9130 0.9146 0.9332 0.7860 0.6470 0.6042 0.7656 0.8741 7.7 353
dleemiller/finecat-nli-v2-xl 1.03B 59.5 0.8263 0.9202 0.9191 0.9356 0.7860 0.6600 0.6067 0.7710 0.8834 6.1 576
dleemiller/finecat-nli-ultra-l 378M 58.6 0.8213 0.9127 0.9132 0.9302 0.7910 0.6400 0.5808 0.7698 0.8708 6.6 1,487
dleemiller/finecat-nli-v2-l 396M 57.8 0.8182 0.9164 0.9122 0.9329 0.7760 0.6040 0.5767 0.7664 0.8775 6.5 1,224
tasksource/ModernBERT-large-nli* 396M 37.8 0.7246 0.8981 0.9223 0.9192 0.7290 0.5130 0.4917 0.6970 0.8497 13.2 1,218
dleemiller/finecat-nli-l* 396M 37.6 0.7292 0.9155 0.9269 0.9162 0.7470 0.5680 0.5425 0.7708 0.8745 20.9 1,225
MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli 435M 27.9 0.6767 0.9123 0.9077 0.8897 0.7950 0.6820 0.6392 0.7696 0.8815 25.3 1,092
sileod/deberta-v3-large-tasksource-nli* 435M 24.8 0.6341 0.9298 0.9447 0.9211 0.7370 0.5770 0.4975 0.6786 0.8673 17.9 1,091

* trained on one or more OpenNLI test sets; those scores are replaced by estimates, as on the leaderboard.


Usage

Label Map:

  • entailment: 0
  • neutral: 1
  • contradiction: 2

Fast mode (Sentence Transformers)

pip install -U sentence-transformers
from sentence_transformers import CrossEncoder
import numpy as np

model = CrossEncoder("dleemiller/AbsoluteZero-nli-9b", trust_remote_code=True)
id2label = model.model.config.id2label  # {0:'entailment', 1:'neutral', 2:'contradiction'}

pairs = [
    ("The glass fell off the counter and shattered on the tile floor.",
     "The glass broke when it hit the floor."),          # E
    ("The store opens at 9 a.m. every day.",
     "The store opens at 7 a.m. on weekdays."),          # C
    ("A researcher presented results at the conference.",
     "The presentation won the best paper award."),       # N
]

logits = model.predict(pairs)  # shape: (3, 3)
for (prem, hyp), row in zip(pairs, logits):
    print(f"[{id2label[int(np.argmax(row))]}]  Premise: {prem}  |  Hypothesis: {hyp}")

The model reads each pair in its training layout (Premise: … / Hypothesis: … / <think></think> / Relation:); the tokenizer shipped with the model builds it, so pass plain (premise, hypothesis) pairs. Inputs are right-padded; the head reads each row's last token.

Scoring multiple hypotheses on one text. The model is causal and the premise comes first, so for tagging or checking many claims against one document, score_many encodes the premise once and scores every hypothesis from that cached state: the same predictions as scoring pair by pair, ~10x faster for 50 labels on a ~500-token document.

doc = "My card was charged twice for the same order, and support hasn't replied in a week."
labels = ["billing", "shipping", "account access", "customer support"]
logits = model.model.score_many(model.tokenizer, doc, [f"This text is about {l}." for l in labels])
print({l: round(p, 3) for l, p in zip(labels, logits.softmax(-1)[:, 0].tolist())})  # P(entailment) per label
# {'billing': 0.964, 'shipping': 0.001, 'account access': 0.005, 'customer support': 0.94}

Think Mode

Think mode is a method on the loaded model; it returns the verdict, its probabilities and the trace.

from sentence_transformers import CrossEncoder

model = CrossEncoder("dleemiller/AbsoluteZero-nli-9b", trust_remote_code=True)
pairs = [
    ("The glass fell off the counter and shattered on the tile floor.", "The glass broke when it hit the floor."),
    ("The store opens at 9 a.m. every day.", "The store opens at 7 a.m. on weekdays."),
    ("A researcher presented results at the conference.", "The presentation won the best paper award."),
]
for r in model.model.think(model.tokenizer, pairs):
    print(r["label"], "|", r["trace"])
entailment | The premise states the glass fell and shattered on the floor, which directly means it broke upon impact. This matches the hypothesis.
contradiction | The premise says the store opens at 9 a.m. every day, which includes weekdays. The hypothesis says it opens at 7 a.m. on weekdays. These times conflict, so the premise contradicts the hypothesis.
neutral | Presenting results does not imply winning an award, nor does it rule it out.

Think vs. fast (test sets: n=300 pairs each; ANLI dev: full dev splits):

Dataset Fast Think Think tokens (avg.)
anli_r2 (test) 0.7233 0.7767 62
anli_r3 (test) 0.6600 0.7033 58
wanli (test) 0.7500 0.7267 47
vitaminc (test) 0.9200 0.8600 49
control (test) 0.8000 0.7800 75
fol (test) 0.8500 0.8600 253
anli_dev_r1 0.8433 0.8651 57
anli_dev_r2 0.7316 0.7572 59
anli_dev_r3 0.6558 0.6811 59

Think mode is clearly better on adversarial NLI (ANLI +2-5 points) and logic; fast mode is better on short factual checks (VitaminC, WANLI). Confidence is calibrated for fast mode, while adding think traces will make the classification head certain toward the think trace, so confidence cannot be used in that mode.


Recommended Applications

Frame each check as a premise (text) and hypothesis (statement), and use the entailment probability.

Application Example hypothesis Supporting results
Zero-shot tagging / topic classification "This text is about billing." zs_meta, fin_topics
Routing and decisions in agent pipelines "The user wants to cancel an order." zs_meta, mnli, snli
RAG grounding / hallucination checks Each answer sentence vs. retrieved passages ragtruth, wice, vitaminc
Contract and policy review "The agreement permits sharing data with third parties." contractnli_a/b, control, docnli
Biomedical and health claim checking Claim vs. abstract or clinical note sdoh_nli, biosift, bionli, scifact, pubhealth
Dialogue fact-checking Assistant reply vs. knowledge source dialfact, vitaminc
Guardrails / policy matching Policy statement vs. user or model text crossingguard
Logical reasoning (think mode) Conclusion vs. premises fol, control, anli

Less suited: financial and general legal verification (findver, legal_domain), and adversarial reasoning (anli_r3, nope_adv).


Extended Evaluation (45 datasets)

Accuracy (fast mode) on the test splits of 45 NLI and adjacent datasets. 2-way datasets (entailed vs. not) are scored as entailment vs. any non-entailment prediction. Scored with the OpenNLI evaluation (DocNLI: 1,000-pair subset; VitaminC: 10,000; FRANK per sentence), plus the FineCat test split.

Dataset n AbsoluteZero-nli-9b AbsoluteZero-nli-4b AbsoluteZero-nli-2b
finecat 18012 0.8598 0.8533 0.8412
mnli 9815 0.9276 0.9211 0.9130
mnli_mismatched 9830 0.9279 0.9227 0.9146
snli 9824 0.9380 0.9366 0.9332
anli_r1 1000 0.8460 0.8250 0.7860
anli_r2 1000 0.7360 0.7060 0.6470
anli_r3 1200 0.6825 0.6517 0.6042
wanli 5000 0.7744 0.7718 0.7656
lingnli 7361 0.8902 0.8829 0.8741
glue_diagnostic 1100 0.7836 0.7573 0.7464
sick 4906 0.9250 0.9244 0.9215
scitail 2126 0.9699 0.9671 0.9647
hans 15000 0.8550 0.8470 0.8655
med 5380 0.8052 0.7944 0.7848
nan_nli 258 0.7481 0.7442 0.7403
scone_test 1000 1.0000 1.0000 0.9980
nmonli_test 200 1.0000 1.0000 1.0000
conjnli 623 0.7368 0.7287 0.7271
nope_adv 345 0.5797 0.5507 0.5159
fol 10277 0.8567 0.8482 0.8189
control 800 0.7913 0.7662 0.7137
vitaminc 10000 0.9095 0.9031 0.8751
dialfact 1499 0.8532 0.8499 0.8359
frank_sent 3176 0.8378 0.8177 0.8130
ragtruth 1497 0.8697 0.8637 0.8584
seahorse 800 0.8512 0.8413 0.8425
wice 958 0.8977 0.8987 0.8967
docnli 1000 0.8650 0.8520 0.8430
pubhealth 600 0.8750 0.8617 0.8533
healthver 230 0.8130 0.7522 0.8000
scifact 300 0.8767 0.8767 0.8767
findver 500 0.7420 0.7240 0.6920
nli4ct 199 0.8794 0.8543 0.8543
sdoh_nli 990 0.9848 0.9818 0.9828
biosift 1000 0.9190 0.9240 0.9190
bionli 600 0.9483 0.9167 0.9117
scientsbank 971 0.7703 0.7693 0.7539
beetle 819 0.6777 0.6667 0.6654
legal_domain 1163 0.6019 0.5804 0.5365
contractnli_a 1707 0.9350 0.9291 0.9268
contractnli_b 2074 0.9045 0.9026 0.8992
vast 1500 0.8093 0.7913 0.7813
zs_meta 5400 0.9356 0.9341 0.9254
fin_topics 600 0.9733 0.9700 0.9583
crossingguard 5397 0.9301 0.9248 0.9209

Macro average over the 45 datasets: 0.8510 (4B: 0.8397, 2B: 0.8288).


Training Details

This model is a fine-tune of Qwen/Qwen3.5-9B (text backbone), trained in two stages.

Stage 1: distillation. Full fine-tune (all weights) & two classifier heads, trained on the FineCat-NLI-Reason compilation relabelled by an NLI fine-tune of Qwen3.8-27B.

Stage 2: reinforcement learning on think mode (GRPO). Think mode was further trained with 32 sampled traces per prompt, reward = correctness of the verdict with a soft overlong penalty, a repetition penalty and a floor that discourages trace collapse.

Stage 2 made think mode more accurate on adversarial NLI and shorter: ANLI dev 0.758 → 0.765 (composite), and on first-order-logic validation problems traces were cut from ~330 to ~240 tokens at equal accuracy (0.874 → 0.872).

The released weights are the stage-2 GRPO LoRA merged into the stage-1 weights, stored in bfloat16.


Limitations

  • English only.
  • Throughput: 84 pairs/s on short pairs (25 on long documents) in fast mode (RTX PRO 6000, 250 W); use the FineCat encoders for very high volume or tight latency. Think mode generates a trace per pair (~1-10 pairs/s with transformers, depending on trace length; vLLM is faster).
  • Trained on inputs up to 8k tokens; scored at up to 32k, but accuracy on documents longer than 8k is not separately tested.
  • Weaker on financial and general legal verification (findver, legal_domain) and adversarial reasoning (anli_r3, nope_adv).

Acknowledgments

We thank the Qwen team for making their work available. This model would not be possible without their efforts and open source contributions.

Citation

@misc{absolutezero-2026,
  title = {AbsoluteZero: Observable NLI},
  author = {Lee Miller},
  year = {2026},
  howpublished = {\url{https://huggingface.co/dleemiller/AbsoluteZero-nli-9b}}
}
Downloads last month
8
Safetensors
Model size
9B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for dleemiller/AbsoluteZero-nli-9b

Finetuned
Qwen/Qwen3.5-9B
Finetuned
(1,442)
this model

Datasets used to train dleemiller/AbsoluteZero-nli-9b

Collection including dleemiller/AbsoluteZero-nli-9b

Evaluation results