openjev-flash: openjev in 150M parameters

openjev-flash is openjev rebuilt on a 150M-parameter encoder, Ettin-150M, and distilled from openjev's 4B model. Like openjev, it is a single cross-encoder. It reads a premise and a hypothesis and returns contradiction, entailment or neutral. That one call covers reranking, grading against a reference, hallucination checks and typed decisions. Nothing is trained per task.

• 4.4–5.2x faster than openjev-0.8B on the same GPU and 8–10x faster than openjev-2B.
• Matches openjev-0.8B on NLI and grounded faithfulness (MNLI 87.9 vs 86.2, RAGTruth 0.896 vs 0.899, AggreFact 0.725 vs 0.695).
• Behind on knowledge-heavy judging and hard typed decisions. It scores 0.602 on JevBench vs 0.736 for openjev-2B. If the answer is not in the premise, a 150M model usually does not know it.

path	contents
/ (repo root)	default. Ettin-150M distilled from openjev qwen3.5-4b-nli-v5. Same weights as ettin150mdistill-nli/
ettin150m-nli/	Ettin-150M trained on gold labels only, with no teacher
checkpoints/	final fp32 Trainer checkpoints (weights, optimizer, scheduler), for continued training
openjev_flash.py	small inference wrapper: predict, rerank, grade, decide
results/, logs/, code/	raw eval JSON for every number below, training and eval logs, all training and eval code

Use the default model unless you judge answers to false-premise or nonsense questions. There, the non-distilled ettin150m-nli is clearly better (BullshitBench judge AUROC 0.839 vs 0.716; see Results).

Quick start

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "xxparthparekhxx/openjev-flash"          # add subfolder="ettin150m-nli" for the non-distilled model
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

premise = "A man is playing a guitar on stage."
hypotheses = ["Someone is making music.", "The stage is empty.", "The man is a professional musician."]
enc = tok([premise] * len(hypotheses), hypotheses, truncation="only_first", max_length=2048,
          padding=True, return_tensors="pt")
with torch.no_grad():
    probs = model(**enc).logits.softmax(-1)
for h, p in zip(hypotheses, probs):
    print(h, {model.config.id2label[i]: round(x, 3) for i, x in enumerate(p.tolist())})
# Someone is making music. {'contradiction': 0.001, 'entailment': 0.969, 'neutral': 0.031}
# The stage is empty. {'contradiction': 0.996, 'entailment': 0.001, 'neutral': 0.004}
# The man is a professional musician. {'contradiction': 0.001, 'entailment': 0.018, 'neutral': 0.98}

Input format differs from openjev. openjev's Qwen models read the template Premise: …\nHypothesis: …. This model reads a plain tokenizer pair, [CLS] premise [SEP] hypothesis [SEP], with no template. Truncate the premise and never the hypothesis (truncation="only_first"). Training used up to 2048 tokens per pair. Labels are 0 = contradiction, 1 = entailment, 2 = neutral.

With pipeline:

from transformers import pipeline
clf = pipeline("text-classification", model="xxparthparekhxx/openjev-flash", top_k=None)
clf({"text": "A man is playing a guitar on stage.", "text_pair": "Someone is making music."})
# [{'label': 'entailment', 'score': 0.968}, {'label': 'neutral', 'score': 0.031}, {'label': 'contradiction', 'score': 0.001}]

Reranking, grading and typed decisions

openjev_flash.py wraps openjev's task recipes, including the typed-decision call from openjev's openjev_decide.py. decide builds one hypothesis per option, normalises P(entailment) over the options and generates nothing.

from huggingface_hub import hf_hub_download
hf_hub_download("xxparthparekhxx/openjev-flash", "openjev_flash.py", local_dir=".")
from openjev_flash import FlashJev

jev = FlashJev.from_pretrained("xxparthparekhxx/openjev-flash")   # bf16, GPU if available

jev.predict([("The meeting was moved from Tuesday to Thursday at 3 pm because the CFO is travelling.",
              "The meeting now takes place on Thursday.")])           # ≈ [[0.01, 0.95, 0.04]]  (con, ent, neu)

jev.decide("Customer: I was charged twice for my March invoice and want the extra charge refunded.",
           [{"type": "choice", "instructions": "Which team should handle this ticket?",
             "options": ["billing", "shipping", "technical support"]}])
# [{'probabilities': {'billing': 0.963, 'shipping': 0.019, 'technical support': 0.018}}]

doc = "Australia's capital is Canberra. Sydney is its largest city and Melbourne is the second largest."
jev.rerank(f"{doc}\nQuestion: What is the capital of Australia?", ["Sydney", "Canberra", "Melbourne"])  # 1

jev.grade("What is 12 * 12?", reference="144", candidate="The answer is 144.")   # 'entailment'
jev.grade("What is 12 * 12?", reference="144", candidate="It is 124.")          # 'contradiction'

decide splits states longer than 24k characters into overlapping windows and keeps the most-entailed window per option.

Speed

Throughput was measured with code/bench_speed.py on one RTX 5060 Laptop GPU (8 GB): bf16, batch 16, pairs per second at three pair lengths. All models ran on the same machine, and reruns matched within 3%. The 150M needs 0.41 GiB of VRAM at batch 16, so it also fits next to a generator LLM on a small GPU.

Gemma 4 E2B was the first backbone considered. Its 2.3B parameters of per-layer embedding tables had to be offloaded to the CPU to fit in 8 GB, and it was 2.7x slower than openjev-0.8B, so it was dropped.

Exact numbers (pairs / second)

model	params	44 tokens	105 tokens	300 tokens	peak VRAM	vs openjev-0.8B
openjev-flash (Ettin-150M)	150M	1425	627	172	0.41 GiB	4.4–5.2x
Ettin-400M (not trained here)	400M	593	215	70	0.91 GiB	1.8–1.9x
openjev Qwen3.5-0.8B	0.8B	310	122	39	2.28 GiB	1x
openjev Qwen3.5-2B	2B	145	62	22	4.84 GiB	0.5x
Gemma 4 E2B as a cross-encoder	1.9B + 2.3B PLE	114	45	14	4.84 GiB + CPU	0.4x

Raw output: results/bench_speed*.json.

Results

All numbers come from openjev's own eval scripts (code/evals/), run by us with the same harness on one GPU for all four models. The harness reproduces openjev's published numbers for the 0.8B (MNLI 86.2 / 87.2, AggreFact 0.695). Raw outputs are in results/, and every chart is drawn from them by code/make_card_plots.py.

• NLI: at or above the 0.8B on MNLI, WANLI, SciTail and ConTRoL. Behind on all three adversarial ANLI rounds, most on rounds 2 and 3.
• Grounded faithfulness (AggreFact, RAGTruth): level with or above the 0.8B.
• Hallucination and judging: clearly weaker. On HaluBench, it is level only on the RAGTruth subset (0.858 vs 0.861) and trails on every other subset. Its FinanceBench and DROP scores (numeric reasoning) are at chance: 0.50 and 0.54. BullshitBench judging and LLMBar, which need judgement more than reading, show the largest gaps.

Exact numbers: NLI (accuracy, %)

model	MNLI m / mm	ANLI r1 / r2 / r3	WANLI	SciTail	ConTRoL
openjev Qwen3.5-0.8B	86.2 / 87.2	65.1 / 50.8 / 48.5	73.8	93.5	51.6
openjev Qwen3.5-2B	85.8 / 87.0	68.1 / 56.3 / 54.5	74.6	91.9	65.3
openjev-flash, no teacher (ettin150m-nli)	87.4 / 87.3	64.9 / 44.8 / 43.8	74.3	94.6	59.3
openjev-flash (default)	87.9 / 87.6	63.7 / 44.4 / 45.2	74.7	94.6	61.2

Exact numbers: faithfulness, hallucination and judging

AggreFact is balanced accuracy averaged over its subsets. LLMBar is accuracy. All other columns are AUROC.

model	AggreFact	RAGTruth	HaluBench	FalseQA	BSB detect	BSB judge	IFEval (instr.)	LLMBar
openjev Qwen3.5-0.8B	0.695	0.899	0.879	0.854	0.747	0.907	0.867	0.607
openjev Qwen3.5-2B	0.746	0.918	0.902	0.926	0.836	0.966	0.933	0.757
openjev-flash, no teacher	0.722	0.902	0.771	0.819	0.790	0.839	0.811	0.533
openjev-flash (default)	0.725	0.896	0.807	0.820	0.778	0.716	0.823	0.531

JevBench (typed decisions)

openjev's 0.8B v2s-long checkpoint (hatched) was trained on the public JevBench items, so its score is not a clean comparison; openjev-2B is the clean reference. openjev-flash saw no JevBench item, because its mix was built with --no-jevbench.

Scores for the other small encoders are self-reported, from the leaderboard snapshot bundled with openjev's eval_jevbench.py. We did not rerun them.

Exact numbers: JevBench (231 public items, accuracy)

model	easy (48)	standard (72)	hard (111)	all public
openjev Qwen3.5-0.8B v2s-long (saw these items)	1.000	0.972	0.874	0.931
openjev Qwen3.5-2B	1.000	0.944	0.486	0.736
openjev-flash, no teacher	0.979	0.694	0.333	0.580
openjev-flash (default)	1.000	0.764	0.324	0.602

What distillation changed

Compared with the same recipe without a teacher:

• Gains: JevBench standard tier +6.9 points, HaluBench +3.7, JevBench overall +2.2, ConTRoL +2.0.
• Loss: BullshitBench judge AUROC fell from 0.839 to 0.716 (−12.3). Everything else moved by 1.3 points or less.
• Distillation sharpens the model on the inputs it already sees, but does not add knowledge the 150M lacks.

Training

• Base: jhu-clsp/ettin-encoder-150m (ModernBERT architecture, 22 layers, hidden size 768) with a 3-way ModernBertForSequenceClassification head (mean pooling).
• Data: xxparthparekhxx/openjev-flash-mix: 1,298,070 training pairs and 4,000 validation pairs (359M tokens), rebuilt with openjev's own data_mix.py (code/data_mix.py). Parts used: text, faith, ifollow, bullshit, agentic, agentic2, jevfmt --no-jevbench and longdoc, class-balanced by build.
  • Sources: SNLI, MultiNLI, ANLI, WANLI, FEVER-NLI, LingNLI, ConTRoL, SciTail, QNLI, bAbI-NLI, MiniCheck C2D/D2C, RAGTruth (train split), ifeval-like, FalseQA (train and valid splits), AgentTraj-L, AgentInstruct, When2Call, Mind2Web, CLINC-150, Banking77, SciQ, GovReport and Qasper.
  • Every eval test split is on data_mix.py's ban list.
  • Not included: openjev's image rows, xLAM tool calls (gated) and openjev's unpublished v4/v5 rows.
• Teacher (default model only): AlexWortega/openjev qwen3.5-4b-nli-v5.
  • It produced logits for 1,161,975 training rows (89.5%) and agrees with the gold labels on 92.3% of them.
  • The long-document rows (gov_*, qasper*, haystack*) were not labelled and train on gold labels only.
  • Loss: 0.5 · CE(gold) + 0.5 · KL(teacher ‖ student), temperature 1.
• Hyperparameters: 1 epoch, learning rate 5e-5, 3% warmup, effective batch size 32, max length 2048, length-grouped batches, bf16 autocast, seed 42.
• Compute: one RTX PRO 6000 Blackwell (96 GB). Training took 75 minutes without the teacher and 82 minutes with it. Teacher labelling took about 2.5 hours.
• Held-out mix accuracy: 88.75% without the teacher, 89.05% distilled.

The distilled run's loss includes the KL term, so its curve is not directly comparable with the no-teacher run's.

To reproduce, either download the mix with teacher logits or rebuild it with code/data_mix.py (each part, then build) and code/distill_label.py. Then run code/train_nli.py (see code/run_encoders.sh and code/run_distill.sh), then code/run_evals.sh, then code/make_card_plots.py for the charts.

Limitations

• English only.
• Little world knowledge. Without context, rerank("What is the capital of Australia?", ["Sydney", "Canberra", "Melbourne"]) picks Melbourne. Put the facts in the premise. The model is built to check text against text.
• Surface-form sensitivity. Graded against the reference "144", the candidate "one hundred forty-four" comes out as contradiction. Normalise numbers and units before grading.
• Hard typed decisions (multi-hop, long policies, temporal and numeric reasoning) are weak: 0.32 on JevBench hard, vs 0.49 for openjev-2B. Use a larger judge where those matter.
• The distilled model is a poor judge of answers to false-premise questions (BullshitBench judge). Use ettin150m-nli for that.
• Scores are not calibrated across tasks (JevBench ECE is 0.14–0.18). Pick thresholds per task on your own data.
• Pairs longer than the 2048 training tokens lose premise text. The encoder supports about 8k positions, but the NLI head never saw pairs that long.

License and credits

MIT, the same as Ettin and openjev. The training data keeps its upstream licences.

All of the recipe comes from openjev by Alex Wortega: the data mix, the eval harness, JevBench and the teacher. The backbone is Ettin from JHU CLSP:

@misc{weller2025seqvsseqopen,
  title={Seq vs Seq: An Open Suite of Paired Encoders and Decoders},
  author={Orion Weller and Kathryn Ricci and Marc Marone and Antoine Chaffin and Dawn Lawrie and Benjamin Van Durme},
  year={2025},
  eprint={2507.11412},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2507.11412},
}