jev-27b

An unofficial JEV-style decision model trained with jev-torch. It reads a state, a question and a list of options, runs one forward pass, and returns a calibrated probability for every option.

Usage

pip install git+https://github.com/smha1012/jev-torch.git

from jev import JEVPredictor

jev = JEVPredictor("smha1012/jev-27b")
jev.predict(
    kind="noul",
    state="The canary shows p99 latency up 40% after the deploy.",
    question="Should the rollout be paused?",
    options=["false", "true"],
)

Question kinds: noul (exactly 2 options, false-like then true-like), choice (2–16 options), score (ordered levels 0–5).

Serving with vLLM

adapter_vllm/ holds the same model as a vLLM LoRA module: the decision head is a LoRA on lm_head (the verbalizer rows only) and the bias and temperatures are in decision_head.json. Requests to jev-decision return decisions; requests to the base model name get Qwen/Qwen3.8-27B unchanged.

hf download smha1012/jev-27b --include "adapter_vllm/*" --local-dir jev-27b
vllm serve Qwen/Qwen3.8-27B --tensor-parallel-size 2 \
    --enable-lora --max-lora-rank 32 \
    --lora-modules jev-decision=jev-27b/adapter_vllm \
    --logprobs-mode processed_logprobs \
    --language-model-only

from jev.vllm_client import JEVVLLMClient

jev = JEVVLLMClient("http://localhost:8000", "jev-27b/adapter_vllm/decision_head.json")
jev.predict(kind="noul", state="...", question="...", options=["false", "true"])

vLLM loads the Qwen/Qwen3.8-27B weights once and applies this module on top; nothing else needs to be served. In bf16 the backbone needs about 54 GB, so serve it on two 48 GB GPUs (--tensor-parallel-size 2) or one 80 GB GPU. Served this way (vLLM 0.31.0, 2x L40S, tensor parallel) it matches the checkpoint: on 2,000 test_set_30k rows the per-option probabilities differ from PyTorch by 0.0025 on average (max 0.017) and 98.6% keep the same top option. On the full test_set_30k teacher rows: KL 0.0162 (checkpoint 0.0161), choice top-1 90.8% / 90.4% (same as below), ECE 0.0011 (0.0008). JevBench: 80.1% accuracy, ECE 0.016 (checkpoint 80.1%, 0.015).

Training

• Training data: jev_distill (https://huggingface.co/datasets/SargeDev/jev-distill-corpus-v3 @ fc99c6357a9f89f7512c4a987314352addead049)
• Base model: Qwen/Qwen3.8-27B (frozen) + LoRA r=16, α=32 + 24-slot fp32 decision head
• Loss: 1·kl + 0.5·rps[score]
• Optimization: 5000 steps × 128 rows, LoRA lr 0.0001, head lr 0.0002
• Temperatures: noul 1.000, choice 0.993, score 1.008

Evaluation

Agreement with the teacher

On the 25,376 rows of test_set_30k labelled by TypeSafe Jev 1.13, the same rows the autotrust/JEV cards report on. These measure how closely the model copies Jev, not accuracy on ground truth (see JevBench below).

Metric	This model	autotrust JEV-9B	autotrust JEV-27B
Mean KL to teacher (lower is better)	0.0161	0.0190	0.0170
Choice top-1 agreement, teacher-labelled rows	90.8%	90.2%	–
Choice top-1 agreement, all choice rows	90.4%	89.8%	90.3%
ECE vs teacher probabilities	0.0008	0.0007	0.0009

Per kind (measured on the vLLM-served model below): choice 90.8%, noul 95.7%, score 88.9% top-1 agreement.

JevBench (Leanmcp v0.1)

Accuracy and calibration on 6,516 public benchmark cases with gold answers, next to TypeSafe Jev 1.13.0 on the same cases: 80.1% vs 84.9% accuracy (94.4% of Jev's), ECE 0.015 vs 0.020.

Limitations

• Distilled, not RLCD. The model learns to reproduce TypeSafe Jev's output distributions (supervised KL + RPS) and is calibrated afterwards with temperature scaling. It does not reproduce Jev's own training method or architecture, which are unpublished, and cannot exceed the teacher.
• Knowledge comes from the backbone. Gaps to Jev on JevBench are largest on knowledge-heavy questions (MMLU-Pro −17 points, MedQA −4), though much smaller than jev-9b's (−24 and −15).
• Long inputs. Agent trajectories (atbench500), often longer than the 1,024-token training context, trail Jev by 9 points.
• At most 16 options per choice question; trained mostly on short, synthetic English scenarios. Validate on your own domain before relying on it.

License and attribution

Weights: CC BY-NC 4.0, non-commercial use only. Most training labels are outputs of the closed TypeSafe Jev 1.13 model, so the weights are not offered for commercial use. For other uses, contact the author.

The code that trains and loads the model, jev-torch, is Apache-2.0. This is an independent project, not affiliated with TypeSafe AI or autotrust.