AgentDecision-4B

AgentDecision-4B is the decision model of the paper JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs. It is a LoRA adapter on Qwen/Qwen3-4B.

A decision model answers the multiple-choice questions that an agent harness asks at each step: which tool to call next, which value an argument takes, or a yes/no question such as "is the task done?". It gives a probability for every option. The harness uses the answer when the model's confidence clears a threshold fitted on the agent's own runs, and lets the LLM decide otherwise. One answer costs one forward pass of a small model instead of an LLM call.

It was trained on the states that a customer-service agent reached in DecisionGym, including the states after its own mistakes, with the commercial decision model Jev as the teacher. GRPO then trained it inside the agent loop, and a last epoch added public tool-use data. This is the released version of the model the paper calls AgentDecision-4B. The model is served in Jev's System-One wire format (POST /v1/systemone), so harness code written for Jev works with it unchanged.

• Code: github.com/JevAgentHarness/JevAgentHarness (the research code is in research/)
• Paper: PDF (arXiv identifier to be added)
• Training data: AgentDecision-train. Benchmark: AgentDecisionBench. Logs of the routing proxy on coding agents: coding-agent-decisions

The models

Model	Base	Training	Use it for
AgentDecision-4B	Qwen3-4B	SFT on agent states, GRPO in the agent loop, then one epoch on agent states mixed with public tool-use data	the paper's main model
AgentDecision-4B-v1	Qwen3-4B	SFT on agent states, then GRPO in the agent loop. Frozen before the paper's Stage-2 tests	reproducing the registered Stage-2 results
AgentDecision-0.8B	StartLux-Decision-0.8B	agent states with the tools renamed in half of them, mixed with public tool-use data	a smaller and cheaper model (non-commercial base)

The code and the research records use older names: AgentDecision-4B is e12 ("mixed"), AgentDecision-4B-v1 is rl (JevAgentRL-4B, "frozen") and AgentDecision-0.8B is e28. DecisionGym is JevGym in the code.

Files

File	What it is
adapter_model.safetensors, adapter_config.json	the LoRA adapter (r 32, alpha 64, all linear layers) in PEFT format
tokenizer.json, tokenizer_config.json, chat_template.jinja	the base model's tokenizer and chat template, as used in training
calibration.json	temperatures for the option probabilities: one per question kind and number of options (choice:3-5, choice:17-52, noul:2) and a global one
thresholds.json	the confidence thresholds the paper's harness used with this model and Gemini 3.5 Flash, for tool choice (tool_select) and argument choice (tool_params), with their source
decision_config.json	the training record: base model and revision, loss, the adapter training started from
NOTICE	licences and attributions

How the model reads a question

Each question is one prompt. The model answers with one token, the label of an option, and the probabilities are read from the next-token logits at the labels:

system: You are a fast decision module inside an AI agent. Read the state and answer the question with exactly one
        token: the label of your answer.
user:   STATE:
        TASK: <the customer's request>
        HISTORY: <earlier tool calls and their results>
        CURRENT OBSERVATION: <the latest result>

        QUESTION: Which tool should the agent call next?
        OPTIONS:
        A. find_user -- Look up a customer by full name; returns user_id and tier.
        B. ship_order -- Ship a pending order.
        ...

        Answer with the option label only.

The labels are A-Z and a-z, so one prompt holds up to 52 options. A tool option is written name -- description. A yes/no question has the options yes and no. The chat template is applied with thinking turned off, and the prompt is cut to its last 4,096 tokens. The probabilities are a softmax over the label logits divided by the temperature from calibration.json. With more than 52 options (DecisionGym has 69 tools), the research server runs a knockout over chunks of 52 and combines the reads into one distribution over all options.

Usage

With transformers and peft

import json
import torch
from huggingface_hub import snapshot_download
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE, BASE_REVISION = "Qwen/Qwen3-4B", "1cfa9a7208912126459214e8b04321603b3df60c"
path = snapshot_download("JevAgentHarness/AgentDecision-4B")   # adapter, tokenizer, calibration.json, thresholds.json

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(BASE, revision=BASE_REVISION, dtype=dtype)
model = PeftModel.from_pretrained(model, path).merge_and_unload().to(device).eval()
cal = json.load(open(f"{path}/calibration.json"))

SYSTEM = ("You are a fast decision module inside an AI agent. Read the state and answer the question "
          "with exactly one token: the label of your answer.")
LABELS = [chr(c) for c in range(65, 91)] + [chr(c) for c in range(97, 123)]   # A-Z, a-z: up to 52 options
CODE_IDS = [tok.encode(c, add_special_tokens=False)[0] for c in LABELS]


def bucket(k):
    return "2" if k == 2 else "3-5" if k <= 5 else "6-16" if k <= 16 else "17-52"


@torch.no_grad()
def decide(state, question, options, kind="choice"):
    """Probability per option. Tool options are written 'name -- description'; a yes/no question uses
    options ['yes', 'no'] with kind='noul'."""
    assert 2 <= len(options) <= len(LABELS), "more than 52 options: use the server (knockout over chunks of 52)"
    lines = "\n".join(f"{LABELS[i]}. {o}" for i, o in enumerate(options))
    prompt = f"STATE:\n{state}\n\nQUESTION: {question}\nOPTIONS:\n{lines}\n\nAnswer with the option label only."
    text = tok.apply_chat_template([{"role": "system", "content": SYSTEM}, {"role": "user", "content": prompt}],
                                   tokenize=False, add_generation_prompt=True, enable_thinking=False)
    ids = tok(text, add_special_tokens=False, return_tensors="pt").input_ids[:, -4096:].to(device)
    logits = model(input_ids=ids, logits_to_keep=1).logits[0, -1].float()
    t = cal["buckets"].get(f"{kind}:{bucket(len(options))}", cal["global"])   # temperature scaling
    p = torch.softmax(logits[CODE_IDS[:len(options)]] / t, -1)
    return dict(zip(options, p.tolist()))


# a state from AgentDecisionBench's common states (DecisionGym), with 6 of its 69 tools
state = ("TASK: A customer wrote in asking us to ship their pending order (express if they are gold tier, otherwise "
         "standard), but did not say who they are.\n\nHISTORY:\n[0] ask_user({\"question\": \"Could you please provide "
         "your name or customer ID so I can locate your pending order?\", \"missing\": \"customer_name\"}) -> Customer: "
         "Oh, sorry -- my name is Ivan Kim.\n\nCURRENT OBSERVATION:\nCustomer: Oh, sorry -- my name is Ivan Kim.")
options = ["find_user -- Look up a customer by full name; returns user_id and tier.",
           "search_users -- List customers whose last name starts with a prefix (display names only, no ids).",
           "get_orders -- List a customer's orders filtered by status.",
           "ship_order -- Ship a pending order.",
           "ask_user -- Ask the customer a clarifying question and wait for the reply. Use it only when information "
           "needed to act is missing or ambiguous.",
           "finish -- Call when the task is complete. Provide the final answer (or a short summary)."]
probs = decide(state, "Which tool should the agent call next?", options)
print(max(probs, key=probs.get), probs)

This prints find_user with most of the probability: the customer just gave their name, so the agent should look them up. Tested with torch 2.14, transformers 5.18 and peft 0.21. We checked this code against the research server: the adapter loads onto Qwen3-4B with every tensor matched, and the prompt tokens and option labels are identical to the server's. The same code ran end to end, with identical probabilities to the server, for AgentDecision-0.8B.

As a decision server in the Jev wire format

The research code serves the model behind POST /v1/systemone, the format Jev uses:

git clone https://github.com/JevAgentHarness/JevAgentHarness.git && cd JevAgentHarness
uv venv -p 3.11 .venv && source .venv/bin/activate
uv pip install -e "research[serve,train]"
hf download JevAgentHarness/AgentDecision-4B --local-dir agentdecision-4b
cd research
python -m jevharness.training.serve_jev --model Qwen/Qwen3-4B --adapter ../agentdecision-4b \
    --calibration ../agentdecision-4b/calibration.json --name agentdecision-4b --port 8090

The server keeps a cache of each episode's prompt prefix, answers all questions of a request in one batch, and runs the knockout for more than 52 options. --fast turns on the faster exact paths the paper's final harness used. One request can carry several questions about the same state:

curl -s localhost:8090/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": "TASK: Ship the pending order of the customer Ivan Kim.\n\nHISTORY:\n\nCURRENT OBSERVATION:\n(none)",
  "questions": {
    "tool": {"type": "choice", "instructions": "Which tool should the agent call next?",
             "criteria": {"find_user": "Look up a customer by full name; returns user_id and tier.",
                          "ship_order": "Ship a pending order.",
                          "ask_user": "Ask the customer a clarifying question and wait for the reply."}},
    "done": {"type": "noul", "instructions": "Is the task complete?"}}}'

The answer has a probability per option ("tool": {"choice": ..., "confidence": ..., "probabilities": {...}}) and "done": {"noul": <p(yes)>}.

In the research harness, the served model is a decision-model spec like any other:

S1='{"type": "jev", "name": "agentdecision-4b", "base_url": "http://127.0.0.1:8090", "model": "agentdecision-4b", "api_key": ""}'
jevbench ping --s1 "$S1"
python scripts/stage2_corpus.py replay --corpus results/stage2/s23 --name agentdecision-4b --s1 "$S1"   # answer the common states

The harness profile with this model's thresholds is research/configs/harness/explore_e21b_e12_final.yaml. research/README.md maps every table of the paper to its command.

In the JevAgentHarness proxy

The proxy in the top level of the repository (jah) takes any server that speaks the Jev wire format:

# ~/.jah/config.yaml
router:
  decision: {type: jev, provider: local, base_url: "http://127.0.0.1:8090", model: agentdecision-4b}

jah decide sends one example state and prints the probabilities, which checks the connection. Keep in mind that the proxy's default question (is the agent's next step routine or hard?) and coding-agent states are outside this model's training data. Its tool_select point asks the question the model was trained on ("Which tool should the agent call next?"), but over a coding agent's tools. Run the proxy with --preset shadow first, fit the thresholds on your own logs with jah fit, and compare jah-strong with jah-auto on your tasks before relying on it (see docs/modes.md in the repository).

Calibration and thresholds

calibration.json holds temperatures, not thresholds. The server applies them with --calibration. The snippet above looks up the key <kind>:<bucket> (kind choice or noul, bucket 2, 3-5, 6-16 or 17-52 options) and falls back to global.

thresholds.json holds the thresholds of the paper's harness for this model with Gemini 3.5 Flash as the agent LLM: the model's answer is used when its calibrated confidence is at least tau, and the LLM decides otherwise (tool choice 0.445, argument choice 0.996). They were fitted in shadow mode, where the model answers on the agent's states without acting, on 150 DecisionGym calibration tasks. They hold only for that setting. For another agent LLM, other tools or another task family, record the model's answers in shadow mode and refit: python -m jevharness.bench.threshold_theory in the research code, or jah fit in the proxy.

Training

AgentDecision-4B is AgentDecision-4B-v1 trained one more epoch. The full chain:

1. SFT on agent states. Qwen3-4B with LoRA (r 32, alpha 64, all linear layers) on AgentDecision-train v1: 15,694 decision states from 587 DecisionGym training tasks. A scripted policy repeated mistakes seen during development, so 30.9% of the states follow a mistake. Where the teacher Jev ranks the correct answer first, the target is 0.3 x the one-hot label + 0.7 x Jev's probabilities. Otherwise it is the one-hot label. The loss is cross-entropy plus 0.5 x a Brier term, for 2 epochs at learning rate 1e-4 (2.2 GPU-hours; the data cost $5.28). Script: research/scripts/train/12_sft_v1.sh.
2. GRPO in the agent loop. The SFT model makes the decisions on training tasks while Gemini 3.5 Flash-Lite does the rest. Each task runs a group of 8 episodes. The reward is task success x (1 - 0.5 min(1, LLM calls / 10)), minus 0.2 for an irreversible wrong call, with a KL penalty to the SFT model. Iteration 15 of 30 was kept by a rule fixed in advance (+4.5 GPU-hours, +$26.53 of LLM calls). Scripts: research/scripts/train/70_grpo_online.sh with research/configs/train/grpo_online_round5.yaml, stage choice research/scripts/train/r5_chain.py. The result is AgentDecision-4B-v1.
3. Public tool-use data. One more epoch from v1's adapter on 14,000 rows: 7,000 public items (When2Call's training split and ToolACE: tool choice, arguments, asking for missing information, saying no tool fits) mixed one to one with 7,000 of its own SFT states, after a leakage audit against every test pool. Same loss, learning rate 5e-5 (+2.4 GPU-hours). Scripts: research/scripts/explore_e12_train.py build (pass the ToolACE JSON you downloaded with --toolace), then research/scripts/aws/e12_train.sh. Record: research/results/explore/e12/.
4. Calibration. Temperatures fitted on 2,000 held-out development questions (1,000 agent states, 1,000 public items) with research/src/jevharness/training/calibrate.py.

The public tool-use rows of step 3 are not part of AgentDecision-train. The build script recreates them from the public datasets. Paper: Section 5 and Appendix D (D.9 for step 3).

Results

All numbers are from the paper, in %. Agent LLM: Gemini 3.5 Flash. "Every tool choice" means the model picks every tool with no threshold, so the agent depends on it fully.

Level	What is measured	AgentDecision-4B	Jev (teacher)	Paper
L1 isolated	accuracy on 3,991 standalone questions (BFCL v4 tool choice and arguments, When2Call)	85.1	86.4	Table 3
L1 isolated	BFCL tool-choice accuracy	89.3	89.9	Figure 4b
L2 open-loop	accuracy on the common states: recorded tool-choice states of 200 new DecisionGym tasks	99.1	91.5	Table 39 (App. G.9)
L2 open-loop	accuracy on DecisionGym-Shift states (renamed tools, more distractors, new task kinds)	94.9	91.8	Table 3
L2 open-loop	the same, must-ask states only (the only right answer is to ask the customer)	67.4	58.3	Table 3
L3 closed-loop	task success, every tool choice by the model, 200 new DecisionGym tasks	100.0	83.0	Table 39 (App. G.9)
L3 closed-loop	task success, every tool choice by the model, 200 shifted tasks	91.5	82.0	Table 3, Figure 4b

In the harness. With fitted thresholds for tool and argument choices, on its own GPU, on 200 new DecisionGym tasks, the agent's task success was 0.5 points higher than with the LLM alone, its cost 86.3% lower, and it ran 2.5 times faster (Section 7.3, Table 2). The best decision model on isolated questions, StartLux-Decision-9B (92.2% on BFCL tool choice), solved only 69.5% of the shifted tasks (Section 7.1).

Serving cost. On one NVIDIA L4 with vLLM and FP8 weights: $0.0045 per 1,000 decisions and a median of 31 ms per answer (App. G.7, Table 33), about 4 times less per decision than Jev (Section 7.1). These numbers come from vLLM with the merged adapter, not from the research server below.

Compared with AgentDecision-4B-v1, the public data raised isolated accuracy from 78.8% to 85.1%, while success on the shifted tasks fell from 96.0% to 91.5% (Section 7.4).

The benchmark data behind the L1 and L2 numbers, every model's answers and the paper's numbers per model are in AgentDecisionBench (model id e12).

Limitations

• Trained for DecisionGym. The agent states come from DecisionGym's customer-service tasks with one agent LLM (Gemini 3.5 Flash-Lite for the training episodes, Flash for evaluation), in English. Other tools, task families or agent LLMs need new thresholds and, for the best accuracy, training on that agent's own states.
• Under shift the threshold did not protect success. On 200 shifted tasks, at the thresholds fitted on the original tasks, the harness with this model solved 92.5% of tasks, 7.5 points below the LLM alone: it still took every tool choice and accepted 22 wrong ones (App. G.9, Table 40).
• Irreversible first errors. A confident wrong choice at an irreversible step (for example cancelling where the agent should have asked) ends the task. The must-ask states are where this model is weakest (67.4% on the shifted states). Never make a decision model the only check on an irreversible action.
• New rules. On DecisionGym-Hard tasks with an order rule it was not trained on, the 4B model before this last epoch (AgentDecision-4B-v1) got 81.6% of the cancel-or-ship decisions right and the agent solved 3.0% of tasks, against 88.0% for the LLM alone (Section 7.5).
• Coding harnesses are not validated. The model was never trained or tested on coding-agent states or on the routing question of the JevAgentHarness proxy.
• Not tested against adversarial input such as injected instructions in tool output.

Licence

CC BY-NC 4.0 for the adapter, calibration.json, thresholds.json and decision_config.json.

• Qwen/Qwen3-4B is licensed under Apache-2.0. The tokenizer files (tokenizer.json, tokenizer_config.json, chat_template.jinja) come from it and stay under Apache-2.0.
• The training data contain outputs of commercial models: Jev's probabilities (TypeSafe) as soft labels, and agent text written by Gemini models (Google). The data are shared under CC BY-NC 4.0 for that reason. Check those providers' terms before any use beyond non-commercial research, in particular before training a model that competes with theirs.
• The public tool-use data come from When2Call (NVIDIA, CC BY 4.0) and ToolACE (Team-ACE, Apache-2.0).

Citation

@misc{zhu2026jevagentharnessbenchmarkingdeployingtraining,
      title={JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs},
      author={Kunlun Zhu and Tao Feng and Xu Han and Heng Wang and Ling Yang and Xiangru Tang and Jiaxuan You and Heng Ji},
      year={2026},
      eprint={2610.21412},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2610.21412},
}

The arXiv identifier will be added when it is available.