AgentDecision-0.8B

AgentDecision-0.8B is the small decision model of the paper JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs. It is a LoRA adapter on startlux-models/StartLux-Decision-v1-0.8B, a decision model with open weights under CC BY-NC 4.0, so this adapter is for non-commercial use only. The paper calls the base StartLux-Decision-0.8B, the name of that repository until StartLux Labs renamed it in October 2026.

A decision model answers the multiple-choice questions that an agent harness asks at each step: which tool to call next, which value an argument takes, or a yes/no question such as "is the task done?". It gives a probability for every option. The harness uses the answer when the model's confidence clears a threshold fitted on the agent's own runs, and lets the LLM decide otherwise. One answer costs one forward pass of a small model instead of an LLM call.

It was trained with the recipe of AgentDecision-4B: states that a customer-service agent reached in DecisionGym, with Jev's answers as soft labels, mixed with public tool-use data. In half of the agent states the tools were renamed and the text reworded, so that the model learns the decisions rather than the tool names. The model is served in Jev's System-One wire format (POST /v1/systemone), so harness code written for Jev works with it unchanged.

• Code: github.com/JevAgentHarness/JevAgentHarness (the research code is in research/)
• Paper: PDF (arXiv identifier to be added)
• Training data: AgentDecision-train. Benchmark: AgentDecisionBench. Logs of the routing proxy on coding agents: coding-agent-decisions

The models

Model	Base	Training	Use it for
AgentDecision-4B	Qwen3-4B	SFT on agent states, GRPO in the agent loop, then one epoch on agent states mixed with public tool-use data	the paper's main model
AgentDecision-4B-v1	Qwen3-4B	SFT on agent states, then GRPO in the agent loop. Frozen before the paper's Stage-2 tests	reproducing the registered Stage-2 results
AgentDecision-0.8B	StartLux-Decision-0.8B	agent states with the tools renamed in half of them, mixed with public tool-use data	a smaller and cheaper model (non-commercial base)

The code and the research records use older names: AgentDecision-4B is e12 ("mixed"), AgentDecision-4B-v1 is rl (JevAgentRL-4B, "frozen") and AgentDecision-0.8B is e28. DecisionGym is JevGym in the code.

Files

File	What it is
adapter_model.safetensors, adapter_config.json	the LoRA adapter (r 32, alpha 64, all linear layers) in PEFT format
tokenizer.json, tokenizer_config.json, chat_template.jinja	the base model's tokenizer and chat template, as used in training
calibration.json	temperatures for the option probabilities: one per question kind and number of options (choice:3-5, choice:17-52, noul:2) and a global one
thresholds.json	the confidence thresholds the paper's harness used with this model and Gemini 3.5 Flash, for tool choice (tool_select) and argument choice (tool_params), with their source
decision_config.json	the training record: base model and revision, loss, the adapter training started from
NOTICE	licences and attributions

How the model reads a question

Each question is one prompt. The model answers with one token, the label of an option, and the probabilities are read from the next-token logits at the labels:

system: You are a fast decision module inside an AI agent. Read the state and answer the question with exactly one
        token: the label of your answer.
user:   STATE:
        TASK: <the customer's request>
        HISTORY: <earlier tool calls and their results>
        CURRENT OBSERVATION: <the latest result>

        QUESTION: Which tool should the agent call next?
        OPTIONS:
        A. find_user -- Look up a customer by full name; returns user_id and tier.
        B. ship_order -- Ship a pending order.
        ...

        Answer with the option label only.

The labels are A-Z and a-z, so one prompt holds up to 52 options. A tool option is written name -- description. A yes/no question has the options yes and no. The chat template is applied with thinking turned off, and the prompt is cut to its last 4,096 tokens. The probabilities are a softmax over the label logits divided by the temperature from calibration.json. With more than 52 options (DecisionGym has 69 tools), the research server runs a knockout over chunks of 52 and combines the reads into one distribution over all options.

Usage

With transformers and peft

import json
import torch
from huggingface_hub import snapshot_download
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE, BASE_REVISION = "startlux-models/StartLux-Decision-v1-0.8B", "bd4f76a600e23227547fee7bfc1825e12a32764c"
path = snapshot_download("JevAgentHarness/AgentDecision-0.8B")   # adapter, tokenizer, calibration.json, thresholds.json

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(BASE, revision=BASE_REVISION, dtype=dtype)
model = PeftModel.from_pretrained(model, path).merge_and_unload().to(device).eval()
cal = json.load(open(f"{path}/calibration.json"))

SYSTEM = ("You are a fast decision module inside an AI agent. Read the state and answer the question "
          "with exactly one token: the label of your answer.")
LABELS = [chr(c) for c in range(65, 91)] + [chr(c) for c in range(97, 123)]   # A-Z, a-z: up to 52 options
CODE_IDS = [tok.encode(c, add_special_tokens=False)[0] for c in LABELS]


def bucket(k):
    return "2" if k == 2 else "3-5" if k <= 5 else "6-16" if k <= 16 else "17-52"


@torch.no_grad()
def decide(state, question, options, kind="choice"):
    """Probability per option. Tool options are written 'name -- description'; a yes/no question uses
    options ['yes', 'no'] with kind='noul'."""
    assert 2 <= len(options) <= len(LABELS), "more than 52 options: use the server (knockout over chunks of 52)"
    lines = "\n".join(f"{LABELS[i]}. {o}" for i, o in enumerate(options))
    prompt = f"STATE:\n{state}\n\nQUESTION: {question}\nOPTIONS:\n{lines}\n\nAnswer with the option label only."
    text = tok.apply_chat_template([{"role": "system", "content": SYSTEM}, {"role": "user", "content": prompt}],
                                   tokenize=False, add_generation_prompt=True, enable_thinking=False)
    ids = tok(text, add_special_tokens=False, return_tensors="pt").input_ids[:, -4096:].to(device)
    logits = model(input_ids=ids, logits_to_keep=1).logits[0, -1].float()
    t = cal["buckets"].get(f"{kind}:{bucket(len(options))}", cal["global"])   # temperature scaling
    p = torch.softmax(logits[CODE_IDS[:len(options)]] / t, -1)
    return dict(zip(options, p.tolist()))


# a state from AgentDecisionBench's common states (DecisionGym), with 6 of its 69 tools
state = ("TASK: A customer wrote in asking us to ship their pending order (express if they are gold tier, otherwise "
         "standard), but did not say who they are.\n\nHISTORY:\n[0] ask_user({\"question\": \"Could you please provide "
         "your name or customer ID so I can locate your pending order?\", \"missing\": \"customer_name\"}) -> Customer: "
         "Oh, sorry -- my name is Ivan Kim.\n\nCURRENT OBSERVATION:\nCustomer: Oh, sorry -- my name is Ivan Kim.")
options = ["find_user -- Look up a customer by full name; returns user_id and tier.",
           "search_users -- List customers whose last name starts with a prefix (display names only, no ids).",
           "get_orders -- List a customer's orders filtered by status.",
           "ship_order -- Ship a pending order.",
           "ask_user -- Ask the customer a clarifying question and wait for the reply. Use it only when information "
           "needed to act is missing or ambiguous.",
           "finish -- Call when the task is complete. Provide the final answer (or a short summary)."]
probs = decide(state, "Which tool should the agent call next?", options)
print(max(probs, key=probs.get), probs)

This prints find_user with most of the probability: the customer just gave their name, so the agent should look them up. Tested with torch 2.14, transformers 5.18 and peft 0.21. The base model needs transformers 5.8 or newer. We ran this code on CPU (float32) and compared it with the research server (serve_jev.Engine) on four questions: the probabilities were identical, and the server reproduced the model's recorded AgentDecisionBench answer to a 69-option question. On CPU, without flash-linear-attention and causal-conv1d, transformers falls back to slower PyTorch kernels; install both on a GPU.

As a decision server in the Jev wire format

The research code serves the model behind POST /v1/systemone, the format Jev uses:

git clone https://github.com/JevAgentHarness/JevAgentHarness.git && cd JevAgentHarness
uv venv -p 3.11 .venv && source .venv/bin/activate
uv pip install -e "research[serve,train]"
uv pip install "transformers>=5.8"                # the base model's architecture (Qwen3.5)
hf download JevAgentHarness/AgentDecision-0.8B --local-dir agentdecision-0.8b
cd research
python -m jevharness.training.serve_jev --model startlux-models/StartLux-Decision-v1-0.8B --adapter ../agentdecision-0.8b \
    --calibration ../agentdecision-0.8b/calibration.json --name agentdecision-0.8b --port 8090

The server keeps a cache of each episode's prompt prefix, answers all questions of a request in one batch, and runs the knockout for more than 52 options. --fast turns on the faster exact paths the paper's final harness used. One request can carry several questions about the same state:

curl -s localhost:8090/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": "TASK: Ship the pending order of the customer Ivan Kim.\n\nHISTORY:\n\nCURRENT OBSERVATION:\n(none)",
  "questions": {
    "tool": {"type": "choice", "instructions": "Which tool should the agent call next?",
             "criteria": {"find_user": "Look up a customer by full name; returns user_id and tier.",
                          "ship_order": "Ship a pending order.",
                          "ask_user": "Ask the customer a clarifying question and wait for the reply."}},
    "done": {"type": "noul", "instructions": "Is the task complete?"}}}'

The answer has a probability per option ("tool": {"choice": ..., "confidence": ..., "probabilities": {...}}) and "done": {"noul": <p(yes)>}.

In the research harness, the served model is a decision-model spec like any other:

S1='{"type": "jev", "name": "agentdecision-0.8b", "base_url": "http://127.0.0.1:8090", "model": "agentdecision-0.8b", "api_key": ""}'
jevbench ping --s1 "$S1"
python scripts/stage2_corpus.py replay --corpus results/stage2/s23 --name agentdecision-0.8b --s1 "$S1"   # answer the common states

The harness profile with this model's thresholds is research/configs/harness/explore_e28_final.yaml. research/README.md maps every table of the paper to its command.

In the JevAgentHarness proxy

The proxy in the top level of the repository (jah) takes any server that speaks the Jev wire format:

# ~/.jah/config.yaml
router:
  decision: {type: jev, provider: local, base_url: "http://127.0.0.1:8090", model: agentdecision-0.8b}

jah decide sends one example state and prints the probabilities, which checks the connection. Keep in mind that the proxy's default question (is the agent's next step routine or hard?) and coding-agent states are outside this model's training data. Its tool_select point asks the question the model was trained on ("Which tool should the agent call next?"), but over a coding agent's tools. Run the proxy with --preset shadow first, fit the thresholds on your own logs with jah fit, and compare jah-strong with jah-auto on your tasks before relying on it (see docs/modes.md in the repository).

Calibration and thresholds

calibration.json holds temperatures, not thresholds. The server applies them with --calibration. The snippet above looks up the key <kind>:<bucket> (kind choice or noul, bucket 2, 3-5, 6-16 or 17-52 options) and falls back to global.

thresholds.json holds the thresholds of the paper's harness for this model with Gemini 3.5 Flash as the agent LLM: the model's answer is used when its calibrated confidence is at least tau, and the LLM decides otherwise (tool choice 0.952, argument choice 0.999). They were fitted in shadow mode, where the model answers on the agent's states without acting, on 150 DecisionGym calibration tasks. They hold only for that setting. For another agent LLM, other tools or another task family, record the model's answers in shadow mode and refit: python -m jevharness.bench.threshold_theory in the research code, or jah fit in the proxy.

Training

• Data (14,000 rows): 7,000 agent-state rows from AgentDecision-train v1 (Jev soft labels mixed with the gold label, as for AgentDecision-4B), of which 3,463 rows on 2,014 states have their surface varied (core tools renamed, text reworded, near-copies of the renamed tools added as distractors, with a vocabulary disjoint from the shifted test split), plus 7,000 public tool-use items (When2Call's training split and ToolACE) after a leakage audit against every test pool.
• Recipe. LoRA (r 32, alpha 64) on every linear layer of StartLux-Decision-0.8B at revision bd4f76a6, cross-entropy plus 0.5 x a Brier term, 2 epochs at learning rate 1e-4. The checkpoint after epoch 2 was selected on development data only.
• Calibration. Temperatures fitted on 2,000 held-out development questions with research/src/jevharness/training/calibrate.py.
• Scripts. research/scripts/explore_e28_sl08b.py build ..., research/scripts/aws/e28_sl08b.sh, configs in research/configs/explore/e28_sl08b/. Record: research/results/explore/e28_sl08b/. Paper: App. G.9 ("Our recipe on a decision-tuned 0.8B base") and Table 39. The recipe is the one of App. D.9.

The augmented and public rows are not part of AgentDecision-train. The build script recreates them.

Results

All numbers are from the paper, in %. Agent LLM: Gemini 3.5 Flash. "Every tool choice" means the model picks every tool with no threshold, so the agent depends on it fully.

Level	What is measured	AgentDecision-0.8B	StartLux-Decision-0.8B (base)	Jev	Paper
L1 isolated	accuracy on 3,991 standalone questions (BFCL v4 tool choice and arguments, When2Call)	80.5	70.0	86.4	Table 3
L1 isolated	BFCL tool-choice accuracy	87.1	88.2	89.9	Figure 4b, Table 31 (App. G.6)
L2 open-loop	accuracy on the common states: recorded tool-choice states of 200 new DecisionGym tasks	98.5	48.3	91.5	Table 39 (App. G.9)
L2 open-loop	accuracy on DecisionGym-Shift states (renamed tools, more distractors, new task kinds)	95.6	51.7	91.8	Table 3
L2 open-loop	the same, must-ask states only (the only right answer is to ask the customer)	93.5	27.2	58.3	Table 3
L3 closed-loop	task success, every tool choice by the model, 200 new DecisionGym tasks	100.0	17.5	83.0	Table 39 (App. G.9)
L3 closed-loop	task success, every tool choice by the model, 200 shifted tasks	82.0	13.0	82.0	Table 3

On the shifted tasks, the model solved 96.4% of the tasks of the training generator's kinds and 48.3% of the new task shapes (App. G.9).

In the harness (App. G.9). With fitted thresholds, on 200 new DecisionGym tasks, the model took 35.9% of the tool choices without a wrong one, and the agent was non-inferior to the LLM alone (100.0% against 99.5% task success, p = 0.002). It saved 9.6% of the API cost, but our slow serving stack made the run 6.0% more expensive once GPU time is counted. On the shifted tasks the threshold did not hand control back: the model took 61.7% of the tool choices, 6 of them wrong, and non-inferiority was not shown (98.0% against 100.0%).

Serving cost. $0.010 per 1,000 decisions on the base model's own engine (App. G.9). On one NVIDIA L4 with vLLM and FP8 weights: $0.0018 per 1,000 decisions and a median of 27 ms per answer (App. G.7, Table 33), about 9 times less per decision than Jev (Section 7.1).

The benchmark data behind the L1 and L2 numbers, every model's answers and the paper's numbers per model are in AgentDecisionBench (model id e28).

Limitations

• Non-commercial. The base model's weights are CC BY-NC 4.0, and the adapter is a derivative under the same terms. Commercial use of the base needs a licence from StartLux Labs.
• Trained for DecisionGym. The agent states come from DecisionGym's customer-service tasks, in English. Other tools, task families or agent LLMs need new thresholds and, for the best accuracy, training on that agent's own states.
• Task success under shift. Step-level accuracy held on the shifted states (95.6%), but task success did not (82.0% when the model picks every tool, 48.3% on the new task shapes). In the harness at the original thresholds the model kept 61.7% of the shifted tool choices and accepted 6 wrong ones.
• Irreversible first errors. A confident wrong choice at an irreversible step ends the task. Never make a decision model the only check on an irreversible action.
• Coding harnesses are not validated. The model was never trained or tested on coding-agent states or on the routing question of the JevAgentHarness proxy.
• Not tested against adversarial input such as injected instructions in tool output.

Licence

CC BY-NC 4.0 for the adapter, calibration.json, thresholds.json and decision_config.json.

• StartLux-Decision-v1-0.8B by StartLux Labs is licensed under CC BY-NC 4.0. Its weights are a modified version of a model released under Apache-2.0 (Copyright 2026 Alibaba Cloud), and its NOTICE keeps the files taken from that model, such as the tokenizer files, under Apache-2.0. The tokenizer files here (tokenizer.json, tokenizer_config.json, chat_template.jinja) come from the base model. This adapter is a derivative of the StartLux-Decision weights and is shared under the same licence, with attribution to StartLux Labs.
• The training data contain outputs of commercial models: Jev's probabilities (TypeSafe) as soft labels, and agent text written by Gemini models (Google). The data are shared under CC BY-NC 4.0 for that reason. Check those providers' terms before any use beyond non-commercial research, in particular before training a model that competes with theirs.
• The public tool-use data come from When2Call (NVIDIA, CC BY 4.0) and ToolACE (Team-ACE, Apache-2.0).

Citation

@misc{zhu2026jevagentharnessbenchmarkingdeployingtraining,
      title={JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs},
      author={Kunlun Zhu and Tao Feng and Xu Han and Heng Wang and Ling Yang and Xiangru Tang and Jiaxuan You and Heng Ji},
      year={2026},
      eprint={2610.21412},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2610.21412},
}

The arXiv identifier will be added when it is available.