KnowLine-4B-Gen5.5

KnowLine-4B-Gen5 with a new way of serving it: the same weights, and a front end that reads short requests twice before it answers.

Inference guide · 中文说明 · weights Apache-2.0 · previous version: KnowLine-4B-Gen5 · siblings: KnowLine-35B-A3B-Gen1.5, KnowLine-26B-A4B-Gen2.5

We raise the minor version when only inference changes and the major version (Gen) when the weights change. Every weight, config and tokenizer file of Gen5.5 is byte-identical to Gen5.

Highlights:

• Same weights, read twice. For a text-only request of up to 3,000 tokens, the front end writes the whole input a second time before the model answers. Nothing was trained; see What changed in Gen5.5.
• Decision Index 0.3 public suite: 66.75 (self-run, fresh full run), up from 65.67 for Gen5 with the same weights. The gains are mostly in reasoning and inference over long inputs (knowledge and reasoning 45.0 → 47.3, language 66.8 → 68.2); tools and agents are unchanged (87.4) and arts and taste slightly lower (61.2 → 61.0). See Comparison.
• Decision tasks from new domains: on our 18 real-text decision tasks from domains not in training it scores 55.0, against 51.4 for Gen5 with the same weights on the same harness; on our 34-task set 48.4 (Gen5: 42.5), and on our older 10-task set 51.3 (Gen5: 49.5).
• Same skills on other data: on our sibling sets, which test the skill of a public benchmark on other data, it averages 66.1 (Gen5: 63.6).
• Private scores: the board's full score comes from the maintainers' private run. Gen5 is listed at 53.68. Gen5.5 has the same weights as Gen5; its private scores will be produced by the maintainers' run.
• Our other evaluations: Open-Jev 1.1 test / OOD 92.2 / 90.7 (Gen5 on the same files: 90.0 / 88.0), C-Eval 80.5 (79.2), JevBench / JevBench-hard 85.3 / 70.3 (83.1 / 66.7), and slightly higher scores on our in-house held-out sets. Planted instructions changed 1.2% of answers (Gen5: 0.6%), and 2.7% with injection templates not seen in training (Gen5: 2.5%); neither difference is significant.
• Calibration: the temperature folded into the weights is unchanged (T = 1.50). Computed with the board's method, ECE is about 0.035-0.046 (Gen5: 0.037-0.049) and the Brier score about 0.27-0.30 (Gen5: 0.28-0.31); see Calibration.
• Cost: about 1.56 times the prompt tokens of Gen5 on the Decision Index suite. One request at a time on one H20: 42.1 ms median and 83 ms mean, against 31.6 ms and 58.8 ms for Gen5 (our measurement, not the board's reference hardware); see INFERENCE.md.
• Images: unchanged. Requests with images are sent exactly as in Gen5, so Gen5's image results apply; see Images.

KnowLine is an independent model. It is not affiliated with, endorsed by, or derived from TypeSafe or Jev.

What it does

You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model answers every question with a probability distribution over its options:

• one forward pass per question, with no text generation;
• the output is the probability of each option's label token;
• existing Jev clients only need to change the base URL, and users of earlier versions only need to change the model name, because the interface is unchanged.

Base model	Qwen/Qwen3.5-4B (Apache-2.0)
Weights	byte-identical to KnowLine-4B-Gen5: a weight average of two LoRA SFT models (rank 32, alpha 64, language model only), merged into bf16 weights, with a readout temperature T = 1.50 folded into the final norm weight
Serving	serve_knowline.sh: SGLang with FP8 at load, as for Gen5; knowline_server.py reads short requests twice (on by default; --no-read-twice gives the Gen5 prompts). We ran every evaluation with one server on one 96 GB H20.
Languages	English, Simplified Chinese, Traditional Chinese

Quickstart

pip install "sglang==0.5.21" "transformers==5.12.1" requests
bash serve_knowline.sh PelaAI/KnowLine-4B-Gen5.5 0 8080     # SGLang (FP8 at load) on :9080 + /v1/systemone on :8080
curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "m",
  "state": "Customer: my order arrived broken, I want my money back.",
  "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"},
                "tone": {"type": "choice", "instructions": "Customer tone?",
                         "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'

• Front end: knowline_server.py with knowline_layout.py next to it; together they depend only on transformers and requests.
• Full settings: the exact settings used for our Decision Index run are in INFERENCE.md.

What changed in Gen5.5

1. Read twice. Gen5 saw each request once: the state, then the instruction, the question and its options. Gen5.5 writes all of that a second time, introduced by the line "Read the same input again before answering:", and then reads the answer from the label token as before. The rule is mechanical: it applies to every text-only request of at most 3,000 tokens (94.6% of the Decision Index requests); longer requests and requests with images are sent as in Gen5.
2. A likely reason. With causal attention, the first copy of a long state is read before the model knows the question; in the second copy every token can also attend to the first copy, question included.
3. No training. The weights, the temperature and the label-token readout are those of Gen5. We checked the change on our own evaluation sets and on a fresh full run of the Decision Index public suite before release. It is the same change as in KnowLine-35B-A3B-Gen1.5 and KnowLine-26B-A4B-Gen2.5.
4. Credit. The read-twice layout follows FlexDM (github.com/1000luca/FlexDM, MIT); repeating the input goes back to RE2 (Xu et al., 2023, arXiv:2309.06275). knowline_layout.py is our own implementation.

Comparison

Scores on the Decision Index. Since Decision Index 0.4, the board's full score is 60% private tests of the same skills and 40% private tasks from new domains; the public suite is shown but no longer counts. Gen5.5 has no board entry yet.

model	size	DI 0.3 public	DI 0.4.1 full	source
KnowLine-4B-Gen5.5	4B	66.75	not yet scored	self-run, official kit
KnowLine-4B-Gen5 (same weights)	4B	65.67	53.68	board
StartLux-Decision v1.1 4B	4.7B	67.81	54.80	board
Perplexity Decider Flash v1.2	4.5B	60.06	54.29	board
vLLM-SR d3-mini	4.5B	54.90	49.97	board
OneJev 1.6	4.5B	61.95	48.88	board
Jev 1.14	(API)	60.70	60.24	board
KnowLine-35B-A3B-Gen1	35B-A3B	69.49	61.20	board

Board rows from the board's data of 2026-10-21 (22:34 UTC, Decision Index 0.4.1).

Public-suite skill by area (DI 0.3, same weights):

area	Gen5.5	Gen5
Knowledge & reasoning	47.3	45.0
Language	68.2	66.8
Retrieval & routing	74.0	73.3
Tools & agents	87.4	87.4
Arts & taste	61.0	61.2

Releases

The model is trained in a self-evolving loop, and new versions will follow.

model	date	DI 0.3 public	golden held-out (en / zh-Hans / zh-Hant)	notes
KnowLine-4B-Gen5.5 (this model)	2026-10-22	66.75	70.3 / 72.1 / 67.5	Gen5 weights, read-twice serving
KnowLine-4B-Gen5	2026-10-21	65.67	69.6 / 71.7 / 67.1	fifth release; rebalanced data, aimed at generalization
KnowLine-4B-Gen4	2026-10-10	64.90	69.7 / 70.1 / 65.1	fourth release
KnowLine-4B-Gen3	2026-10-09	63.11	69.4 / 70.8 / 65.0	third release
KnowLine-4B-Gen2	2026-10-08	62.54	69.7 / 71.2 / 64.8	second release
KnowLine-4B-Gen1	2026-10-07	60.47	68.7 / 69.9 / 64.1	first release

Evaluation

All results are self-run and have not been verified by a third party. Every Gen5.5 number below was measured through the Gen5.5 front end (read twice on). Unless noted, the Gen5 numbers were measured again on the same files.

Held-out, Chinese and generalization evaluations

suite	Gen5.5	Gen5
In-house evaluation set, English	70.3	69.6
In-house evaluation set, Simplified Chinese	72.1	71.7
In-house evaluation set, Traditional Chinese	67.5	67.1
C-Eval (4 categories, macro)	80.5	79.2
Open-Jev 1.1 test / OOD	92.2 / 90.7	90.0 / 88.0
JevBench / JevBench-hard	85.3 / 70.3	83.1 / 66.7
Decision tasks from new domains (our own, 18 real-text tasks)¹	55.0	51.4
Decision tasks from new domains (our own, 34 tasks)¹	48.4	42.5
Decision tasks from new domains (our own, 10 tasks)¹	51.3	49.5
Sibling sets (same skill, other data; mean skill)¹	66.1	63.6
Prompt injection: answers changed (lower is better)	1.2%	0.6%
Prompt injection with templates not seen in training (lower is better)	2.7%	2.5%

• In a paired test against Gen5, p < 0.001 for Open-Jev 1.1 test and OOD and p = 0.02 for C-Eval; the differences on the in-house sets (p = 0.20, 0.44 and 0.61), JevBench (p = 0.23) and JevBench-hard (p = 0.34) are not significant. Planted instructions changed 12 of 1,000 answers against 6 for Gen5 (p = 0.24), and 16 against 15 of 600 with unseen templates.
• The Gen5 column matches the Gen5 card.
• ¹ Re-run for both models under the same serving (FP8 at load, read twice through our evaluation front end, which uses the same layout); the Gen5 card's values (51.3, 43.0, 49.4 and 63.7) came from the serving of the time.

Web operation (Mind2Web official test splits, evaluation only)

split	Gen5.5 element selection	Gen5.5 operation (balanced)	Gen5 element selection	Gen5 operation (balanced)
test_task (websites seen in training, new tasks)	92.7	96.9	92.6	97.1
test_website (new websites)	91.2	96.3	90.9	96.9
test_domain (new domains)	92.0	98.0	91.8	98.4

• Within noise of Gen5 on all three splits (paired test, p ≥ 0.06); no request failed.

• The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web leaderboard.

Images (zero-shot)

• Unchanged from Gen5: read twice applies only to text-only requests. A request with images is sent exactly as in Gen5, byte for byte, to the same weights, so Gen5's image results hold for Gen5.5 and we did not rerun them.
• Gen5's results: 66.0 on our reconstruction of the Decision Index 0.3.1 vision public suite (Gen4: 66.4; stock Qwen3.5-4B: 59.4); see the Gen5 card for the per-benchmark table.

Calibration

Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are used.

metric	Gen5.5	Gen5
ECE (lower is better)	0.035-0.046	0.037-0.049
Brier score (lower is better)	0.27-0.30	0.28-0.31
Confidence ≥95% but wrong	1.5-1.6%	1.5-1.6%
Mean confidence / accuracy	0.83 / 0.78-0.80	0.82-0.83 / 0.78-0.80

• Temperature in the weights: unchanged from Gen5: the final norm weight includes T = 1.50, so every logit is divided by 1.50 before the softmax over the option labels.
• Why ranges: the board does not publish which benchmarks it uses for calibration, so we report ranges over the plausible benchmark sets, as for Gen5.
• Recommendation: if your use depends on probability thresholds, check calibration on your own data.

Disclosures

• Training data: the weights are those of Gen5, so Gen5's disclosures on training data, Decision Index format data and decontamination apply unchanged; see the Gen5 card.
• Selection on evaluations: we adopted read twice after checking it on our own evaluation sets and on a fresh full run of the Decision Index 0.3 public suite (the 66.75 above).
• No model outputs as labels: no output of Jev or any other decision model was used as a training label.

Limitations

• Slower than Gen5: about 1.56 times the prompt tokens on the Decision Index suite; one request at a time on one H20, 42.1 ms median and 83 ms mean (Gen5: 31.6 ms and 58.8 ms), and 297 ms at the 95th percentile (Gen5: 144 ms). Use --no-read-twice if latency matters more than these gains.
• Only short requests change: text-only requests over 3,000 tokens, and requests with images, are answered exactly as by Gen5.
• Not better everywhere: arts and taste on the public suite is slightly lower than with Gen5 (61.2 → 61.0), and planted instructions changed 1.2% of answers against 0.6% (not significant).
• Everything else as Gen5: knowledge-heavy reasoning is still the weakest area and a 4B model is limited there; new domains are better than with Gen5 on our own sets but still well below larger models; other prompt-injection attacks may still work; the image scores are zero-shot and self-run on our own reconstruction of the suite.

Citation

@misc{knowline4bgen55,
  title  = {KnowLine-4B-Gen5.5: a 4B decision model served with read twice},
  author = {PelaAI},
  year   = {2026},
  url    = {https://huggingface.co/PelaAI/KnowLine-4B-Gen5.5}
}

License

• Weights: Apache-2.0, the same as the base model.
• Code: knowline_server.py, knowline_layout.py and knowline_engine.py are MIT (see the file headers).