KnowLine-31B-Gen1: serving and Decision Index reproduction

These settings reproduce our self-run Decision Index 0.3 evaluation of KnowLine-31B-Gen1: the full 0.3 public suite sent fresh (no rows carried over from another run) through the release front end with read twice on, scored 2026-10-24 04:28 CST.

Weights

bf16, about 63 GB. They were made in three steps:
1. Fine-tuning: gemma-4-31B-it trained for one epoch with a LoRA adapter (rank 32, alpha 64, language model only), merged into the weights.
2. Averaging with the base model: every tensor is 0.75 × the fine-tuned tensor + 0.25 × the gemma-4-31B-it tensor (computed in fp32, stored in bf16). Tensors that fine-tuning did not change, such as the vision tower, stay those of the base model.
3. Readout temperature T = 1.44, folded in two places so that every final logit is divided exactly by T:
  - the final norm weight model.language_model.norm.weight is divided by 1.44 (Gemma 4's RMSNorm applies the weight as x * w, not (1 + w) * x);
  - text_config.final_logit_softcapping in config.json is 20.833333333333336 instead of 30.0. Gemma 4 caps its logits as cap * tanh(z / cap); with both changes the output is 30 * tanh(z / 30) / T, the capped logit divided by T. Do not change the soft-cap back to 30, or the temperature is only partly applied.
  - T was fitted by negative log-likelihood on our in-training evaluation sets, served read twice through this front end (4,965 questions); refitting on the released weights gives T = 1.00, and ECE on our held-out sets (6,995 questions) is 0.044 (0.102 at T = 1 before folding).
The architecture is Gemma4ForConditionalGeneration. Apart from final_logit_softcapping, config.json is identical to that of google/gemma-4-31B-it; the tokenizer files, the chat template, processor_config.json and generation_config.json are byte-identical to the base model.
SHA256SUMS lists every file.

Software

component	version
SGLang	0.5.21 (torch 2.13.0, CUDA 13.0)
transformers	5.12.1
front end	`knowline_server.py` + `knowline_layout.py` (this repository; need only transformers and requests)
decision-index kit	0.3 at commit 62d2f51de34a2de64906345b6bc3e98e27ff55c7

About the front end

knowline_server.py is the /v1/systemone front end of our runs; the prompt layout lives in knowline_layout.py next to it. Text requests are rendered exactly as by the KnowLine-26B-A4B-Gen2.5 front end, prompt for prompt (checked on every backend call of our test requests). It renders with the model's own chat template. The front end contains:

rendering: the chat template with thinking off; the state is followed by one user turn with the instruction, the question and all options labelled A, B, ...; the assistant turn opens with Answer:;
read twice (on by default, see below);
label-token scoring: one prefill per question, then a softmax over the option labels only;
the HTTP server.

Read twice

What it does: for a text-only request of at most 3,000 tokens (counted with the model's tokenizer over the whole request read once), every turn after the system turn is written a second time before the answer is read: the state, the instruction, the question and its options. The second copy starts with the line "Read the same input again before answering:". The answer is still read from the label token after Answer:, one prefill per question.
What it leaves alone: requests over 3,000 tokens and requests with images are sent once. In our Decision Index run, 132,376 of the 140,620 requests (94.1%) were read twice (the rows and the Gemma 4 tokenizer are those of KnowLine-26B-A4B-Gen2.5's run, so the same requests fall under the cut).
Switches: --no-read-twice reads every request once; --twice-max-tokens sets the cut. knowline_engine.py takes --option read_twice=0 and --option twice_max_tokens=<n> for the same. All our evaluations of this model used read twice.
Cost: about 1.54 times the prompt tokens of reading once over the whole suite (323 million against 209 million, the same rows and tokenizer as KnowLine-26B-A4B-Gen2.5's run); see Hardware and latency.
Credit: the read-twice layout follows FlexDM (github.com/1000luca/FlexDM, MIT), including its "Read the same input again before answering:" line; repeating the input goes back to RE2 (Xu et al., 2023, arXiv:2309.06275). knowline_layout.py is our own implementation.

The front end uses temperature 1 and the readout temperature is in the weights, so both backends get it. One setting is new in this release: temperatures.json ({"score": 0.6944}, shipped in this repository) makes the front end divide the logits of score (rubric) questions by a further 0.6944, so they are read at an effective temperature of 1.44 × 0.6944 = 1.0, while yes/no and choice questions stay at 1.44. Score questions are often read as an expected value over the scale, which a higher temperature pulls toward the middle. serve_knowline.sh passes the file and knowline_engine.py loads it by default (--option temperatures=none turns it off). The earlier KnowLine releases did not use per-type temperatures. All our evaluations of this model used them, except where noted.

Serve

serve_knowline.sh starts both processes, with the settings of KnowLine-26B-A4B-Gen2.5 plus temperatures.json and the box-question flags.

SGLang engine, bf16.
- Served in bf16, without --quantization fp8, as the 26B-A4B (for which two FP8 runs of the same questions agreed on only 95.7% of the decisions, against 99.35% in bf16). We did not test FP8 at load for this model.
- There is no --mamba-radix-cache-strategy flag (Gemma 4 has no recurrent layers).
- SGLang reserves --mem-fraction-static (default 0.85) of the GPU's memory for the weights and its KV cache: about 82 GB on a 96 GB card, of which the weights take about 63 GB. The value only sizes the KV cache; scores do not depend on it. Every run below used one 96 GB H20 per server; we have not run it on smaller GPUs.
- SGLang treats the model as multimodal by itself. The Decision Index run sent text only.
/v1/systemone front end: knowline_server.py, chat style, temperature 1, 16 scoring threads, read twice up to 3,000 tokens, per-type temperatures from temperatures.json (score questions / 0.6944), box questions with --box-prompt gemma --box-coords yxyx (Gemma's box_2d format, y first) and box_calibration.json when present.

pip install "sglang==0.5.21" "transformers==5.12.1" requests   # torch / CUDA per SGLang's install docs
bash serve_knowline.sh PelaAI/KnowLine-31B-Gen1 0 8080            # GPU 0; SGLang on :9080, /v1/systemone on :8080
curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "m",
  "state": "Customer: my order arrived broken, I want my money back.",
  "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"},
                "tone": {"type": "choice", "instructions": "Customer tone?",
                         "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'

Equivalent manual launch, with the exact flags of our run:

CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --model-path PelaAI/KnowLine-31B-Gen1 --served-model-name m \
  --tp 1 --mem-fraction-static 0.85 --enable-fp32-lm-head --port 9080 &
python knowline_server.py --model PelaAI/KnowLine-31B-Gen1 --backend sglang --url http://127.0.0.1:9080 \
  --served-model-name m --temperature 1 --workers 16 --port 8080 --temperatures temperatures.json \
  --box-prompt gemma --box-coords yxyx          # read twice is on by default

Without SGLang (through transformers; text only, no prefix cache, bf16):

pip install torch "transformers==5.12.1" requests
python knowline_server.py --model PelaAI/KnowLine-31B-Gen1 --backend hf --port 8080 --temperatures temperatures.json

Loaded in bf16, the weights take about 62.5 GB (58.3 GiB) of GPU memory.
Checked on one H20, read twice on: on 30 random Decision Index 0.3 rows, 80 of the 82 answers were the same as in our SGLang run; the other two are near-ties (the top two options within 0.06 of each other in one of the two runs). Requests took 576.5 ms median one at a time.
Images need the SGLang backend.

Decision Index run

Suite rows: the 140,620 rows that decision-index run --edition 0.3 sends: selected-rows.jsonl.gz, added-rows.jsonl.gz and gsm8k-rows.jsonl.gz of suite-0.3, with ToolRet, BRIGHT and ACOS cut to their scoring subsets.
Four identical servers ran on four GPUs, one each, with the settings above (--mem-fraction-static 0.90, which only sizes the KV cache). The rows are split round-robin into 16 shards, 4 per server, so that each server's prefix cache keeps the shared prefixes of its few concurrent requests. Each shard is a resumable decision-index run against its server.
The run used the released knowline_server.py (read twice on by default) from a local copy of this repository before the upload. All 140,178 scoreable requests were answered; none was unsupported or truncated, and none failed.

decision-index run --edition 0.3 --engine http --option base_url=http://127.0.0.1:<port> --option model=m --no-verify \
  --compact --rows shards/<i>.jsonl.gz --out shards/<i>          # i = 0..63, run in parallel
cat shards/*/results.jsonl > results.jsonl
decision-index score --suite-dir suite-0.3 --results results.jsonl --engine http --out score

Hardware and latency

Four NVIDIA H20 (96 GB each), one server per GPU.
Latency: we sent the same 300 random Decision Index 0.3 rows (477 questions) as for KnowLine-26B-A4B-Gen2 and Gen2.5, one at a time, to one server on one H20 (after a warm-up).
- Requests took 340.3 ms median, 910.0 ms mean, 1,185.4 ms at the 80th percentile and 3,976.6 ms at the 95th.
- 473 of the 477 answers were the same as in our Decision Index run; the other four are near-ties (the top two options within 0.06 of each other in one of the two runs). A second pass matched the first on 476 of 477.
- KnowLine-26B-A4B-Gen2.5 on the same rows and hardware: 66.3 ms median, 151.1 ms mean and 628.6 ms at the 95th percentile.
Latency was not measured on the board's reference hardware (1x RTX PRO 6000, latency-v1).

One-command Decision Index run

knowline_engine.py (in this repository) is a Decision Index engine. It runs the same front end in process, so there is no server to start by hand:

pip install "sglang==0.5.21" "transformers==5.12.1" requests        # plus the decision-index kit
git clone https://huggingface.co/PelaAI/KnowLine-31B-Gen1 && cd KnowLine-31B-Gen1   # puts the three files on the path
python -m decision_index pipeline --engine knowline_engine:KnowLine \
    --option model=PelaAI/KnowLine-31B-Gen1 --option revision=<commit> --out runs/KnowLine-31B-Gen1

Default backend and layout (the setting of our runs): read twice is on; the engine starts SGLang with the flags above (bf16, --mem-fraction-static 0.85) on a free local port and stops it when the run ends. Use --option gpu=<n> to pick a GPU, and --option sglang_python=<python> if SGLang lives in another environment.
--option backend=hf: transformers only, bf16, no server; slower, and not the setting of our runs.
Limits: up to 64 questions per request and 255 options per question. Larger requests are reported as unsupported; nothing is truncated.
Temperature in both backends: transformers and SGLang both read the final norm weight and final_logit_softcapping from these files, so both backends apply the folded temperature; the per-type temperatures of temperatures.json are applied by the front end, the same for both. Our Decision Index run and calibration numbers use SGLang.

Acknowledgements

Read-twice layout: FlexDM (github.com/1000luca/FlexDM, MIT licence).
Input repetition: Xu et al., 2023, "Re-Reading Improves Reasoning in Large Language Models" (arXiv:2309.06275).