# KnowLine-31B-Gen1: serving and Decision Index reproduction These settings reproduce our self-run Decision Index 0.3 evaluation of KnowLine-31B-Gen1: the full 0.3 public suite sent fresh (no rows carried over from another run) through the release front end with read twice on, scored 2026-10-24 04:28 CST. ## Weights - bf16, about 63 GB. They were made in three steps: 1. **Fine-tuning:** gemma-4-31B-it trained for one epoch with a LoRA adapter (rank 32, alpha 64, language model only), merged into the weights. 2. **Averaging with the base model:** every tensor is 0.75 × the fine-tuned tensor + 0.25 × the gemma-4-31B-it tensor (computed in fp32, stored in bf16). Tensors that fine-tuning did not change, such as the vision tower, stay those of the base model. 3. **Readout temperature T = 1.44**, folded in two places so that every final logit is divided exactly by T: - the final norm weight `model.language_model.norm.weight` is divided by 1.44 (Gemma 4's RMSNorm applies the weight as `x * w`, not `(1 + w) * x`); - `text_config.final_logit_softcapping` in `config.json` is 20.833333333333336 instead of 30.0. Gemma 4 caps its logits as `cap * tanh(z / cap)`; with both changes the output is `30 * tanh(z / 30) / T`, the capped logit divided by T. **Do not change the soft-cap back to 30**, or the temperature is only partly applied. - T was fitted by negative log-likelihood on our in-training evaluation sets, served read twice through this front end (4,965 questions); refitting on the released weights gives T = 1.00, and ECE on our held-out sets (6,995 questions) is 0.044 (0.102 at T = 1 before folding). - The architecture is `Gemma4ForConditionalGeneration`. Apart from `final_logit_softcapping`, `config.json` is identical to that of [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it); the tokenizer files, the chat template, `processor_config.json` and `generation_config.json` are byte-identical to the base model. - `SHA256SUMS` lists every file. ## Software | component | version | |---|---| | SGLang | 0.5.21 (torch 2.13.0, CUDA 13.0) | | transformers | 5.12.1 | | front end | `knowline_server.py` + `knowline_layout.py` (this repository; need only transformers and requests) | | decision-index kit | 0.3 at commit 62d2f51de34a2de64906345b6bc3e98e27ff55c7 | ### About the front end `knowline_server.py` is the `/v1/systemone` front end of our runs; the prompt layout lives in `knowline_layout.py` next to it. Text requests are rendered exactly as by the KnowLine-26B-A4B-Gen2.5 front end, prompt for prompt (checked on every backend call of our test requests). It renders with the model's own chat template. The front end contains: - rendering: the chat template with thinking off; the state is followed by one user turn with the instruction, the question and all options labelled A, B, ...; the assistant turn opens with `Answer:`; - read twice (on by default, see below); - label-token scoring: one prefill per question, then a softmax over the option labels only; - the HTTP server. ### Read twice - **What it does:** for a text-only request of at most 3,000 tokens (counted with the model's tokenizer over the whole request read once), every turn after the system turn is written a second time before the answer is read: the state, the instruction, the question and its options. The second copy starts with the line "Read the same input again before answering:". The answer is still read from the label token after `Answer:`, one prefill per question. - **What it leaves alone:** requests over 3,000 tokens and requests with images are sent once. In our Decision Index run, 132,376 of the 140,620 requests (94.1%) were read twice (the rows and the Gemma 4 tokenizer are those of KnowLine-26B-A4B-Gen2.5's run, so the same requests fall under the cut). - **Switches:** `--no-read-twice` reads every request once; `--twice-max-tokens` sets the cut. `knowline_engine.py` takes `--option read_twice=0` and `--option twice_max_tokens=` for the same. All our evaluations of this model used read twice. - **Cost:** about 1.54 times the prompt tokens of reading once over the whole suite (323 million against 209 million, the same rows and tokenizer as KnowLine-26B-A4B-Gen2.5's run); see [Hardware and latency](#hardware-and-latency). - **Credit:** the read-twice layout follows FlexDM ([github.com/1000luca/FlexDM](https://github.com/1000luca/FlexDM), MIT), including its "Read the same input again before answering:" line; repeating the input goes back to RE2 (Xu et al., 2023, [arXiv:2309.06275](https://arxiv.org/abs/2309.06275)). `knowline_layout.py` is our own implementation. The front end uses temperature 1 and the readout temperature is in the weights, so both backends get it. One setting is new in this release: `temperatures.json` (`{"score": 0.6944}`, shipped in this repository) makes the front end divide the logits of score (rubric) questions by a further 0.6944, so they are read at an effective temperature of 1.44 × 0.6944 = 1.0, while yes/no and choice questions stay at 1.44. Score questions are often read as an expected value over the scale, which a higher temperature pulls toward the middle. `serve_knowline.sh` passes the file and `knowline_engine.py` loads it by default (`--option temperatures=none` turns it off). The earlier KnowLine releases did not use per-type temperatures. All our evaluations of this model used them, except where noted. ## Serve `serve_knowline.sh` starts both processes, with the settings of KnowLine-26B-A4B-Gen2.5 plus `temperatures.json` and the box-question flags. 1. **SGLang engine, bf16.** - Served in bf16, without `--quantization fp8`, as the 26B-A4B (for which two FP8 runs of the same questions agreed on only 95.7% of the decisions, against 99.35% in bf16). We did not test FP8 at load for this model. - There is no `--mamba-radix-cache-strategy` flag (Gemma 4 has no recurrent layers). - SGLang reserves `--mem-fraction-static` (default 0.85) of the GPU's memory for the weights and its KV cache: about 82 GB on a 96 GB card, of which the weights take about 63 GB. The value only sizes the KV cache; scores do not depend on it. Every run below used one 96 GB H20 per server; we have not run it on smaller GPUs. - SGLang treats the model as multimodal by itself. The Decision Index run sent text only. 2. **`/v1/systemone` front end:** `knowline_server.py`, `chat` style, temperature 1, 16 scoring threads, read twice up to 3,000 tokens, per-type temperatures from `temperatures.json` (score questions / 0.6944), box questions with `--box-prompt gemma --box-coords yxyx` (Gemma's `box_2d` format, y first) and `box_calibration.json` when present. ```bash pip install "sglang==0.5.21" "transformers==5.12.1" requests # torch / CUDA per SGLang's install docs bash serve_knowline.sh PelaAI/KnowLine-31B-Gen1 0 8080 # GPU 0; SGLang on :9080, /v1/systemone on :8080 curl -s http://127.0.0.1:8080/health curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{ "model": "m", "state": "Customer: my order arrived broken, I want my money back.", "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"}, "tone": {"type": "choice", "instructions": "Customer tone?", "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}' ``` Equivalent manual launch, with the exact flags of our run: ```bash CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --model-path PelaAI/KnowLine-31B-Gen1 --served-model-name m \ --tp 1 --mem-fraction-static 0.85 --enable-fp32-lm-head --port 9080 & python knowline_server.py --model PelaAI/KnowLine-31B-Gen1 --backend sglang --url http://127.0.0.1:9080 \ --served-model-name m --temperature 1 --workers 16 --port 8080 --temperatures temperatures.json \ --box-prompt gemma --box-coords yxyx # read twice is on by default ``` Without SGLang (through transformers; text only, no prefix cache, bf16): ```bash pip install torch "transformers==5.12.1" requests python knowline_server.py --model PelaAI/KnowLine-31B-Gen1 --backend hf --port 8080 --temperatures temperatures.json ``` - Loaded in bf16, the weights take about 62.5 GB (58.3 GiB) of GPU memory. - Checked on one H20, read twice on: on 30 random Decision Index 0.3 rows, 80 of the 82 answers were the same as in our SGLang run; the other two are near-ties (the top two options within 0.06 of each other in one of the two runs). Requests took 576.5 ms median one at a time. - Images need the SGLang backend. ## Decision Index run - Suite rows: the 140,620 rows that `decision-index run --edition 0.3` sends: `selected-rows.jsonl.gz`, `added-rows.jsonl.gz` and `gsm8k-rows.jsonl.gz` of `suite-0.3`, with ToolRet, BRIGHT and ACOS cut to their scoring subsets. - Four identical servers ran on four GPUs, one each, with the settings above (`--mem-fraction-static 0.90`, which only sizes the KV cache). The rows are split round-robin into 16 shards, 4 per server, so that each server's prefix cache keeps the shared prefixes of its few concurrent requests. Each shard is a resumable `decision-index run` against its server. - The run used the released `knowline_server.py` (read twice on by default) from a local copy of this repository before the upload. All 140,178 scoreable requests were answered; none was unsupported or truncated, and none failed. ```bash decision-index run --edition 0.3 --engine http --option base_url=http://127.0.0.1: --option model=m --no-verify \ --compact --rows shards/.jsonl.gz --out shards/ # i = 0..63, run in parallel cat shards/*/results.jsonl > results.jsonl decision-index score --suite-dir suite-0.3 --results results.jsonl --engine http --out score ``` ## Hardware and latency - Four NVIDIA H20 (96 GB each), one server per GPU. - **Latency:** we sent the same 300 random Decision Index 0.3 rows (477 questions) as for KnowLine-26B-A4B-Gen2 and Gen2.5, one at a time, to one server on one H20 (after a warm-up). - Requests took 340.3 ms median, 910.0 ms mean, 1,185.4 ms at the 80th percentile and 3,976.6 ms at the 95th. - 473 of the 477 answers were the same as in our Decision Index run; the other four are near-ties (the top two options within 0.06 of each other in one of the two runs). A second pass matched the first on 476 of 477. - KnowLine-26B-A4B-Gen2.5 on the same rows and hardware: 66.3 ms median, 151.1 ms mean and 628.6 ms at the 95th percentile. - Latency was not measured on the board's reference hardware (1x RTX PRO 6000, latency-v1). ## One-command Decision Index run `knowline_engine.py` (in this repository) is a Decision Index engine. It runs the same front end in process, so there is no server to start by hand: ```bash pip install "sglang==0.5.21" "transformers==5.12.1" requests # plus the decision-index kit git clone https://huggingface.co/PelaAI/KnowLine-31B-Gen1 && cd KnowLine-31B-Gen1 # puts the three files on the path python -m decision_index pipeline --engine knowline_engine:KnowLine \ --option model=PelaAI/KnowLine-31B-Gen1 --option revision= --out runs/KnowLine-31B-Gen1 ``` - **Default backend and layout (the setting of our runs):** read twice is on; the engine starts SGLang with the flags above (bf16, `--mem-fraction-static 0.85`) on a free local port and stops it when the run ends. Use `--option gpu=` to pick a GPU, and `--option sglang_python=` if SGLang lives in another environment. - **`--option backend=hf`:** transformers only, bf16, no server; slower, and not the setting of our runs. - **Limits:** up to 64 questions per request and 255 options per question. Larger requests are reported as unsupported; nothing is truncated. - **Temperature in both backends:** transformers and SGLang both read the final norm weight and `final_logit_softcapping` from these files, so both backends apply the folded temperature; the per-type temperatures of `temperatures.json` are applied by the front end, the same for both. Our Decision Index run and calibration numbers use SGLang. ## Acknowledgements - Read-twice layout: FlexDM ([github.com/1000luca/FlexDM](https://github.com/1000luca/FlexDM), MIT licence). - Input repetition: Xu et al., 2023, "Re-Reading Improves Reasoning in Large Language Models" ([arXiv:2309.06275](https://arxiv.org/abs/2309.06275)).