Instructions to use PelaAI/KnowLine-9B-Gen1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use PelaAI/KnowLine-9B-Gen1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="PelaAI/KnowLine-9B-Gen1")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("PelaAI/KnowLine-9B-Gen1")
model = AutoModelForMultimodalLM.from_pretrained("PelaAI/KnowLine-9B-Gen1", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use PelaAI/KnowLine-9B-Gen1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "PelaAI/KnowLine-9B-Gen1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PelaAI/KnowLine-9B-Gen1",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/PelaAI/KnowLine-9B-Gen1

SGLang

How to use PelaAI/KnowLine-9B-Gen1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "PelaAI/KnowLine-9B-Gen1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PelaAI/KnowLine-9B-Gen1",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "PelaAI/KnowLine-9B-Gen1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PelaAI/KnowLine-9B-Gen1",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use PelaAI/KnowLine-9B-Gen1 with Docker Model Runner:
```
docker model run hf.co/PelaAI/KnowLine-9B-Gen1
```

KnowLine-9B-Gen1 / INFERENCE.md

PEScn

Report full DI 0.3 results and serial consistency validation

a421fc4 verified 10 days ago

preview code

Raw History Blame Contribute Delete

4.45 kB

# KnowLine-9B-Gen1: serving and Decision Index reproduction

This file describes the serving path used for the KnowLine-9B-Gen1 evaluation. The model reads option-label probabilities directly; it does not generate free-form answers.

## Weights

- Base: Qwen3.5-9B.
- One epoch of LoRA SFT, rank 32, alpha 64, language-model linear layers, bf16.
- Training data: `mixU_clean_qwen35.jsonl`, 1,143,656 request rows after cleaning.
- Maximum training length: 12,288 tokens.
- Candidate: `a3-9b-u-ck4644-wise90-t118`.
- Readout temperature `T=1.18` is folded into `model.language_model.norm.weight`. The exact metadata is in `CALIBRATION.json`.

Do not apply `T=1.18` a second time. The front end reads at temperature 1 after the folded readout temperature.

## Software

The included front end consists of:

- `knowline_server.py`: `/v1/systemone` HTTP server;
- `knowline_engine.py`: backend and label-probability readout;
- `knowline_layout.py`: prompt layout and read-twice logic;
- `serve_knowline.sh`: SGLang launch helper.

The earlier internal candidate evaluation used FP8-at-load SGLang. The release launcher uses bf16 SGLang 0.5.21 with Qwen3.5's chat template, thinking disabled, 16 scoring workers, and read-twice enabled for short text-only requests. The full public DI run below completed on 2026-10-25 using this release package.

## Read twice

For a text-only request whose one-pass length is at most 3,000 tokens, the front end writes the user input a second time before the answer position. The second copy starts with:

```text
Read the same input again before answering:
```

Requests above the threshold and requests with images are read once. Disable this behavior with the front end's `--no-read-twice` option when comparing serving formats.

## Launch

```bash
bash serve_knowline.sh /path/to/KnowLine-9B-Gen1 0 8080
```

Equivalent shape:

```bash
CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server \
  --model-path /path/to/KnowLine-9B-Gen1 \
  --served-model-name m \
  --tp 1 --mem-fraction-static 0.85 --enable-fp32-lm-head \
  --mamba-radix-cache-strategy extra_buffer \
  --port 9080
python knowline_server.py \
  --model /path/to/KnowLine-9B-Gen1 \
  --backend sglang \
  --url http://127.0.0.1:9080 \
  --served-model-name m \
  --temperature 1 \
  --workers 16 \
  --port 8080
```

## Decision Index result

The official Decision Index 0.3 scorer reports **65.67 public index / 73.88 raw index**. All 140,620 requests were sent fresh; 140,178 are scoreable, and 442 are excluded by the official suite. Final scoreable counts: 140,178 ok, zero errors and zero unsupported; no truncation.

Two identical H20 servers processed 16 round-robin shards (eight per server). The initial pass had two HTTP504 transport errors, one SATA-Bench and one RouterBench request. Only those failed requests were retried, using the official runner and identical payload hashes, weights and serving. The original first-pass file and retry provenance were retained. The other 140,618 successful request responses were not changed or selected by correctness.

A separate 300-request serial pass compared 477 answers with this model's own final run: 476 agreed. The only disagreement had new/old margins 0.006/0.021, both below 0.06. All disagreements therefore qualify as near-ties; agreement after excluding near-ties is 100%. Single-request H20 latency: median 96.7 ms, mean 207.7 ms, p80 261.9 ms, p95 826.7 ms. Initial parallel-run request times include queueing and should not be substituted for these serial values.

The earlier 65.68 sample index and 59.55 predicted private Full were internal candidate-selection estimates, not this full public result. Public index and private leaderboard Full are different metrics and must not be ranked against each other.

## Limitations

- Images were not used in 9B Gen1 training.
- No block-bidirectional attention was used.
- Do not silently truncate long requests.
- The front end is for structured decisions, not open-ended chat.

`SHA256SUMS` covers every file in the repository.

## Serving correction (2026-10-25)

The launcher now uses Qwen's `locate`/`xyxy` box settings and no longer references the absent Gemma-specific `temperatures.json`. The folded T=1.18 remains the only readout scaling. Choice, yes/no and score requests passed an actual two-server bf16 smoke run before the full DI run was started. Image and box output quality is not established by that text-only smoke.

	# KnowLine-9B-Gen1: serving and Decision Index reproduction

	This file describes the serving path used for the KnowLine-9B-Gen1 evaluation. The model reads option-label probabilities directly; it does not generate free-form answers.

	## Weights

	- Base: Qwen3.5-9B.
	- One epoch of LoRA SFT, rank 32, alpha 64, language-model linear layers, bf16.
	- Training data: `mixU_clean_qwen35.jsonl`, 1,143,656 request rows after cleaning.
	- Maximum training length: 12,288 tokens.
	- Candidate: `a3-9b-u-ck4644-wise90-t118`.
	- Readout temperature `T=1.18` is folded into `model.language_model.norm.weight`. The exact metadata is in `CALIBRATION.json`.

	Do not apply `T=1.18` a second time. The front end reads at temperature 1 after the folded readout temperature.

	## Software

	The included front end consists of:

	- `knowline_server.py`: `/v1/systemone` HTTP server;
	- `knowline_engine.py`: backend and label-probability readout;
	- `knowline_layout.py`: prompt layout and read-twice logic;
	- `serve_knowline.sh`: SGLang launch helper.

	The earlier internal candidate evaluation used FP8-at-load SGLang. The release launcher uses bf16 SGLang 0.5.21 with Qwen3.5's chat template, thinking disabled, 16 scoring workers, and read-twice enabled for short text-only requests. The full public DI run below completed on 2026-10-25 using this release package.

	## Read twice

	For a text-only request whose one-pass length is at most 3,000 tokens, the front end writes the user input a second time before the answer position. The second copy starts with:

	```text
	Read the same input again before answering:
	```

	Requests above the threshold and requests with images are read once. Disable this behavior with the front end's `--no-read-twice` option when comparing serving formats.

	## Launch

	```bash
	bash serve_knowline.sh /path/to/KnowLine-9B-Gen1 0 8080
	```

	Equivalent shape:

	```bash
	CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server \
	--model-path /path/to/KnowLine-9B-Gen1 \
	--served-model-name m \
	--tp 1 --mem-fraction-static 0.85 --enable-fp32-lm-head \
	--mamba-radix-cache-strategy extra_buffer \
	--port 9080

	python knowline_server.py \
	--model /path/to/KnowLine-9B-Gen1 \
	--backend sglang \
	--url http://127.0.0.1:9080 \
	--served-model-name m \
	--temperature 1 \
	--workers 16 \
	--port 8080
	```

	## Decision Index result

	The official Decision Index 0.3 scorer reports 65.67 public index / 73.88 raw index. All 140,620 requests were sent fresh; 140,178 are scoreable, and 442 are excluded by the official suite. Final scoreable counts: 140,178 ok, zero errors and zero unsupported; no truncation.

	Two identical H20 servers processed 16 round-robin shards (eight per server). The initial pass had two HTTP504 transport errors, one SATA-Bench and one RouterBench request. Only those failed requests were retried, using the official runner and identical payload hashes, weights and serving. The original first-pass file and retry provenance were retained. The other 140,618 successful request responses were not changed or selected by correctness.

	A separate 300-request serial pass compared 477 answers with this model's own final run: 476 agreed. The only disagreement had new/old margins 0.006/0.021, both below 0.06. All disagreements therefore qualify as near-ties; agreement after excluding near-ties is 100%. Single-request H20 latency: median 96.7 ms, mean 207.7 ms, p80 261.9 ms, p95 826.7 ms. Initial parallel-run request times include queueing and should not be substituted for these serial values.

	The earlier 65.68 sample index and 59.55 predicted private Full were internal candidate-selection estimates, not this full public result. Public index and private leaderboard Full are different metrics and must not be ranked against each other.

	## Limitations

	- Images were not used in 9B Gen1 training.
	- No block-bidirectional attention was used.
	- Do not silently truncate long requests.
	- The front end is for structured decisions, not open-ended chat.

	`SHA256SUMS` covers every file in the repository.

	## Serving correction (2026-10-25)

	The launcher now uses Qwen's `locate`/`xyxy` box settings and no longer references the absent Gemma-specific `temperatures.json`. The folded T=1.18 remains the only readout scaling. Choice, yes/no and score requests passed an actual two-server bf16 smoke run before the full DI run was started. Image and box output quality is not established by that text-only smoke.