Instructions to use PelaAI/KnowLine-9B-Gen1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use PelaAI/KnowLine-9B-Gen1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="PelaAI/KnowLine-9B-Gen1")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("PelaAI/KnowLine-9B-Gen1")
model = AutoModelForMultimodalLM.from_pretrained("PelaAI/KnowLine-9B-Gen1", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use PelaAI/KnowLine-9B-Gen1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "PelaAI/KnowLine-9B-Gen1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PelaAI/KnowLine-9B-Gen1",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/PelaAI/KnowLine-9B-Gen1

SGLang

How to use PelaAI/KnowLine-9B-Gen1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "PelaAI/KnowLine-9B-Gen1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PelaAI/KnowLine-9B-Gen1",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "PelaAI/KnowLine-9B-Gen1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PelaAI/KnowLine-9B-Gen1",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use PelaAI/KnowLine-9B-Gen1 with Docker Model Runner:
```
docker model run hf.co/PelaAI/KnowLine-9B-Gen1
```

KnowLine-9B-Gen1 / serve_knowline.sh

PEScn

Fix Qwen 9B launcher and document verified bf16 serving

d8f7d9a verified 10 days ago

Raw History Blame Contribute Delete

1.58 kB

#!/bin/bash
# Serve KnowLine-9B-Gen1 behind /v1/systemone (see INFERENCE.md).
#   bash serve_knowline.sh <model dir or HF repo id> [gpu=0] [port=8080]
# Starts SGLang in bf16 (port = front-end port + 1000), then knowline_server.py.
# Read twice is enabled by default; temperature 1 preserves the folded T=1.18.
# No Gemma-specific temperatures file is used. Untrained boxes use Qwen's xyxy format.
# MEM (default 0.85) is SGLang's --mem-fraction-static: it only sizes the KV cache, scores do not depend on it.
# HOST (default 127.0.0.1) is the front end's bind address.
# Without SGLang: python knowline_server.py --model <model> --backend hf --port 8080 (transformers, slower).
set -euo pipefail
MODEL=${1:?usage: serve_knowline.sh <model dir or HF repo id> [gpu] [port]}
GPU=${2:-0}; PORT=${3:-8080}; SGL_PORT=$((PORT + 1000)); MEM=${MEM:-0.85}; HOST=${HOST:-127.0.0.1}
HERE=$(cd "$(dirname "$0")" && pwd)
CUDA_VISIBLE_DEVICES=$GPU python -m sglang.launch_server --model-path "$MODEL" --served-model-name m --tp 1 \
  --mem-fraction-static "$MEM" --enable-fp32-lm-head --mamba-radix-cache-strategy extra_buffer \
  --port "$SGL_PORT" &
SGL=$!
trap 'kill "$SGL" 2>/dev/null || true' EXIT
until curl -sf -m 3 "http://127.0.0.1:$SGL_PORT/health" > /dev/null; do
  sleep 5; kill -0 "$SGL" 2>/dev/null || { echo "SGLang exited" >&2; exit 1; }
done
python "$HERE/knowline_server.py" --model "$MODEL" --backend sglang --url "http://127.0.0.1:$SGL_PORT" \
  --served-model-name m --temperature 1 --workers 16 --host "$HOST" --port "$PORT" \
  --box-prompt locate --box-coords xyxy

	#!/bin/bash
	# Serve KnowLine-9B-Gen1 behind /v1/systemone (see INFERENCE.md).
	# bash serve_knowline.sh <model dir or HF repo id> [gpu=0] [port=8080]
	# Starts SGLang in bf16 (port = front-end port + 1000), then knowline_server.py.
	# Read twice is enabled by default; temperature 1 preserves the folded T=1.18.
	# No Gemma-specific temperatures file is used. Untrained boxes use Qwen's xyxy format.
	# MEM (default 0.85) is SGLang's --mem-fraction-static: it only sizes the KV cache, scores do not depend on it.
	# HOST (default 127.0.0.1) is the front end's bind address.
	# Without SGLang: python knowline_server.py --model <model> --backend hf --port 8080 (transformers, slower).
	set -euo pipefail
	MODEL=${1:?usage: serve_knowline.sh <model dir or HF repo id> [gpu] [port]}
	GPU=${2:-0}; PORT=${3:-8080}; SGL_PORT=$((PORT + 1000)); MEM=${MEM:-0.85}; HOST=${HOST:-127.0.0.1}
	HERE=$(cd "$(dirname "$0")" && pwd)
	CUDA_VISIBLE_DEVICES=$GPU python -m sglang.launch_server --model-path "$MODEL" --served-model-name m --tp 1 \
	--mem-fraction-static "$MEM" --enable-fp32-lm-head --mamba-radix-cache-strategy extra_buffer \
	--port "$SGL_PORT" &
	SGL=$!
	trap 'kill "$SGL" 2>/dev/null \|\| true' EXIT
	until curl -sf -m 3 "http://127.0.0.1:$SGL_PORT/health" > /dev/null; do
	sleep 5; kill -0 "$SGL" 2>/dev/null \|\| { echo "SGLang exited" >&2; exit 1; }
	done
	python "$HERE/knowline_server.py" --model "$MODEL" --backend sglang --url "http://127.0.0.1:$SGL_PORT" \
	--served-model-name m --temperature 1 --workers 16 --host "$HOST" --port "$PORT" \
	--box-prompt locate --box-coords xyxy