Instructions to use ServiceNow-AI/Apriel-1.6-15b-Thinker with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ServiceNow-AI/Apriel-1.6-15b-Thinker with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ServiceNow-AI/Apriel-1.6-15b-Thinker")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ServiceNow-AI/Apriel-1.6-15b-Thinker")
model = AutoModelForMultimodalLM.from_pretrained("ServiceNow-AI/Apriel-1.6-15b-Thinker")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ServiceNow-AI/Apriel-1.6-15b-Thinker with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ServiceNow-AI/Apriel-1.6-15b-Thinker"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ServiceNow-AI/Apriel-1.6-15b-Thinker",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ServiceNow-AI/Apriel-1.6-15b-Thinker

SGLang

How to use ServiceNow-AI/Apriel-1.6-15b-Thinker with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ServiceNow-AI/Apriel-1.6-15b-Thinker" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ServiceNow-AI/Apriel-1.6-15b-Thinker",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ServiceNow-AI/Apriel-1.6-15b-Thinker" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ServiceNow-AI/Apriel-1.6-15b-Thinker",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ServiceNow-AI/Apriel-1.6-15b-Thinker with Docker Model Runner:
```
docker model run hf.co/ServiceNow-AI/Apriel-1.6-15b-Thinker
```

Working Ollama template

#16

by MobbyButcher - opened 1 day ago

Discussion

MobbyButcher

1 day ago

After trying for hours, I've managed to make the model with this template that has been taken from Ollama repository's GGUF file:

Working template

{{- $reasoningPrompt := "You are a thoughtful, systematic AI assistant from ServiceNow Language Models (SLAM) lab. Analyze each question carefully, present your reasoning step-by-step, then provide the final response after the marker [BEGIN FINAL RESPONSE]." -}}

<|begin_system|>
{{ $reasoningPrompt }}
{{- if .System }}
{{ .System }}
{{- end }}
{{- if .Tools }}

You are provided with function signatures within <available_tools></available_tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about the arguments. You should infer the argument values from previous user responses and the system message. Here are the available tools:
<available_tools>
{{ .Tools }}
</available_tools>

Return all function calls as a list of JSON objects within <tool_calls></tool_calls> XML tags. Each JSON object should contain a function name and arguments as follows:
<tool_calls>[
    {"name": <function-name-1>, "arguments": <args-dict-1>}, 
    {"name": <function-name-2>, "arguments": <args-dict-2>},
    ...
]</tool_calls>
{{- end }}

{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 -}}

{{- if eq .Role "user" }}<|begin_user|>
{{ .Content }}{{ range .Images }}[IMG]{{ end }}
{{- end }}

{{- if eq .Role "assistant" }}

<|begin_assistant|>
{{- if .Content }}
{{ .Content }}
{{- end }}
{{- if .ToolCalls }}

<tool_calls>[{{ range $j, $tc := .ToolCalls }}{{ if $j }}, {{ end }}{"name": "{{ $tc.Function.Name }}", "arguments": {{ $tc.Function.Arguments }}{{ if $tc.ID }}, "id": "{{ $tc.ID }}"{{ end }}}{{ end }}]</tool_calls>
{{- end }}
{{- if not $last }}
<|end|>
{{ end }}
{{- end }}

{{- if false }}Here are my reasoning steps:
{{ .Thinking }}
[BEGIN FINAL RESPONSE]
{{ end }}

{{- if eq .Role "tool" }}
<|begin_tool_result|>
{{ .Content }}
{{ end }}

{{- if eq .Role "content" }}<|begin_content|>
{{ .Content }}
{{ end }}

{{- if and (ne .Role "assistant") $last }}

<|begin_assistant|>
Here are my reasoning steps:
{{ end }}

{{- end -}}

Usage:

Download the GGUF you want to use, make sure it's somewhere you can find it.
Create a new file (Modelfile, but you can use any name) with the following contents:

Modelfile

FROM ./model.gguf
TEMPLATE """{{- $reasoningPrompt := "You are a thoughtful, systematic AI assistant from ServiceNow Language Models (SLAM) lab. Analyze each question carefully, present your reasoning step-by-step, then provide the final response after the marker [BEGIN FINAL RESPONSE]." -}}

<|begin_system|>
{{ $reasoningPrompt }}
{{- if .System }}
{{ .System }}
{{- end }}
{{- if .Tools }}

You are provided with function signatures within <available_tools></available_tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about the arguments. You should infer the argument values from previous user responses and the system message. Here are the available tools:
<available_tools>
{{ .Tools }}
</available_tools>

Return all function calls as a list of JSON objects within <tool_calls></tool_calls> XML tags. Each JSON object should contain a function name and arguments as follows:
<tool_calls>[
    {"name": <function-name-1>, "arguments": <args-dict-1>}, 
    {"name": <function-name-2>, "arguments": <args-dict-2>},
    ...
]</tool_calls>
{{- end }}

{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 -}}

{{- if eq .Role "user" }}<|begin_user|>
{{ .Content }}{{ range .Images }}[IMG]{{ end }}
{{- end }}

{{- if eq .Role "assistant" }}

<|begin_assistant|>
{{- if .Content }}
{{ .Content }}
{{- end }}
{{- if .ToolCalls }}

<tool_calls>[{{ range $j, $tc := .ToolCalls }}{{ if $j }}, {{ end }}{"name": "{{ $tc.Function.Name }}", "arguments": {{ $tc.Function.Arguments }}{{ if $tc.ID }}, "id": "{{ $tc.ID }}"{{ end }}}{{ end }}]</tool_calls>
{{- end }}
{{- if not $last }}
<|end|>
{{ end }}
{{- end }}

{{- if false }}Here are my reasoning steps:
{{ .Thinking }}
[BEGIN FINAL RESPONSE]
{{ end }}

{{- if eq .Role "tool" }}
<|begin_tool_result|>
{{ .Content }}
{{ end }}

{{- if eq .Role "content" }}<|begin_content|>
{{ .Content }}
{{ end }}

{{- if and (ne .Role "assistant") $last }}

<|begin_assistant|>
Here are my reasoning steps:
{{ end }}

{{- end -}}"""

Run ollama create Apriel-1.6-15B:Q4_K_M -f ./Modelfile

This should create a new model in Ollama.

Replace model path and name on the first line of your Modelfile (FROM ./model.gguf) to your model filename and path.
Replace model name (Apriel-1.6-15B:Q4_K_M) in ollama create Apriel-1.6-15B:Q4_K_M -f ./Modelfile command to whatever you want. I'd recommend keeping the same quantization tag (Q4_K_M) after the colon (:) as the quantization on your GGUF file, but it's not necessary.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment