Instructions to use KoarAI/LFM2.5-350M-Thinking with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use KoarAI/LFM2.5-350M-Thinking with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="KoarAI/LFM2.5-350M-Thinking")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("KoarAI/LFM2.5-350M-Thinking")
model = AutoModelForCausalLM.from_pretrained("KoarAI/LFM2.5-350M-Thinking", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use KoarAI/LFM2.5-350M-Thinking with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "KoarAI/LFM2.5-350M-Thinking"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KoarAI/LFM2.5-350M-Thinking",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/KoarAI/LFM2.5-350M-Thinking

SGLang

How to use KoarAI/LFM2.5-350M-Thinking with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "KoarAI/LFM2.5-350M-Thinking" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KoarAI/LFM2.5-350M-Thinking",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "KoarAI/LFM2.5-350M-Thinking" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KoarAI/LFM2.5-350M-Thinking",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use KoarAI/LFM2.5-350M-Thinking with Docker Model Runner:
```
docker model run hf.co/KoarAI/LFM2.5-350M-Thinking
```

KoarAI commited on about 21 hours ago

Commit

dbeb0b4

verified ·

1 Parent(s): 0a0e4be

Upload chat_template.jinja with huggingface_hub

Browse files

Files changed (1) hide show

chat_template.jinja +122 -125

chat_template.jinja CHANGED Viewed

@@ -1,125 +1,122 @@
-{{- bos_token -}}
-{%- set preserve_thinking = preserve_thinking | default(keep_past_thinking | default(false)) -%}
-{%- macro format_arg_value(arg_value) -%}
-    {%- if arg_value is string -%}
-        {{- "'" + (arg_value | replace("\\", "\\\\") | replace("'", "\\'") | replace("\n", "\\n") | replace("\r", "\\r")) + "'" -}}
-    {%- elif arg_value is mapping or arg_value is iterable -%}
-        {{- arg_value | tojson -}}
-    {%- else -%}
-        {{- arg_value | string -}}
-    {%- endif -%}
-{%- endmacro -%}
-{%- macro parse_content(content) -%}
-    {%- if content is string -%}
-        {{- content -}}
-    {%- elif content is mapping -%}
-        {{- content | tojson -}}
-    {%- elif content is iterable -%}
-        {%- set _ns = namespace(result="") -%}
-        {%- for item in content -%}
-            {%- if item is string -%}
-                {%- set _ns.result = _ns.result + item -%}
-            {%- elif item is mapping and item.get("type") == "image" -%}
-                {%- set _ns.result = _ns.result + "<image>" -%}
-            {%- elif item is mapping and item.get("type") == "text" -%}
-                {%- set _ns.result = _ns.result + ((item.get("text") or "") | string) -%}
-            {%- else -%}
-                {%- set _ns.result = _ns.result + (item | tojson) -%}
-            {%- endif -%}
-        {%- endfor -%}
-        {{- _ns.result -}}
-    {%- endif -%}
-{%- endmacro -%}
-{%- macro render_tool_calls(tool_calls) -%}
-    {%- set tool_calls_ns = namespace(tool_calls=[]) -%}
-    {%- for tool_call in tool_calls -%}
-        {%- set func = tool_call["function"] if "function" in tool_call else tool_call -%}
-        {%- set func_name = func["name"] -%}
-        {%- set func_args = func.get("arguments") -%}
-        {%- set args_ns = namespace(arg_strings=[]) -%}
-        {%- if func_args is mapping -%}
-            {%- for arg_name, arg_value in func_args.items() -%}
-                {%- set args_ns.arg_strings = args_ns.arg_strings + [arg_name + "=" + format_arg_value(arg_value)] -%}
-            {%- endfor -%}
-        {%- elif func_args is string and (func_args | trim) not in ["", "{}", "null"] -%}
-            {{- raise_exception("Tool call arguments must be a mapping, got a JSON-encoded string: parse arguments with json.loads() before applying the chat template") -}}
-        {%- endif -%}
-        {%- set tool_calls_ns.tool_calls = tool_calls_ns.tool_calls + [func_name + "(" + (args_ns.arg_strings | join(", ")) + ")"] -%}
-    {%- endfor -%}
-    {{- "<|tool_call_start|>[" + (tool_calls_ns.tool_calls | join(", ")) + "]<|tool_call_end|>" -}}
-{%- endmacro -%}
-{%- set ns = namespace(system_prompt="", last_assistant_index=-1) -%}
-{%- if messages and messages[0]["role"] == "system" -%}
-    {%- if messages[0].get("content") -%}
-        {%- set ns.system_prompt = parse_content(messages[0]["content"]) -%}
-    {%- endif -%}
-    {%- set messages = messages[1:] -%}
-{%- endif -%}
-{%- if tools -%}
-    {%- set ns.system_prompt = ns.system_prompt + ("\n" if ns.system_prompt else "") + "List of tools: [" -%}
-    {%- for tool in tools -%}
-        {%- if tool is not string -%}
-            {%- set tool = tool | tojson -%}
-        {%- endif -%}
-        {%- set ns.system_prompt = ns.system_prompt + tool -%}
-        {%- if not loop.last -%}
-            {%- set ns.system_prompt = ns.system_prompt + ", " -%}
-        {%- endif -%}
-    {%- endfor -%}
-    {%- set ns.system_prompt = ns.system_prompt + "]" -%}
-{%- endif -%}
-{%- if ns.system_prompt -%}
-    {{- "<|im_start|>system\n" + ns.system_prompt + "<|im_end|>\n" -}}
-{%- endif -%}
-{%- for message in messages -%}
-    {%- if message["role"] == "assistant" -%}
-        {%- set ns.last_assistant_index = loop.index0 -%}
-    {%- endif -%}
-{%- endfor -%}
-{%- for message in messages -%}
-    {{- "<|im_start|>" + message.role + "\n" -}}
-    {%- if message.role == "assistant" -%}
-        {%- generation -%}
-        {%- set keep_thinking = preserve_thinking or loop.index0 == ns.last_assistant_index -%}
-        {%- set thinking = message.thinking or message.reasoning or message.reasoning_content -%}
-        {%- set thinking = thinking if thinking is string else "" -%}
-        {%- if thinking and keep_thinking -%}
-            {{- "<think>" + thinking + "</think>" -}}
-        {%- endif -%}
-        {%- set _cfm_tag = "CONTINUE_FINAL_MESSAGE_TAG " -%}
-        {%- set _has_cfm = false -%}
-        {%- set content = "" -%}
-        {%- if message.get("content") -%}
-            {%- set content = parse_content(message.content) -%}
-        {%- endif -%}
-        {%- if not keep_thinking and "</think>" in content -%}
-            {%- set content = content.split("</think>")[-1] | trim -%}
-        {%- endif -%}
-        {%- if content.endswith(_cfm_tag) -%}
-            {%- set _has_cfm = true -%}
-            {%- set _trunc_len = (content | length) - (_cfm_tag | length) -%}
-            {%- set content = content[:_trunc_len] -%}
-        {%- endif -%}
-        {{- content -}}
-        {%- if message.tool_calls -%}
-            {{- render_tool_calls(message.tool_calls) -}}
-        {%- endif -%}
-        {%- if _has_cfm -%}
-            {{- _cfm_tag -}}
-        {%- endif -%}
-        {{- "<|im_end|>\n" -}}
-        {%- endgeneration -%}
-    {%- else %}
-        {%- if message.get("content") -%}
-            {{- parse_content(message["content"]) -}}
-        {%- endif -%}
-        {{- "<|im_end|>\n" -}}
-    {%- endif %}
-{%- endfor -%}
-{%- if add_generation_prompt -%}
-    {{- "<|im_start|>assistant\n" -}}
-{%- endif -%}

+{{- bos_token -}}
+{%- set preserve_thinking = preserve_thinking | default(keep_past_thinking | default(false)) -%}
+{%- macro format_arg_value(arg_value) -%}
+	{%- if arg_value is string -%}
+		{{- "'" + arg_value | replace("\\", "\\\\") | replace("'", "\\'") | replace("\n", "\\n") | replace("\r", "\\r") + "'" -}}
+	{%- elif arg_value is mapping or arg_value is iterable -%}
+		{{- arg_value | tojson -}}
+	{%- else -%}
+		{{- arg_value | string -}}
+	{%- endif -%}
+{%- endmacro -%}
+{%- macro parse_content(content) -%}
+	{%- if content is string -%}
+		{{- content -}}
+	{%- elif content is mapping -%}
+		{{- content | tojson -}}
+	{%- elif content is iterable -%}
+		{%- set _ns = namespace(result="") -%}
+		{%- for item in content -%}
+			{%- if item is string -%}
+				{%- set _ns.result = _ns.result + item -%}
+			{%- elif item is mapping and item.get("type") == "image" -%}
+				{%- set _ns.result = _ns.result + "<image>" -%}
+			{%- elif item is mapping and item.get("type") == "text" -%}
+				{%- set _ns.result = _ns.result + (item.get("text") or "") | string -%}
+			{%- else -%}
+				{%- set _ns.result = _ns.result + item | tojson -%}
+			{%- endif -%}
+		{%- endfor -%}
+		{{- _ns.result -}}
+	{%- endif -%}
+{%- endmacro -%}
+{%- macro render_tool_calls(tool_calls) -%}
+	{%- set tool_calls_ns = namespace(tool_calls=[]) -%}
+	{%- for tool_call in tool_calls -%}
+		{%- set func = tool_call["function"] if "function" in tool_call else tool_call -%}
+		{%- set func_name = func["name"] -%}
+		{%- set func_args = func.get("arguments") -%}
+		{%- set args_ns = namespace(arg_strings=[]) -%}
+		{%- if func_args is mapping -%}
+			{%- for (arg_name, arg_value) in func_args.items() -%}
+				{%- set args_ns.arg_strings = args_ns.arg_strings + [arg_name + "=" + format_arg_value(arg_value)] -%}
+			{%- endfor -%}
+		{%- elif func_args is string and func_args | trim not in ["", "{}", "null"] -%}
+			{{- raise_exception("Tool call arguments must be a mapping, got a JSON-encoded string: parse arguments with json.loads() before applying the chat template") -}}
+		{%- endif -%}
+		{%- set tool_calls_ns.tool_calls = tool_calls_ns.tool_calls + [func_name + "(" + args_ns.arg_strings | join(", ") + ")"] -%}
+	{%- endfor -%}
+	{{- "<|tool_call_start|>[" + tool_calls_ns.tool_calls | join(", ") + "]<|tool_call_end|>" -}}
+{%- endmacro -%}
+{%- set ns = namespace(system_prompt="", last_assistant_index=-1) -%}
+{%- if messages and messages[0]["role"] == "system" -%}
+	{%- if messages[0].get("content") -%}
+		{%- set ns.system_prompt = parse_content(messages[0]["content"]) -%}
+	{%- endif -%}
+	{%- set messages = messages[1:] -%}
+{%- endif -%}
+{%- if tools -%}
+	{%- set ns.system_prompt = ns.system_prompt + ("\n" if ns.system_prompt else "") + "List of tools: [" -%}
+	{%- for tool in tools -%}
+		{%- if tool is not string -%}
+			{%- set tool = tool | tojson -%}
+		{%- endif -%}
+		{%- set ns.system_prompt = ns.system_prompt + tool -%}
+		{%- if not loop.last -%}
+			{%- set ns.system_prompt = ns.system_prompt + ", " -%}
+		{%- endif -%}
+	{%- endfor -%}
+	{%- set ns.system_prompt = ns.system_prompt + "]" -%}
+{%- endif -%}
+{%- if ns.system_prompt -%}
+	{{- "<|im_start|>system\n" + ns.system_prompt + "<|im_end|>\n" -}}
+{%- endif -%}
+{%- for message in messages -%}
+	{%- if message["role"] == "assistant" -%}
+		{%- set ns.last_assistant_index = loop.index0 -%}
+	{%- endif -%}
+{%- endfor -%}
+{%- for message in messages -%}
+	{{- "<|im_start|>" + message.role + "\n" -}}
+	{%- if message.role == "assistant" -%}
+		{%- set keep_thinking = preserve_thinking or loop.index0 == ns.last_assistant_index -%}
+		{%- set thinking = message.thinking or message.reasoning or message.reasoning_content -%}
+		{%- set thinking = thinking if thinking is string else "" -%}
+		{%- if thinking and keep_thinking -%}
+			{{- "<think>" + thinking + "</think>" -}}
+		{%- endif -%}
+		{%- set _cfm_tag = "CONTINUE_FINAL_MESSAGE_TAG " -%}
+		{%- set _has_cfm = false -%}
+		{%- set content = "" -%}
+		{%- if message.get("content") -%}
+			{%- set content = parse_content(message.content) -%}
+		{%- endif -%}
+		{%- if not keep_thinking and "</think>" in content -%}
+			{%- set content = content.split("</think>")[-1] | trim -%}
+		{%- endif -%}
+		{%- if content.endswith(_cfm_tag) -%}
+			{%- set _has_cfm = true -%}
+			{%- set _trunc_len = content | length - _cfm_tag | length -%}
+			{%- set content = content[:_trunc_len] -%}
+		{%- endif -%}
+		{{- content -}}
+		{%- if message.tool_calls -%}
+			{{- render_tool_calls(message.tool_calls) -}}
+		{%- endif -%}
+		{%- if _has_cfm -%}
+			{{- _cfm_tag -}}
+		{%- endif -%}
+		{{- "<|im_end|>\n" -}}
+	{%- else -%}
+		{%- if message.get("content") -%}
+			{{- parse_content(message["content"]) -}}
+		{%- endif -%}
+		{{- "<|im_end|>\n" -}}
+	{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+	{{- "<|im_start|>assistant\n" -}}
+	{%- if enable_thinking is not defined or enable_thinking -%}
+		{{- "<think>\n" -}}
+	{%- endif -%}
+{%- endif -%}