Instructions to use LiquidAI/LFM2.5-8B-A1B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LiquidAI/LFM2.5-8B-A1B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="LiquidAI/LFM2.5-8B-A1B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("LiquidAI/LFM2.5-8B-A1B")
model = AutoModelForCausalLM.from_pretrained("LiquidAI/LFM2.5-8B-A1B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use LiquidAI/LFM2.5-8B-A1B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LiquidAI/LFM2.5-8B-A1B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LiquidAI/LFM2.5-8B-A1B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/LiquidAI/LFM2.5-8B-A1B

SGLang

How to use LiquidAI/LFM2.5-8B-A1B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LiquidAI/LFM2.5-8B-A1B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LiquidAI/LFM2.5-8B-A1B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LiquidAI/LFM2.5-8B-A1B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LiquidAI/LFM2.5-8B-A1B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use LiquidAI/LFM2.5-8B-A1B with Docker Model Runner:
```
docker model run hf.co/LiquidAI/LFM2.5-8B-A1B
```

Potential cause for tool calling failures

#22

by liebedan - opened Jun 4

Discussion

liebedan

Jun 4

Hi,
I'm running a GGUF version of this model with llama.cpp and did some tests with VS Code and the GitHub Copilot Chat talking to this instance.
Like many other people on different sites I have huge issues with tool calls. But I might have found the root cause.

A quite simple prompt with a question about the code base triggers the VSC tool "grep_search". The thinking points that it wants to use it. And I see in llama.cpp logs that it actually generates the tool call, but in the VS Code Chat it just stops.

This is the generated tool call according to llama.cpp verbose logs:
Parsed message: {"role":"assistant","content":"\n","tool_calls":[{"type":"function","function":{"name":"grep_search","arguments":"{\"query\":\"garbage_schedule\",\"isRegexp\":True,\"includePattern\":\"*.py\"}"}}]}

Look at it closely! It puts True to the isRegexp argument. This is not valid JSON! (RFC8256) It is python-esque. JSON would be lowercase true. I suspect that VS Code silently fails the parsing and just stops. I had instances where the model left out this very argument, VS Code executed the tool call but it failed for the missing argument. In these cases the model retried with the argument and the same "silent error" happened.

According to the very detailed llama.cpp verbose looks there is no doubt the model generates this mistake:
slot process_toke: id 0 | task 4277 | n_decoded = 16, n_remaining = 772, next token: 27438 '=True'

It also makes sense to me that the developers might not run into these problems, when their harness simply tolerates "True" and "False" in JSON. Maybe others can try to validate this.

Any feedback would be appreciated!

Best Regards
Daniel

tarek-liquid

Liquid AI org Jun 5

•

edited Jun 5

Thanks for reporting, @liebedan . You are on point. The fix https://github.com/ggml-org/llama.cpp/pull/24178 for llama.cpp is ready, and I hope it lands in upstream soon.

liebedan

Jun 5

Thank you, @tarek-liquid ! That's good news. I feared such an issue could only be solved with fine-tuning. I'm looking forward to trying the merged changes to the tool parser.

tarek-liquid

Liquid AI org Jun 5

The PR has been merged, and the issue is now resolved.

tarek-liquid changed discussion status to closed Jun 5

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment