README.md

metadata

title: Agent Inference — llama-swap Dual Specialist (CPU)
emoji: 🔀
colorFrom: purple
colorTo: blue
sdk: docker
pinned: false
license: mit

Agent Inference: llama-swap Dual-Specialist Router

Two 1.2B specialists behind one OpenAI-compatible endpoint. llama-swap loads the right model for each request based on its model field — decision work goes to the Thinking model, execution goes to the Tool model.

model value	Serves	Use for
lfm-thinking (aliases: thinking, decision)	LFM2.5-1.2B-Thinking Q8_0	reasoning, planning, conditional decisions
lfm-tool (aliases: tool, agent)	LFM2-1.2B-Tool Q8_0	tool/function calling, agent execution steps

Only one model is resident at a time (2-vCPU tier); a swap is a 1.3 GB mmap load (5–10 s). Both use the shared CPU tuning: 32K context / 2 slots, split KV cache quant (K=q8_0, V=q4_0), flash attention, --cache-reuse 256, --jinja.

Usage

# Decision request -> Thinking model
curl -X POST https://Leon4gr45-agent-inference.hf.space/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"lfm-thinking","messages":[{"role":"user","content":"Should I ..."}]}'

# Tool-calling request -> Tool model
curl -X POST https://Leon4gr45-agent-inference.hf.space/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"lfm-tool","messages":[...],"tools":[...]}'

GET /health returns OK; llama-swap's UI at / shows model state and logs.

Client contract

• lfm-thinking ignores your max_tokens: the proxy strips it and the server enforces a 2048 default — a thinking model on a small budget dies mid-<think> before answering (measured at 512).
• Strip <think>…</think> client-side for lfm-thinking: llama.cpp's reasoning parser doesn't recognize the LFM template, so reasoning arrives inside content. One regex: re.sub(r"<think>.*?</think>", "", content, flags=re.DOTALL).
• Routing is the caller's job: llama-swap only loads whatever the model field names. Recommended policy — default to lfm-tool (fast); use lfm-thinking for conditional/multi-step decisions, or as escalation when a lfm-tool response fails verification.
• Only one model is resident at a time; batch same-model requests to avoid swap thrash under concurrency.

License

MIT