README.md

---
title: Agent Inference — llama-swap Dual Specialist (CPU)
emoji: 🔀
colorFrom: purple
colorTo: blue
sdk: docker
pinned: false
license: mit
---

# Agent Inference: llama-swap Dual-Specialist Router

Two 1.2B specialists behind one OpenAI-compatible endpoint. **llama-swap**
loads the right model for each request based on its `model` field — decision
work goes to the Thinking model, execution goes to the Tool model.

| `model` value | Serves | Use for |
|---|---|---|
| `lfm-thinking` (aliases: `thinking`, `decision`) | LFM2.5-1.2B-Thinking Q8_0 | reasoning, planning, conditional decisions |
| `lfm-tool` (aliases: `tool`, `agent`) | LFM2-1.2B-Tool Q8_0 | tool/function calling, agent execution steps |

Only one model is resident at a time (2-vCPU tier); a swap is a ~1.3 GB mmap
load (~5–10 s). Both use the shared CPU tuning: 32K context / 2 slots, split
KV cache quant (K=q8_0, V=q4_0), flash attention, `--cache-reuse 256`, `--jinja`.

## Usage

```bash
# Decision request -> Thinking model
curl -X POST https://Leon4gr45-agent-inference.hf.space/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"lfm-thinking","messages":[{"role":"user","content":"Should I ..."}]}'

# Tool-calling request -> Tool model
curl -X POST https://Leon4gr45-agent-inference.hf.space/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"lfm-tool","messages":[...],"tools":[...]}'
```

`GET /health` returns OK; llama-swap's UI at `/` shows model state and logs.

## Client contract

- **`lfm-thinking` ignores your `max_tokens`**: the proxy strips it and the
  server enforces a 2048 default — a thinking model on a small budget dies
  mid-`<think>` before answering (measured at 512).
- **Strip `<think>…</think>` client-side** for `lfm-thinking`: llama.cpp's
  reasoning parser doesn't recognize the LFM template, so reasoning arrives
  inside `content`. One regex: `re.sub(r"<think>.*?</think>", "", content, flags=re.DOTALL)`.
- **Routing is the caller's job**: llama-swap only loads whatever the `model`
  field names. Recommended policy — default to `lfm-tool` (fast); use
  `lfm-thinking` for conditional/multi-step decisions, or as escalation when
  a `lfm-tool` response fails verification.
- Only one model is resident at a time; batch same-model requests to avoid
  swap thrash under concurrency.

## License
MIT