Spaces:
Paused
Paused
File size: 2,380 Bytes
ec7c073 e7a144d 191b101 ec7c073 191b101 ec7c073 e7a144d c5c5d80 e7a144d c5c5d80 e7a144d c5c5d80 e7a144d c5c5d80 e7a144d c5c5d80 e7a144d 21cc5dd e7a144d 191b101 e7a144d 191b101 e7a144d 191b101 e7a144d 191b101 3675bfd 191b101 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 | ---
title: Agent Inference — llama-swap Dual Specialist (CPU)
emoji: 🔀
colorFrom: purple
colorTo: blue
sdk: docker
pinned: false
license: mit
---
# Agent Inference: llama-swap Dual-Specialist Router
Two 1.2B specialists behind one OpenAI-compatible endpoint. **llama-swap**
loads the right model for each request based on its `model` field — decision
work goes to the Thinking model, execution goes to the Tool model.
| `model` value | Serves | Use for |
|---|---|---|
| `lfm-thinking` (aliases: `thinking`, `decision`) | LFM2.5-1.2B-Thinking Q8_0 | reasoning, planning, conditional decisions |
| `lfm-tool` (aliases: `tool`, `agent`) | LFM2-1.2B-Tool Q8_0 | tool/function calling, agent execution steps |
Only one model is resident at a time (2-vCPU tier); a swap is a ~1.3 GB mmap
load (~5–10 s). Both use the shared CPU tuning: 32K context / 2 slots, split
KV cache quant (K=q8_0, V=q4_0), flash attention, `--cache-reuse 256`, `--jinja`.
## Usage
```bash
# Decision request -> Thinking model
curl -X POST https://Leon4gr45-agent-inference.hf.space/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"lfm-thinking","messages":[{"role":"user","content":"Should I ..."}]}'
# Tool-calling request -> Tool model
curl -X POST https://Leon4gr45-agent-inference.hf.space/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"lfm-tool","messages":[...],"tools":[...]}'
```
`GET /health` returns OK; llama-swap's UI at `/` shows model state and logs.
## Client contract
- **`lfm-thinking` ignores your `max_tokens`**: the proxy strips it and the
server enforces a 2048 default — a thinking model on a small budget dies
mid-`<think>` before answering (measured at 512).
- **Strip `<think>…</think>` client-side** for `lfm-thinking`: llama.cpp's
reasoning parser doesn't recognize the LFM template, so reasoning arrives
inside `content`. One regex: `re.sub(r"<think>.*?</think>", "", content, flags=re.DOTALL)`.
- **Routing is the caller's job**: llama-swap only loads whatever the `model`
field names. Recommended policy — default to `lfm-tool` (fast); use
`lfm-thinking` for conditional/multi-step decisions, or as escalation when
a `lfm-tool` response fails verification.
- Only one model is resident at a time; batch same-model requests to avoid
swap thrash under concurrency.
## License
MIT
|