README.md

---
title: Ornith 1.0 Inference API (CPU Agentic)
emoji: 🦅
colorFrom: green
colorTo: blue
sdk: docker
pinned: false
license: mit
---

# Qwythos 9B Claude Mythos 5 1M MTP Inference API — CPU Agentic (v2.0.0)

CPU-only LLM inference for **agentic / tool-calling** workloads. A thin FastAPI
reverse proxy in front of a single, fully-optimized `llama.cpp` server that speaks
the **native OpenAI API** — including function/tool calling, streaming, and the
model's real chat template.

**Model**: Qwythos 9B Claude Mythos 5 1M MTP (Q6_K GGUF) — a reasoning model (`<think>`).

## What changed from v1.0 (and why)

v1.0 ran **two** llama.cpp instances and hand-rolled the prompt. On a shared CPU
that was a net loss: 2 instances × 4 threads oversubscribed the cores (per-token
speed collapsed ~4–5× under load) while doubling RAM, and the hand-built
`System:/User:/Assistant:` prompt bypassed the chat template (looping output, no
tool calling).

v2.0:
- **One instance, all cores**, concurrency via `llama.cpp` continuous-batching
  slots (`--parallel`). No thread oversubscription, one model in RAM.
- **Native OpenAI passthrough** — `tools`, `tool_choice`, `response_format`,
  streaming and the model's jinja chat template all handled by llama.cpp
  (grammar-constrained tool calls instead of brittle string parsing).
- **CPU tuning**: flash-attention, q4_0 K/V cache, `--threads-batch`, native SIMD.
- Pooled HTTP client; no per-request health pre-flight; real SSE streaming.

## Performance (honest, CPU basic tier ≈ 2 vCPU)

CPU generation of a 9B model is memory-bandwidth bound. Expect **~1–2 tok/s**
single-request on the free 2-vCPU tier. For snappy agents, use **CPU-Upgrade
(8 vCPU)** or a GPU tier — agent latency multiplies by step count.

## Configuration (env vars)

| Variable | Default | Description |
|----------|---------|-------------|
| `NUM_INSTANCES` | `1` | llama.cpp processes (keep 1 on shared CPU) |
| `CPU_THREADS` | all vCPUs | generation threads |
| `PARALLEL` | `2` | continuous-batching slots (concurrent requests/subagents) |
| `CONTEXT_SIZE` | `32768` | total context (per-slot = ctx / parallel) |
| `KV_CACHE_QUANT_K` | `q8_0` | K cache type — K is more quality-sensitive than V |
| `KV_CACHE_QUANT_V` | `q4_0` | V cache type (needs flash-attn when quantized) |
| `KV_CACHE_QUANT` | – | legacy: overrides both K and V |
| `CACHE_REUSE` | `256` | min tokens for prompt-prefix reuse (0 = off) |
| `FLASH_ATTN` | `true` | flash attention (enables V-cache quant) |
| `MMAP_ENABLED` / `MLOCK_ENABLED` | `true` / `false` | memory mapping / locking |
| `REASONING_FORMAT` | `auto` | how `<think>` reasoning is surfaced |

## API

### Chat completion (OpenAI-compatible)
```bash
curl -X POST https://Leon4gr45-fable5-inference.hf.space/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwythos-9b","messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'
```

### Tool / function calling (native)
```bash
curl -X POST https://Leon4gr45-fable5-inference.hf.space/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model":"qwythos-9b",
    "messages":[{"role":"user","content":"What is the weather in Paris?"}],
    "tools":[{"type":"function","function":{
      "name":"get_weather",
      "description":"Get current weather for a city",
      "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
    "tool_choice":"auto"
  }'
```
The response contains a proper `tool_calls` array with grammar-constrained JSON
arguments — feed the tool result back as a `role:"tool"` message to continue the chain.

### Streaming
Add `"stream": true` to receive SSE token deltas.

### Health / config
```bash
curl https://Leon4gr45-fable5-inference.hf.space/health
curl https://Leon4gr45-fable5-inference.hf.space/v1/config
```

## License
MIT