feat: Implement llama.cpp backend for local inference with GGUF models

docs/adr/0032-llama-cpp-local-backend.md

@@ -0,0 +1,71 @@
+# ADR-0032: A Third Inference Backend — Local llama.cpp (GGUF)
+
+## Status
+
+Accepted (extends ADR-0024 *second inference backend / unified registry*, amends
+ADR-0015 *LiteLLM gateway*, ADR-0022 *per-agent explicit model binding*)
+
+## Context
+
+The engine had two live backends: vLLM endpoints we deploy on Modal (ADR-0015/0019)
+and Hugging Face's serverless router (ADR-0024). Both run *somewhere else* — Modal needs
+warmed GPUs, HF needs a token and a provider that serves the model. There was no way to
+run a cast **entirely on the operator's own machine**, with no account, no token, and no
+network after the first download.
+
+llama.cpp's `llama-server` is exactly that: it loads a quantized **GGUF** model, runs it
+on whatever hardware is present (Apple Metal, NVIDIA CUDA, or CPU), and exposes an
+**OpenAI-compatible** API on `/v1`. Because it speaks the same REST surface as Modal/HF,
+it slots into the existing LiteLLM gateway with no new transport code — the same seam
+ADR-0024 was designed to leave open ("adding a third backend later touches only
+`inference._BACKENDS`").
+
+This also stacks hackathon lanes from one local server: the **Llama Champion** badge
+(a real llama.cpp runtime in the cast), the **NVIDIA Nemotron Quest** (Nemotron 3 Nano
+4B), and the **OpenBMB** track (MiniCPM 4.1 8B) — plus a JetBrains Mellum 2 thinking
+model on the balanced tier. Every model stays within the ≤32B "small minds" rule, and the
+4B Nemotron honours the ≤4B Tiny-Titan band.
+
+## Decision
+
+**A third backend = one more catalogue + a registry entry.** Add
+`src/models/llamacpp_catalogue.py`, stdlib-only and offline-safe like its siblings, listing
+the GGUF models with both engine-facing fields (`key` / `profile` / `params_b` /
+`served_id`) and serving fields (`hf_repo` / `quant` / `ctx_size` / sampling /
+`flash_attn` / `reasoning`). Its `binding_for()` yields the LiteLLM custom-endpoint form
+`model = openai/<served_id>`, `base_url = $LLAMACPP_BASE_URL` (default
+`http://127.0.0.1:8080/v1`), `api_key = $LLAMACPP_API_KEY` (a placeholder — llama-server
+ignores it). Register it in `inference._BACKENDS` under the prefix `llamacpp`; qualified
+keys are `llamacpp:<slug>` (e.g. `llamacpp:nemotron-3-nano-4b`). Nothing above the registry
+changes — the router, the config loader's `endpoint:` expansion, the live/offline gate, and
+the Lab picker all derive from the façade.
+
+**The serving side is a separate, pure-where-it-matters launcher.** Add
+`src/models/llamacpp_server.py`. `detect_accelerator(platform, probe)` returns
+`metal` on macOS, `cuda` when `nvidia-smi` reports a GPU, else `cpu`;
+`build_command(model, accelerator, …)` assembles the `llama-server` argv — pulling the
+model by its `-hf` spec (downloaded on first run), serving it under `--alias <key>` so the
+running server reports the stable id the engine binds to, and offloading **every layer to
+the GPU** (`-ngl 999`) when one is present, omitting the flag on CPU. Both take their
+environment as arguments so the GPU/CPU branches are testable with no GPU and no binary.
+The `__main__` CLI launches a model by key and prints the matching `LLAMACPP_BASE_URL`
+export.
+
+**Opt-in by base URL, not a token.** A local server needs no auth, so the live/offline
+gate (`has_credentials`) keys on `LLAMACPP_BASE_URL` being *set* — the launcher sets it,
+or you export it to point at an already-running or remote server. With it unset the
+backend never claims to be live, so the deterministic stub still owns the no-config demo.
+
+## Consequences
+
+- A cast can run fully local: `uv run python -m src.models.llamacpp_server
+  nemotron-3-nano-4b`, export the printed URL, and the engine routes to it through the
+  unchanged LiteLLM transport — no account, no token, GPU used automatically when present.
+- Llama Champion + Nemotron + OpenBMB lanes are reachable from one server; the catalogue
+  is plain data, so adding a GGUF or retuning a tier is a one-line `LlamaCppModel(...)` edit.
+- Backward compatible by construction: bare keys still mean Modal, and the `llamacpp`
+  prefix is new — existing config, manifests, and the green test baseline are unaffected.
+- The engine still never names a vendor on a hot path: routing is by qualified key through
+  one façade; the GGUF/quant churn is hidden behind `--alias <key>`.
+- llama-server and the GGUF download are operator-side, never a Python dependency — the
+  offline stub remains the default with no binary, no network, and extras uninstalled.

docs/architecture/model-routing.md

@@ -77,6 +77,38 @@ Profiles map to the OpenAI-compatible vLLM endpoints served on Modal
 The LiteLLM model string for an OpenAI-compatible custom endpoint is
 `openai/<served_model_id>` with `api_base` pointing at the endpoint's `/v1` URL.
 
+### Backends: Modal · Hugging Face · llama.cpp
+
+A *backend* is just a catalogue + a binding rule, unified behind one registry
+(`src/models/inference.py`, ADR-0024). Models are named by a **backend-qualified
+key** `"<backend>:<raw>"`; a bare key means Modal, so all existing config keeps
+working. Three backends ship today:
+
+| Backend | Prefix | Where it runs | Opt-in |
+|---|---|---|---|
+| Modal | *(bare)* | vLLM endpoints you deploy on Modal GPUs | `MODAL_WORKSPACE` / `MODAL_LLM_BASE_URL` |
+| Hugging Face | `hf:` | serverless Inference Providers router | `HF_TOKEN` |
+| llama.cpp | `llamacpp:` | **local** GGUF via `llama-server`, GPU when present | `LLAMACPP_BASE_URL` |
+
+The llama.cpp backend (ADR-0032) runs a cast fully on your own machine. Launch a
+model and export the URL it prints:
+
+```bash
+uv run python -m src.models.llamacpp_server nemotron-3-nano-4b   # GPU auto-detected
+export LLAMACPP_BASE_URL=http://127.0.0.1:8080/v1
+```
+
+The launcher detects an accelerator — Apple Metal on macOS, NVIDIA CUDA via
+`nvidia-smi`, else CPU — and offloads every layer to the GPU (`-ngl 999`) when one
+is present. `llama-server` downloads the GGUF on first run (`-hf`) and serves it
+under `--alias <key>`, so the engine binds to a stable id (`openai/<key>`) through
+the same LiteLLM transport. Bind a tier to a local model with a qualified key:
+
+```yaml
+profiles:
+  tiny: { endpoint: "llamacpp:nemotron-3-nano-4b", temperature: 0.7, max_tokens: 192 }
+```
+
 ### Real cost → Governor
 
 LiteLLM prices each call (`response._hidden_params["response_cost"]`, falling back
@@ -150,6 +182,9 @@ everything on the big model.
 - `src/core/registry.py` — `Registry.from_world()` (a UI/LLM-composed run on the same path)
 - `src/models/litellm_provider.py` — `LiteLLMProvider` (live transport, real cost)
 - `src/models/modal_catalogue.py` — engine view of the catalogue (key → binding)
+- `src/models/inference.py` — unified backend registry (Modal · HF · llama.cpp); qualified keys
+- `src/models/llamacpp_catalogue.py` — local GGUF catalogue (key → binding)
+- `src/models/llamacpp_server.py` — `llama-server` launcher: GPU detection + command building
 - `src/core/manifest.py` — `resolve_model()` (env → catalogue default)
 - `src/core/registry.py` — `build_router()`, `_resolve_model_endpoints()`, `_expand_env()`
 - `src/models/provider.py` — `ModelProvider.last_usage`, `estimate_tokens()`

src/models/inference.py

@@ -24,7 +24,7 @@ from __future__ import annotations
 import os
 from dataclasses import dataclass
 
-from src.models import hf_catalogue, modal_catalogue
+from src.models import hf_catalogue, llamacpp_catalogue, modal_catalogue
 
 # Separator between a backend prefix and the backend-local key. A raw HF repo id can
 # contain ``/`` but never a leading ``<backend>:`` prefix, so a single split is safe.
@@ -55,6 +55,12 @@ _BACKENDS: dict[str, Backend] = {
         blurb="serverless Inference Providers — many small models, just a token",
         catalogue=hf_catalogue,
     ),
+    "llamacpp": Backend(
+        key="llamacpp",
+        label="llama.cpp",
+        blurb="local GGUF models you run yourself — llama-server, GPU when present",
+        catalogue=llamacpp_catalogue,
+    ),
 }
 
 DEFAULT_BACKEND = "modal"

src/models/llamacpp_catalogue.py

@@ -0,0 +1,227 @@
+"""llama.cpp local-inference catalogue — the third backend, next to Modal and HF.
+
+Where ``modal/catalogue.py`` describes models the project deploys itself (vLLM on
+Modal GPUs) and ``hf_catalogue.py`` describes models reachable on Hugging Face's
+serverless router, this module describes **GGUF models you run on your own machine**
+through ``llama-server`` — llama.cpp's OpenAI-compatible HTTP server. It is the
+"Llama Champion" lane: a real llama.cpp runtime in the cast, and the same data also
+carries the **NVIDIA Nemotron** and **OpenBMB MiniCPM** small models, so one local
+server can qualify several sponsor tracks at once.
+
+Like the other catalogues this file is **stdlib-only** and reaches no network: it is
+pure data plus string building, so the engine and the Lab picker can read it offline.
+The engine never imports a vendor SDK from here — calls route through the same LiteLLM
+gateway as the Modal/HF paths, using the OpenAI-compatible custom-endpoint form
+``openai/<served_id>`` + ``api_base`` (the local server's ``/v1`` URL). The *serving*
+side — picking a GGUF, detecting a GPU, and launching ``llama-server`` — lives in the
+sibling :mod:`src.models.llamacpp_server`; this module only describes the models and
+how to reach a running server.
+
+Tier mapping mirrors the four logical profiles the cast routes by:
+``tiny`` ≤4B · ``fast`` ≤8B · ``balanced`` ≤13B · ``strong`` ≤32B. Every model stays
+within the project's ≤32B "small minds" rule; ``tiny`` honours the Tiny-Titan ≤4B band.
+
+Add a model = append one :class:`LlamaCppModel`. Nothing downstream needs editing —
+the unified backend registry (:mod:`src.models.inference`), the router, the Lab picker,
+and the launcher all derive from this data.
+"""
+
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass
+
+# The local llama.cpp server's OpenAI-compatible base URL. llama-server's default bind
+# is 127.0.0.1:8080; the launcher prints the matching ``LLAMACPP_BASE_URL`` export so the
+# engine and the server agree. Overridable for a remote box or a non-default port.
+DEFAULT_BASE_URL = "http://127.0.0.1:8080/v1"
+
+# Base-URL env var (the opt-in seam: set this and the backend goes live). A llama.cpp
+# server needs no real token, but LiteLLM/OpenAI clients require *some* bearer string —
+# ``LLAMACPP_API_KEY`` overrides the harmless default below.
+_BASE_URL_ENV = "LLAMACPP_BASE_URL"
+_API_KEY_ENV = "LLAMACPP_API_KEY"
+_DEFAULT_API_KEY = "llama.cpp"  # any non-empty string; llama-server ignores its value
+
+
+@dataclass(frozen=True)
+class LlamaCppModel:
+    """One GGUF model servable locally through ``llama-server``.
+
+    Engine-facing fields (``key`` / ``profile`` / ``params_b`` / ``served_id``) mirror
+    the other catalogues so the registry treats all three backends identically. The
+    serving fields (``hf_repo`` / ``quant`` / ``gguf_file`` / ``ctx_size`` / sampling /
+    ``flash_attn`` / ``reasoning``) are read only by :mod:`src.models.llamacpp_server`
+    when it assembles the ``llama-server`` command — the engine never touches them.
+    """
+
+    # Identity / engine-facing
+    key: str  # stable slug + served-model id, e.g. "nemotron-3-nano-4b"
+    hf_repo: str  # Hugging Face GGUF repo, e.g. "nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF"
+    profile: str | None = None  # default tier (tiny/fast/balanced/strong) or None
+    params_b: float | None = None  # total params in billions (docs / Tiny-Titan checks)
+    source: str = "llama.cpp"  # friendly family/org label for the picker
+
+    # Serving / launcher-facing
+    quant: str | None = "Q4_K_M"  # GGUF quant tag for the ``-hf repo:QUANT`` form; None
+    #                               when the quant is already baked into ``hf_repo``.
+    gguf_file: str | None = None  # explicit GGUF filename (for ``-m``/download); optional
+    ctx_size: int = 8192  # default --ctx-size; 0 means "use the model's trained context"
+    temperature: float = 0.7
+    top_p: float = 0.95
+    top_k: int = 40
+    flash_attn: bool = True  # --flash-attn (-fa): faster + lower memory where supported
+    reasoning: bool = False  # a "thinking" model — budget more tokens; nudge sampling
+
+    @property
+    def served_model_id(self) -> str:
+        """Model id clients pass. We launch with ``--alias <key>`` so the running server
+        reports this exact id, keeping the binding's ``openai/<id>`` stable across the
+        repo/quant churn of GGUF names."""
+        return self.key
+
+    @property
+    def hf_spec(self) -> str:
+        """The argument for ``llama-server -hf`` — ``repo`` or ``repo:QUANT``.
+
+        Some repos bake the quant into the repo name (e.g. JetBrains' ``…-Q4_K_M``); for
+        those ``quant`` is None and the bare repo is used. Others publish many quants in
+        one repo and need the ``:QUANT`` selector.
+        """
+        return f"{self.hf_repo}:{self.quant}" if self.quant else self.hf_repo
+
+
+# --- The catalogue: small GGUF models, one per default tier ---------------------------
+# A deliberate spread across sponsor families so a single local server fills the cast and
+# stacks prize lanes (Llama Champion + Nemotron + OpenBMB). Plain data — retuning a tier
+# or adding a quant is a one-line edit.
+
+LLAMACPP_MODELS: tuple[LlamaCppModel, ...] = (
+    # NVIDIA Nemotron 3 Nano 4B — tiny tier, ≤4B Tiny-Titan band. Repo publishes several
+    # quants, so the quant is selected via the ``:Q4_K_M`` form.
+    LlamaCppModel(
+        key="nemotron-3-nano-4b",
+        hf_repo="nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF",
+        profile="tiny",
+        params_b=4.0,
+        source="NVIDIA Nemotron",
+        quant="Q4_K_M",
+        temperature=0.7,
+    ),
+    # OpenBMB MiniCPM 4.1 8B — fast tier (≤8B). Multi-quant repo → ``:Q4_K_M``.
+    LlamaCppModel(
+        key="minicpm-4-1-8b",
+        hf_repo="openbmb/MiniCPM4.1-8B-GGUF",
+        profile="fast",
+        params_b=8.0,
+        source="OpenBMB MiniCPM",
+        quant="Q4_K_M",
+        gguf_file="MiniCPM4.1-8B-Q4_K_M.gguf",
+        temperature=0.8,
+    ),
+    # JetBrains Mellum 2 12B-A2.5B — balanced tier (≤13B; MoE, ~2.5B active). A "thinking"
+    # model: budget more tokens and use its recommended sampling. The quant is baked into
+    # the repo name, so ``quant`` is None (the bare ``-hf`` repo is used).
+    LlamaCppModel(
+        key="mellum2-12b-thinking",
+        hf_repo="JetBrains/Mellum2-12B-A2.5B-Thinking-GGUF-Q4_K_M",
+        profile="balanced",
+        params_b=12.0,
+        source="JetBrains Mellum",
+        quant=None,
+        gguf_file="Mellum2-12B-A2.5B-Thinking-Q4_K_M.gguf",
+        ctx_size=16384,
+        temperature=0.6,
+        top_p=0.95,
+        top_k=20,
+        reasoning=True,
+    ),
+)
+
+
+# --- engine-facing read view (mirrors modal_catalogue / hf_catalogue dict shape) ------
+
+
+def _build_entry(m: LlamaCppModel) -> dict:
+    """One model as a plain dict, shaped like ``modal_catalogue.entries()``."""
+    return {
+        "key": m.key,
+        "provider": m.source,
+        "app": "llama.cpp",
+        "endpoint_name": m.key,
+        "served_model_id": m.served_model_id,
+        "profile": m.profile,
+        "params_b": m.params_b,
+    }
+
+
+# Built once at import (the catalogue is static): callers that mutate copy first.
+_ENTRIES: tuple[dict, ...] = tuple(_build_entry(m) for m in LLAMACPP_MODELS)
+_ENTRY_BY_KEY: dict[str, dict] = {e["key"]: e for e in _ENTRIES}
+_MODEL_BY_KEY: dict[str, LlamaCppModel] = {m.key: m for m in LLAMACPP_MODELS}
+
+
+def entries() -> list[dict]:
+    """Every local model as a plain dict, shaped like the other catalogues:
+
+    ``{key, provider, app, endpoint_name, served_model_id, profile, params_b}`` — so the
+    unified registry and the Lab picker treat all three backends identically.
+    """
+    return list(_ENTRIES)
+
+
+def entry_by_key(key: str) -> dict | None:
+    """The catalogue entry whose key is *key*, or None."""
+    return _ENTRY_BY_KEY.get(key)
+
+
+def model_by_key(key: str) -> LlamaCppModel | None:
+    """The full :class:`LlamaCppModel` for *key* (serving fields included), or None.
+
+    The launcher uses this; the engine path only needs :func:`binding_for`.
+    """
+    return _MODEL_BY_KEY.get(key)
+
+
+def default_key_for_profile(profile: str) -> str | None:
+    """The key of the model tagged for *profile* (first match), or None."""
+    return next((m.key for m in LLAMACPP_MODELS if m.profile == profile), None)
+
+
+def base_url(env: dict[str, str] | None = None) -> str:
+    """The configured local server base URL, or the llama-server default."""
+    source = os.environ if env is None else env
+    return source.get(_BASE_URL_ENV, "").strip() or DEFAULT_BASE_URL
+
+
+def has_credentials(env: dict[str, str] | None = None) -> bool:
+    """True when the llama.cpp backend is opted in — an explicit ``LLAMACPP_BASE_URL``.
+
+    Unlike the hosted backends there is no token to gate on: a local ``llama-server``
+    needs no auth. We gate on the base URL being *set* so the backend never silently
+    claims to be live when nothing is running — the launcher sets this for you, or you
+    export it by hand to point at an already-running (or remote) server.
+    """
+    source = os.environ if env is None else env
+    return bool(source.get(_BASE_URL_ENV, "").strip())
+
+
+def binding_for(key: str, env: dict[str, str] | None = None) -> dict:
+    """Resolve a catalogue *key* into a concrete profile binding.
+
+    Returns ``{"model", "base_url", "api_key"}`` where ``model`` is the LiteLLM
+    OpenAI-compatible string ``openai/<served_id>``, ``base_url`` is ``LLAMACPP_BASE_URL``
+    (or the llama-server default), and ``api_key`` is ``LLAMACPP_API_KEY`` (or a harmless
+    placeholder — llama-server ignores it). Raises ``KeyError`` for an unknown key.
+    """
+    source = os.environ if env is None else env
+    model = _MODEL_BY_KEY.get(key)
+    if model is None:
+        known = sorted(_MODEL_BY_KEY)
+        raise KeyError(f"unknown llama.cpp model {key!r}; known: {known}")
+    api_key = source.get(_API_KEY_ENV, "").strip() or _DEFAULT_API_KEY
+    return {
+        "model": f"openai/{model.served_model_id}",
+        "base_url": base_url(source),
+        "api_key": api_key,
+    }

src/models/llamacpp_server.py

@@ -0,0 +1,212 @@
+"""Launch a local ``llama-server`` for a catalogue model — GPU when one is present.
+
+This is the *serving* side of the llama.cpp backend (the read/binding side lives in
+:mod:`src.models.llamacpp_catalogue`). It assembles the ``llama-server`` command for a
+GGUF model, detects an accelerator (Apple Metal on macOS, NVIDIA CUDA elsewhere) and
+offloads every layer to it when found, and otherwise serves on CPU — so the same command
+works on a laptop and a GPU box. ``llama-server`` downloads the GGUF on first run via its
+``-hf`` flag and exposes an OpenAI-compatible API on ``/v1``, which the engine reaches
+through the same LiteLLM gateway as the Modal/HF backends.
+
+Usage::
+
+    # launch the tiny-tier Nemotron (GPU auto-detected), then export the URL it prints
+    uv run python -m src.models.llamacpp_server nemotron-3-nano-4b
+    export LLAMACPP_BASE_URL=http://127.0.0.1:8080/v1
+
+    uv run python -m src.models.llamacpp_server --list          # show available models
+    uv run python -m src.models.llamacpp_server minicpm-4-1-8b --cpu --print-only
+
+``build_command`` and ``detect_accelerator`` are pure and take their inputs as arguments
+(platform string, a probe callable) so tests can exercise the GPU/CPU branches without a
+GPU or the ``llama-server`` binary present.
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import shutil
+import subprocess
+import sys
+from collections.abc import Callable
+
+from src.models import llamacpp_catalogue
+from src.models.llamacpp_catalogue import LlamaCppModel
+
+# Default bind. 127.0.0.1 keeps the server local; pass --host 0.0.0.0 to expose it (e.g.
+# a remote GPU box). The port matches llama-server's own default so the catalogue's
+# DEFAULT_BASE_URL lines up with a no-flag launch.
+DEFAULT_HOST = "127.0.0.1"
+DEFAULT_PORT = 8080
+DEFAULT_BINARY = "llama-server"
+
+# Offload-all sentinel: llama.cpp treats a large -ngl as "every layer on the GPU".
+_OFFLOAD_ALL_LAYERS = 999
+
+
+def detect_accelerator(
+    platform: str | None = None,
+    probe: Callable[[], bool] | None = None,
+) -> str:
+    """Return the accelerator to offload to: ``"metal"``, ``"cuda"``, or ``"cpu"``.
+
+    macOS (``darwin``) ships llama.cpp with Metal, so Apple Silicon always offloads.
+    Elsewhere we offload only when an NVIDIA GPU is visible — probed with ``nvidia-smi``
+    by default (injectable so tests don't depend on the host). Anything else → CPU.
+    """
+    plat = platform if platform is not None else sys.platform
+    if plat == "darwin":
+        return "metal"
+    has_cuda = probe() if probe is not None else _nvidia_smi_present()
+    return "cuda" if has_cuda else "cpu"
+
+
+def _nvidia_smi_present() -> bool:
+    """True when ``nvidia-smi`` exists and exits cleanly (a usable NVIDIA GPU)."""
+    if shutil.which("nvidia-smi") is None:
+        return False
+    try:
+        result = subprocess.run(
+            ["nvidia-smi"],
+            capture_output=True,
+            timeout=10,
+            check=False,
+        )
+    except (OSError, subprocess.SubprocessError):  # pragma: no cover - host-dependent
+        return False
+    return result.returncode == 0
+
+
+def gpu_layers(accelerator: str) -> int:
+    """How many layers to offload for *accelerator*: all on a GPU, none on CPU."""
+    return _OFFLOAD_ALL_LAYERS if accelerator in ("metal", "cuda") else 0
+
+
+def build_command(
+    model: LlamaCppModel,
+    *,
+    accelerator: str,
+    host: str = DEFAULT_HOST,
+    port: int = DEFAULT_PORT,
+    ctx_size: int | None = None,
+    binary: str = DEFAULT_BINARY,
+) -> list[str]:
+    """Assemble the ``llama-server`` argv for *model*. Returned as a list so the caller
+    launches with ``subprocess`` and no shell (no quoting pitfalls).
+
+    The model is pulled by its ``-hf`` spec (downloaded on first run) and served under
+    ``--alias <key>`` so the running server reports the stable id the engine binds to.
+    On a GPU (``metal``/``cuda``) every layer is offloaded (``-ngl 999``); on CPU the
+    flag is omitted. Sampling defaults and flash-attention come from the model's
+    catalogue entry; ``ctx_size`` overrides the per-model context window when given.
+    """
+    ctx = model.ctx_size if ctx_size is None else ctx_size
+    cmd: list[str] = [
+        binary,
+        "-hf",
+        model.hf_spec,
+        "--alias",
+        model.key,
+        "--host",
+        host,
+        "--port",
+        str(port),
+        "--ctx-size",
+        str(ctx),
+        "--temp",
+        str(model.temperature),
+        "--top-p",
+        str(model.top_p),
+        "--top-k",
+        str(model.top_k),
+    ]
+    layers = gpu_layers(accelerator)
+    if layers:
+        cmd += ["-ngl", str(layers)]
+    if model.flash_attn:
+        cmd += ["--flash-attn", "on"]
+    return cmd
+
+
+def base_url_for(host: str, port: int) -> str:
+    """The OpenAI-compatible URL clients should use for a server bound to *host:port*.
+
+    A server bound to ``0.0.0.0`` listens on every interface but is reached locally via
+    the loopback address, so we advertise ``127.0.0.1`` in that case.
+    """
+    reachable = "127.0.0.1" if host in ("0.0.0.0", "::") else host
+    return f"http://{reachable}:{port}/v1"
+
+
+def _format_models() -> str:
+    lines = ["Available llama.cpp models (key → repo · tier · params):"]
+    for m in llamacpp_catalogue.LLAMACPP_MODELS:
+        tier = m.profile or "—"
+        params = f"{m.params_b:g}B" if m.params_b else "?"
+        lines.append(f"  {m.key:<24} {m.hf_spec}  ·  {tier:<8} ·  {params}")
+    return "\n".join(lines)
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(
+        prog="python -m src.models.llamacpp_server",
+        description="Launch a local llama-server for a catalogue model (GPU auto-detected).",
+    )
+    parser.add_argument("key", nargs="?", help="catalogue model key (see --list)")
+    parser.add_argument("--list", action="store_true", help="list available models and exit")
+    parser.add_argument("--host", default=DEFAULT_HOST, help=f"bind host (default {DEFAULT_HOST})")
+    parser.add_argument("--port", type=int, default=DEFAULT_PORT, help=f"bind port (default {DEFAULT_PORT})")
+    parser.add_argument("--ctx-size", type=int, default=None, help="override the model's context window")
+    parser.add_argument("--cpu", action="store_true", help="force CPU (skip GPU offload)")
+    parser.add_argument("--binary", default=DEFAULT_BINARY, help=f"llama-server binary (default {DEFAULT_BINARY})")
+    parser.add_argument("--print-only", action="store_true", help="print the command and export line, do not launch")
+    args = parser.parse_args(argv)
+
+    if args.list or not args.key:
+        print(_format_models())
+        return 0 if args.list else 2
+
+    model = llamacpp_catalogue.model_by_key(args.key)
+    if model is None:
+        print(f"unknown model {args.key!r}.\n\n{_format_models()}", file=sys.stderr)
+        return 2
+
+    accelerator = "cpu" if args.cpu else detect_accelerator()
+    cmd = build_command(
+        model,
+        accelerator=accelerator,
+        host=args.host,
+        port=args.port,
+        ctx_size=args.ctx_size,
+        binary=args.binary,
+    )
+    url = base_url_for(args.host, args.port)
+
+    where = {"metal": "Apple Metal GPU", "cuda": "NVIDIA GPU", "cpu": "CPU"}[accelerator]
+    print(f"▶ {model.key} ({model.hf_spec}) on {where}")
+    print(f"  {' '.join(cmd)}")
+    print(f"\nPoint the engine at it:\n  export {llamacpp_catalogue._BASE_URL_ENV}={url}\n")
+
+    if args.print_only:
+        return 0
+
+    if shutil.which(args.binary) is None:
+        print(
+            f"'{args.binary}' not found on PATH. Install llama.cpp "
+            "(https://github.com/ggml-org/llama.cpp) or pass --binary /path/to/llama-server.",
+            file=sys.stderr,
+        )
+        return 127
+
+    # Export the URL into this process's children too, so a wrapper that launches the
+    # app in the same shell session sees it; the printed line covers the manual case.
+    os.environ[llamacpp_catalogue._BASE_URL_ENV] = url
+    try:
+        return subprocess.call(cmd)
+    except KeyboardInterrupt:  # pragma: no cover - interactive
+        return 0
+
+
+if __name__ == "__main__":  # pragma: no cover - CLI entry
+    raise SystemExit(main())

tests/test_inference_backends.py

@@ -38,10 +38,12 @@ def test_entries_are_tagged_and_qualified():
     assert modal_keys == {e["key"] for e in modal_catalogue.entries()}
     assert all(k.startswith("hf:") for k in hf_keys)
     assert modal_keys.isdisjoint(hf_keys)
-    # The unqualified call returns both backends' models, each tagged with its backend.
+    # The unqualified call returns every backend's models, each tagged with its backend.
     everything = inference.entries()
-    assert {e["backend"] for e in everything} == {"modal", "hf"}
-    assert len(everything) == len(modal_keys) + len(hf_keys)
+    assert {"modal", "hf"} <= {e["backend"] for e in everything}
+    assert len(everything) == len(inference.entries("modal")) + len(inference.entries("hf")) + len(
+        inference.entries("llamacpp")
+    )
 
 
 def test_entry_by_key_round_trips_both_backends():

tests/test_llamacpp_backend.py

@@ -0,0 +1,154 @@
+"""Tests for the llama.cpp local backend — catalogue, registry integration, launcher.
+
+llama.cpp is the third inference backend (next to Modal and Hugging Face): GGUF models
+served locally by ``llama-server`` behind an OpenAI-compatible API. These tests cover the
+read/binding side (catalogue + unified registry) and the serving side (the launcher's
+pure command-building and GPU detection) without a GPU or the binary present — the
+launcher's pure functions take platform/probe as arguments precisely so they're testable.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from src.models import inference, llamacpp_catalogue, llamacpp_server
+
+
+# ── catalogue ────────────────────────────────────────────────────────────────────────
+
+
+def test_catalogue_covers_three_sponsor_tiers():
+    by_profile = {m.profile: m for m in llamacpp_catalogue.LLAMACPP_MODELS}
+    # Nemotron (tiny ≤4B), MiniCPM (fast ≤8B), Mellum (balanced ≤13B) — the three lanes.
+    assert by_profile["tiny"].params_b <= 4
+    assert by_profile["fast"].params_b <= 8
+    assert by_profile["balanced"].params_b <= 13
+    # Every model stays within the ≤32B "small minds" rule.
+    assert all(m.params_b <= 32 for m in llamacpp_catalogue.LLAMACPP_MODELS)
+
+
+def test_hf_spec_appends_quant_only_when_not_baked_into_repo():
+    nemotron = llamacpp_catalogue.model_by_key("nemotron-3-nano-4b")
+    mellum = llamacpp_catalogue.model_by_key("mellum2-12b-thinking")
+    # Multi-quant repo → the quant is selected with the ``:QUANT`` form.
+    assert nemotron.hf_spec == "nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF:Q4_K_M"
+    # Quant already baked into the repo name → bare repo, no ``:QUANT`` suffix.
+    assert mellum.quant is None
+    assert mellum.hf_spec == "JetBrains/Mellum2-12B-A2.5B-Thinking-GGUF-Q4_K_M"
+
+
+def test_served_id_is_the_stable_key_not_the_gguf_name():
+    # The launcher serves under --alias <key>, so the engine binds to a stable id even as
+    # GGUF repo/quant names churn.
+    m = llamacpp_catalogue.model_by_key("minicpm-4-1-8b")
+    assert m.served_model_id == "minicpm-4-1-8b"
+
+
+def test_binding_uses_local_url_and_placeholder_key_by_default():
+    binding = llamacpp_catalogue.binding_for("nemotron-3-nano-4b", env={})
+    assert binding["model"] == "openai/nemotron-3-nano-4b"
+    assert binding["base_url"] == llamacpp_catalogue.DEFAULT_BASE_URL
+    # llama-server ignores the token but OpenAI clients require a non-empty one.
+    assert binding["api_key"]
+
+
+def test_binding_honours_env_overrides():
+    env = {"LLAMACPP_BASE_URL": "http://gpu-box:9000/v1", "LLAMACPP_API_KEY": "sekret"}
+    binding = llamacpp_catalogue.binding_for("minicpm-4-1-8b", env=env)
+    assert binding["base_url"] == "http://gpu-box:9000/v1"
+    assert binding["api_key"] == "sekret"
+
+
+def test_binding_unknown_key_raises():
+    with pytest.raises(KeyError):
+        llamacpp_catalogue.binding_for("does-not-exist", env={})
+
+
+def test_has_credentials_gates_on_explicit_base_url():
+    # No silent "live": the backend is opted in only when the URL is set (the launcher
+    # sets it, or you export it to point at a running/remote server).
+    assert llamacpp_catalogue.has_credentials(env={}) is False
+    assert llamacpp_catalogue.has_credentials(env={"LLAMACPP_BASE_URL": "http://x/v1"}) is True
+
+
+# ── unified registry integration ──────────────────────────────────────────────────────
+
+
+def test_registered_as_third_backend():
+    assert "llamacpp" in {b.key for b in inference.backends()}
+    keys = {e["key"] for e in inference.entries("llamacpp")}
+    assert all(k.startswith("llamacpp:") for k in keys)
+
+
+def test_registry_dispatches_binding_and_availability():
+    key = inference.default_key_for_profile("tiny", "llamacpp")
+    assert key == "llamacpp:nemotron-3-nano-4b"
+    binding = inference.binding_for(key, env={"LLAMACPP_BASE_URL": "http://127.0.0.1:8080/v1"})
+    assert binding["base_url"] == "http://127.0.0.1:8080/v1"
+    assert inference.backend_available("llamacpp", env={"LLAMACPP_BASE_URL": "http://x/v1"}) is True
+    assert inference.backend_available("llamacpp", env={}) is False
+
+
+def test_configured_backends_includes_llamacpp_when_url_set():
+    configured = inference.configured_backends(env={"LLAMACPP_BASE_URL": "http://x/v1"})
+    assert "llamacpp" in configured
+    assert inference.configured_backends(env={}) == []
+
+
+# ── launcher: GPU detection ────────────────────────────────────────────────────────────
+
+
+def test_detect_accelerator_metal_on_macos():
+    assert llamacpp_server.detect_accelerator(platform="darwin") == "metal"
+
+
+def test_detect_accelerator_cuda_when_gpu_present():
+    assert llamacpp_server.detect_accelerator(platform="linux", probe=lambda: True) == "cuda"
+
+
+def test_detect_accelerator_cpu_when_no_gpu():
+    assert llamacpp_server.detect_accelerator(platform="linux", probe=lambda: False) == "cpu"
+
+
+def test_gpu_layers_offloads_all_on_gpu_none_on_cpu():
+    assert llamacpp_server.gpu_layers("metal") == 999
+    assert llamacpp_server.gpu_layers("cuda") == 999
+    assert llamacpp_server.gpu_layers("cpu") == 0
+
+
+# ── launcher: command building ─────────────────────────────────────────────────────────
+
+
+def test_build_command_offloads_layers_on_gpu():
+    model = llamacpp_catalogue.model_by_key("nemotron-3-nano-4b")
+    cmd = llamacpp_server.build_command(model, accelerator="cuda")
+    assert cmd[0] == "llama-server"
+    assert "-hf" in cmd and model.hf_spec in cmd
+    assert cmd[cmd.index("--alias") + 1] == "nemotron-3-nano-4b"
+    assert "-ngl" in cmd and cmd[cmd.index("-ngl") + 1] == "999"
+
+
+def test_build_command_omits_offload_on_cpu():
+    model = llamacpp_catalogue.model_by_key("nemotron-3-nano-4b")
+    cmd = llamacpp_server.build_command(model, accelerator="cpu")
+    assert "-ngl" not in cmd
+
+
+def test_build_command_carries_model_sampling_and_ctx():
+    model = llamacpp_catalogue.model_by_key("mellum2-12b-thinking")
+    cmd = llamacpp_server.build_command(model, accelerator="metal")
+    assert cmd[cmd.index("--temp") + 1] == "0.6"
+    assert cmd[cmd.index("--top-k") + 1] == "20"
+    assert cmd[cmd.index("--ctx-size") + 1] == "16384"
+    assert "--flash-attn" in cmd
+
+
+def test_build_command_ctx_override_wins():
+    model = llamacpp_catalogue.model_by_key("mellum2-12b-thinking")
+    cmd = llamacpp_server.build_command(model, accelerator="cpu", ctx_size=2048)
+    assert cmd[cmd.index("--ctx-size") + 1] == "2048"
+
+
+def test_base_url_for_advertises_loopback_when_bound_to_all_interfaces():
+    assert llamacpp_server.base_url_for("0.0.0.0", 8080) == "http://127.0.0.1:8080/v1"
+    assert llamacpp_server.base_url_for("127.0.0.1", 9000) == "http://127.0.0.1:9000/v1"