docs/adr/0032-llama-cpp-local-backend.md

# ADR-0032: A Third Inference Backend — Local llama.cpp (GGUF)

## Status

Superseded by ADR-0033. llama.cpp's persistent `llama-server` cannot hold a GPU under
ZeroGPU's per-call grant model; replaced by an in-process `transformers` backend that
works on any HF Space hardware (ADR-0024 *second inference backend / unified registry*,
ADR-0015 *LiteLLM gateway*, ADR-0022 *per-agent explicit model binding* remain in
force; this ADR is retained as a historical record only).

## Context

The engine had two live backends: vLLM endpoints we deploy on Modal (ADR-0015/0019)
and Hugging Face's serverless router (ADR-0024). Both run *somewhere else* — Modal needs
warmed GPUs, HF needs a token and a provider that serves the model. There was no way to
run a cast **entirely on the operator's own machine**, with no account, no token, and no
network after the first download.

llama.cpp's `llama-server` is exactly that: it loads a quantized **GGUF** model, runs it
on whatever hardware is present (Apple Metal, NVIDIA CUDA, or CPU), and exposes an
**OpenAI-compatible** API on `/v1`. Because it speaks the same REST surface as Modal/HF,
it slots into the existing LiteLLM gateway with no new transport code — the same seam
ADR-0024 was designed to leave open ("adding a third backend later touches only
`inference._BACKENDS`").

This also stacks hackathon lanes from one local server: the **Llama Champion** badge
(a real llama.cpp runtime in the cast), the **NVIDIA Nemotron Quest** (Nemotron 3 Nano
4B), and the **OpenBMB** track (MiniCPM 4.1 8B) — plus a JetBrains Mellum 2 thinking
model on the balanced tier. Every model stays within the ≤32B "small minds" rule, and the
4B Nemotron honours the ≤4B Tiny-Titan band.

## Decision

**A third backend = one more catalogue + a registry entry.** Add
`src/models/llamacpp_catalogue.py`, stdlib-only and offline-safe like its siblings, listing
the GGUF models with both engine-facing fields (`key` / `profile` / `params_b` /
`served_id`) and serving fields (`hf_repo` / `quant` / `ctx_size` / sampling /
`flash_attn` / `reasoning`). Its `binding_for()` yields the LiteLLM custom-endpoint form
`model = openai/<served_id>`, `base_url = $LLAMACPP_BASE_URL` (default
`http://127.0.0.1:8080/v1`), `api_key = $LLAMACPP_API_KEY` (a placeholder — llama-server
ignores it). Register it in `inference._BACKENDS` under the prefix `llamacpp`; qualified
keys are `llamacpp:<slug>` (e.g. `llamacpp:nemotron-3-nano-4b`). Nothing above the registry
changes — the router, the config loader's `endpoint:` expansion, the live/offline gate, and
the Lab picker all derive from the façade.

**The serving side is a separate, pure-where-it-matters launcher.** Add
`src/models/llamacpp_server.py`. `detect_accelerator(platform, probe)` returns
`metal` on macOS, `cuda` when `nvidia-smi` reports a GPU, else `cpu`;
`build_command(model, accelerator, …)` assembles the `llama-server` argv — pulling the
model by its `-hf` spec (downloaded on first run), serving it under `--alias <key>` so the
running server reports the stable id the engine binds to, and offloading **every layer to
the GPU** (`-ngl 999`) when one is present, omitting the flag on CPU. Both take their
environment as arguments so the GPU/CPU branches are testable with no GPU and no binary.
The `__main__` CLI launches a model by key and prints the matching `LLAMACPP_BASE_URL`
export.

**Opt-in by base URL, not a token.** A local server needs no auth, so the live/offline
gate (`has_credentials`) keys on `LLAMACPP_BASE_URL` being *set* — the launcher sets it,
or you export it to point at an already-running or remote server. With it unset the
backend never claims to be live, so the deterministic stub still owns the no-config demo.

## Consequences

- A cast can run fully local: `uv run python -m src.models.llamacpp_server
  nemotron-3-nano-4b`, export the printed URL, and the engine routes to it through the
  unchanged LiteLLM transport — no account, no token, GPU used automatically when present.
- Llama Champion + Nemotron + OpenBMB lanes are reachable from one server; the catalogue
  is plain data, so adding a GGUF or retuning a tier is a one-line `LlamaCppModel(...)` edit.
- Backward compatible by construction: bare keys still mean Modal, and the `llamacpp`
  prefix is new — existing config, manifests, and the green test baseline are unaffected.
- The engine still never names a vendor on a hot path: routing is by qualified key through
  one façade; the GGUF/quant churn is hidden behind `--alias <key>`.
- llama-server and the GGUF download are operator-side, never a Python dependency — the
  offline stub remains the default with no binary, no network, and extras uninstalled.