add: Hosted generation for the Space, and retire the ZeroGPU attempt

The Space moves to plain CPU hardware with generation over Inference Providers.
Everything that makes ControlAI what it is stays in-process -- all 29
deterministic solvers, the verifier, the full 80,370-chunk hybrid retriever --
because those cost milliseconds on a CPU. Only token generation leaves, to a GPU
behind an API. That is a real trade: the demo is no longer self-contained, which
is the local app's whole point, so the Space page says so.

engine_api.py implements the same engine contract. It renders the tool schemas
through the model's own chat template locally and passes the result as an
ordinary system message, never as tools=: a provider given tools= applies its own
template and returns structured tool_calls, which the agent does not speak. This
way the model emits <tool_call> as text and toolcall.parse sees what it sees
locally. Every provider serving Qwen3 offers conversational only, not raw
text-generation, so render() returns a message list instead of a string -- the
agent treats that value as opaque, so nothing downstream changes. Thinking is
disabled via Qwen3's /no_think soft switch rather than chat_template_kwargs,
which is an extra_body passthrough not every provider forwards.

app_space.py drops from 200 lines to 60: no spaces, no Gradio probe, no
import-window choreography, and the /api/gpudiag debug route is gone with it.
requirements-space.txt uses CPU torch wheels, still needed for the 0.6B
retrieval embedder. Embedder now treats every non-MLX backend as transformers,
since "api" describes where generation happens and says nothing about embedding.

engine_torch.py stays: it works on an ordinary CUDA box, and it is ZeroGPU
specifically that does not fit. CLAUDE.md records all seven incompatibilities,
ending with the structural one -- ZeroGPU schedules GPU workers by forking the
Gradio server process and assumes the Gradio app is the Space, while this is a
FastAPI app that merely borrows the process. Written down so the attempt is not
repeated blind.

Verified locally up to the auth boundary: the engine renders, folds the tool
block in, applies /no_think, and reaches the provider endpoint. It returns 403
because the current token lacks the Inference Providers permission, which is a
token setting, not code.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CLAUDE.md

@@ -179,88 +179,53 @@ formats correctly. `CONTROLAI_ADAPTER=<path>` loads one anyway for A/B work.
 discovery → extraction → dataset generation → training → evaluation). That pipeline is independent
 of the serving path and uses `requirements-training.txt`/`requirements-corpus.txt`.
 
-### Deployment (`app_space.py`, `engine_torch.py`, `requirements-space.txt`)
-The demo Space (huggingface.co/spaces/atakankahya/ControlAI-Agent) runs Linux/NVIDIA on ZeroGPU,
-where MLX does not exist. `CONTROLAI_BACKEND` makes `app.py::_make_engine` build
-`TorchEngine` instead of `LocalEngine`; `Embedder` switches on the same variable. That is the whole
-switch — two branches, no orchestrator. It accepts `torch`, `pytorch` or `cuda`, and **raises on
-anything it does not recognise rather than falling back to MLX**. The deployed Space still carries
-`CONTROLAI_BACKEND=pytorch` as a variable from the old orchestrator; a check for `"torch"` alone
-silently selected MLX on a box with no MLX, and the failure surfaced as `ModuleNotFoundError:
-mlx_lm` several frames from the real cause.
-
-`engine_torch.py` mirrors `engine.py` rather than calling `model.generate`, because `generate`
-cannot express either of the two things that matter: `DynamicCache.crop()` for prefix reuse across
-tool steps, and injecting `</think>` to close an overrunning reasoning block.
-
-**It loads bf16 with an explicit `.to("cuda")`, and `app_space.py` constructs it at module import
-scope. Do not reintroduce `device_map` or bitsandbytes, and do not move the construction into a
-lifespan or a request.** All of those fail identically with `RuntimeError: Low-level CUDA init
-(torch._C._cuda_init) reached`. ZeroGPU patches torch during the import of the Space's entry module
-and attaches real hardware only inside a `@spaces.GPU` call; only CUDA operations inside that import
-window are intercepted. `app.py` normally builds the model in FastAPI's `lifespan`, on a
-`ThreadPoolExecutor` worker — after import, on another thread — and `.to("cuda")` there reaches real
-CUDA init and raises. `app_space.py::_build_agent_at_import` builds the agent during import and
-assigns `app._agent`, so `lifespan`'s `get_agent()` is a no-op. **The retrieval embedder is a second
-model with the same problem** — it loads lazily on the first query, which is a request, outside the
-window — so the same function embeds one throwaway string to force it onto the GPU during import.
-Without that, the Space starts fine and every answer carries `[agent] retrieval failed`. `device_map` fails independently, by
-routing transformers through `caching_allocator_warmup` and its direct
-`torch.empty(..., device="cuda")`; bitsandbytes requires `device_map`, so **4-bit quantisation is
-unavailable**, which is what forces a model small enough to carry in bf16: `Qwen/Qwen3-8B`, ~16GB
-against ~28GB for the 14B run locally.
-
-**All inference must run inside a `@spaces.GPU` call.** Loading the model in the import window is
-necessary but not sufficient: ZeroGPU creates it under CUDA *emulation*, reports `cuda:0`, and packs
-its tensors — only a `@spaces.GPU` call materialises them on real hardware. Forward passes anywhere
-else do not fail. They read unmaterialised tensors and return fluent-looking multilingual noise at
-roughly fifteen minutes a turn, which is the worst possible failure mode: a demo that is up,
-responsive, and confidently wrong. `app.py` exposes a `stream_hook`, `None` locally; `app_space.py`
-sets it to `_gpu_stream`, a `@spaces.GPU(duration=300)` generator wrapping one whole agent turn.
-Wrapping the *turn* rather than each `engine.stream()` call is deliberate — a turn is several
-generations sharing one KV cache, and splitting them across separate calls would put that shared
-state on the far side of a process boundary each time. `_collect()` exists so `/api/chat` honours the
-hook too; `ControlAgent.run()` consumes `self.stream` directly and would bypass it.
-
-**The dedicated inference thread must never carry a CUDA call.** `spaces` intercepts CUDA only
-inside the context it manages, and `app.py`'s `ThreadPoolExecutor` is outside it — a `@spaces.GPU`
-call made from there fails in its own worker with `RuntimeError: No CUDA GPUs are available`, even
-with a GPU genuinely attached (`hardware.current: zero-a10g`). `USE_INFERENCE_THREAD` gates the
-executor on the backend: MLX keeps its single pinned thread, torch gets direct calls under the lock
-and hands `/api/chat/stream` a plain sync generator for Starlette's own threadpool. **This was found
-and fixed once before, in 4de16e3, and the MLX rewrite reintroduced it** — the executor was made
-unconditional because the CUDA path had been deleted. Read that commit before touching threading
-here.
-
-**ZeroGPU platform gotchas, each learned by having the Space fail:**
-- A `@spaces.GPU` function is only detected if wired to a real Gradio event handler. One called
-  solely from a FastAPI route fails startup with "No @spaces.GPU function detected". Hence the
-  hidden probe button.
-- That function must be a module-level `def`; nested inside `with gr.Blocks():` the detection fails.
-- Don't `mount_gradio_app()` the probe and then run uvicorn on the same port — Gradio's own server
-  setup collides ("address already in use"). The probe launches on `port + 1`, non-blocking.
-- **Never pass the model-holding object as an argument** to a `@spaces.GPU` function. ZeroGPU
-  marshals arguments across a process boundary and tries to share the model's CUDA tensors, failing
-  with `_share_cuda_: only available on CUDA` after emitting nothing — which reads exactly like
-  "just slow". Reach the agent through the module global.
-
-**Do not put an upper bound in `requirements-space.txt`.** The platform appends its own
-`gradio[oauth,mcp]`, `spaces`, `uvicorn` and a `torch` ceiling to whatever that file asks for, and
-one extra constraint can make the resolve impossible. Pinning `transformers<4.56` — to keep using
-the `torch_dtype=` kwarg it renamed — failed the build outright, because gradio 6.x requires
-`huggingface-hub>=1.16` and every `transformers<4.56` requires `<1.0`. `engine_torch.dtype_kwarg()`
-detects the spelling instead, and `_align_cache` falls back to a full re-prefill if `DynamicCache`
-has no `crop()`, so an unpinned transformers costs speed rather than correctness.
-
-The Space needs `HF_TOKEN` as a secret: the retrieval index is in a private dataset repo and
-`app_space.py::_fetch_index` pulls it at startup. Without it the Space still boots, and answers
-from model knowledge alone.
-
-**The Space's queries are embedded with the bf16 `Qwen/Qwen3-Embedding-0.6B` while the index was
-built with the MLX 4-bit checkpoint.** Measured, the two produce vectors agreeing at cosine 0.96,
-and against the real 80,370-chunk index the gate behaves the same: in-domain worst best-match 0.712
-(MLX 0.708), off-domain best 0.528 (MLX 0.541). `MIN_COSINE = 0.62` therefore holds unchanged on
-both. `CONTROLAI_MIN_COSINE` overrides it if that ever drifts.
+### Deployment (`app_space.py`, `engine_api.py`, `requirements-space.txt`)
+The demo Space runs **plain CPU hardware with hosted generation**, not ZeroGPU.
+`CONTROLAI_BACKEND=api` makes `app.py::_make_engine` build `RemoteEngine`; everything else —
+all 29 solvers, the verifier, the full 80,370-chunk retriever — runs in-process on the Space's CPU,
+where it costs milliseconds. Only token generation leaves the machine. The Space page says so
+plainly, because the local app's whole point is that nothing does.
+
+`engine_api.py` renders the tool schemas through the model's own chat template locally and hands
+the result over as an ordinary **system message** — never as a `tools=` argument, which would make
+the provider apply its own template and return structured `tool_calls` the agent does not speak.
+Every provider serving Qwen3 offers `conversational` only, not raw text-generation, so `render()`
+returns a *message list* rather than a string; the agent treats that value as opaque, so nothing
+downstream cares. Thinking is disabled with Qwen3's `/no_think` soft switch rather than
+`chat_template_kwargs`, which is an `extra_body` passthrough not every provider forwards.
+
+`HF_TOKEN` must be a Space secret with **two** permissions: read on the private index dataset repo,
+and *Make calls to Inference Providers*. Missing the first disables retrieval silently; missing the
+second fails generation with `403 ... does not have sufficient permissions to call Inference
+Providers`.
+
+**ZeroGPU was tried at length and abandoned.** `engine_torch.py` is kept and works on an ordinary
+CUDA box; it is ZeroGPU specifically that does not fit. Seven distinct problems, recorded so the
+attempt is not repeated blind:
+1. **Dependency resolution.** The platform appends its own `gradio[oauth,mcp]`, `spaces`, `uvicorn`
+   and a `torch` ceiling to `requirements.txt`. Any upper bound of your own can make the resolve
+   impossible — `transformers<4.56` did, because gradio 6 needs `huggingface-hub>=1.16` and every
+   `transformers<4.56` needs `<1.0`.
+2. **Import-window loading.** ZeroGPU patches torch during the entry module's import and only
+   intercepts CUDA inside that window. Building the model in FastAPI's `lifespan`, on a worker
+   thread, reaches real CUDA init and raises.
+3. **`device_map` and bitsandbytes.** `device_map` routes transformers through
+   `caching_allocator_warmup`'s direct `torch.empty(..., device="cuda")`, which trips the same
+   guard; bitsandbytes requires `device_map`, so 4-bit is unavailable.
+4. **Inference must be inside `@spaces.GPU`.** Otherwise the packed tensors are never materialised
+   and forward passes return fluent multilingual noise at ~15 min/turn — up, responsive, and
+   confidently wrong.
+5. **Thread context.** `spaces` intercepts CUDA only in the context it manages; a call from a
+   custom `ThreadPoolExecutor` thread fails with `No CUDA GPUs are available` even with a GPU
+   attached. Found and fixed once before in 4de16e3, then reintroduced by the MLX rewrite.
+6. **The probe must be `launch()`ed, not mounted.** `gr.mount_gradio_app` fails startup with
+   "No @spaces.GPU function detected". Launch it on its own port with `ssr_mode=False` (Gradio 6's
+   SSR spawns a Node subprocess into the process ZeroGPU forks from).
+7. **The structural one.** With all six fixed, the GPU is scheduled and acquired and ZeroGPU's own
+   forked worker still dies in `torch.init()`. A no-op `@spaces.GPU` function fails identically, so
+   it is not the model or the payload. Meanwhile the platform probes the public port for Gradio's
+   `/api/predict` and gets 404, because FastAPI owns it. ZeroGPU assumes the Gradio app *is* the
+   Space; this one is not.
 
 ### Benchmark (`benchmarks/`)
 `controlbench_v1.jsonl` is the eval set; `SCOPE.md` defines the taxonomy and `README.md` the

app.py

@@ -51,7 +51,7 @@ for directory in (STATIC_DIR, PLOTS_DIR, UPLOADS_DIR):
 # One thread, for the lifetime of the process: see the module docstring.
 def _uses_mlx() -> bool:
     return os.environ.get("CONTROLAI_BACKEND", "mlx").strip().lower() not in (
-        "torch", "pytorch", "cuda",
+        "torch", "pytorch", "cuda", "api", "remote", "hosted",
     )
 
 
@@ -87,6 +87,10 @@ def _make_engine():
     registry, every tool -- is identical either way.
     """
     backend = os.environ.get("CONTROLAI_BACKEND", "mlx").strip().lower()
+    if backend in ("api", "remote", "hosted"):
+        from controlai_agent.engine_api import RemoteEngine
+
+        return RemoteEngine()
     # "pytorch" is what the old orchestrator called this and it survives as a
     # variable on the deployed Space; "cuda" is the obvious other guess.
     if backend in ("torch", "pytorch", "cuda"):
@@ -100,7 +104,7 @@ def _make_engine():
         # MLX and died in an import several frames deeper than the real cause.
         raise ValueError(
             f"CONTROLAI_BACKEND={backend!r} is not a known backend "
-            f"(expected one of: mlx, torch/pytorch/cuda)"
+            f"(expected one of: mlx, torch/pytorch/cuda, api/remote/hosted)"
         )
     return None  # ControlAgent's own default, LocalEngine
 

app_space.py

@@ -1,90 +1,43 @@
-"""Hugging Face Space entry point: the same FastAPI app, on ZeroGPU.
-
-Not used for local runs -- `./run.sh` runs `app.py` directly. This module exists
-because the Space's platform imposes requirements that have nothing to do with
-ControlAI, and each of the four below was learned by having the Space fail:
-
-1.  A `@spaces.GPU`-decorated function is only recognised by the platform's
-    startup validation if it is wired to an actual Gradio event handler. A
-    function called only from a FastAPI route fails startup with "No @spaces.GPU
-    function detected". Hence the hidden probe button.
-2.  That function must be a module-level `def`. Nested inside a `with
-    gr.Blocks():` block, the same detection fails.
-3.  The probe must be `launch()`ed, on its own port, and **not** merely mounted.
-    Mounting it with `gr.mount_gradio_app` was tried: the Space fails to start
-    with "No @spaces.GPU function detected during startup", so mounting does not
-    register the handler the way the platform's validation looks for. Do not
-    mount it into the FastAPI app and *also* run uvicorn on the same port,
-    though -- that is two servers and collides with "address already in use".
-    Launch with `ssr_mode=False`: Gradio 6 defaults to SSR, which spawns a Node
-    subprocess into the very process ZeroGPU then `fork()`s its CUDA worker
-    from, and a fork parent holding a subprocess is a plausible cause of the
-    worker's "No CUDA GPUs are available".
-4.  Never pass the model-holding object as an *argument* to a `@spaces.GPU`
-    function. ZeroGPU marshals arguments across a process boundary and will try
-    to share the model's CUDA tensors, failing with `_share_cuda_: only
-    available on CUDA` after producing no output at all -- which reads exactly
-    like "just slow". Reach the agent through the module-level global instead.
-
-The retrieval index is not in the repo (see CLAUDE.md), so it is pulled from the
-private Hub dataset repo at startup using the Space's HF_TOKEN secret. Without
-it the retriever has no corpus and every answer falls back to model knowledge.
+"""Hugging Face Space entry point: FastAPI, CPU hardware, hosted generation.
+
+`./run.sh` runs `app.py` directly; this exists only for the Space.
+
+**This deliberately does not use ZeroGPU.** Seven separate incompatibilities came
+out of trying to, and the last one is structural rather than a bug: ZeroGPU is
+built around "the Gradio app *is* the Space", scheduling GPU workers by forking
+the Gradio server process, while ControlAI is a FastAPI app with its own console
+that merely borrows the process. The findings are catalogued in CLAUDE.md so the
+attempt is not repeated blind, and `engine_torch.py` is kept for anyone running
+this on an ordinary CUDA box, where all of it works.
+
+What runs where: everything that makes ControlAI what it is stays in-process on
+the Space's CPU -- all 29 deterministic solvers, the verifier, the full
+80,370-chunk hybrid retriever. Those cost milliseconds. Only token generation
+leaves, to a GPU behind an API, via `RemoteEngine`.
+
+The Space needs `HF_TOKEN` as a secret, with **two** permissions: read access to
+the private dataset repo holding the index, and "Make calls to Inference
+Providers". Without the first, retrieval is silently disabled; without the
+second, generation fails with 403.
 """
 
 from __future__ import annotations
 
 import os
-from typing import Iterator
 
-# Assigned, not setdefault: this module is the CUDA entry point by definition,
-# and a stale Space variable must not be able to select something else. One did
-# -- CONTROLAI_BACKEND=pytorch, left over from the old orchestrator -- and the
-# Space booted into the MLX engine and died on `import mlx_lm`.
-os.environ["CONTROLAI_BACKEND"] = "torch"
-# Qwen3-14B in 4-bit NF4. Override with CONTROLAI_MODEL_TORCH in Space settings.
+# Must precede the `app` import: it decides which engine gets built. Assigned,
+# not setdefault-ed, because a stale Space variable must not select something
+# else -- CONTROLAI_BACKEND=pytorch, left from the old orchestrator, once did.
+os.environ["CONTROLAI_BACKEND"] = "api"
 
-import gradio as gr
-import spaces
 import uvicorn
 
 import app as app_module
 from app import app
 
-# Filled in at import, before and after the model load, so the parent's CUDA
-# state can be compared against the worker's. See /api/gpudiag.
-_parent_state: dict = {}
-
-
-def _torch_state(label: str) -> dict:
-    import os as _os
-
-    import torch
-
-    state = {
-        "where": label,
-        "torch": torch.__version__,
-        "torch.version.cuda": torch.version.cuda,
-        "CUDA_VISIBLE_DEVICES": _os.environ.get("CUDA_VISIBLE_DEVICES"),
-        "ZERO_GPU_PATCH_TORCH": _os.environ.get("ZERO_GPU_PATCH_TORCH"),
-        "pid": _os.getpid(),
-    }
-    # is_initialized() is the one that matters: if the parent has really
-    # initialised CUDA before ZeroGPU forks its worker, the child cannot use the
-    # GPU and reports "No CUDA GPUs are available" -- which is our exact error.
-    for name, fn in (
-        ("cuda.is_initialized", lambda: torch.cuda.is_initialized()),
-        ("cuda.is_available", lambda: torch.cuda.is_available()),
-        ("cuda.device_count", lambda: torch.cuda.device_count()),
-    ):
-        try:
-            state[name] = fn()
-        except Exception as exc:  # noqa: BLE001 - the message is the datum
-            state[name] = f"{type(exc).__name__}: {exc}"
-    return state
-
 
 def _fetch_index() -> None:
-    """Pull the retrieval index into data/rag_index/ before the agent loads."""
+    """Pull the retrieval index in before the agent starts."""
     from controlai_rag.fetch_index import fetch
 
     if not os.environ.get("HF_TOKEN"):
@@ -97,140 +50,10 @@ def _fetch_index() -> None:
         print(f"[space] could not fetch the index ({exc}); retrieval disabled")
 
 
-def _build_agent_at_import() -> None:
-    """Construct the agent *now*, while this module is still being imported.
-
-    ZeroGPU installs its CUDA emulation by patching torch during the import of
-    the Space's entry module, and only operations that happen inside that window
-    are intercepted. `app.py` normally builds the model in FastAPI's `lifespan`,
-    on a ThreadPoolExecutor worker -- long after import, on another thread, where
-    the patching does not apply. `.to("cuda")` there reaches the real
-    `torch._C._cuda_init` and raises:
-
-        RuntimeError: Low-level CUDA init (`torch._C._cuda_init`) reached.
-
-    Building here and handing the finished agent to `app.py` keeps the load
-    inside the window. `lifespan` then finds `_agent` already set and its
-    `get_agent()` is a no-op.
-    """
-    from controlai_agent.agent import ControlAgent
-    from controlai_agent.engine_torch import TorchEngine
-    from controlai_rag.embeddings import get_embedder
-
-    _parent_state["before_model_load"] = _torch_state("parent-before-load")
-    print("[space] building agent at import scope (ZeroGPU CUDA window)")
-    app_module._agent = ControlAgent(engine=TorchEngine())
-
-    # The retrieval embedder is a second model, and it loads lazily on first
-    # query -- which is a request, outside the import window, so it hit exactly
-    # the same _cuda_init wall and every answer came back with
-    # "[agent] retrieval failed". Force it onto the GPU here too. Embedding one
-    # throwaway string is what actually triggers the load.
-    get_embedder().encode_query("warmup")
-    # Route every turn through the GPU-decorated generator above.
-    app_module.stream_hook = _gpu_stream
-    _parent_state["after_model_load"] = _torch_state("parent-after-load")
-    print(f"[space] parent CUDA state after load: {_parent_state['after_model_load']}")
-    print("[space] agent and embedder ready, GPU stream hook installed")
-
-
-# One allocation per user turn, not per engine call. A turn is up to
-# MAX_TOOL_STEPS generations plus the tool executions between them, and they all
-# mutate the same KV cache; splitting them across separate @spaces.GPU calls
-# would put that shared state on the far side of a process boundary each time.
-# 300s is the ceiling for a turn that actually calls two tools.
-@spaces.GPU(duration=300)
-def _gpu_stream(message: str, history: list) -> Iterator[dict]:
-    """Run one whole agent turn with real hardware attached.
-
-    Everything about ControlAI's inference has to happen inside a call like this
-    one. ZeroGPU creates the model under CUDA *emulation* at import and packs its
-    tensors; only a @spaces.GPU call materialises them on a real device. Running
-    the forward passes anywhere else does not fail loudly -- it reads
-    unmaterialised tensors and returns fluent-looking multilingual noise at a few
-    minutes per turn, which is exactly what the Space did before this existed.
-
-    `message` and `history` are plain data. The agent is reached through the
-    module global and never passed in: ZeroGPU marshals arguments across a
-    process boundary and would try to share the model's CUDA tensors, hanging
-    with no output at all.
-    """
-    yield from app_module.get_agent().stream(message, history)
-
-
-@spaces.GPU(duration=60)
-def _gpu_diagnostics() -> dict:
-    """Report CUDA state from inside the ZeroGPU worker, where it fails."""
-    import subprocess
-
-    state = _torch_state("gpu-worker")
-    try:
-        state["nvidia-smi"] = subprocess.run(
-            ["nvidia-smi", "--query-gpu=name,driver_version", "--format=csv,noheader"],
-            capture_output=True, text=True, timeout=20,
-        ).stdout.strip() or "(no output)"
-    except Exception as exc:  # noqa: BLE001
-        state["nvidia-smi"] = f"{type(exc).__name__}: {exc}"
-    return state
-
-
-@app.get("/api/gpudiag")
-def gpudiag() -> dict:
-    """Compare parent-process CUDA state with the @spaces.GPU worker's.
-
-    A plain `def`, so Starlette runs it on its own threadpool -- the same
-    context the real inference path uses.
-    """
-    try:
-        worker = _gpu_diagnostics()
-    except Exception as exc:  # noqa: BLE001
-        worker = {"error": f"{type(exc).__name__}: {exc}"}
-    return {
-        "parent_at_import": _parent_state,
-        "parent_now": _torch_state("parent-now"),
-        "worker": worker,
-    }
-
-
-@spaces.GPU(duration=60)
-def _gpu_probe(text: str) -> str:
-    """Satisfies ZeroGPU's startup validation.
-
-    A @spaces.GPU function is only detected if it is wired to a real Gradio event
-    handler, and _gpu_stream is driven by FastAPI rather than by Gradio, so it
-    cannot serve that purpose itself. This one exists to be wired to the hidden
-    button below.
-    """
-    return "ready"
-
-
-# Both run at import, in this order: the index first (the agent touches retrieval
-# as it starts), then the agent -- which installs _gpu_stream, so it has to come
-# after that function exists.
-_fetch_index()
-_build_agent_at_import()
-
-
-with gr.Blocks() as _gpu_demo:
-    _probe_in = gr.Textbox(visible=False)
-    _probe_out = gr.Textbox(visible=False)
-    _probe_btn = gr.Button("probe", visible=False)
-    _probe_btn.click(fn=_gpu_probe, inputs=_probe_in, outputs=_probe_out)
-
-
 def main() -> None:
-    port = int(os.environ.get("PORT", 7860))
-    # Launched, not mounted: mounting fails startup validation ("No @spaces.GPU
-    # function detected"). Separate port, non-blocking, and ssr_mode=False so no
-    # Node subprocess lands in the process ZeroGPU forks its CUDA worker from.
-    _gpu_demo.launch(
-        server_name="0.0.0.0",
-        server_port=port + 1,
-        prevent_thread_lock=True,
-        share=False,
-        ssr_mode=False,
-    )
-    uvicorn.run(app, host="0.0.0.0", port=port, log_level="info")
+    _fetch_index()
+    uvicorn.run(app, host="0.0.0.0", port=int(os.environ.get("PORT", 7860)),
+                log_level="info")
 
 
 if __name__ == "__main__":

controlai_agent/engine_api.py

@@ -0,0 +1,249 @@
+"""Remote inference engine: the same contract as `LocalEngine`, over an HTTP API.
+
+Why this exists: ZeroGPU is built around "the Gradio app *is* the Space", and
+ControlAI is a FastAPI app with its own console. Seven distinct incompatibilities
+came out of trying to bridge that (they are catalogued in CLAUDE.md); the last is
+structural rather than a bug. This sidesteps the whole category. The Space runs
+on plain CPU hardware and keeps everything that makes ControlAI what it is --
+all 29 deterministic solvers, the verifier, the 80,370-chunk hybrid retriever --
+running locally in-process, where they cost milliseconds. Only token generation
+leaves the machine, to a real GPU behind an API.
+
+That is a deliberate trade and it is not free: the demo is no longer
+self-contained, which is the local app's whole point. The Space page says so.
+Nothing about `engine.py` or the local Apple Silicon path changes.
+
+**The tool schemas go through the model's own chat template locally, and the
+result is handed over as an ordinary system message** -- never as a `tools=`
+argument. That matters: a provider given `tools=` applies its own template and
+returns structured `tool_calls` objects, which the agent does not speak. Doing it
+this way, the model emits `<tool_call>` as ordinary text and `toolcall.parse` and
+`_StreamGate` see the same input they see locally.
+
+Every provider serving Qwen3 offers `conversational` only, not raw
+text-generation, so `render()` returns a *message list* rather than a prompt
+string. The agent treats that value as opaque -- it renders and passes it
+straight to `stream()` -- so nothing downstream cares which it is.
+"""
+
+from __future__ import annotations
+
+import os
+import time
+from typing import Any, Generator, Iterable, Sequence
+
+from controlai_agent.engine import Chunk, SamplingConfig, Stats
+
+DEFAULT_API_MODEL = os.environ.get("CONTROLAI_MODEL_API", "Qwen/Qwen3-14B")
+# "auto" lets the Hub pick whichever provider currently serves the model.
+DEFAULT_PROVIDER = os.environ.get("CONTROLAI_PROVIDER", "auto")
+
+
+class RemoteEngine:
+    """Streaming generation against a hosted model, over huggingface_hub."""
+
+    def __init__(
+        self,
+        model_id: str = DEFAULT_API_MODEL,
+        adapter_path: str | None = None,
+        sampling: SamplingConfig | None = None,
+        provider: str = DEFAULT_PROVIDER,
+        token: str | None = None,
+    ) -> None:
+        from huggingface_hub import InferenceClient
+        from transformers import AutoTokenizer
+
+        if adapter_path:
+            raise ValueError(
+                "RemoteEngine cannot load a LoRA adapter: the weights are not local. "
+                "Serve a merged model, or use LocalEngine."
+            )
+
+        self.model_id = model_id
+        self.adapter_path = None
+        self.sampling = sampling or SamplingConfig()
+        self.provider = provider
+
+        t0 = time.time()
+        # Tokenizer only -- no weights. This is what renders the chat template and
+        # counts tokens for history truncation, so it must be the served model's.
+        self.tokenizer = AutoTokenizer.from_pretrained(model_id)
+        self.client = InferenceClient(
+            model=model_id,
+            provider=provider,
+            token=token or os.environ.get("HF_TOKEN"),
+            timeout=300,
+        )
+        self.load_seconds = time.time() - t0
+        self.last_stats = Stats()
+        self.supports_thinking = self._probe_thinking_support()
+        print(f"[remote] {model_id} via provider={provider} ({self.load_seconds:.1f}s)")
+
+    # ------------------------------------------------------------------ setup
+
+    def _probe_thinking_support(self) -> bool:
+        try:
+            self.tokenizer.apply_chat_template(
+                [{"role": "user", "content": "x"}],
+                tokenize=False,
+                add_generation_prompt=True,
+                enable_thinking=False,
+            )
+            return True
+        except (TypeError, ValueError):
+            return False
+
+    def render(
+        self,
+        messages: Sequence[dict[str, Any]],
+        tools: Sequence[dict[str, Any]] | None = None,
+        enable_thinking: bool = False,
+    ) -> list[dict[str, Any]]:
+        """Messages ready for a chat-completions call, tool block folded in.
+
+        The tool block is not hand-written: the model's own template renders it,
+        and it is lifted back out of the rendered system turn. That keeps the
+        wording identical to the local path even if the template changes.
+        """
+        out = [dict(m) for m in messages]
+        if tools:
+            rendered = self.tokenizer.apply_chat_template(
+                list(messages), tools=list(tools), tokenize=False, add_generation_prompt=True
+            )
+            head, tail = "<|im_start|>system\n", "<|im_end|>"
+            start = rendered.find(head)
+            end = rendered.find(tail, start) if start >= 0 else -1
+            if start >= 0 and end > start:
+                system = rendered[start + len(head):end]
+                out = [m for m in out if m.get("role") != "system"]
+                out.insert(0, {"role": "system", "content": system})
+
+        if not enable_thinking and self.supports_thinking:
+            # Qwen3's documented soft switch. `chat_template_kwargs` would be the
+            # direct equivalent, but it is an `extra_body` passthrough that not
+            # every provider forwards; this is in the prompt and always arrives.
+            for m in reversed(out):
+                if m.get("role") == "user":
+                    m["content"] = f"{m['content']} /no_think"
+                    break
+        return out
+
+    def encode(self, text: str) -> list[int]:
+        return self.tokenizer.encode(text)
+
+    def count_tokens(self, text: str) -> int:
+        return len(self.tokenizer.encode(text))
+
+    # ------------------------------------------------------------------ cache
+
+    def reset_cache(self) -> None:
+        """No-op. The KV cache lives on the provider's side, out of reach."""
+
+    def prewarm(self, text: str) -> int:
+        """No-op beyond reporting size.
+
+        `LocalEngine.prewarm` prefills the shared prefix into a persistent cache.
+        There is no cache to prefill here, and the provider's own prefix caching
+        is not ours to manage, so this only reports what the prefix costs.
+        """
+        return self.count_tokens(text)
+
+    # ------------------------------------------------------------- generation
+
+    def _request(self, messages: list[dict[str, Any]], cfg: SamplingConfig,
+                 stop: list[str], budget: int):
+        return self.client.chat_completion(
+            messages=messages,
+            stream=True,
+            max_tokens=max(budget, 1),
+            temperature=cfg.temperature if cfg.temperature > 0 else None,
+            top_p=cfg.top_p if 0 < cfg.top_p < 1 else None,
+            # Natively supported here, so this matches engine.py exactly rather
+            # than approximating it -- and a flat repetition_penalty stays out,
+            # for the reason engine.py gives.
+            presence_penalty=cfg.presence_penalty or None,
+            stop=stop or None,
+        )
+
+    def stream(
+        self,
+        prompt: str | list[int],
+        sampling: SamplingConfig | None = None,
+        stop: Iterable[str] = (),
+        think_budget: int | None = None,
+    ) -> Generator[Chunk, None, None]:
+        """Yield output chunks as they arrive. See `LocalEngine.stream`."""
+        cfg = sampling or self.sampling
+        convo = prompt if isinstance(prompt, list) and prompt and isinstance(prompt[0], dict) else [
+            {"role": "user", "content": prompt if isinstance(prompt, str) else self.tokenizer.decode(prompt)}
+        ]
+        stops = [s for s in stop if s]
+
+        t0 = time.time()
+        self.last_stats = Stats(
+            prompt_tokens=sum(self.count_tokens(str(m.get("content", ""))) for m in convo)
+        )
+        first_token_at: float | None = None
+
+        emitted = 0
+        thinking = False
+        think_tokens = 0
+        seen = ""
+        # Runs at most twice: once normally, and again if the reasoning budget
+        # was overrun. `LocalEngine` closes an overrunning <think> block by
+        # injecting the closing token mid-stream; there is no mid-stream here, so
+        # the equivalent is to stop, hand back what was generated with `</think>`
+        # appended, and let a continuation produce the answer. Same transcript,
+        # one extra round trip.
+        for attempt in (0, 1):
+            overran = False
+            for event in self._request(convo, cfg, stops, cfg.max_tokens - emitted):
+                try:
+                    piece = event.choices[0].delta.content
+                except (AttributeError, IndexError, TypeError):
+                    piece = None
+                if not piece:
+                    continue
+                if first_token_at is None:
+                    first_token_at = time.time()
+                    self.last_stats.prefill_seconds = first_token_at - t0
+
+                seen += piece
+                if "<think>" in piece:
+                    thinking, think_tokens = True, 0
+                elif "</think>" in piece:
+                    thinking = False
+                elif thinking:
+                    think_tokens += 1
+
+                emitted += 1
+                yield Chunk(
+                    text=piece,
+                    token=-1,  # the API returns text, not ids
+                    thinking=thinking,
+                    tool_call="<tool_call>" in piece,
+                )
+
+                if thinking and think_budget and think_tokens >= think_budget:
+                    overran = True
+                    break
+                if emitted >= cfg.max_tokens:
+                    break
+
+            if not (overran and attempt == 0):
+                break
+            yield Chunk(text="</think>", token=-1, thinking=False)
+            convo = convo + [{"role": "assistant", "content": seen + "</think>"}]
+            thinking = False
+
+        self.last_stats.generated_tokens = emitted
+        self.last_stats.decode_seconds = time.time() - (first_token_at or t0)
+
+    def generate(
+        self,
+        prompt: str | list[int],
+        sampling: SamplingConfig | None = None,
+        stop: Iterable[str] = (),
+        think_budget: int | None = None,
+    ) -> str:
+        return "".join(c.text for c in self.stream(prompt, sampling, stop, think_budget))

controlai_rag/embeddings.py

@@ -44,7 +44,12 @@ class Embedder:
         # retrieval on the Space looks over- or under-eager, MIN_COSINE is the
         # knob (CONTROLAI_MIN_COSINE), not this.
         self._backend = (backend or os.environ.get("CONTROLAI_BACKEND", "mlx")).lower()
-        if self._backend != "torch":
+        # Anything that is not MLX embeds through transformers. On the Space that
+        # is CPU torch, which is ample for one query at a time; "api" refers to
+        # where *generation* happens, and says nothing about the embedder.
+        if self._backend in ("torch", "pytorch", "cuda", "api", "remote", "hosted"):
+            self._backend = "torch"
+        else:
             self._backend = "mlx"
         self.model_id = model_id or (TORCH_MODEL_ID if self._backend == "torch" else MODEL_ID)
         self._model = None

requirements-space.txt

@@ -1,39 +1,38 @@
-# Dependencies for the Hugging Face Space only -- see app_space.py.
+# Hugging Face Space dependencies -- see app_space.py.
 #
-# The Space runs on Linux/NVIDIA, where MLX does not exist, so inference goes
-# through controlai_agent/engine_torch.py instead. Do NOT install this file
-# locally: requirements.txt is the Apple Silicon runtime and is MLX-only.
+# The Space runs on plain CPU hardware. Token generation is remote (RemoteEngine
+# -> Inference Providers), so there is no CUDA and no local LLM here. torch is
+# still needed, on CPU, for the 0.6B retrieval embedder: one query is a few
+# hundred milliseconds, which is fine.
+#
+# Do NOT install this locally: requirements.txt is the Apple Silicon MLX runtime.
 
-# Deliberately unpinned above the floor. The Space platform force-installs its
-# own gradio, spaces, uvicorn and a torch ceiling on top of this file, and any
-# upper bound here can make the resolve impossible -- pinning transformers <4.56
-# did exactly that, because gradio 6.x needs huggingface-hub >=1.16 and every
-# transformers <4.56 needs <1.0.
-# Left unconstrained deliberately. Pinning <2.9 to force CUDA 12 wheels was
-# tried, on the theory that torch 2.11's CUDA 13 wheels were behind ZeroGPU's
-# "No CUDA GPUs are available". It was not, and the pin was strictly worse:
-# unpinned, the log reaches "Waiting for a GPU" then "Successfully acquired a
-# GPU" before the worker dies (14.2s); pinned, neither line appears at all and
-# it fails in 1.8s, never reaching GPU acquisition. Do not retry that bound.
+# CPU wheels only. The default PyPI torch wheel drags several GB of CUDA
+# libraries that would never be used here.
+--extra-index-url https://download.pytorch.org/whl/cpu
 torch>=2.4.0
-transformers>=4.51.0
-accelerate>=0.30.0
-bitsandbytes>=0.43.0          # 4-bit NF4; CUDA-only
-sentencepiece>=0.2.0
 
-# gradio, spaces and huggingface-hub are injected by the platform at build
-# time; listing them here only adds constraints that can conflict with it.
+# No upper bound: the platform appends its own gradio/spaces/uvicorn on top of
+# this file, and one extra constraint can make the resolve impossible. Pinning
+# transformers<4.56 did exactly that once -- gradio 6 needs huggingface-hub>=1.16
+# and every transformers<4.56 needs <1.0.
+transformers>=4.51.0
+huggingface-hub>=0.34.0        # InferenceClient.chat_completion
 
-# Everything below is shared with requirements.txt (minus mlx-lm).
+# Deterministic numerics -- every number in an answer comes from these, locally.
 numpy>=1.24.0
 scipy>=1.11.0
 control>=0.9.4
 cvxpy>=1.4.0
 matplotlib>=3.7.0
 jsonschema>=4.20.0
+
+# Retrieval over the local control-engineering library.
 rank-bm25>=0.2.2
 pypdf>=3.17.0
 pymupdf>=1.23.0
+
+# Web console.
 fastapi>=0.110.0
 uvicorn>=0.28.0
 python-multipart>=0.0.9