Fix multi-minute UI latency: disable reasoning, preload model, kill hidden task calls

Browser-measured TTFT through Open WebUI went from ~210s to ~2.9s:

- llama-swap macro: --reasoning off (unbounded thinking ate the whole
budget with zero visible output; budget 0 leaks thinking prose into
content), -n 1024 output cap, --cache-reuse 256 (multi-turn prompt
cache, ~70% less prompt cost on follow-ups).
- Preload DEFAULT_MODEL at startup (hooks.on_startup) and raise ttl
600->86400 so the first message never pays the ~60s cold load.
- Open WebUI: disable title/tags/follow-up/autocomplete/query
generation - each fired an extra LLM call that queued the real chat
on the single CPU slot. NB these are PersistentConfig: env vars are
ignored once values exist in webui.db (fixed live via
/api/v1/tasks/config/update; envs kept for fresh installs).
- README: measured perf on the 4c/16GB box + capacity planning
estimates for 8c/16GB and 16c/64GB tiers (speed, TTFT, RPS, RAM
budgets, parallel-slot notes).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

README.md

@@ -21,7 +21,8 @@ Open WebUI ──┬─► llama-swap ─► llama-server (Gemma 4 omni + BF16 m
 
 Requests flow **User → Open WebUI → {llama-swap, tei-embed, tei-rerank}**; llama-swap in turn
 spawns `llama-server` on demand for whichever Gemma 4 profile is selected (only one resident at a
-time, unloaded on idle-ttl to keep RAM flat). The audio Pipe bypasses Open WebUI's built-in STT
+time; the profile's `DEFAULT_MODEL` is preloaded at startup and kept warm for 24 h so the first
+message never pays the ~60 s cold load — switching models still swaps on demand). The audio Pipe bypasses Open WebUI's built-in STT
 and talks to llama-swap directly so Gemma hears raw audio natively. Dotted arrows are read/write
 volume mounts, not network calls.
 
@@ -90,8 +91,6 @@ docker compose up -d                  # boots TEI + Open WebUI + llama-swap
 ```
 Open <http://localhost:3000>, create the first (admin) account.
 
-Open <http://localhost:3000>, create the first (admin) account.
-
 **Default Open WebUI login:**
 - **Email:** `UI@gmail.com`
 - **Password:** `G4H!dg!R!4EdjkR`
@@ -149,6 +148,66 @@ B64=$(base64 -w0 clip.wav); curl localhost:8080/v1/chat/completions -H 'Content-
 to sweep `--threads`. Set the winner as `THREADS` in `.env`. Long context? add KV quant
 (`-ctk q8_0 -ctv q8_0`) to the macro in `llama-swap/config.tmpl.yaml`.
 
+## Performance & capacity planning (CPU serving)
+
+**Measured** on the 4-core / 16 GB dev box (Xeon 2.6 GHz, E4B Q4_K_M, warm model,
+via the real Open WebUI through the public proxy — not just raw llama-bench):
+
+| Metric (E4B, warm) | 4c / 16 GB — measured |
+|---|---|
+| TTFT in the browser, short prompt | **~3 s** |
+| Generation speed | **~3.4–4.0 tok/s** |
+| Prompt processing | **~8–15 tok/s** |
+| ~150-token chat answer, end to end | **~45–70 s** |
+| Follow-up turn (prompt cache hit) | prompt cost ≈ **–70 %** |
+| TEI embed / rerank (per request) | **0.7 s / 1.7 s** |
+
+**Back-of-envelope formula** (holds well in practice):
+
+```
+TTFT   ≈ prompt_tokens / pp_speed          (+ ~0.5 s overhead)
+total  ≈ TTFT + output_tokens / gen_speed
+RPS    ≈ parallel_slots / total            (sequential queue otherwise)
+```
+
+**Estimates for bigger CPU tiers** (prompt processing is compute-bound → scales ~linearly
+with physical cores; token generation is memory-bandwidth-bound → scales sublinearly,
+roughly ×1.7 at 8c and ×2.5–3 at 16c vs this box). Assumes a ~300-token prompt and
+~150-token answer per request:
+
+| | 4c / 16 GB (this box) | 8c / 16 GB (est.) | 16c / 64 GB (est.) |
+|---|---|---|---|
+| **E4B** gen / pp (tok/s) | 3.5–4 / 8–15 ✓ | ~6–7 / ~25 | ~9–12 / ~50 |
+| **E4B** TTFT / total per chat | ~3 s / ~60 s ✓ | ~1.5 s / ~35 s | ~1 s / ~20 s |
+| **E4B** chat throughput | ~1 req/min (RPS ≈ 0.015) | ~1.7 req/min (0.03) | ~3 req/min (0.05) |
+| **12B** gen / pp (tok/s) | ~1.2–1.5 / ~4 (RAM-tight) | ~2–2.5 / ~9 (tight) | ~4–5 / ~20 |
+| **12B** chat total / throughput | not recommended | ~2 min/req | ~55 s/req, ~1 req/min |
+| **26B-A4B** (MoE, ~18 GB) | ✗ no RAM | ✗ no RAM | ~7–9 / ~40; ~30 s/req |
+| Comfortable interactive users | 1 | 1–2 | 2–4 |
+
+Practical read of that table:
+
+- **16 GB RAM caps you at E4B** as the daily driver (12B fits but leaves TEI+UI almost no
+  headroom). **The 16c / 64 GB tier is the first one where 12B and 26B-A4B are pleasant.**
+- **Concurrency**: `llama-server` runs **one inference slot** by default — a second chat
+  *queues*, it doesn't run alongside. On the 16c/64 GB tier add `--parallel 2` (and note each
+  slot gets `CTX / N` context) to serve 2 streams; aggregate tok/s barely changes
+  (bandwidth-bound), it just shares fairly instead of queueing.
+- **CPU serving is single-digit-users territory.** For ~10+ concurrent chat users at
+  interactive speeds, use the GPU profile — no CPU tier gets you there.
+- **RAM budget** (Q4_K_M + BF16 mmproj + 8k ctx): E4B ≈ 5 GB · 12B ≈ 10 GB · 26B-A4B ≈ 18 GB;
+  plus TEI embed+rerank ≈ 4 GB, Open WebUI ≈ 0.7 GB, OS ≈ 1 GB.
+- Embedding RAG load is cheap next to chat: TEI on this box sustains ≈ 1.4 embeds/s and
+  ≈ 0.6 reranks/s per core-set — RAG retrieval adds ~2–3 s per message, not minutes.
+
+**Latency defaults already baked into this stack** (the difference between these numbers and
+minutes-long hangs): `--reasoning off` (Gemma 4 otherwise burns the whole budget thinking),
+`-n 1024` output cap, `--cache-reuse 256` (multi-turn prompt cache), startup **preload** of
+`DEFAULT_MODEL` + 24 h idle-ttl (no cold load on first message), and Open WebUI background
+task generation (title/tags/follow-up/query) **disabled** — note those are PersistentConfig:
+once in `webui.db`, env vars are ignored; change them in Admin Settings → Interface or via
+`POST /api/v1/tasks/config/update`.
+
 ## Troubleshooting (the known traps)
 - **Audio crashes / asserts** → keep `-b 2048 -ub 2048` (already in the macro). Default 512 crashes.
 - **Audio is garbage/repetitive** → the mmproj must be **BF16** (we pin it). Q8_0/F16 break audio.
@@ -163,9 +222,11 @@ to sweep `--threads`. Set the winner as `THREADS` in `.env`. Long context? add K
   `RAG_RERANKING_ENGINE=external` + URL envs and set `RAG_RERANKING_MODEL=BAAI/bge-reranker-v2-m3`.
 
 ## Operational notes (validated live on a 15 GB CPU box)
-- **Gemma 4 is a reasoning model** — it emits `reasoning_content` before `content`. Give it a
-  generous `max_tokens` (≥256) or short prompts will return empty `content` (all budget spent
-  thinking). The Pipe/UI handle this; raw API callers should read both fields.
+- **Gemma 4 is a reasoning model** — by default it emits `reasoning_content` before `content`
+  and, unbounded, can think for *minutes* before a visible answer. The macro therefore ships
+  `--reasoning off` (clean direct answers). If you re-enable it, prefer `--reasoning-budget N`
+  over unbounded, give `max_tokens` ≥ 256, and read both fields in raw API callers
+  (`--reasoning-budget 0` leaks "Thinking Process:" prose into `content` — use `off`).
 - **Send media via a file, not inline** — base64 audio/large images exceed the shell arg limit
   (`argument list too long`). Use `curl -d @payload.json` (the e2e script and Pipe already do).
 - **TEI memory on CPU**: the ONNX backend is heavy (bge-m3 ≈ 4.75 GB). Setting `TEI_DTYPE=float16`

docker-compose.yml

@@ -32,6 +32,7 @@ services:
       NGL: ${NGL:-0}              # GPU layers to offload (0 = pure CPU)
       THREADS: ${THREADS:-4}      # physical cores, not hyperthreads
       CTX: ${CTX:-8192}           # context window
+      DEFAULT_MODEL: ${DEFAULT_MODEL:-gemma-e4b}  # preloaded at startup (no cold first message)
       HF_TOKEN: ${HF_TOKEN:-}     # for gated Gemma 4 pulls via -hf
       HF_HOME: /models/hf-cache
       LLAMA_CACHE: /models/llama-cache   # persist GGUFs fetched by -hf across restarts
@@ -131,6 +132,13 @@ services:
       RAG_TOP_K_RERANKER: "4"
       # --- native audio is the Pipe's job; turn STT off ----------------------
       AUDIO_STT_ENGINE: ""
+      # --- kill hidden background LLM calls (title/tags/follow-up/autocomplete
+      #     each fire an extra chat completion on the same slow CPU backend) ---
+      ENABLE_TITLE_GENERATION: "false"
+      ENABLE_TAGS_GENERATION: "false"
+      ENABLE_FOLLOW_UP_GENERATION: "false"
+      ENABLE_AUTOCOMPLETE_GENERATION: "false"
+      ENABLE_RETRIEVAL_QUERY_GENERATION: "false"
       WEBUI_NAME: ${WEBUI_NAME:-Gemma 4 Local}
     volumes:
       - ./openwebui/data:/app/backend/data

llama-swap/config.tmpl.yaml

@@ -16,12 +16,21 @@
 healthCheckTimeout: 300
 logLevel: info
 
+# Warm the profile's default model as soon as the stack starts, so the first
+# chat message doesn't pay the ~60s cold-load.
+hooks:
+  on_startup:
+    preload:
+      - "${DEFAULT_MODEL}"
+
 macros:
   base: >-
     llama-server --host 0.0.0.0 --port ${PORT}
     --jinja --no-mmproj-offload -ngl ${NGL}
     --threads ${THREADS} -b 2048 -ub 2048
     --temp 1.0 --top-k 64 --top-p 0.95
+    --reasoning off -n 1024
+    --cache-reuse 256
     --metrics
 
 models:
@@ -32,7 +41,7 @@ models:
       -m /models/gemma-4-E4B-it-Q4_K_M.gguf
       --mmproj /models/mmproj-gemma-4-E4B-it-bf16.gguf
       -c ${CTX}
-    ttl: 600
+    ttl: 86400
 
   # ---- 12B: audio + image + text (server default, ~10-12 GB) ----------------
   "gemma-12b":
@@ -41,7 +50,7 @@ models:
       -hf ggml-org/gemma-4-12B-it-GGUF:Q4_K_M
       --mmproj /models/mmproj-gemma-4-12B-it-bf16.gguf
       -c ${CTX}
-    ttl: 600
+    ttl: 86400
 
   # ---- 26B-A4B MoE: image + text ONLY, no audio (~16-18 GB) -----------------
   "gemma-26b-a4b":
@@ -50,4 +59,4 @@ models:
       -hf ggml-org/gemma-4-26B-A4B-it-GGUF:Q4_K_M
       --mmproj /models/mmproj-gemma-4-26B-A4B-it-bf16.gguf
       -c ${CTX}
-    ttl: 300
+    ttl: 86400

llama-swap/entrypoint.sh

@@ -7,6 +7,7 @@ set -eu
 NGL="${NGL:-0}"
 THREADS="${THREADS:-4}"
 CTX="${CTX:-8192}"
+DEFAULT_MODEL="${DEFAULT_MODEL:-gemma-e4b}"
 
 TMPL="/config/config.tmpl.yaml"
 OUT="/tmp/config.yaml"
@@ -15,6 +16,7 @@ sed \
   -e "s|\${NGL}|${NGL}|g" \
   -e "s|\${THREADS}|${THREADS}|g" \
   -e "s|\${CTX}|${CTX}|g" \
+  -e "s|\${DEFAULT_MODEL}|${DEFAULT_MODEL}|g" \
   "$TMPL" > "$OUT"
 
 echo "===================================================================="