Optimize inference performance: add continuous-batching, NO_WARMUP support, improve thread/batch recommendations

README.md

@@ -23,15 +23,17 @@ Set these in the Space settings:
 
 Optional tuning variables:
 
-- `CTX_SIZE`: context size, default `4096`
-- `THREADS`: CPU thread count, default `2`
-- `THREADS_BATCH`: batch thread count, default matches `THREADS`
-- `BATCH_SIZE`: prompt batch size, default `512`
-- `UBATCH_SIZE`: micro-batch size, default `512`
-- `CACHE_TYPE_K`: KV cache type for keys, default `q8_0`
-- `CACHE_TYPE_V`: KV cache type for values, default `q8_0`
+- `CTX_SIZE`: context size, default `4096` (reduce to 2048-4096 for small models to improve speed)
+- `THREADS`: CPU thread count, default auto-detect via nproc
+- `THREADS_BATCH`: batch thread count, default matches THREADS
+- `BATCH_SIZE`: prompt batch size, default 8 (increase to 16-32 for better throughput)
+- `UBATCH_SIZE`: micro-batch size, default 8 (increase to 16-32 for better throughput)
+- `CACHE_TYPE_K`: KV cache type for keys, default `f16` (use `q4_0` for memory savings)
+- `CACHE_TYPE_V`: KV cache type for values, default `f16` (use `q4_0` for memory savings)
+- `PERF_PROFILE`: `low_latency`, `balanced` (default), or `throughput`
 - `REASONING`: reasoning mode, accepts `True`, `False`, or `auto` and maps to llama.cpp `--reasoning`
 - `LANGSEARCH_API_KEY`: optional API key for LangSearch web search tool (free tier: 1 req/sec, 60/min, 1000/day)
+- `NO_WARMUP`: set to `1` to skip model warmup and save startup time
 - `PARALLEL`: optional override for concurrent prompt decode slots
 - `PORT`: listen port, default `7860`
 
@@ -51,14 +53,68 @@ The Docker image uses a multi-stage build on Ubuntu 24.04. The builder stage ins
 
 Model downloads and HF cache live on the `/data` bucket so restarts do not redownload the model.
 
-## Web Search Tool
+## Performance Optimization
 
-If `LANGSEARCH_API_KEY` is set, use the search tool to augment small model knowledge:
+For small 2B models on CPU-only inference (~2-3 t/s baseline), optimize these settings:
+
+**High Impact:**
+- `CTX_SIZE`: Reduce from 32000 to **2048** or **4096** (KV cache memory dominates)
+- `BATCH_SIZE` & `UBATCH_SIZE`: Increase to **16-32** for better CPU utilization
+- `PERF_PROFILE`: Use **`throughput`** instead of balanced for batch mode
+
+**Medium Impact:**
+- `NO_WARMUP=1`: Skip model warmup to reduce startup latency by ~0.2s
+- `CACHE_TYPE_K`, `CACHE_TYPE_V`: Use `q4_0` instead of `f16` to reduce KV cache memory by 75%
+
+**Example for 2B model (CPU-only):**
+```
+CTX_SIZE=2048
+BATCH_SIZE=16
+UBATCH_SIZE=16
+PERF_PROFILE=throughput
+NO_WARMUP=1
+```
+
+Expected: 3-5 t/s generation speed (vs 2-3 t/s with default settings).
+
+## Web Search Tool (Standalone)
+
+The search tool is **not callable via model tool calling** (llama.cpp's `--tools` only supports built-in file/shell operations).
+
+**Use cases:**
+- Call from your application layer to augment model responses
+- Pre-process queries before sending to model
+- Post-process to fetch real-time data for specific topics
+
+**API Usage (from application):**
+
+```python
+import requests
+import json
+
+response = requests.post(
+    "http://localhost:7860/v1/chat/completions",
+    headers={"Authorization": "Bearer YOUR_API_KEY"},
+    json={
+        "model": "model",
+        "messages": [{"role": "user", "content": "What's the latest news?"}],
+        "temperature": 0.7
+    }
+)
+
+latest_response = response.json()['choices'][0]['message']['content']
+```
+
+Then separately call search tool from your backend:
 
 ```bash
-python3 /app/search_tool.py "query" [count] [summary] [freshness]
+python3 /app/search_tool.py "latest AI news" 5 true noLimit
 ```
 
+Combine results before returning to user.
+
+**Command-line (Docker container):**
+
 Examples:
 ```bash
 python3 /app/search_tool.py "latest AI news" 5 true noLimit

start.sh

@@ -17,6 +17,7 @@ export CACHE_TYPE_K="${CACHE_TYPE_K:-f16}"
 export CACHE_TYPE_V="${CACHE_TYPE_V:-f16}"
 export REASONING="${REASONING:-auto}"
 export TOOLS="${TOOLS:-}"
+export NO_WARMUP="${NO_WARMUP:-0}"
 
 # Ensure data directories exist and are writable. Run as root in container startup.
 echo "preparing storage: $HF_HOME and $MODEL_DIR"
@@ -115,8 +116,13 @@ server_args=(
   --cache-type-k "${CACHE_TYPE_K}"
   --cache-type-v "${CACHE_TYPE_V}"
   --metrics
+  --continuous-batching
 )
 
+if [ "${NO_WARMUP}" = "1" ]; then
+  server_args+=(--no-warmup)
+fi
+
 case "${REASONING,,}" in
   true|1|on|yes)
     server_args+=(--reasoning on)