Spaces:
Running on Zero
Running on Zero
atakan Claude Sonnet 5 commited on
Commit ·
f22161c
1
Parent(s): ad33173
perf: Warm up the model's KV cache during startup, not on the first user request
Browse filesThe system prompt + tool schemas prefix (~6.4k tokens) is identical on
every request and llama.cpp caches/reuses it automatically, but only
after it's been processed once. Without a warm-up, that first prefill
(50s+ on this machine, likely longer on Spaces CPU) landed on whichever
user happened to send the first message after a cold start, looking
like the app was unresponsive. Now it happens during startup instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
app.py
CHANGED
|
@@ -29,7 +29,12 @@ from controlai_rag.index import ControlRAGIndex
|
|
| 29 |
@asynccontextmanager
|
| 30 |
async def lifespan(app: FastAPI):
|
| 31 |
print("Pre-loading ControlAI Core Engine on startup...")
|
| 32 |
-
get_agent()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
print("ControlAI Core Engine is online and ready for traffic.")
|
| 34 |
yield
|
| 35 |
|
|
|
|
| 29 |
@asynccontextmanager
|
| 30 |
async def lifespan(app: FastAPI):
|
| 31 |
print("Pre-loading ControlAI Core Engine on startup...")
|
| 32 |
+
agent = get_agent()
|
| 33 |
+
print("Warming up KV cache for the system prompt + tool schemas...")
|
| 34 |
+
try:
|
| 35 |
+
agent.run("hi", verbose=False)
|
| 36 |
+
except Exception as exc:
|
| 37 |
+
print(f"Warning: warm-up generation failed (non-fatal): {exc}")
|
| 38 |
print("ControlAI Core Engine is online and ready for traffic.")
|
| 39 |
yield
|
| 40 |
|