atakan Claude Sonnet 5 commited on
Commit
f22161c
·
1 Parent(s): ad33173

perf: Warm up the model's KV cache during startup, not on the first user request

Browse files

The system prompt + tool schemas prefix (~6.4k tokens) is identical on
every request and llama.cpp caches/reuses it automatically, but only
after it's been processed once. Without a warm-up, that first prefill
(50s+ on this machine, likely longer on Spaces CPU) landed on whichever
user happened to send the first message after a cold start, looking
like the app was unresponsive. Now it happens during startup instead.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Files changed (1) hide show
  1. app.py +6 -1
app.py CHANGED
@@ -29,7 +29,12 @@ from controlai_rag.index import ControlRAGIndex
29
  @asynccontextmanager
30
  async def lifespan(app: FastAPI):
31
  print("Pre-loading ControlAI Core Engine on startup...")
32
- get_agent()
 
 
 
 
 
33
  print("ControlAI Core Engine is online and ready for traffic.")
34
  yield
35
 
 
29
  @asynccontextmanager
30
  async def lifespan(app: FastAPI):
31
  print("Pre-loading ControlAI Core Engine on startup...")
32
+ agent = get_agent()
33
+ print("Warming up KV cache for the system prompt + tool schemas...")
34
+ try:
35
+ agent.run("hi", verbose=False)
36
+ except Exception as exc:
37
+ print(f"Warning: warm-up generation failed (non-fatal): {exc}")
38
  print("ControlAI Core Engine is online and ready for traffic.")
39
  yield
40