Spaces:
Running on Zero
Running on Zero
atakan Claude Sonnet 5 commited on
Commit ·
c01ad45
1
Parent(s): f22161c
revert: Don't block app startup on a warm-up generation
Browse filesMeasured on Spaces' CPU tier, the ~6.4k-token warm-up prefill takes
much longer than on dev hardware and was blocking uvicorn from binding
at all, risking a platform startup timeout. Startup is fast again;
send one chat message after a fresh deploy or Space wake-up to warm
the KV cache manually instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
app.py
CHANGED
|
@@ -29,13 +29,12 @@ from controlai_rag.index import ControlRAGIndex
|
|
| 29 |
@asynccontextmanager
|
| 30 |
async def lifespan(app: FastAPI):
|
| 31 |
print("Pre-loading ControlAI Core Engine on startup...")
|
| 32 |
-
|
| 33 |
-
print("Warming up KV cache for the system prompt + tool schemas...")
|
| 34 |
-
try:
|
| 35 |
-
agent.run("hi", verbose=False)
|
| 36 |
-
except Exception as exc:
|
| 37 |
-
print(f"Warning: warm-up generation failed (non-fatal): {exc}")
|
| 38 |
print("ControlAI Core Engine is online and ready for traffic.")
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
yield
|
| 40 |
|
| 41 |
|
|
|
|
| 29 |
@asynccontextmanager
|
| 30 |
async def lifespan(app: FastAPI):
|
| 31 |
print("Pre-loading ControlAI Core Engine on startup...")
|
| 32 |
+
get_agent()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
print("ControlAI Core Engine is online and ready for traffic.")
|
| 34 |
+
# Note: the system prompt + tool schemas (~6.4k tokens) are only cached
|
| 35 |
+
# in llama.cpp after the first generation. Send one chat message after
|
| 36 |
+
# a fresh deploy or after the Space wakes from sleep to warm it up —
|
| 37 |
+
# every request after that reuses the cached prefix and is fast.
|
| 38 |
yield
|
| 39 |
|
| 40 |
|