AdithyaSK HF Staff commited on
Commit
cd5c830
·
verified ·
1 Parent(s): e44a877

Serve Crello corpus aafdb0e86816 from the mounted bucket

Browse files
Files changed (2) hide show
  1. README.md +14 -0
  2. server/router.py +3 -1
README.md CHANGED
@@ -92,6 +92,20 @@ Measured locally (`scripts/loadtest.py`, 4 workers, no errors):
92
 
93
  The router itself saturates one core at about 300–450 tool calls per second, so 4–6 workers is the useful range.
94
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
95
  ## Talk to it
96
 
97
  ```python
 
92
 
93
  The router itself saturates one core at about 300–450 tool calls per second, so 4–6 workers is the useful range.
94
 
95
+ On a private Space (`scripts/deploy_space.py`: cpu-upgrade, 8 vCPU / 32 GB, 4 workers × 48 sessions, the full
96
+ corpus read over HTTP), tested from India, where the network round trip is about 0.22 s:
97
+
98
+ | Load | Result | Throughput | Step p50 / p95 |
99
+ |---|---|---|---|
100
+ | Tools, 192 concurrent | 575/576 ok | 643 episodes/min, 110 calls/s | 0.26 / 0.8 s |
101
+ | Built-in GUI editor, 128 concurrent | 256/256 ok | 75 actions/s | 0.46 / 1.3 s |
102
+
103
+ - The one failure arrived at the capacity edge, just before a closing session was freed.
104
+ - At capacity, cold resets take 7–10 s; each new design is fetched once.
105
+ - More than 192 sessions are refused cleanly ("Server at capacity") after every worker has been tried.
106
+ - OpenEnv's client has no option for auth headers. `scripts/loadtest.py --hf-token-env HF_TOKEN` shows the two
107
+ wrappers a client needs to reach a private Space.
108
+
109
  ## Talk to it
110
 
111
  ```python
server/router.py CHANGED
@@ -39,8 +39,10 @@ from starlette.responses import JSONResponse, Response, StreamingResponse
39
  from starlette.routing import Route, WebSocketRoute
40
  from starlette.websockets import WebSocket, WebSocketDisconnect
41
 
 
 
42
  HOP = {"connection", "keep-alive", "proxy-authenticate", "proxy-authorization", "te", "trailers",
43
- "transfer-encoding", "upgrade", "host", "content-length"}
44
  EPISODE_ID = re.compile(rb'"episode_id"\s*:\s*"([A-Za-z0-9_.-]{1,64})"')
45
  STICKY_PREFIXES = ("/web", "/gradio_api", "/gradio", "/theme.css", "/assets", "/file=", "/upload", "/queue",
46
  "/config", "/info", "/heartbeat")
 
39
  from starlette.routing import Route, WebSocketRoute
40
  from starlette.websockets import WebSocket, WebSocketDisconnect
41
 
42
+ # hop-by-hop headers stay here. Host is forwarded: the playground (Gradio) builds its URLs from it, and with the
43
+ # worker's own address the browser is sent to 127.0.0.1 (an unstyled page on the Space)
44
  HOP = {"connection", "keep-alive", "proxy-authenticate", "proxy-authorization", "te", "trailers",
45
+ "transfer-encoding", "upgrade", "content-length"}
46
  EPISODE_ID = re.compile(rb'"episode_id"\s*:\s*"([A-Za-z0-9_.-]{1,64})"')
47
  STICKY_PREFIXES = ("/web", "/gradio_api", "/gradio", "/theme.css", "/assets", "/file=", "/upload", "/queue",
48
  "/config", "/info", "/heartbeat")