Halve @spaces.GPU duration 60s → 30s to save quota per call
Browse files
llm.py
CHANGED
|
@@ -237,14 +237,17 @@ def format_mistral_prompt(
|
|
| 237 |
|
| 238 |
# ──────────────────────────────────────────────────────────── inference
|
| 239 |
|
| 240 |
-
@spaces.GPU(duration=
|
| 241 |
def generate(prompt: str, **gen_kwargs) -> str:
|
| 242 |
"""Run one generation pass and return only the newly-generated text.
|
| 243 |
|
| 244 |
-
Decorated with @spaces.GPU so ZeroGPU acquires the A100
|
| 245 |
-
|
| 246 |
-
|
| 247 |
-
|
|
|
|
|
|
|
|
|
|
| 248 |
|
| 249 |
On a local CPU host the decorator is a no-op (see the spaces shim
|
| 250 |
at the top of this file) and the call runs on whatever device the
|
|
|
|
| 237 |
|
| 238 |
# ──────────────────────────────────────────────────────────── inference
|
| 239 |
|
| 240 |
+
@spaces.GPU(duration=30)
|
| 241 |
def generate(prompt: str, **gen_kwargs) -> str:
|
| 242 |
"""Run one generation pass and return only the newly-generated text.
|
| 243 |
|
| 244 |
+
Decorated with @spaces.GPU(duration=30) so ZeroGPU acquires the A100
|
| 245 |
+
for up to 30 seconds. ZeroGPU bills the FULL declared duration
|
| 246 |
+
against the caller's daily quota regardless of how long the actual
|
| 247 |
+
call took — so a 60s budget for a 5s response wastes 55s of quota.
|
| 248 |
+
Most chat answers complete in 5-15s; 30s is comfortably above that
|
| 249 |
+
while halving the quota cost vs. the old 60s. If responses ever
|
| 250 |
+
start truncating because they hit the timeout, bump back up.
|
| 251 |
|
| 252 |
On a local CPU host the decorator is a no-op (see the spaces shim
|
| 253 |
at the top of this file) and the call runs on whatever device the
|