WINTER4000 commited on
Commit
7efcef4
·
verified ·
1 Parent(s): d1bcdde

Halve @spaces.GPU duration 60s → 30s to save quota per call

Browse files
Files changed (1) hide show
  1. llm.py +8 -5
llm.py CHANGED
@@ -237,14 +237,17 @@ def format_mistral_prompt(
237
 
238
  # ──────────────────────────────────────────────────────────── inference
239
 
240
- @spaces.GPU(duration=60)
241
  def generate(prompt: str, **gen_kwargs) -> str:
242
  """Run one generation pass and return only the newly-generated text.
243
 
244
- Decorated with @spaces.GPU so ZeroGPU acquires the A100 for the
245
- duration of this call. The 60-second budget covers up to ~1500
246
- tokens at ~25 tok/s; tune via gen_kwargs['max_new_tokens'] if you
247
- need longer answers.
 
 
 
248
 
249
  On a local CPU host the decorator is a no-op (see the spaces shim
250
  at the top of this file) and the call runs on whatever device the
 
237
 
238
  # ──────────────────────────────────────────────────────────── inference
239
 
240
+ @spaces.GPU(duration=30)
241
  def generate(prompt: str, **gen_kwargs) -> str:
242
  """Run one generation pass and return only the newly-generated text.
243
 
244
+ Decorated with @spaces.GPU(duration=30) so ZeroGPU acquires the A100
245
+ for up to 30 seconds. ZeroGPU bills the FULL declared duration
246
+ against the caller's daily quota regardless of how long the actual
247
+ call took — so a 60s budget for a 5s response wastes 55s of quota.
248
+ Most chat answers complete in 5-15s; 30s is comfortably above that
249
+ while halving the quota cost vs. the old 60s. If responses ever
250
+ start truncating because they hit the timeout, bump back up.
251
 
252
  On a local CPU host the decorator is a no-op (see the spaces shim
253
  at the top of this file) and the call runs on whatever device the