docs/archive/gpu-inference-context.md

DenseFeed GPU Inference Problem Context

Current Setup

• Daily tech news podcast pipeline: fetch → rank → extract → script → synthesize → publish
• Deployed on Hugging Face Spaces (Gradio SDK) with ZeroGPU
• LLM stages: rank (BAML filter+score on ~30 items) and script (two-host dialogue generation)
• ZeroGPU backend: @spaces.GPU(duration=120) decorator, 120-second GPU leases
• Model: google/gemma-4-12b-it (bfloat16, loaded via transformers)
• Budget-aware lease manager: stopwatch-based, 25s safety margin, batch processing within lease
• Graceful CPU fallback when GPU lease fails
• BAML for structured prompts (filter, score, script with streaming)
• Gradio app with ThreadPoolExecutor for background pipeline runs

ZeroGPU Problems

1. Timeouts: 120s GPU lease is too short for full pipeline stages (ranking + script gen)
2. Queue slowness: ZeroGPU queue has unpredictable wait times; cold starts are brutal
3. Lease execution failures: GPU leases fail frequently, triggering CPU fallback (slow + degraded quality)
4. Pipeline can't complete: Stages chain together; if one stage's GPU lease expires mid-work, the whole pipeline stalls
5. No persistence: Each cold start reloads model weights (~30s+), burning precious lease time
6. Unpredictable latency: Same request can take 10s or 120s depending on queue position

Requirements

• Must work within HF Spaces ecosystem (or migrate to something better)
• Need reliable GPU for LLM inference (gemma-4-12b or similar)
• Pipeline stages are sequential: rank completes → script starts
• Total LLM inference time per run: ~3-5 minutes of actual GPU compute
• Free or very cheap (this is a side project, not enterprise)
• Minimal ops burden — set it and forget it

Pipeline Compute Needs

• Rank stage: 2 LLM passes × 30 items = ~60 short completions (50-200 tokens each)
• Script stage: 1 outline + 10-15 segment generations (500-1000 tokens each)
• Model size: 12B params in bfloat16 ≈ 24GB VRAM
• Total per run: ~3-5 min GPU compute, runs once daily