# DenseFeed GPU Inference Problem Context

## Current Setup
- Daily tech news podcast pipeline: fetch → rank → extract → script → synthesize → publish
- Deployed on Hugging Face Spaces (Gradio SDK) with ZeroGPU
- LLM stages: rank (BAML filter+score on ~30 items) and script (two-host dialogue generation)
- ZeroGPU backend: `@spaces.GPU(duration=120)` decorator, 120-second GPU leases
- Model: google/gemma-4-12b-it (bfloat16, loaded via transformers)
- Budget-aware lease manager: stopwatch-based, 25s safety margin, batch processing within lease
- Graceful CPU fallback when GPU lease fails
- BAML for structured prompts (filter, score, script with streaming)
- Gradio app with ThreadPoolExecutor for background pipeline runs

## ZeroGPU Problems
1. **Timeouts**: 120s GPU lease is too short for full pipeline stages (ranking + script gen)
2. **Queue slowness**: ZeroGPU queue has unpredictable wait times; cold starts are brutal
3. **Lease execution failures**: GPU leases fail frequently, triggering CPU fallback (slow + degraded quality)
4. **Pipeline can't complete**: Stages chain together; if one stage's GPU lease expires mid-work, the whole pipeline stalls
5. **No persistence**: Each cold start reloads model weights (~30s+), burning precious lease time
6. **Unpredictable latency**: Same request can take 10s or 120s depending on queue position

## Requirements
- Must work within HF Spaces ecosystem (or migrate to something better)
- Need reliable GPU for LLM inference (gemma-4-12b or similar)
- Pipeline stages are sequential: rank completes → script starts
- Total LLM inference time per run: ~3-5 minutes of actual GPU compute
- Free or very cheap (this is a side project, not enterprise)
- Minimal ops burden — set it and forget it

## Pipeline Compute Needs
- **Rank stage**: 2 LLM passes × ~30 items = ~60 short completions (~50-200 tokens each)
- **Script stage**: 1 outline + ~10-15 segment generations (~500-1000 tokens each)
- **Model size**: 12B params in bfloat16 ≈ 24GB VRAM
- **Total per run**: ~3-5 min GPU compute, runs once daily