| # DenseFeed GPU Inference Problem Context |
|
|
| ## Current Setup |
| - Daily tech news podcast pipeline: fetch → rank → extract → script → synthesize → publish |
| - Deployed on Hugging Face Spaces (Gradio SDK) with ZeroGPU |
| - LLM stages: rank (BAML filter+score on ~30 items) and script (two-host dialogue generation) |
| - ZeroGPU backend: `@spaces.GPU(duration=120)` decorator, 120-second GPU leases |
| - Model: google/gemma-4-12b-it (bfloat16, loaded via transformers) |
| - Budget-aware lease manager: stopwatch-based, 25s safety margin, batch processing within lease |
| - Graceful CPU fallback when GPU lease fails |
| - BAML for structured prompts (filter, score, script with streaming) |
| - Gradio app with ThreadPoolExecutor for background pipeline runs |
|
|
| ## ZeroGPU Problems |
| 1. **Timeouts**: 120s GPU lease is too short for full pipeline stages (ranking + script gen) |
| 2. **Queue slowness**: ZeroGPU queue has unpredictable wait times; cold starts are brutal |
| 3. **Lease execution failures**: GPU leases fail frequently, triggering CPU fallback (slow + degraded quality) |
| 4. **Pipeline can't complete**: Stages chain together; if one stage's GPU lease expires mid-work, the whole pipeline stalls |
| 5. **No persistence**: Each cold start reloads model weights (~30s+), burning precious lease time |
| 6. **Unpredictable latency**: Same request can take 10s or 120s depending on queue position |
|
|
| ## Requirements |
| - Must work within HF Spaces ecosystem (or migrate to something better) |
| - Need reliable GPU for LLM inference (gemma-4-12b or similar) |
| - Pipeline stages are sequential: rank completes → script starts |
| - Total LLM inference time per run: ~3-5 minutes of actual GPU compute |
| - Free or very cheap (this is a side project, not enterprise) |
| - Minimal ops burden — set it and forget it |
|
|
| ## Pipeline Compute Needs |
| - **Rank stage**: 2 LLM passes × ~30 items = ~60 short completions (~50-200 tokens each) |
| - **Script stage**: 1 outline + ~10-15 segment generations (~500-1000 tokens each) |
| - **Model size**: 12B params in bfloat16 ≈ 24GB VRAM |
| - **Total per run**: ~3-5 min GPU compute, runs once daily |
|
|
