docs/lessons-zerogpu.md

# Lessons learned: llama.cpp + ZeroGPU + a continuous simulation loop

Raw notes for the blog. The things I wish I had known on day one. They came out of a long debug session against the "RuntimeError: No CUDA GPUs are available" error.

## 1. ZeroGPU is designed for synchronous, user-clicks-a-button workloads

`@spaces.GPU` forks a worker process per call. The scheduler hands you a GPU slot if one is available; if the pool is busy, your fork's `worker_init` fails with `No CUDA GPUs are available` *inside `spaces/zero/wrappers.py:worker_init`*, before any of your code runs. This is the allocator backing off — not a bug, not anything you can catch from app code.

That's fine if your app makes ten GPU calls per recording session (chat demos, generate-an-image apps). It is not fine if your app makes hundreds (a simulation that polls every five seconds). The scheduler punishes high-frequency callers to keep the shared pool fair, and Townlet was the loudest neighbour on the block.

## 2. The `spaces` package is built around torch, not llama.cpp

`import spaces` monkey-patches torch's CUDA APIs so `model.to("cuda")` at module scope succeeds without a real GPU attached. On the first `@spaces.GPU` call the worker streams weights from disk → pinned memory → VRAM through a double-buffered pipeline — that's the cold-start optimisation the docs talk about.

**llama-cpp-python is invisible to that pipeline.** It's a separate C++ runtime; the monkey-patching does nothing for it. Every fork loads the GGUF from scratch every time. None of the warm-worker reuse benefits apply.

Surveying 50+ ZeroGPU Spaces in the same hackathon: nearly every team using LLMs ships transformers + torch on the GPU and keeps llama-cpp as a CPU-only side path. Two Spaces in the org actually ran llama-cpp on ZeroGPU (Dean's Rizz Therapy and 1000-Rooms) and both are click-triggered single-pipeline apps. None of them runs llama-cpp inside a continuous poll loop. Townlet was the only one.

## 3. Keep the `Llama()` constructor minimal

Dean's actual production code (he's the Discord OP whose recipe the whole hackathon copied) calls `Llama()` with **five kwargs**: `model_path`, `n_gpu_layers=-1`, `n_ctx`, `flash_attn=True`, `verbose=False`. That's it. No `type_k`/`type_v` for KV-cache quantisation, no `use_mlock`, no `n_batch` override, no `use_mmap=True`.

I had piled on those kwargs chasing performance. They added latency on every fork because some are version-gated and my `try/except` had to retry on rejection. Stripping back to the minimal five matched Dean's pattern and removed a whole class of failure modes.

## 4. `hf_hub_download` belongs at module scope, not inside `@spaces.GPU`

`Llama.from_pretrained(repo_id=..., filename=...)` is convenient, but it runs the cache-validation roundtrip *inside the fork*, where every second of wall time is precious. Dean's pattern, the 1000-Rooms pattern, and the long-running `gokaygokay/Gemma-2-llamacpp` reference all do `model_path = hf_hub_download(...)` in the parent process at import time, then `Llama(model_path=model_path, ...)` inside the decorated function. Same outcome, no network roundtrip per fork.

## 5. Remaining quota controls queue priority, so credits compound

From the docs: *"Remaining quota directly impacts priority in ZeroGPU queues."* Credits don't just extend daily runtime — they raise priority *while you have them*, which lowers the rate at which the allocator denies your forks. $20 of pre-paid credits buys 200 minutes of GPU time *and* moves you up the queue. It's the cheapest debugging tool a hackathon team has.

## 6. The error message doesn't tell you the cause

`No CUDA GPUs are available` is what the worker reports when `torch.Tensor([0]).cuda()` fails inside `worker_init`. The actual upstream causes can be: quota pre-check rejected the request, no slots in the shared pool, the allocator's fairness signal kicking in, or an HTTP-layer failure that the spaces decorator surfaces as a generic exception. We can't tell which from our side. The right escalation path is the spaces-team discussion board, not the app code.

## 7. The fix wasn't an architecture change. It was being less greedy.

Once diagnosed, the fix was small:
- Strip `Llama()` to Dean's five kwargs
- Move `hf_hub_download` to module scope
- Slow the JS poll interval (~25s — the world still feels alive because the snapshot tick at 800ms keeps characters animating between drains)
- Pre-paid quota as priority insurance

The simulation kept its continuous-loop architecture. It just stopped hammering the allocator.

## Things I'd do differently next time

- **Use transformers, not llama.cpp, on ZeroGPU.** Llama Champion badge is nice but it cost a week of debug time. Most teams shipped transformers + LoRA, which is what the spaces package is actually built for.
- **Measure quota burn before optimising decisions.** I tuned drain duration and decision count before knowing that polling cadence was the dominant variable.
- **Read the spaces source, not just the docs.** `worker_init` raising before your code runs is something the docs underplay. Five minutes in the source tells you more than the entire ZeroGPU page.