docs/lessons-zerogpu.md

Lessons learned: llama.cpp + ZeroGPU + a continuous simulation loop

Raw notes for the blog. The things I wish I had known on day one. They came out of a long debug session against the "RuntimeError: No CUDA GPUs are available" error.

1. ZeroGPU is designed for synchronous, user-clicks-a-button workloads

@spaces.GPU forks a worker process per call. The scheduler hands you a GPU slot if one is available; if the pool is busy, your fork's worker_init fails with No CUDA GPUs are available inside spaces/zero/wrappers.py:worker_init, before any of your code runs. This is the allocator backing off — not a bug, not anything you can catch from app code.

That's fine if your app makes ten GPU calls per recording session (chat demos, generate-an-image apps). It is not fine if your app makes hundreds (a simulation that polls every five seconds). The scheduler punishes high-frequency callers to keep the shared pool fair, and Townlet was the loudest neighbour on the block.

2. The spaces package is built around torch, not llama.cpp

import spaces monkey-patches torch's CUDA APIs so model.to("cuda") at module scope succeeds without a real GPU attached. On the first @spaces.GPU call the worker streams weights from disk → pinned memory → VRAM through a double-buffered pipeline — that's the cold-start optimisation the docs talk about.

llama-cpp-python is invisible to that pipeline. It's a separate C++ runtime; the monkey-patching does nothing for it. Every fork loads the GGUF from scratch every time. None of the warm-worker reuse benefits apply.

Surveying 50+ ZeroGPU Spaces in the same hackathon: nearly every team using LLMs ships transformers + torch on the GPU and keeps llama-cpp as a CPU-only side path. Two Spaces in the org actually ran llama-cpp on ZeroGPU (Dean's Rizz Therapy and 1000-Rooms) and both are click-triggered single-pipeline apps. None of them runs llama-cpp inside a continuous poll loop. Townlet was the only one.

3. Keep the Llama() constructor minimal

Dean's actual production code (he's the Discord OP whose recipe the whole hackathon copied) calls Llama() with five kwargs: model_path, n_gpu_layers=-1, n_ctx, flash_attn=True, verbose=False. That's it. No type_k/type_v for KV-cache quantisation, no use_mlock, no n_batch override, no use_mmap=True.

I had piled on those kwargs chasing performance. They added latency on every fork because some are version-gated and my try/except had to retry on rejection. Stripping back to the minimal five matched Dean's pattern and removed a whole class of failure modes.

4. hf_hub_download belongs at module scope, not inside @spaces.GPU

Llama.from_pretrained(repo_id=..., filename=...) is convenient, but it runs the cache-validation roundtrip inside the fork, where every second of wall time is precious. Dean's pattern, the 1000-Rooms pattern, and the long-running gokaygokay/Gemma-2-llamacpp reference all do model_path = hf_hub_download(...) in the parent process at import time, then Llama(model_path=model_path, ...) inside the decorated function. Same outcome, no network roundtrip per fork.

5. Remaining quota controls queue priority, so credits compound

From the docs: "Remaining quota directly impacts priority in ZeroGPU queues." Credits don't just extend daily runtime — they raise priority while you have them, which lowers the rate at which the allocator denies your forks. $20 of pre-paid credits buys 200 minutes of GPU time and moves you up the queue. It's the cheapest debugging tool a hackathon team has.

6. The error message doesn't tell you the cause

No CUDA GPUs are available is what the worker reports when torch.Tensor([0]).cuda() fails inside worker_init. The actual upstream causes can be: quota pre-check rejected the request, no slots in the shared pool, the allocator's fairness signal kicking in, or an HTTP-layer failure that the spaces decorator surfaces as a generic exception. We can't tell which from our side. The right escalation path is the spaces-team discussion board, not the app code.

7. The fix wasn't an architecture change. It was being less greedy.

Once diagnosed, the fix was small:

• Strip Llama() to Dean's five kwargs
• Move hf_hub_download to module scope
• Slow the JS poll interval (~25s — the world still feels alive because the snapshot tick at 800ms keeps characters animating between drains)
• Pre-paid quota as priority insurance

The simulation kept its continuous-loop architecture. It just stopped hammering the allocator.

Things I'd do differently next time

• Use transformers, not llama.cpp, on ZeroGPU. Llama Champion badge is nice but it cost a week of debug time. Most teams shipped transformers + LoRA, which is what the spaces package is actually built for.
• Measure quota burn before optimising decisions. I tuned drain duration and decision count before knowing that polling cadence was the dominant variable.
• Read the spaces source, not just the docs. worker_init raising before your code runs is something the docs underplay. Five minutes in the source tells you more than the entire ZeroGPU page.