docs/lessons-learned.md

# Lessons Learned — Dead Ends & Disproven Approaches

> **Read this before pursuing performance or architecture changes.**
> Each entry records what was tried, why it failed, and when it might be
> worth revisiting. Detailed reasoning lives in `docs/archive/`.

---

## Transformers + dedicated GPU for workshop load

**Status:** Disproven for >1 concurrent user.

**Outcome:** 0.18 req/s, 25% success rate at 15 concurrent users on a dedicated T4. GPU utilization stuck at ~17%.

**Why it failed:**
- Python/GIL contention in the `model.generate()` loop serializes threads under concurrent load.
- Per-token Python overhead (logit processing, sampling, prompt re-packing) dominates; CUDA kernels are fast but the host can't feed them fast enough.
- Each additional Gradio worker thread makes every other thread slower, until long generations stall past the timeout.

**Detail:** `docs/archive/profiling-analysis.md` §3-5.

**When to revisit:** Never, unless the generation loop moves out of Python (vLLM, process isolation, or a compiled backend that actually works under multithreading).

---

## torch.compile

**Status:** Disproven — 44% throughput regression.

**Outcome:** 0.10 req/s with compile on vs 0.18 req/s off (L4, 15-user stress test).

**Why it failed:**
- Cudagraphs crash under Gradio's multithreaded handlers — Inductor stores cudagraph tree state in thread-local storage, so worker threads trip an `AssertionError`. Without cudagraphs, `reduce-overhead` loses most of its benefit for tiny per-token kernels.
- `dynamic=True` + varying prompt lengths means each new shape pays recompilation cost; at classroom scale those first calls dominate.
- Models are so small (270m-1B) that kernel execution time is already tiny; Dynamo guard + Inductor codegen dispatch overhead outweighs fusion benefit.

**Detail:** `docs/archive/profiling-analysis.md` §"torch.compile measured regression" addendum.

**When to revisit:** Only if the threading model changes (single-threaded inference) or the model size grows large enough that kernel fusion matters.

---

## Quantization (INT8 / AWQ / FP8)

**Status:** Disproven — wrong bottleneck.

**Outcome:** GPU is already idle at 17% utilization. Reducing weight size/compute does not help when the bottleneck is host-side Python overhead.

**Why it failed:**
- Quantization reduces memory bandwidth and VRAM footprint, but the GPU is not memory-bound or compute-bound — it's starved by the Python dispatch loop.
- Pre-quantized checkpoints exist only for some model variants (1B instruct); no coverage for base or 270m.
- Community checkpoints introduce quality, tokenizer, and dependency risks.

**Detail:** `docs/archive/pre-quantized-models-assessment.md`.

**When to revisit:** Only as part of a vLLM migration where quantized weights are paired with a continuous-batching engine that can exploit them.

---

## Raising GRADIO_CONCURRENCY on Transformers

**Status:** Disproven — triggers GIL collapse.

**Outcome:** At concurrency=4, 75% failure rate (all long-generation requests timed out). At concurrency=1, 100% success but serial.

**Why it failed:**
- GIL contention is not a mild slowdown — it's a catastrophic collapse. Each additional thread makes every thread slower until long generations stall past the 30s timeout.
- The correct fix is to reduce per-request server time, not increase concurrency.

**Detail:** `docs/archive/profiling-analysis.md` §5.

**When to revisit:** Never on the Transformers backend. vLLM's continuous batching handles concurrency without Python-level contention.

---

## CUDA_LAUNCH_BLOCKING=1

**Status:** Disproven — made things 10x worse.

**Outcome:** Next-token forward pass went from 0.18s to ~2.0s. All streaming requests timed out. GPU util did not rise.

**Why it failed:**
- Blocking removes the CPU/GPU overlap that was partly masking the GIL problem.
- A 270m forward/decode is hundreds of tiny kernels; blocking forces a CPU↔GPU sync on each one.
- GPU util stayed flat at ~10-17%, confirming the GPU was not secretly busy.

**Detail:** `docs/archive/profiling-analysis.md` §4.

**When to revisit:** Never. The diagnosis is closed: async dispatch was helping, not hurting.

---

## vLLM on T4 (Turing)

**Status:** Incompatible.

**Outcome:** vLLM 0.24 V1 engine requires Ampere+ (compute 8.0). T4 is Turing (compute 7.5).

**Why it failed:**
- V1 engine requires Ampere+ for bf16, FlashAttention 2, and Triton shared memory.
- No workaround; T4 is fundamentally below the floor.

**Detail:** `docs/archive/vllm-spike-results.md` §"Hardware note".

**When to revisit:** Only if vLLM adds Turing support (unlikely) or you move to Ampere+ hardware.

---

## vLLM on CPU (0.24 pinned)

**Status:** Not viable with current pin.

**Outcome:** The standard `vllm==0.24.0` wheel is GPU-only.

**Why it failed:**
- vLLM's CPU backend is experimental with no official prebuilt wheels.
- Requires a third-party `vllm-cpu` PyPI package or a source build.
- Needs AVX512-class instructions (`avx512_bf16` / `avx512_vnni`) for usable performance; plain x86-64 crawls.
- The `AsyncLLMEngine` API used in this repo is untested on CPU.

**Detail:** This spec's research (2026-07-08).

**When to revisit:** If vLLM CPU support matures to official wheels and the `AsyncLLMEngine` path is validated on CPU.

---

## ZeroGPU + vLLM

**Status:** Fundamentally incompatible.

**Why it fails:**
- vLLM engines are long-lived processes that grab VRAM at construction and hold persistent state.
- ZeroGPU allocates GPU per-call from a shared pool with short-lived allocation windows.
- They cannot coexist architecturally.

**Detail:** `AGENTS.md` history; commit `55d2416`.

**When to revisit:** Never. This is an architectural incompatibility, not a version issue.

---

## Process isolation / multiprocessing

**Status:** Considered, deprioritized.

**Outcome:** Would remove GIL contention but at high complexity for modest gain.

**Why it was passed over:**
- Adds IPC and streaming complexity for small models.
- Requires duplicating model weights per process or implementing shared-memory sharing.
- Does not solve the "too many tiny kernel calls" problem — only makes them contend less.
- vLLM solves both the GIL/contention problem AND the tiny-kernel problem, making it the better investment.

**Detail:** `docs/archive/performance-path-forward.md`.

**When to revisit:** If vLLM is blocked by deployment restrictions and throughput must improve.

---

## ZeroGPU workshop load capacity

**Status:** Resolved — measured 2026-07-12. ZeroGPU is viable for a ~15-user workshop.

**Run:** `scripts/stress_test.py --users 15 --mode realistic --duration 180` against the deployed Space (`rectified-snugness/glorified-spellcheck`, ZeroGPU H200, Transformers backend). 100 requests / 187 s wall.

**Outcome:** **0.53 req/s, 100% success** (0 timeouts, 0 errors, 0 rate-limited). Per-endpoint client latency:

| Endpoint | p50 | p95 | p99 |
|---|---|---|---|
| `next_token` | 5.4 s | 12.8 s | 14.8 s |
| `compare_models` | 8.6 s | 28.6 s | 32.7 s |
| `layering` | 8.6 s | 33.0 s | 37.9 s |

**Interpretation:**
- **Reliability fixed vs the T4 baseline** (`docs/archive/profiling-analysis.md`): 25% → 100% success and 0.18 → 0.53 req/s (~3×). The H200 is fast enough that even GIL-contended generations finish within `DEFAULT_TIMEOUT`, so the collapse that crippled the T4 no longer causes failures.
- **The GIL still caps throughput** at ~0.53 req/s — roughly 7× below vLLM's dedicated-L4 ceiling (3.63 req/s, `docs/archive/vllm-spike-results.md`). ZeroGPU did not remove the Python bottleneck; it stopped it from breaking requests. vLLM remains a ZeroGPU dead-end (dedicated GPU only), so this gap is the accepted trade-off.
- **`next_token` latency is wait-bound**, not compute-bound: its forward pass is ~21 ms warm but p50 is 5.4 s — the gap is queueing behind slow generator calls plus ZeroGPU's per-call GPU-grant overhead. On a dedicated GPU this lightweight endpoint interleaved cheaply (0.5 s p50); on ZeroGPU every `@GPU` call pays a fixed allocation cost that dominates a 21 ms forward pass.

**Reproduce:** `scripts/stress_test.py --url <owner/name-or-*.hf.space-app-url> --users 15 --mode realistic --duration 180` (not the `huggingface.co/spaces/...` Hub page URL — `gradio_client` cannot fetch config from it).

**When to revisit:** Before any workshop larger than ~15 users, or if p95 latency (~30 s on the generators) becomes painful. This single run exceeds the free tier's 300 GPU-s/day budget; a PRO account (2400 s/day) absorbs it with headroom for a few repeats. If throughput must improve, the only remaining lever is vLLM on a dedicated GPU (`archive/vllm-backend` branch + `docs/archive/vllm-*.md`); the GIL ceiling cannot be raised on ZeroGPU.