docs/archive/performance-path-forward.md

> **Historical document.** Retained as reference for the strategic reassessment after the torch.compile failure and L4 profiling. See `docs/lessons-learned.md` for the consolidated summary.

# Reassessment: best path forward for workshop throughput

> Synthesizes the torch.compile failure, the L4 profiling numbers, and the earlier pre-quantized investigation to pick a pragmatic next path.

> **Updates since written (2026-07-06):**
> - **ShieldGemma stays eagerly loaded.** Lazy-loading was rejected because the
>   safety guard *will* be used during the workshop; loading everything before
>   the workshop starts is preferable to a mid-workshop cold-start. Mitigation
>   is operational (warm the app once before attendees arrive), not a code
>   change.
> - **`py-spy` is unavailable on Hugging Face Spaces.** The GIL-vs-CPU-saturated
>   gate this doc proposes cannot be run there, and is no longer a blocker:
>   horizontal scaling and vLLM both help regardless of the answer, so the
>   distinction is informational only.

## What we now know

- **Baseline (compile off):** 0.18 req/s under the 15-user/180 s stress test.
- **`torch.compile` enabled:** 0.10 req/s — a ~44 % regression.
- **Cudagraphs are not usable** because the app calls compiled models from multiple Gradio worker threads. They only "work" for the fixed-shape next-token forward, and even there they crash.
- **GPU utilization is low** (~17 % on T4/L4). The bottleneck remains host-side Python/GIL overhead per token, not GPU kernel throughput or model weight size.
- **Pre-quantized checkpoints** exist for `gemma-3-1b-it` but do not address the diagnosed bottleneck, and they do not solve for base/270m variants.

Conclusion: the root-cause diagnosis in `docs/profiling-analysis.md` is still correct, but **`torch.compile` is not the fix**. We need either (a) remove the GIL contention by isolating each generation, or (b) batch requests so fewer, fatter GPU calls amortize Python overhead.

## Decision matrix

| Path | Expected throughput gain | Engineering cost | T4 VRAM feasibility | Recommendation |
|------|--------------------------|--------------------|---------------------|---------------- |
| Tune config (timeout/concurrency/caps) + lazy ShieldGemma | Low (restores reliability) | Very low | Yes | **Do this immediately** |
| Process isolation / multiprocessing | Medium-High (removes GIL) | High | Tight with 1B base + instruct + ShieldGemma | **Fallback if vLLM can't be used** |
| In-process batching | High (fewer, fatter calls) | Very high (new scheduler + streaming) | Yes | Highest ceiling but complex |
| vLLM with continuous batching | High (optimized engine) | High (replace backend) | Yes for 1B models | **Best long-term path** |

## Recommended path

### Immediate (this week): stabilization without compile

Do the minimum to make the current architecture reliable on the L4 (and still deployable on the T4 small):

1. **Set `TORCH_COMPILE=False`** (or disable the feature flag) in production.
2. **Lazy-load ShieldGemma** so the ~5.2 GB safety model is only loaded when the safety guard is toggled on. This was excluded earlier but is now high value and low risk.
3. **Tune Gradio concurrency** on the L4/T4:
   - Start with `GRADIO_CONCURRENCY=2`.
   - Stress-test at 2, 3, 4, and 5 concurrent requests under the realistic workload.
   - Pick the value that maximizes successful requests per second.
4. **Cap response length** for the demo:
   - Lower `DEFAULT_MAX_TOKENS` and `DEFAULT_BASE_MAX_TOKENS` to 128 or 100.
   - This shortens every generation and therefore raises throughput.
5. **Optional: use `MODEL_SIZE=270m`** if the demo content allows it. It is already supported and drastically cheaper.

Target outcome: a stable, predictable workshop at the cost of some output length and parallel capacity.

### Short term (next iteration): prototype a vLLM backend

For a real throughput improvement, the highest-leverage change is replacing the in-process Transformers backend with **vLLM**:

- vLLM uses continuous batching and paged attention, which directly solves the "many tiny GPU calls" problem.
- It handles streaming generation and batched generation out of the box.
- Two `vllm.LLM` engines (one base, one instruct) fit comfortably on a T4 or L4 for 1B/270M models.
- It does not require torch.compile to be fast.

This is a contained backend replacement: `llm_backend/generation.py`, `llm_backend/logit_pipeline.py`, and `llm_backend/safety.py`. The Gradio UI in `app.py` stays mostly the same because the public function signatures (`generate_comparison_response`, `generate_single_response`, `predict_next_token`) can be preserved.

### Contingent: process isolation

If vLLM is not an option (e.g., deployment restrictions, dependency issues on Hugging Face Spaces, or the team does not want to rewrite the backend), then **process isolation** is the next best alternative:

- Run each model variant in its own process.
- Each process has its own Python interpreter and GIL, eliminating GIL contention.
- This is the path already described in the main optimization plan’s Task 3.
- On a T4, load only what fits; on the L4, load base and instruct in two long-lived processes and keep ShieldGemma lazy.

## Why not process isolation first?

Process isolation would remove the GIL contention, but it:

- Adds significant IPC and streaming complexity for a modest model size.
- Requires either duplicating model weights in memory (one process per variant) or implementing shared-memory sharing.
- Does not solve the "too many tiny calls" problem; it only makes those calls contend less.

vLLM solves both the GIL/contention problem *and* the tiny-kernel problem, so it is the better investment of engineering time.

## Pre-quantized checkpoint role

Pre-quantized `gemma-3-1b-it` checkpoints (e.g., `RedHatAI/gemma-3-1b-it-quantized.w8a8`) become relevant again **only if** we move to vLLM and need to squeeze memory on the T4 small. They are not useful for the current Transformers-only backend. Defer decision until the vLLM prototype is running.

## Proposed next steps (ordered)

1. Merge or apply the stabilization changes (disable compile, lazy ShieldGemma, concurrency tuning).
2. Run the 15-user stress test on the L4 to get a new baseline.
3. Start a branch to spike a vLLM backend:
   - Load `google/gemma-3-1b-pt` and `google/gemma-3-1b-it` as two vLLM engines.
   - Implement `generate_single_response` and `generate_comparison_response` using vLLM synchronous/async streaming APIs.
   - Keep `predict_next_token` and `check_safety` functional.
4. Stress-test the vLLM branch under the same 15-user load.
5. Compare vLLM req/s, success rate, and VRAM against the stabilized baseline.

If vLLM delivers a clear win, adopt it and drop the process-isolation plan. If vLLM is blocked by deployment issues, fall back to the process-isolation plan.