docs/learn.md

AI Infrastructure Learnings

Modal — Serverless GPU

Cold Start Anatomy (H200, 31B dense model)

Phase	Duration (cached)	Duration (fresh)	Detail
Container init	~30s	~30s	Image pull, env setup
Weights load	~32s	~5-10 min	from huggingface-cache volume vs fresh download (58.25 GiB)
torch.compile	~8s	~60s+	from vllm-cache volume vs cold compilation
CUDA graph capture	~14s	~14s	51 piecewise + 51 full graphs
Engine init (rest)	~10s	~10s	Profiling, KV cache allocation
Engine init total	~107s	~107s+download	Includes compile + graph capture
Warm-up query	~7s	~10s	Absorbs JIT kernel compilation spikes
Total	~182s	10-15 min

Volumes

Modal Volumes are network-attached persistent storage mounted into containers at runtime. Two are critical:

• huggingface-cache — stores model weights via HF_HOME=/cache. First deploy downloads 58+ GiB; subsequent deploys read from cache. Without this, every cold start pays the full download penalty.
• vllm-cache — stores torch.compile artifacts and AOT compilation outputs via VLLM_CACHE_DIR=/root/.cache/vllm. Reusing compiled graphs saves ~60s+ vs cold compilation.

Volumes persist across deploys; they are NOT wiped when a container scales down.

Image Building

• Dependencies: use .uv_pip_install("package==version") on the image chain. Prefer this over raw pip_install for consistency with the project's uv tooling.
• Bundled files: use .add_local_file(local_path, remote_path, copy=True). Without copy=True, files are mounted at container startup (not baked into the image layer), making them unavailable for subsequent image build steps.
• Decorator params (gpu=, scaledown_window=, volumes=): these are evaluated at Python module load time on the deploy host (your machine), not inside the Modal container. Module-level constants work fine for these.

Path Resolution

Inside a Modal container, __file__ resolves to /root/modal_serve.py (the script is copied flattened into the root). Path(__file__).parent.parent.parent does NOT point to your project root. For runtime config files, bundle them into the image and reference the bundled path:

# At module level — try the bundled image path first, fall back to local project path
for path in (Path("/opt/config.yaml"), Path(__file__).resolve().parent.parent.parent / "config.yaml"):
    if path.exists():
        cfg = yaml.safe_load(path.read_text(encoding="utf-8"))
        break

Idle Management

Two competing knobs:

Knob	Behavior	Cost
keep_warm	Keeps N containers alive permanently	H200: $4.54/hr × N continuously
scaledown_window	Kills container after N minutes of no requests	H200: $4.54/hr for those N idle minutes per session end

For limited budgets (e.g., $240 hackathon credit), 30-minute scaledown_window is the practical sweet spot. Max idle waste per session: ~$2.27. keep_warm is unsustainable (burns credit in ~53 hours).

Cost Model (H200, ~$0.001261/sec → $4.54/hr)

Event	Cost
Cold start (182s from cache)	~$0.23
Per inference	~$0.005-0.01
Idle waste (30 min after last request)	~$2.27
Keep-warm (per hour)	$4.54

Deploy Lifecycle

• modal deploy pushes a new immutable deployment with the current code + image. Existing live containers continue running the OLD deployment.
• Killing a container (modal app stop) restarts it from the same old deployment. Code/image changes require modal deploy to take effect.
• Modal endpoints are public HTTPS URLs with no built-in auth layer. The backend class must skip API key validation (unlike HuggingFace or OpenRouter).

---

HuggingFace Hub

Gated Models

Models like google/gemma-4-31b-it require an accepted license agreement on HuggingFace before the model becomes accessible. Without this, even a valid token returns 401/403.

Token Access

• Read access: a HuggingFace token (HF_TOKEN) with READ scope is sufficient for downloading gated models.
• Inference API: requests require Authorization: Bearer <token> header. Tokens with only READ work for inference endpoints too.
• Environment variables: HF_TOKEN (auth), HF_HUB_ENABLE_HF_TRANSFER=1 (fast downloads via hf_transfer Rust library). HF_HOME controls the cache directory.

Model Identity

• Model IDs follow org/model-name format (e.g., google/gemma-4-31b-it).
• Revisions: optional branch/tag/commit hash pin. An invalid revision causes a 404 from the HF Hub. When in doubt, omit it and use the default (main).
• Checkpoint format: safetensors (.safetensors files), typically sharded. Gemma 4 31B = 2 shards, 58.25 GiB total.

---

vLLM (Runtime Notes)

Relevant Startup Flags

Flag	Value	Reason
--tensor-parallel-size	1	Single GPU (H200). >1 only for multi-GPU.
--enforce-eager	omit (default=False)	Let vLLM use CUDA graphs. Eager mode is a debug fallback and hurts throughput.
--async-scheduling	enabled	Improves throughput for single-request scenarios.
--tool-call-parser	gemma4	Model-specific. Needed for structured output / tool calling.
--reasoning-parser	gemma4	Model-specific. Parses chain-of-thought in responses.
--limit-mm-per-prompt	{"image":0,"video":0,"audio":0}	Force text-only mode. Reduces memory overhead.
--enable-auto-tool-choice	enabled	Allows the model to decide when to use tools.
--max-model-len	auto	vLLM auto-detects. Gemma 4 → 262144.
--gpu-memory-utilization	0.92	Leaves headroom for CUDA graphs and KV cache.
--safetensors-load-strategy	prefetch	Can speed up weight loading on network FS; omitted when on 9P (Modal default).
--generation-config	vllm	Override model's generation_config.json sampling defaults (see Sampling Defaults below).

Gemma4-Specific Architecture Notes

• Heterogeneous head dimensions: head_dim=256, global_head_dim=512. This forces the TRITON_ATTN backend to prevent mixed-backend numerical divergence.
• Multimodal-bidirectional attention: causes vLLM to force --disable_chunked_mm_input automatically.
• Architecture: resolved as Gemma4ForConditionalGeneration.
• Context length: auto-detected as 262,144 tokens.
• Chunked prefill: enabled with max_num_batched_tokens=8192.

Attention Backend

Gemma4's heterogeneous head dimensions trigger automatic selection of TRITON_ATTN. vLLM emits a config-time warning and forces this backend:

Gemma4 model has heterogeneous head dimensions (head_dim=256, global_head_dim=512).
Forcing TRITON_ATTN backend to prevent mixed-backend numerical divergence.

FlashInfer is used only for top-p & top-k sampling (via topk_topp_sampler.py), not for attention.

CUDA Graph Memory Profiling (v0.21.0+)

Since v0.21.0, vLLM profiles CUDA graph memory during startup and subtracts it from the GPU memory budget. The effective --gpu-memory-utilization is lower than the nominal value:

• Nominal: --gpu-memory-utilization=0.9200
• Effective: 0.9145 (i.e., you lose ~0.55pp to CUDA graph overhead)
• To maintain the same KV cache size: increase --gpu-memory-utilization to 0.9255
• To disable profiling: set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0

GPU Memory Breakdown (H200, 31B dense)

Component	Memory
Model weights	57.91 GiB
CUDA graphs (actual)	0.67 GiB
CUDA graphs (estimated)	0.76 GiB (difference: 13.7%)
Available KV cache	65.94 GiB
KV cache capacity	639,184 tokens
Max concurrency (262k-token reqs)	~2.44x

Filesystem & Weight Loading

Modal containers use the 9P filesystem by default. vLLM's auto-prefetch detection skips 9P because it is not a recognized network filesystem (NFS/Lustre):

Auto-prefetch is disabled because the filesystem (9P) is not a recognized network FS (NFS/Lustre).
If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.

Weight loading from huggingface-cache volume takes ~27.65s for a 58.25 GiB model (2 safetensors shards).

Sampling Defaults

vLLM warns that the model's generation_config.json overrides its built-in defaults:

Default vLLM sampling parameters have been overridden by the model's `generation_config.json`:
`{'temperature': 1.0, 'top_k': 64, 'top_p': 0.95}`.
If this is not intended, please relaunch with `--generation-config vllm`.

Chat Template Detection

vLLM auto-detects the chat template format as openai. You can override with --chat-template-content-format.

Warm-Up

Sending a trivial chat completion query ([{"role":"user","content":"Hi"}]) during startup triggers JIT kernel compilation (Triton) for the first-inference shapes. Without this, the first real user request pays a 2-3s latency spike from JIT compilation. Warm-up absorbs this cost before traffic arrives.

Known JIT compilation gaps during inference — even after a warm-up query, some Triton kernels compile on first real use:

• _compute_slot_mapping_kernel
• kernel_unified_attention

Each causes a latency spike. Consider extending the warm-up to cover these shapes/configs if consistent tail latency matters.

Throughput (H200, 31B dense, single request)

Metric	Value
Avg prompt throughput	244.6 tok/s
Avg generation throughput	55.9 tok/s

Startup Timeline (cached)

Phase	Duration
Container init	~30s
Model load	~29s
torch.compile (cached)	~8.8s
Profiling/warmup run	~0.3s
CUDA graph capture	~15s
Engine init total	~117s
Warm-up query	~7s
Total to healthy	~202s