modal_apps/README.md

# Modal apps

These files define separate Modal deployments for the virtual character project.

## Setup

```powershell
python -m pip install -r requirements.txt
modal setup
modal secret create hf-token HF_TOKEN=hf_xxx
```

You must also accept gated model licenses on Hugging Face before Modal can download those weights.

If your Modal Secret uses a different name, set it before running checks or deploys:

```powershell
$env:VC_HF_SECRET_NAME="your-secret-name"
```

## First checks

Run remote method health checks:

```powershell
modal run modal_apps/modal_ping.py
modal run modal_apps/modal_hf_check.py
python scripts/check_modal_connectivity.py --mode remote-methods
```

`modal_ping.py` is CPU-only and only checks login/connectivity. `modal_hf_check.py` is also CPU-only and checks whether Modal can read `hf-token` and query the selected Hugging Face model metadata without downloading weights. The health checks in `check_modal_connectivity.py` start the service containers but do not load model weights. Actual generation tests will load models and consume GPU credits.

Benchmark Gemma on Modal:

```powershell
$env:VC_BENCH_MODEL="google/gemma-4-12B-it"
$env:VC_BENCH_GPU="L40S"
modal run modal_apps/modal_gemma_benchmark.py --max-new-tokens 64
```

Benchmark Gemma through vLLM:

```powershell
$env:VC_VLLM_MODEL="google/gemma-4-12B-it"
$env:VC_VLLM_VERSION="0.22.1"
$env:VC_VLLM_GPU="L40S"
$env:VC_VLLM_FAST_BOOT="1"
modal run modal_apps/modal_vllm_gemma.py --max-tokens 128
```

Note: `vllm==0.22.1` is the latest PyPI stable release checked on 2026-06-12, but it still does not run `google/gemma-4-12B-it` correctly in our Modal test. For Gemma 4 12B, use this only as a regression check.

Benchmark Gemma through vLLM nightly:

```powershell
$env:PYTHONIOENCODING="utf-8"
$env:PYTHONUTF8="1"
$env:VC_SKIP_HF_SECRET="1"
$env:VC_VLLM_MODEL="google/gemma-4-12B-it"
$env:VC_VLLM_PACKAGE="vllm==0.22.1rc1.dev468+gfbc3a1907.cu129"
$env:VC_VLLM_EXTRA_INDEX_URL="https://wheels.vllm.ai/nightly/cu129"
$env:VC_VLLM_UV_EXTRA_OPTIONS="--index-strategy unsafe-best-match"
$env:VC_VLLM_PRE="1"
$env:VC_VLLM_GPU="L40S"
$env:VC_VLLM_FAST_BOOT="1"
modal run modal_apps/modal_vllm_gemma.py --max-tokens 128
```

Deploy the working vLLM nightly endpoint:

```powershell
$env:PYTHONIOENCODING="utf-8"
$env:PYTHONUTF8="1"
$env:VC_SKIP_HF_SECRET="1"
$env:VC_VLLM_MODEL="google/gemma-4-12B-it"
$env:VC_VLLM_PACKAGE="vllm==0.22.1rc1.dev468+gfbc3a1907.cu129"
$env:VC_VLLM_EXTRA_INDEX_URL="https://wheels.vllm.ai/nightly/cu129"
$env:VC_VLLM_UV_EXTRA_OPTIONS="--index-strategy unsafe-best-match"
$env:VC_VLLM_PRE="1"
$env:VC_VLLM_GPU="L40S"
$env:VC_VLLM_FAST_BOOT="1"
modal deploy modal_apps/modal_vllm_gemma.py
```

Keep the deployed vLLM endpoint warm:

```powershell
python scripts/set_modal_vllm_autoscaler.py on
```

This updates the deployed `serve` function to `min_containers=1`, `buffer_containers=0`, and `scaledown_window=1200`. It takes effect without rebuilding the image, but Modal resets this override on the next deploy.

To make warm residency part of the deployment configuration, set these before `modal deploy`:

```powershell
$env:VC_VLLM_MIN_CONTAINERS="1"
$env:VC_VLLM_BUFFER_CONTAINERS="0"
$env:VC_VLLM_SCALEDOWN_WINDOW="1200"
modal deploy modal_apps/modal_vllm_gemma.py
```

Turn warm residency off when the demo window is over:

```powershell
python scripts/set_modal_vllm_autoscaler.py off
```

Current deployed endpoint:

```text
https://veronicaulises0--virtual-characters-vllm-gemma-serve.modal.run
```

This deployment intentionally skips mounting `hf-token` into the nightly vLLM runtime. It depends on the `vc-hf-cache` Modal Volume already containing `google/gemma-4-12B-it`.

To avoid building every service image at once, check one service at a time:

```powershell
python scripts/check_modal_connectivity.py --mode remote-methods --service llm
python scripts/check_modal_connectivity.py --mode remote-methods --service tts
python scripts/check_modal_connectivity.py --mode remote-methods --service image
```

## Character generation spike

The automated character-generation spike is intentionally isolated from the Gradio UI and the deployed image endpoint.

Safe checks:

```powershell
python scripts/run_character_generation_spike.py list-models
python scripts/run_character_generation_spike.py modal-health
```

`modal-health` starts the Modal app but does not load model weights. Generation probes load weights and consume GPU credits, so the CLI requires `--confirm-gpu`:

```powershell
python scripts/run_character_generation_spike.py modal-probe --candidate flux_schnell --batch-size 1 --confirm-gpu
python scripts/run_character_generation_spike.py modal-benchmark --candidates flux_schnell qwen_image --confirm-gpu
python scripts/run_character_generation_spike.py modal-benchmark --candidates qwen_image_edit --init-image path\to\reference.png --include-expressions --confirm-gpu
python scripts/run_character_generation_spike.py modal-benchmark --candidates qwen_controlnet_union --control-image path\to\pose.png --include-expressions --confirm-gpu
```

Candidates:

- `flux_schnell`: speed baseline using `black-forest-labs/FLUX.1-schnell`.
- `qwen_image`: Chinese prompt and high-quality text-to-image candidate using `Qwen/Qwen-Image`.
- `qwen_image_edit`: expression/edit candidate using `Qwen/Qwen-Image-Edit`.
- `qwen_controlnet_union`: pose/canny/depth action candidate using `InstantX/Qwen-Image-ControlNet-Union`.
- `instantid_sdxl`: tracked as identity-preserving candidate, but disabled until the face-analysis/model download path is pinned.

The spike image installs diffusers from GitHub because the Qwen Image and ControlNet pipelines require recent upstream support. If a model is gated or not cached, make sure the `hf-token` secret has accepted the Hugging Face model terms.

If `hf-token` is not created yet and you only want to verify Modal container startup, use:

```powershell
$env:VC_SKIP_HF_SECRET="1"
python scripts/check_modal_connectivity.py --mode remote-methods
```

Only deploy with `VC_SKIP_HF_SECRET=1` when the selected model is public or the required weights are already cached in the mounted Modal Volume. The working vLLM Gemma 4 deployment uses this cache-based route to avoid exposing `hf-token` to a nightly dependency stack.

Deploy services:

```powershell
modal deploy modal_apps/modal_llm.py
modal deploy modal_apps/modal_tts.py
modal deploy modal_apps/modal_image.py
```

Then set endpoint URLs from Modal output:

```powershell
$env:VC_MODAL_LLM_URL="https://...modal.run/persona_events"
$env:VC_MODAL_TTS_URL="https://...modal.run/tts"
$env:VC_MODAL_IMAGE_URL="https://...modal.run/character_image"
python scripts/check_modal_connectivity.py --mode endpoints
```

## Cost defaults

- vLLM LLM: `google/gemma-4-12B-it`, GPU `L40S`, scaledown 5 min unless `VC_VLLM_MIN_CONTAINERS=1`.
- Transformers LLM fallback: `google/gemma-4-12B-it`, GPU `L40S`, scaledown 3 min.
- TTS: Chatterbox, GPU `A10G`, scaledown 3 min.
- Image: FLUX.1-schnell, GPU `H100`, scaledown 1 min.

Modal public GPU prices checked on 2026-06-14:

| GPU | Approx hourly | Approx 7 days |
| --- | ---: | ---: |
| T4 | $0.5904 | $99.19 |
| L4 | $0.7992 | $134.27 |
| A10 | $1.1016 | $185.07 |
| L40S | $1.9512 | $327.80 |
| A100-40GB | $2.0988 | $352.60 |
| A100-80GB | $2.4984 | $419.73 |
| H100 | $3.9492 | $663.47 |

These numbers are GPU-only. CPU, memory, regional multipliers, non-preemptible execution, and storage can add extra cost. With a $240 budget, one L40S can stay warm for about 123 hours, so a full 7-day L40S warm deployment is over budget. A10 fits the 7-day GPU-only budget, but the current Gemma 4 12B vLLM deployment is validated on L40S and should not be moved to A10 without a separate benchmark.

Override with env vars before deploy:

```powershell
$env:VC_LLM_MODEL="google/gemma-4-E4B-it"
$env:VC_LLM_GPU="A10"
$env:VC_TTS_BACKEND="kokoro"
$env:VC_IMAGE_GPU="L40S"
```