modal_apps/README.md

Modal apps

These files define separate Modal deployments for the virtual character project.

Setup

python -m pip install -r requirements.txt
modal setup
modal secret create hf-token HF_TOKEN=hf_xxx

You must also accept gated model licenses on Hugging Face before Modal can download those weights.

If your Modal Secret uses a different name, set it before running checks or deploys:

$env:VC_HF_SECRET_NAME="your-secret-name"

First checks

Run remote method health checks:

modal run modal_apps/modal_ping.py
modal run modal_apps/modal_hf_check.py
python scripts/check_modal_connectivity.py --mode remote-methods

modal_ping.py is CPU-only and only checks login/connectivity. modal_hf_check.py is also CPU-only and checks whether Modal can read hf-token and query the selected Hugging Face model metadata without downloading weights. The health checks in check_modal_connectivity.py start the service containers but do not load model weights. Actual generation tests will load models and consume GPU credits.

Benchmark Gemma on Modal:

$env:VC_BENCH_MODEL="google/gemma-4-12B-it"
$env:VC_BENCH_GPU="L40S"
modal run modal_apps/modal_gemma_benchmark.py --max-new-tokens 64

Benchmark Gemma through vLLM:

$env:VC_VLLM_MODEL="google/gemma-4-12B-it"
$env:VC_VLLM_VERSION="0.22.1"
$env:VC_VLLM_GPU="L40S"
$env:VC_VLLM_FAST_BOOT="1"
modal run modal_apps/modal_vllm_gemma.py --max-tokens 128

Note: vllm==0.22.1 is the latest PyPI stable release checked on 2026-06-12, but it still does not run google/gemma-4-12B-it correctly in our Modal test. For Gemma 4 12B, use this only as a regression check.

Benchmark Gemma through vLLM nightly:

$env:PYTHONIOENCODING="utf-8"
$env:PYTHONUTF8="1"
$env:VC_SKIP_HF_SECRET="1"
$env:VC_VLLM_MODEL="google/gemma-4-12B-it"
$env:VC_VLLM_PACKAGE="vllm==0.22.1rc1.dev468+gfbc3a1907.cu129"
$env:VC_VLLM_EXTRA_INDEX_URL="https://wheels.vllm.ai/nightly/cu129"
$env:VC_VLLM_UV_EXTRA_OPTIONS="--index-strategy unsafe-best-match"
$env:VC_VLLM_PRE="1"
$env:VC_VLLM_GPU="L40S"
$env:VC_VLLM_FAST_BOOT="1"
modal run modal_apps/modal_vllm_gemma.py --max-tokens 128

Deploy the working vLLM nightly endpoint:

$env:PYTHONIOENCODING="utf-8"
$env:PYTHONUTF8="1"
$env:VC_SKIP_HF_SECRET="1"
$env:VC_VLLM_MODEL="google/gemma-4-12B-it"
$env:VC_VLLM_PACKAGE="vllm==0.22.1rc1.dev468+gfbc3a1907.cu129"
$env:VC_VLLM_EXTRA_INDEX_URL="https://wheels.vllm.ai/nightly/cu129"
$env:VC_VLLM_UV_EXTRA_OPTIONS="--index-strategy unsafe-best-match"
$env:VC_VLLM_PRE="1"
$env:VC_VLLM_GPU="L40S"
$env:VC_VLLM_FAST_BOOT="1"
modal deploy modal_apps/modal_vllm_gemma.py

Keep the deployed vLLM endpoint warm:

python scripts/set_modal_vllm_autoscaler.py on

This updates the deployed serve function to min_containers=1, buffer_containers=0, and scaledown_window=1200. It takes effect without rebuilding the image, but Modal resets this override on the next deploy.

To make warm residency part of the deployment configuration, set these before modal deploy:

$env:VC_VLLM_MIN_CONTAINERS="1"
$env:VC_VLLM_BUFFER_CONTAINERS="0"
$env:VC_VLLM_SCALEDOWN_WINDOW="1200"
modal deploy modal_apps/modal_vllm_gemma.py

Turn warm residency off when the demo window is over:

python scripts/set_modal_vllm_autoscaler.py off

Current deployed endpoint:

https://veronicaulises0--virtual-characters-vllm-gemma-serve.modal.run

This deployment intentionally skips mounting hf-token into the nightly vLLM runtime. It depends on the vc-hf-cache Modal Volume already containing google/gemma-4-12B-it.

To avoid building every service image at once, check one service at a time:

python scripts/check_modal_connectivity.py --mode remote-methods --service llm
python scripts/check_modal_connectivity.py --mode remote-methods --service tts
python scripts/check_modal_connectivity.py --mode remote-methods --service image

Character generation spike

The automated character-generation spike is intentionally isolated from the Gradio UI and the deployed image endpoint.

Safe checks:

python scripts/run_character_generation_spike.py list-models
python scripts/run_character_generation_spike.py modal-health

modal-health starts the Modal app but does not load model weights. Generation probes load weights and consume GPU credits, so the CLI requires --confirm-gpu:

python scripts/run_character_generation_spike.py modal-probe --candidate flux_schnell --batch-size 1 --confirm-gpu
python scripts/run_character_generation_spike.py modal-benchmark --candidates flux_schnell qwen_image --confirm-gpu
python scripts/run_character_generation_spike.py modal-benchmark --candidates qwen_image_edit --init-image path\to\reference.png --include-expressions --confirm-gpu
python scripts/run_character_generation_spike.py modal-benchmark --candidates qwen_controlnet_union --control-image path\to\pose.png --include-expressions --confirm-gpu

Candidates:

• flux_schnell: speed baseline using black-forest-labs/FLUX.1-schnell.
• qwen_image: Chinese prompt and high-quality text-to-image candidate using Qwen/Qwen-Image.
• qwen_image_edit: expression/edit candidate using Qwen/Qwen-Image-Edit.
• qwen_controlnet_union: pose/canny/depth action candidate using InstantX/Qwen-Image-ControlNet-Union.
• instantid_sdxl: tracked as identity-preserving candidate, but disabled until the face-analysis/model download path is pinned.

The spike image installs diffusers from GitHub because the Qwen Image and ControlNet pipelines require recent upstream support. If a model is gated or not cached, make sure the hf-token secret has accepted the Hugging Face model terms.

If hf-token is not created yet and you only want to verify Modal container startup, use:

$env:VC_SKIP_HF_SECRET="1"
python scripts/check_modal_connectivity.py --mode remote-methods

Only deploy with VC_SKIP_HF_SECRET=1 when the selected model is public or the required weights are already cached in the mounted Modal Volume. The working vLLM Gemma 4 deployment uses this cache-based route to avoid exposing hf-token to a nightly dependency stack.

Deploy services:

modal deploy modal_apps/modal_llm.py
modal deploy modal_apps/modal_tts.py
modal deploy modal_apps/modal_image.py

Then set endpoint URLs from Modal output:

$env:VC_MODAL_LLM_URL="https://...modal.run/persona_events"
$env:VC_MODAL_TTS_URL="https://...modal.run/tts"
$env:VC_MODAL_IMAGE_URL="https://...modal.run/character_image"
python scripts/check_modal_connectivity.py --mode endpoints

Cost defaults

• vLLM LLM: google/gemma-4-12B-it, GPU L40S, scaledown 5 min unless VC_VLLM_MIN_CONTAINERS=1.
• Transformers LLM fallback: google/gemma-4-12B-it, GPU L40S, scaledown 3 min.
• TTS: Chatterbox, GPU A10G, scaledown 3 min.
• Image: FLUX.1-schnell, GPU H100, scaledown 1 min.

Modal public GPU prices checked on 2026-06-14:

GPU	Approx hourly	Approx 7 days
T4	$0.5904	$99.19
L4	$0.7992	$134.27
A10	$1.1016	$185.07
L40S	$1.9512	$327.80
A100-40GB	$2.0988	$352.60
A100-80GB	$2.4984	$419.73
H100	$3.9492	$663.47

These numbers are GPU-only. CPU, memory, regional multipliers, non-preemptible execution, and storage can add extra cost. With a $240 budget, one L40S can stay warm for about 123 hours, so a full 7-day L40S warm deployment is over budget. A10 fits the 7-day GPU-only budget, but the current Gemma 4 12B vLLM deployment is validated on L40S and should not be moved to A10 without a separate benchmark.

Override with env vars before deploy:

$env:VC_LLM_MODEL="google/gemma-4-E4B-it"
$env:VC_LLM_GPU="A10"
$env:VC_TTS_BACKEND="kokoro"
$env:VC_IMAGE_GPU="L40S"