deploy/MODAL_GEMMA_VLLM.md

JackAILocal Gemma 4 12B IT on Modal vLLM

This deploys an OpenAI-compatible vLLM endpoint used by the hosted Hugging Face Space packaging agent.

Setup

pip install modal
modal setup
modal secret create gemma-secrets \
  HF_TOKEN=hf_xxxxxxxxxxxxxxxxx \
  VLLM_API_KEY=change_this_long_random_value

Deploy

modal deploy deploy/modal_gemma_vllm.py

The deployed URL is used by the Hugging Face Space:

MODAL_BASE_URL=https://<workspace>--jackailocal-gemma-4-12b-it-vllm-serve.modal.run
MODAL_API_KEY=change_this_long_random_value
MODEL_ID=google/gemma-4-12B-it

Test

curl "$MODAL_BASE_URL/v1/chat/completions" \
  -H "Authorization: Bearer $MODAL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-12B-it",
    "messages": [
      {"role": "system", "content": "Return only JSON."},
      {"role": "user", "content": "Return {\"ok\":true}."}
    ],
    "temperature": 0.1,
    "max_tokens": 128,
    "response_format": {"type": "json_object"},
    "chat_template_kwargs": {"enable_thinking": false}
  }'

The endpoint is for packaging intelligence only. The generated JackAILocal runtime remains local/offline.