# Modal llms.txt

> Modal is a platform for running Python code in the cloud with minimal
> configuration, especially for serving AI models and high-performance batch
> processing. It supports fast prototyping, serverless APIs, scheduled jobs,
> GPU inference, distributed volumes, and sandboxes.

> Local mirror of Modal's llms.txt, kept in-repo so the serving layer stays
> grounded in current Modal guidance. Refresh from https://modal.com/docs and
> re-sync when bumping the vLLM/Modal versions in ../service.py.

Important notes:

- Modal's primitives are embedded in Python and tailored for AI/GPU use cases,
  but they can be used for general-purpose cloud compute.
- Modal is a serverless platform, meaning you are only billed for resources used
  and can spin up containers on demand in seconds.

You can sign up for free at [https://modal.com] and get $30/month of credits.

## Patterns this repo relies on

- OpenAI-compatible vLLM serving via `@modal.web_server(port=8000)` launching
  `vllm serve` as a subprocess (see ../service.py).
- `@modal.concurrent(max_inputs=...)` to multiplex requests onto one container.
- Shared `modal.Volume`s for the Hugging Face weight cache and vLLM compile
  cache, so weights are pulled once and warm across every provider app.
- `modal.Secret` (`huggingface-secret`, key `HF_TOKEN`) for gated repos.
- `scaledown_window` / `min_containers` for cold-start vs. cost trade-offs.

## Guide

- [Introduction](https://modal.com/docs/guide)
- Custom container images
  - [Defining Images](https://modal.com/docs/guide/images.md)
  - [Using existing container images](https://modal.com/docs/guide/existing-images.md)
  - [Fast pull from registry](https://modal.com/docs/guide/fast-pull-from-registry.md)
- GPUs and other resources
  - [GPU acceleration](https://modal.com/docs/guide/gpu.md)
  - [Using CUDA on Modal](https://modal.com/docs/guide/cuda.md)
  - [Configuring CPU, memory, and disk](https://modal.com/docs/guide/resources.md)
- Scaling out
  - [Scaling out](https://modal.com/docs/guide/scale.md)
  - [Input concurrency](https://modal.com/docs/guide/concurrent-inputs.md)
  - [Batch processing](https://modal.com/docs/guide/batch-processing.md)
  - [Job queues](https://modal.com/docs/guide/job-queue.md)
  - [Dynamic batching](https://modal.com/docs/guide/dynamic-batching.md)
- Deployment
  - [Apps, Functions, and entrypoints](https://modal.com/docs/guide/apps.md)
  - [Managing deployments](https://modal.com/docs/guide/managing-deployments.md)
  - [Invoking deployed functions](https://modal.com/docs/guide/trigger-deployed-functions.md)
  - [Continuous deployment](https://modal.com/docs/guide/continuous-deployment.md)
- Secrets and environment variables
  - [Secrets](https://modal.com/docs/guide/secrets.md)
  - [Environment variables](https://modal.com/docs/guide/environment_variables.md)
- Web Functions
  - [Web Functions](https://modal.com/docs/guide/webhooks.md)
  - [Streaming endpoints](https://modal.com/docs/guide/streaming-endpoints.md)
  - [Web Function URLs](https://modal.com/docs/guide/webhook-urls.md)
  - [Request timeouts](https://modal.com/docs/guide/webhook-timeouts.md)
  - [Proxy Auth Tokens](https://modal.com/docs/guide/webhook-proxy-auth.md)
- Data sharing and storage
  - [Volumes](https://modal.com/docs/guide/volumes.md)
  - [Storing model weights](https://modal.com/docs/guide/model-weights.md)
- Performance
  - [Cold start performance](https://modal.com/docs/guide/cold-start.md)
  - [Memory Snapshots](https://modal.com/docs/guide/memory-snapshots.md)
  - [High-performance LLM inference](https://modal.com/docs/guide/high-performance-llm-inference.md)
- Reliability and robustness
  - [Failures and retries](https://modal.com/docs/guide/retries.md)
  - [Timeouts](https://modal.com/docs/guide/timeouts.md)
  - [GPU health](https://modal.com/docs/guide/gpu-health.md)

## Examples

- Large language models (LLMs)
  - [Deploy an OpenAI-compatible LLM service with vLLM](https://modal.com/docs/examples/llm_inference.md)
  - [Maximize tokens per second in batch processing with vLLM](https://modal.com/docs/examples/vllm_throughput.md)
  - [Serve an ultra-low-latency chatbot with SGLang](https://modal.com/docs/examples/sglang_low_latency.md)
  - [Deploy Nemotron 3](https://modal.com/docs/examples/nemotron_inference.md)
  - [Run a multimodal RAG chatbot to answer questions about PDFs](https://modal.com/docs/examples/chat_with_pdf_vision.md)

## API Reference

- Application construction
  - [`App`](https://modal.com/docs/reference/modal.App.md)
  - [`App.function`](https://modal.com/docs/reference/modal.App.md)
  - [`App.cls`](https://modal.com/docs/reference/modal.App.md)
- Web integrations
  - [`web_server`](https://modal.com/docs/reference/modal.web_server.md)
  - [`fastapi_endpoint`](https://modal.com/docs/reference/modal.fastapi_endpoint.md)
  - [`asgi_app`](https://modal.com/docs/reference/modal.asgi_app.md)
- Function semantics
  - [`concurrent`](https://modal.com/docs/reference/modal.concurrent.md)
  - [`batched`](https://modal.com/docs/reference/modal.batched.md)
- Container configuration
  - [`Image`](https://modal.com/docs/reference/modal.Image.md)
  - [`Secret`](https://modal.com/docs/reference/modal.Secret.md)
- Data primitives
  - [`Volume`](https://modal.com/docs/reference/modal.Volume.md)