diff --git a/.agents/skills/hf-cli/.hf-skill-manifest.json b/.agents/skills/hf-cli/.hf-skill-manifest.json deleted file mode 100644 index 993fc3f78039ee644f3fd466308ff42ecffcae2a..0000000000000000000000000000000000000000 --- a/.agents/skills/hf-cli/.hf-skill-manifest.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "installed_revision": "7bf59b7f85b79c74207b10d5e425934514e8b089", - "schema_version": 1 -} diff --git a/.agents/skills/hf-cli/SKILL.md b/.agents/skills/hf-cli/SKILL.md deleted file mode 100644 index e927eca6cb3ff84b6d86558366b2864c05dbd999..0000000000000000000000000000000000000000 --- a/.agents/skills/hf-cli/SKILL.md +++ /dev/null @@ -1,218 +0,0 @@ ---- -name: hf-cli -description: "Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing models, datasets, spaces, buckets, repos, papers, jobs, and more on the Hugging Face Hub. Use when: handling authentication; managing local cache; managing Hugging Face Buckets; running or scheduling jobs on Hugging Face infrastructure; managing Hugging Face repos; discussions and pull requests; browsing models, datasets and spaces; reading, searching, or browsing academic papers; managing collections; querying datasets; configuring spaces; setting up webhooks; or deploying and managing HF Inference Endpoints. Make sure to use this skill whenever the user mentions 'hf', 'huggingface', 'Hugging Face', 'huggingface-cli', or 'hugging face cli', or wants to do anything related to the Hugging Face ecosystem and to AI and ML in general. Also use for cloud storage needs like training checkpoints, data pipelines, or agent traces. Use even if the user doesn't explicitly ask for a CLI command. Replaces the deprecated `huggingface-cli`." ---- - -Install: `curl -LsSf https://hf.co/cli/install.sh | bash -s`. - -The Hugging Face Hub CLI tool `hf` is available. IMPORTANT: The `hf` command replaces the deprecated `huggingface-cli` command. - -Use `hf --help` to view available functions. Note that auth commands are now all under `hf auth` e.g. `hf auth whoami`. - -Generated with `huggingface_hub v1.19.0`. Run `hf skills add --force` to regenerate. - -## Commands - -- `hf cp SRC` — Copy files between local paths, repositories, and buckets. `[--format [auto|human|agent|json|quiet]]` -- `hf download REPO_ID` — Download files from the Hub. `[--type [model|dataset|space] --revision TEXT --include TEXT --exclude TEXT --cache-dir TEXT --local-dir TEXT --force-download --dry-run --max-workers INTEGER --format [auto|human|agent|json|quiet]]` -- `hf env` — Print information about the environment. `[--format [auto|human|agent|json|quiet]]` -- `hf sync` — Sync files between local directory and a bucket. `[--delete --ignore-times --ignore-sizes --plan TEXT --apply TEXT --dry-run --include TEXT --exclude TEXT --filter-from TEXT --existing --ignore-existing --verbose --format [auto|human|agent|json|quiet]]` -- `hf update` — Update the `hf` CLI to the latest version. `[--format [auto|human|agent|json|quiet]]` -- `hf upload REPO_ID` — Upload a file or a folder to the Hub. Recommended for single-commit uploads. `[--type [model|dataset|space] --revision TEXT --private --include TEXT --exclude TEXT --delete TEXT --commit-message TEXT --commit-description TEXT --create-pr --every FLOAT --format [auto|human|agent|json|quiet]]` -- `hf upload-large-folder REPO_ID LOCAL_PATH` — Upload a large folder to the Hub. Recommended for resumable uploads. `[--type [model|dataset|space] --revision TEXT --private --include TEXT --exclude TEXT --num-workers INTEGER --no-report --no-bars --format [auto|human|agent|json|quiet]]` -- `hf version` — Print information about the hf version. `[--format [auto|human|agent|json|quiet]]` - -### `hf auth` — Manage authentication (login, logout, etc.). - -- `hf auth list` — List all stored access tokens. `[--format [auto|human|agent|json|quiet]]` -- `hf auth login` — Login using a token from huggingface.co/settings/tokens. `[--add-to-git-credential --force --format [auto|human|agent|json|quiet]]` -- `hf auth logout` — Logout from a specific token. `[--token-name TEXT --format [auto|human|agent|json|quiet]]` -- `hf auth switch` — Switch between access tokens. `[--token-name TEXT --add-to-git-credential --format [auto|human|agent|json|quiet]]` -- `hf auth token` — Print the current access token to stdout. `[--format [auto|human|agent|json|quiet]]` -- `hf auth whoami` — Find out which huggingface.co account you are logged in as. `[--format [auto|human|agent|json|quiet]]` - -### `hf buckets` — Commands to interact with buckets. - -- `hf buckets cp SRC` — Copy files between local paths, repositories, and buckets. `[--format [auto|human|agent|json|quiet]]` -- `hf buckets create BUCKET_ID` — Create a new bucket. `[--private --region [us|eu] --exist-ok --format [auto|human|agent|json|quiet]]` -- `hf buckets delete BUCKET_ID` — Delete a bucket. `[--yes --missing-ok --format [auto|human|agent|json|quiet]]` -- `hf buckets info BUCKET_ID` — Get info about a bucket. `[--format [auto|human|agent|json|quiet]]` -- `hf buckets list` — List buckets or files in a bucket. `[--human-readable --tree --recursive --search TEXT --format [auto|human|agent|json|quiet]]` -- `hf buckets move FROM_ID TO_ID` — Move (rename) a bucket to a new name or namespace. `[--format [auto|human|agent|json|quiet]]` -- `hf buckets remove ARGUMENT` — Remove files from a bucket. `[--recursive --yes --dry-run --include TEXT --exclude TEXT --format [auto|human|agent|json|quiet]]` -- `hf buckets sync` — Sync files between local directory and a bucket. `[--delete --ignore-times --ignore-sizes --plan TEXT --apply TEXT --dry-run --include TEXT --exclude TEXT --filter-from TEXT --existing --ignore-existing --verbose --format [auto|human|agent|json|quiet]]` - -### `hf cache` — Manage local cache directory. - -- `hf cache list` — List cached repositories or revisions. `[--cache-dir TEXT --revisions --filter TEXT --sort [accessed|accessed:asc|accessed:desc|modified|modified:asc|modified:desc|name|name:asc|name:desc|size|size:asc|size:desc] --limit INTEGER --format [auto|human|agent|json|quiet]]` -- `hf cache prune` — Remove detached revisions from the cache. `[--cache-dir TEXT --yes --dry-run --format [auto|human|agent|json|quiet]]` -- `hf cache rm TARGETS` — Remove cached repositories or revisions. `[--cache-dir TEXT --yes --dry-run --format [auto|human|agent|json|quiet]]` -- `hf cache verify REPO_ID` — Verify checksums for a single repo revision from cache or a local directory. `[--type [model|dataset|space] --revision TEXT --cache-dir TEXT --local-dir TEXT --fail-on-missing-files --fail-on-extra-files --format [auto|human|agent|json|quiet]]` - -### `hf collections` — Interact with collections on the Hub. - -- `hf collections add-item COLLECTION_SLUG ITEM_ID ITEM_TYPE` — Add an item to a collection. `[--note TEXT --exists-ok --format [auto|human|agent|json|quiet]]` -- `hf collections create TITLE` — Create a new collection on the Hub. `[--namespace TEXT --description TEXT --private --exists-ok --format [auto|human|agent|json|quiet]]` -- `hf collections delete COLLECTION_SLUG` — Delete a collection from the Hub. `[--missing-ok --format [auto|human|agent|json|quiet]]` -- `hf collections delete-item COLLECTION_SLUG ITEM_OBJECT_ID` — Delete an item from a collection. `[--missing-ok --format [auto|human|agent|json|quiet]]` -- `hf collections info COLLECTION_SLUG` — Get info about a collection on the Hub. `[--format [auto|human|agent|json|quiet]]` -- `hf collections list` — List collections on the Hub. `[--owner TEXT --item TEXT --sort [lastModified|trending|upvotes] --limit INTEGER --format [auto|human|agent|json|quiet]]` -- `hf collections update COLLECTION_SLUG` — Update a collection's metadata on the Hub. `[--title TEXT --description TEXT --position INTEGER --private --theme TEXT --format [auto|human|agent|json|quiet]]` -- `hf collections update-item COLLECTION_SLUG ITEM_OBJECT_ID` — Update an item in a collection. `[--note TEXT --position INTEGER --format [auto|human|agent|json|quiet]]` - -### `hf datasets` — Interact with datasets on the Hub. - -- `hf datasets card DATASET_ID` — Get the dataset card (README) for a dataset on the Hub. `[--metadata --text --format [auto|human|agent|json|quiet]]` -- `hf datasets info DATASET_ID` — Get info about a dataset on the Hub. `[--revision TEXT --expand TEXT --format [auto|human|agent|json|quiet]]` -- `hf datasets leaderboard DATASET_ID` — List model scores from a dataset leaderboard. This command helps find the best models for a task or compare models by benchmark scores. Use 'hf datasets ls --filter benchmark:official' to list available leaderboards. `[--limit INTEGER --format [auto|human|agent|json|quiet]]` -- `hf datasets list` — List datasets on the Hub, or files in a dataset repo. `[--search TEXT --author TEXT --filter TEXT --sort [created_at|downloads|last_modified|likes|trending_score] --limit INTEGER --expand TEXT --human-readable --tree --recursive --revision TEXT --format [auto|human|agent|json|quiet]]` -- `hf datasets parquet DATASET_ID` — List parquet file URLs available for a dataset. `[--subset TEXT --split TEXT --format [auto|human|agent|json|quiet]]` -- `hf datasets sql SQL` — Execute a raw SQL query with DuckDB against dataset parquet URLs. `[--format [auto|human|agent|json|quiet]]` - -### `hf discussions` — Manage discussions and pull requests on the Hub. - -- `hf discussions close REPO_ID NUM` — Close a discussion or pull request. `[--comment TEXT --yes --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` -- `hf discussions comment REPO_ID NUM` — Comment on a discussion or pull request. `[--body TEXT --body-file PATH --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` -- `hf discussions create REPO_ID --title TEXT` — Create a new discussion or pull request on a repo. `[--body TEXT --body-file PATH --pull-request --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` -- `hf discussions diff REPO_ID NUM` — Show the diff of a pull request. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]` -- `hf discussions info REPO_ID NUM` — Get info about a discussion or pull request. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]` -- `hf discussions list REPO_ID` — List discussions and pull requests on a repo. `[--status [open|closed|merged|draft|all] --kind [all|discussion|pull_request] --author TEXT --limit INTEGER --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` -- `hf discussions merge REPO_ID NUM` — Merge a pull request. `[--comment TEXT --yes --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` -- `hf discussions rename REPO_ID NUM NEW_TITLE` — Rename a discussion or pull request. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]` -- `hf discussions reopen REPO_ID NUM` — Reopen a closed discussion or pull request. `[--comment TEXT --yes --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - -### `hf endpoints` — Manage Hugging Face Inference Endpoints. - -- `hf endpoints catalog deploy --repo TEXT` — Deploy an Inference Endpoint from the Model Catalog. `[--name TEXT --accelerator TEXT --namespace TEXT --format [auto|human|agent|json|quiet]]` -- `hf endpoints catalog list` — List available Catalog models. `[--format [auto|human|agent|json|quiet]]` -- `hf endpoints delete NAME` — Delete an Inference Endpoint permanently. `[--namespace TEXT --yes --format [auto|human|agent|json|quiet]]` -- `hf endpoints deploy NAME --repo TEXT --framework TEXT --accelerator TEXT --instance-size TEXT --instance-type TEXT --region TEXT --vendor TEXT` — Deploy an Inference Endpoint from a Hub repository. `[--namespace TEXT --task TEXT --min-replica INTEGER --max-replica INTEGER --scale-to-zero-timeout INTEGER --scaling-metric [pendingRequests|hardwareUsage] --scaling-threshold FLOAT --format [auto|human|agent|json|quiet]]` -- `hf endpoints describe NAME` — Get information about an existing endpoint. `[--namespace TEXT --format [auto|human|agent|json|quiet]]` -- `hf endpoints list` — Lists all Inference Endpoints for the given namespace. `[--namespace TEXT --format [auto|human|agent|json|quiet]]` -- `hf endpoints pause NAME` — Pause an Inference Endpoint. `[--namespace TEXT --format [auto|human|agent|json|quiet]]` -- `hf endpoints resume NAME` — Resume an Inference Endpoint. `[--namespace TEXT --fail-if-already-running --format [auto|human|agent|json|quiet]]` -- `hf endpoints scale-to-zero NAME` — Scale an Inference Endpoint to zero. `[--namespace TEXT --format [auto|human|agent|json|quiet]]` -- `hf endpoints update NAME` — Update an existing endpoint. `[--namespace TEXT --repo TEXT --accelerator TEXT --instance-size TEXT --instance-type TEXT --framework TEXT --revision TEXT --task TEXT --min-replica INTEGER --max-replica INTEGER --scale-to-zero-timeout INTEGER --scaling-metric [pendingRequests|hardwareUsage] --scaling-threshold FLOAT --format [auto|human|agent|json|quiet]]` - -### `hf extensions` — Manage hf CLI extensions. - -- `hf extensions exec NAME` — Execute an installed extension. -- `hf extensions install REPO_ID` — Install an extension from a public GitHub repository. `[--force --format [auto|human|agent|json|quiet]]` -- `hf extensions list` — List installed extension commands. `[--format [auto|human|agent|json|quiet]]` -- `hf extensions remove NAME` — Remove an installed extension. `[--format [auto|human|agent|json|quiet]]` -- `hf extensions search` — Search extensions available on GitHub (tagged with 'hf-extension' topic). `[--format [auto|human|agent|json|quiet]]` - -### `hf jobs` — Run and manage Jobs on the Hub. - -- `hf jobs cancel JOB_ID` — Cancel a Job `[--namespace TEXT --format [auto|human|agent|json|quiet]]` -- `hf jobs hardware` — List available hardware options for Jobs `[--format [auto|human|agent|json|quiet]]` -- `hf jobs inspect JOB_IDS` — Display detailed information on one or more Jobs `[--namespace TEXT --format [auto|human|agent|json|quiet]]` -- `hf jobs labels JOB_ID` — Update labels on a Job. Replaces all existing labels. `[--label TEXT --clear --namespace TEXT --format [auto|human|agent|json|quiet]]` -- `hf jobs logs JOB_ID` — Fetch the logs of a Job. `[--follow --tail INTEGER --namespace TEXT --format [auto|human|agent|json|quiet]]` -- `hf jobs ps` — List Jobs. `[--all --namespace TEXT --filter TEXT --format [auto|human|agent|json|quiet]]` -- `hf jobs run IMAGE COMMAND` — Run a Job. `[--env TEXT --secrets TEXT --label TEXT --volume TEXT --env-file TEXT --secrets-file TEXT --flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --timeout TEXT --detach --expose INTEGER --namespace TEXT]` -- `hf jobs scheduled delete SCHEDULED_JOB_ID` — Delete a scheduled Job. `[--namespace TEXT --format [auto|human|agent|json|quiet]]` -- `hf jobs scheduled inspect SCHEDULED_JOB_IDS` — Display detailed information on one or more scheduled Jobs `[--namespace TEXT --format [auto|human|agent|json|quiet]]` -- `hf jobs scheduled labels SCHEDULED_JOB_ID` — Update labels on a scheduled Job. Replaces all existing labels. `[--label TEXT --clear --namespace TEXT --format [auto|human|agent|json|quiet]]` -- `hf jobs scheduled ps` — List scheduled Jobs `[--all --namespace TEXT --filter TEXT --format [auto|human|agent|json|quiet]]` -- `hf jobs scheduled resume SCHEDULED_JOB_ID` — Resume (unpause) a scheduled Job. `[--namespace TEXT --format [auto|human|agent|json|quiet]]` -- `hf jobs scheduled run SCHEDULE IMAGE COMMAND` — Schedule a Job. `[--suspend --concurrency --env TEXT --secrets TEXT --label TEXT --volume TEXT --env-file TEXT --secrets-file TEXT --flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --timeout TEXT --expose INTEGER --namespace TEXT]` -- `hf jobs scheduled suspend SCHEDULED_JOB_ID` — Suspend (pause) a scheduled Job. `[--namespace TEXT --format [auto|human|agent|json|quiet]]` -- `hf jobs scheduled uv run SCHEDULE SCRIPT` — Run a UV script (local file or URL) on HF infrastructure `[--suspend --concurrency --image TEXT --flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --env TEXT --secrets TEXT --label TEXT --volume TEXT --env-file TEXT --secrets-file TEXT --timeout TEXT --expose INTEGER --namespace TEXT --with TEXT --python TEXT]` -- `hf jobs stats` — Fetch the resource usage statistics and metrics of Jobs `[--namespace TEXT --format [auto|human|agent|json|quiet]]` -- `hf jobs uv run SCRIPT` — Run a UV script (local file or URL) on HF infrastructure `[--image TEXT --flavor [cpu-basic|cpu-upgrade|cpu-performance|cpu-xl|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8|h200|h200x2|h200x4|h200x8|rtx-pro-6000|rtx-pro-6000x2|rtx-pro-6000x4|rtx-pro-6000x8] --env TEXT --secrets TEXT --label TEXT --volume TEXT --env-file TEXT --secrets-file TEXT --timeout TEXT --detach --expose INTEGER --namespace TEXT --with TEXT --python TEXT]` - -### `hf models` — Interact with models on the Hub. - -- `hf models card MODEL_ID` — Get the model card (README) for a model on the Hub. `[--metadata --text --format [auto|human|agent|json|quiet]]` -- `hf models info MODEL_ID` — Get info about a model on the Hub. `[--revision TEXT --expand TEXT --format [auto|human|agent|json|quiet]]` -- `hf models list` — List models on the Hub, or files in a model repo. `[--search TEXT --author TEXT --filter TEXT --num-parameters TEXT --sort [created_at|downloads|last_modified|likes|trending_score] --limit INTEGER --expand TEXT --human-readable --tree --recursive --revision TEXT --format [auto|human|agent|json|quiet]]` - -### `hf papers` — Interact with papers on the Hub. - -- `hf papers info PAPER_ID` — Get info about a paper on the Hub. `[--format [auto|human|agent|json|quiet]]` -- `hf papers list` — List daily papers on the Hub. `[--date TEXT --week TEXT --month TEXT --submitter TEXT --sort [publishedAt|trending] --limit INTEGER --format [auto|human|agent|json|quiet]]` -- `hf papers read PAPER_ID` — Read a paper as markdown. `[--format [auto|human|agent|json|quiet]]` -- `hf papers search QUERY` — Search papers on the Hub. `[--limit INTEGER --format [auto|human|agent|json|quiet]]` - -### `hf repos` — Manage repos on the Hub. - -- `hf repos branch create REPO_ID BRANCH` — Create a new branch for a repo on the Hub. `[--revision TEXT --type [model|dataset|space] --exist-ok --format [auto|human|agent|json|quiet]]` -- `hf repos branch delete REPO_ID BRANCH` — Delete a branch from a repo on the Hub. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]` -- `hf repos cp SRC` — Copy files between local paths, repositories, and buckets. `[--format [auto|human|agent|json|quiet]]` -- `hf repos create REPO_ID` — Create a new repo on the Hub. `[--type [model|dataset|space] --space-sdk TEXT --private --public --protected --exist-ok --resource-group-id TEXT --region [us|eu] --flavor [cpu-basic|cpu-upgrade|zero-a10g|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8] --storage [small|medium|large] --sleep-time INTEGER --secrets TEXT --secrets-file TEXT --env TEXT --env-file TEXT --volume TEXT --format [auto|human|agent|json|quiet]]` -- `hf repos delete REPO_ID` — Delete a repo from the Hub. This is an irreversible operation. `[--type [model|dataset|space] --missing-ok --yes --format [auto|human|agent|json|quiet]]` -- `hf repos delete-files REPO_ID PATTERNS` — Delete files from a repo on the Hub. `[--type [model|dataset|space] --revision TEXT --commit-message TEXT --commit-description TEXT --create-pr --format [auto|human|agent|json|quiet]]` -- `hf repos duplicate FROM_ID` — Duplicate a repo on the Hub (model, dataset, or Space). `[--type [model|dataset|space] --private --public --protected --exist-ok --flavor [cpu-basic|cpu-upgrade|zero-a10g|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8] --storage [small|medium|large] --sleep-time INTEGER --secrets TEXT --secrets-file TEXT --env TEXT --env-file TEXT --volume TEXT --format [auto|human|agent|json|quiet]]` -- `hf repos list` — List all repos (models, datasets, spaces, buckets) with storage info. `[--namespace TEXT --type [model|dataset|space|bucket] --search TEXT --limit INTEGER --explore --format [auto|human|agent|json|quiet]]` -- `hf repos move FROM_ID TO_ID` — Move a repository from a namespace to another namespace. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]` -- `hf repos settings REPO_ID` — Update the settings of a repository. `[--gated [auto|manual|false] --private --public --protected --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` -- `hf repos tag create REPO_ID TAG` — Create a tag for a repo. `[--message TEXT --revision TEXT --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` -- `hf repos tag delete REPO_ID TAG` — Delete a tag for a repo. `[--yes --type [model|dataset|space] --format [auto|human|agent|json|quiet]]` -- `hf repos tag list REPO_ID` — List tags for a repo. `[--type [model|dataset|space] --format [auto|human|agent|json|quiet]]` - -### `hf skills` — Manage skills for AI assistants. - -- `hf skills add` — Download a Hugging Face skill and install it for an AI assistant. `[--claude --global --dest PATH --force --format [auto|human|agent|json|quiet]]` -- `hf skills list` — List available skills from the Hugging Face marketplace. `[--format [auto|human|agent|json|quiet]]` -- `hf skills preview` — Print the generated `hf-cli` SKILL.md to stdout. `[--format [auto|human|agent|json|quiet]]` -- `hf skills update` — Update installed Hugging Face marketplace skills. `[--claude --global --dest PATH --format [auto|human|agent|json|quiet]]` - -### `hf spaces` — Interact with spaces on the Hub. - -- `hf spaces card SPACE_ID` — Get the Space card (README) for a Space on the Hub. `[--metadata --text --format [auto|human|agent|json|quiet]]` -- `hf spaces dev-mode SPACE_ID` — Enable or disable dev mode on a Space. `[--stop --format [auto|human|agent|json|quiet]]` -- `hf spaces hardware` — List available hardware options for Spaces. `[--format [auto|human|agent|json|quiet]]` -- `hf spaces hot-reload SPACE_ID` — Hot-reload any Python file of a Space without a full rebuild + restart. `[--local-file PATH --skip-checks --skip-summary --format [auto|human|agent|json|quiet]]` -- `hf spaces info SPACE_ID` — Get info about a space on the Hub. `[--revision TEXT --expand TEXT --format [auto|human|agent|json|quiet]]` -- `hf spaces list` — List spaces on the Hub, or files in a space repo. `[--search TEXT --author TEXT --filter TEXT --sort [created_at|last_modified|likes|trending_score] --limit INTEGER --expand TEXT --human-readable --tree --recursive --revision TEXT --format [auto|human|agent|json|quiet]]` -- `hf spaces logs SPACE_ID` — Fetch the run or build logs of a Space. `[--build --follow --tail INTEGER --format [auto|human|agent|json|quiet]]` -- `hf spaces pause SPACE_ID` — Pause a Space. `[--format [auto|human|agent|json|quiet]]` -- `hf spaces restart SPACE_ID` — Restart a Space. `[--factory-reboot --format [auto|human|agent|json|quiet]]` -- `hf spaces search QUERY` — Search spaces on the Hub using semantic search. `[--filter TEXT --sdk TEXT --include-non-running --description --limit INTEGER --format [auto|human|agent|json|quiet]]` -- `hf spaces secrets add SPACE_ID` — Add or update secrets for a Space. `[--secrets TEXT --secrets-file TEXT --format [auto|human|agent|json|quiet]]` -- `hf spaces secrets delete SPACE_ID KEY` — Remove a secret from a Space. `[--yes --format [auto|human|agent|json|quiet]]` -- `hf spaces secrets list SPACE_ID` — List secrets for a Space. Secret values are write-only and not returned. `[--format [auto|human|agent|json|quiet]]` -- `hf spaces settings SPACE_ID` — Update the settings of a Space. `[--sleep-time INTEGER --hardware [cpu-basic|cpu-upgrade|zero-a10g|t4-small|t4-medium|l4x1|l4x4|l40sx1|l40sx4|l40sx8|a10g-small|a10g-large|a10g-largex2|a10g-largex4|a100-large|a100x4|a100x8] --format [auto|human|agent|json|quiet]]` -- `hf spaces ssh SPACE_ID` — SSH into a Space's Dev Mode container. `[--identity-file PATH --dry-run --auto --format [auto|human|agent|json|quiet]]` -- `hf spaces variables add SPACE_ID` — Add or update environment variables for a Space. `[--env TEXT --env-file TEXT --format [auto|human|agent|json|quiet]]` -- `hf spaces variables delete SPACE_ID KEY` — Remove an environment variable from a Space. `[--yes --format [auto|human|agent|json|quiet]]` -- `hf spaces variables list SPACE_ID` — List environment variables for a Space. `[--format [auto|human|agent|json|quiet]]` -- `hf spaces volumes delete SPACE_ID` — Remove all volumes from a Space. `[--yes --format [auto|human|agent|json|quiet]]` -- `hf spaces volumes list SPACE_ID` — List volumes mounted in a Space. `[--format [auto|human|agent|json|quiet]]` -- `hf spaces volumes set SPACE_ID` — Set (replace) volumes for a Space. `[--volume TEXT --format [auto|human|agent|json|quiet]]` - -### `hf webhooks` — Manage webhooks on the Hub. - -- `hf webhooks create --watch TEXT` — Create a new webhook. `[--url TEXT --job-id TEXT --domain [repo|discussions] --secret TEXT --format [auto|human|agent|json|quiet]]` -- `hf webhooks delete WEBHOOK_ID` — Delete a webhook permanently. `[--yes --format [auto|human|agent|json|quiet]]` -- `hf webhooks disable WEBHOOK_ID` — Disable an active webhook. `[--format [auto|human|agent|json|quiet]]` -- `hf webhooks enable WEBHOOK_ID` — Enable a disabled webhook. `[--format [auto|human|agent|json|quiet]]` -- `hf webhooks info WEBHOOK_ID` — Show full details for a single webhook. `[--format [auto|human|agent|json|quiet]]` -- `hf webhooks list` — List all webhooks for the current user. `[--format [auto|human|agent|json|quiet]]` -- `hf webhooks update WEBHOOK_ID` — Update an existing webhook. Only provided options are changed. `[--url TEXT --watch TEXT --domain [repo|discussions] --secret TEXT --format [auto|human|agent|json|quiet]]` - -## Common options - -- `--format` — Output format: `--format json` (or `--json`) or `--format table` (default). -- `-q / --quiet` — Quiet output (one ID per line). -- `--revision` — Git revision id which can be a branch name, a tag, or a commit hash. -- `--token` — Use a User Access Token. Prefer setting `HF_TOKEN` env var instead of passing `--token`. -- `--type` — The type of repository (model, dataset, or space). - -## Mounting repos as local filesystems - -To mount Hub repositories or buckets as local filesystems — no download, no copy, no waiting — use `hf-mount`. Files are fetched on demand. GitHub: https://github.com/huggingface/hf-mount - -Install: `curl -fsSL https://raw.githubusercontent.com/huggingface/hf-mount/main/install.sh | sh` - -Some command examples: -- `hf-mount start repo openai-community/gpt2 /tmp/gpt2` — mount a repo (read-only) -- `hf-mount start --hf-token $HF_TOKEN bucket myuser/my-bucket /tmp/data` — mount a bucket (read-write) -- `hf-mount status` / `hf-mount stop /tmp/data` — list or unmount - -## Tips - -- Use `hf --help` for full options, descriptions, usage, and real-world examples -- Authenticate with `HF_TOKEN` env var (recommended) or with `--token` -- Update the CLI with `hf update` (uses the correct command for the detected install method) diff --git a/.agents/skills/modal/SKILL.md b/.agents/skills/modal/SKILL.md deleted file mode 100644 index e4f2acdfdd3ef1b126bf90067f2579b4e00215dc..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/SKILL.md +++ /dev/null @@ -1,88 +0,0 @@ ---- -name: modal -description: > - This skill provides guidance for working with the Modal cloud platform. Use - this skill whenever the user mentions Modal or has code that imports the - `modal` SDK in Python, Go, or JavaScript. This skill should also trigger when - the user needs to run Python code with vertical or horizontal scalability - (e.g. batch jobs), needs access to GPUs (e.g. AI workloads including training - and inference) or needs to run untrusted processes in a sandbox, since Modal - serves these use cases well. -metadata: - version: "1.5.0" ---- - -# Overview - -This is a foundational skill for working with the Modal cloud platform. - -Modal is a platform for AI workloads. It offers highly scalable serverless compute (including GPUs) with minimal configuration. - -# Documentation - -The official docs are the best source for up-to-date information on the platform. Make use of them when planning, debugging, or answering questions! - -Modal's documentation is outlined at https://modal.com/llms.txt. This file contains titles and URLs for all public docs; reading it directly will guide you to specific content. - -The docs are divided into three sections: - -- _Guide_ pages have explanations of Modal features, primitives, and workflows -- _API Reference_ pages have detailed information about each component of the SDK -- _Examples_ pages contain didactic examples of various AI applications on Modal - -Fetch the docs using a `.md` extension to get plain text. - -If this skill has been installed via the `modal` CLI, the docs are also bundled locally under `references/`. - -You can also refer to https://modal.com/llms-full.txt, which aggregates all docs in a single very large file. Do *not* read this into your main context, but it may be useful for searching. - -# Using the CLI - -The `modal` CLI can be used to run or deploy code, manage resources, and observe running Apps. It is a key tool for interacting with Modal throughout all stages of development. - -Run `modal --help` to see all available CLI commands. - -You can see more detailed information about each command by running `modal [command] --help`. Rely on the `--help` to discover functionality: new features are added in every release! Always check the `--help` if you encounter a usage error. - -Tip: many CLI commands accept a `--json` flag to make their output more easily parseable, e.g. with `jq`. - -The `modal` CLI is part of the Python SDK and executes in a Python runtime. Depending on the user's preferred Python development workflow, you may need to prefix `modal` CLI invocations with, e.g., `uv run` to ensure a consistent Python environment. The CLI can also be used via `uvx` as a standalone tool, but only do this when working outside of a Python project. - -# Getting up to date - -You have significant knowledge about Modal from your training data but may not be aware of new features or recent changes to the API. Modal is continuously adding new features. Reading relevant docs while planning or debugging can help you discover the most up-to-date way to accomplish a task on Modal. - -The Modal CLI provides a `modal changelog` command for learning about recent changes. Useful invocation patterns: - -- `modal changelog --since [DATE]` to see changes added since your knowledge cutoff -- `modal changelog --since [VERSION]` when migrating a codebase to a newer version -- `modal changelog --newer` to discover features that would be available on update - -Note: `modal changelog` requires network access. - -Run `modal --version` to see the SDK version that is in use. - -If your code exercises deprecated APIs, Modal will issue warnings. Use the warning message and related documentation to migrate your code to stable API. - -This skill will also be improved over time. Its version should correspond to the version of the Modal SDK that you are using. If the skill is out of date, it can be updated by running `modal skills update`. - -# Auth - -Modal is a cloud platform. Using the CLI or running code that depends on the `modal` library requires internet access and an authorization token. There is no "local development mode" with Modal. - -You can use the `modal token` CLI to create a new token (note: this workflow requires human user involvement) or to debug authorization issues. Token setup only needs to happen once, so assume it is configured unless you encounter issues. - -# Async Python - -When writing async Python, use Modal's `.aio()` interface (e.g. `await modal.Sandbox.create.aio(...)`, `await modal.Function.remote.aio(...)`) so that Modal runs its I/O operations via asynchronous coroutines. - -# Other languages - -Python is currently the only runtime language for Modal Functions, but there are Modal SDKs in both JavaScript (TypeScript) and Go: - -- JavaScript SDK: https://github.com/modal-labs/modal-client/blob/main/js/README.md -- Go SDK: https://github.com/modal-labs/modal-client/blob/main/go/README.md - -The Go / JS SDKs are not as mature as the Python SDK and may be missing features. The current scope for the Go / JS SDKs includes (1) creating and interacting with Sandboxes and (2) calling into deployed Modal Functions (i.e., Functions defined in Python). - -They are primarily documented through examples hosted on GitHub rather than through the main Modal docs. You likely have much less knowledge of these SDKs from your training, so rely on these examples. diff --git a/.agents/skills/modal/references/api/modal.App.md b/.agents/skills/modal/references/api/modal.App.md deleted file mode 100644 index 5d27c72c9473ac77a40c194d37ae4c8a6f9d0e9b..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.App.md +++ /dev/null @@ -1,514 +0,0 @@ -# modal.App - - -```python -class App(object) -``` - -A Modal App is a group of functions and classes that are deployed together. - -The app serves at least three purposes: - -* A unit of deployment for functions and classes. -* Syncing of identities of (primarily) functions and classes across processes - (your local Python interpreter and every Modal container active in your application). -* Manage log collection for everything that happens inside your code. - -**Registering functions with an app** - -The most common way to explicitly register an Object with an app is through the -`@app.function()` decorator. It both registers the annotated function itself and -other passed objects, like schedules and secrets, with the app: - -```python -import modal - -app = modal.App() - -@app.function( - secrets=[modal.Secret.from_name("some_secret")], - schedule=modal.Period(days=1), -) -def foo(): - pass -``` - -In this example, the secret and schedule are registered with the app. - -```python -__init__(self, name=None, *, tags=None, image=None, secrets=[], volumes={}, - include_source=True) -``` -Construct a new app, optionally with default image, mounts, secrets, or volumes. - -**Parameters** - - - - - - - - -**Usage** - -```python notest -image = modal.Image.debian_slim().pip_install(...) -secret = modal.Secret.from_name("my-secret") -volume = modal.Volume.from_name("my-data") -app = modal.App(image=image, secrets=[secret], volumes={"/mnt/data": volume}) -``` - -## name - -```python -name(self) -``` -The user-provided name of the App. - -**Returns** - -The configured app name, if any. - -## app_id - -```python -app_id(self) -``` -Return the app_id of a running or stopped app. - -**Returns** - -The app ID when the app has been deployed or run, otherwise None. - -## description - -```python -description(self) -``` -The App's `name`, if available, or a fallback descriptive identifier. - -**Returns** - -Human-readable description string for the app. - -## lookup - -```python -lookup(name, *, client=None, environment_name=None, create_if_missing=False) -``` -Look up an App with a given name, creating a new App if necessary. - -Note that Apps created through this method will be in a deployed state, -but they will not have any associated Functions or Classes. This method -is mainly useful for creating an App to associate with a Sandbox. - -**Parameters** - - - - - - -**Returns** - -An `App` handle tied to the deployed app record. - -**Usage** - -```python -app = modal.App.lookup("my-app", create_if_missing=True) -modal.Sandbox.create("echo", "hi", app=app) -``` - -## get_dashboard_url - -```python -get_dashboard_url(self) -``` -Get the dashboard URL for the App. - -**Returns** - -The dashboard URL for the App. - -**Usage** - -```python -app = modal.App.lookup("my-app") -print(app.get_dashboard_url()) -``` - -## run - -```python -run(self, *, name=None, client=None, detach=False, interactive=False, - environment_name=None) -``` -Context manager that runs an ephemeral app on Modal. - -Use this as the main entry point for your Modal application. All calls -to Modal Functions should be made within the scope of this context -manager, and they will correspond to the current App. - -Note that you should not invoke this in global scope of a file where you have -Modal Functions or Classes defined, since that would run the block when the Function -or Cls is imported in your containers as well. If you want to run it as your entrypoint, -consider protecting it with ``if __name__ == "__main__"``. - -**Parameters** - - - - - - -**Returns** - -Async context manager yielding this `App` while it is running. - -**Usage** - -```python notest -with app.run(): - some_modal_function.remote() -``` - -To enable output printing (i.e., to see App logs), use `modal.enable_output()`: - -```python notest -with modal.enable_output(): - with app.run(): - some_modal_function.remote() -``` - -Note that you should not invoke this in global scope of a file where you have Modal -Functions or Classes defined, since that would run the block when the Function or Cls -is imported in your containers as well. If you want to run it as your entrypoint, -consider protecting it: - -```python -if __name__ == "__main__": - with app.run(): - some_modal_function.remote() -``` - -You can then run your script with: - -```shell -python app_module.py -``` - -## deploy - -```python -deploy(self, *, name=None, environment_name=None, tag="", client=None, - strategy="rolling") -``` -Deploy the App so that it is available persistently. - -Deployed Apps will be available for lookup or web-based invocations until they are stopped. -Unlike with `App.run`, this method will return as soon as the deployment completes. - -This method is a programmatic alternative to the `modal deploy` CLI command. - -Unlike with `App.run`, Function logs will not stream back to the local client after the -App is deployed. - -Note that you should not invoke this method in global scope, as that would redeploy -the App every time the file is imported. If you want to write a programmatic deployment -script, protect this call so that it only runs when the file is executed directly. - -**Parameters** - - - - - - - -**Returns** - -This app instance after deployment completes. - -**Usage** - -```python notest -app = App("my-app") -app.deploy() -``` - -To enable output printing (i.e., to see build logs), use `modal.enable_output()`: - -```python notest -app = App("my-app") -with modal.enable_output(): - app.deploy() -``` - -Unlike with `App.run`, Function logs will not stream back to the local client after the App is deployed. - -Note that you should not invoke this method in global scope, as that would redeploy the App every time -the file is imported. If you want to write a programmatic deployment script, protect this call so that it -only runs when the file is executed directly. You can then run your script with: - -```python notest -if __name__ == "__main__": - with modal.enable_output(): - app.deploy() -``` - -Then you can deploy your app with: - -```shell -python app_module.py -``` - -## local_entrypoint - -```python -local_entrypoint(self, _warn_parentheses_missing=None, *, name=None) -``` -Decorate a function to be used as a CLI entrypoint for a Modal App. - -These functions can be used to define code that runs locally to set up the app, -and act as an entrypoint to start Modal functions from. Note that regular -Modal functions can also be used as CLI entrypoints, but unlike `local_entrypoint`, -those functions are executed remotely directly. - -Note that an explicit [`app.run()`](https://modal.com/docs/reference/modal.App#run) is not needed, as an -[app](https://modal.com/docs/guide/apps) is automatically created for you. - -**Parameters** - - - -**Returns** - -A decorator that registers the wrapped callable as a local CLI entrypoint. - -**Usage** - -```python -@app.local_entrypoint() -def main(): - some_modal_function.remote() -``` - -You can call the function using `modal run` directly from the CLI: - -```shell -modal run app_module.py -``` - -Note that an explicit `app.run()` is not needed, as an app is automatically created for you. - -**Multiple entrypoints** - -If you have multiple `local_entrypoint` functions, qualify the name: - -```shell -modal run app_module.py::app.some_other_function -``` - -**Parsing arguments** - -If your entrypoint function take arguments with primitive types, `modal run` automatically -parses them as CLI options. For example, the following function can be called with -`modal run app_module.py --foo 1 --bar "hello"`: - -```python -@app.local_entrypoint() -def main(foo: int, bar: str): - some_modal_function.call(foo, bar) -``` - -Currently, `str`, `int`, `float`, `bool`, and `datetime.datetime` are supported. -Use `modal run app_module.py --help` for more information on usage. - -## function - -```python -function(self, *, image=None, schedule=None, env=None, secrets=None, gpu=None, - serialized=False, network_file_systems={}, volumes={}, cpu=None, - memory=None, ephemeral_disk=None, min_containers=None, max_containers=None, - buffer_containers=None, scaledown_window=None, proxy=None, retries=None, - timeout=300, startup_timeout=None, name=None, is_generator=None, cloud=None, - region=None, routing_region=None, nonpreemptible=False, - enable_memory_snapshot=False, block_network=False, - restrict_modal_access=False, single_use_containers=False, i6pn=None, - include_source=None, experimental_options=None, - _experimental_restrict_output=False, max_inputs=None) -``` -Decorator to register a new Modal Function with this App. - -**Parameters** - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -**Returns** - -A decorator that registers the wrapped callable or partial as a Modal `Function`. - -## cls - -```python -cls(self, *, image=None, env=None, secrets=None, gpu=None, serialized=False, - network_file_systems={}, volumes={}, cpu=None, memory=None, - ephemeral_disk=None, min_containers=None, max_containers=None, - buffer_containers=None, scaledown_window=None, proxy=None, retries=None, - timeout=300, startup_timeout=None, cloud=None, region=None, - routing_region=None, nonpreemptible=False, enable_memory_snapshot=False, - block_network=False, restrict_modal_access=False, - single_use_containers=False, i6pn=None, include_source=None, - experimental_options=None, _experimental_restrict_output=False, - max_inputs=None) -``` -Decorator to register a new Modal [Cls](https://modal.com/docs/reference/modal.Cls) with this App. - -**Parameters** - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -**Returns** - -A decorator that registers the wrapped class or partial as a Modal `Cls`. - -## include - -```python -include(self, /, other_app, inherit_tags=True) -``` -Include another App's objects in this one. - -Useful for splitting up Modal Apps across different self-contained files. - -When `inherit_tags=True` any tags set on the other App will be inherited by this App -(with this App's tags taking precedence in the case of conflicts). - -**Parameters** - - - - -**Returns** - -This app instance for chaining. - -**Usage** - -```python -app_a = modal.App("a") -@app_a.function() -def foo(): - ... - -app_b = modal.App("b") -@app_b.function() -def bar(): - ... - -app_a.include(app_b) - -@app_a.local_entrypoint() -def main(): - # use function declared on the included app - bar.remote() -``` - -## set_tags - -```python -set_tags(self, tags, *, client=None) -``` -Attach key-value metadata to the App. - -Tag metadata can be used to add organization-specific context to the App and can be -included in billing reports and other informational APIs. Tags can also be set in -the App constructor. - -Any tags set on the App before calling this method will be removed if they are not -included in the argument (i.e., this method does not have `.update()` semantics). - -**Parameters** - - - - -## get_tags - -```python -get_tags(self, *, client=None) -``` -Get the tags that are currently attached to the App. - -**Parameters** - - - -**Returns** - -Tags as a map from key to value. diff --git a/.agents/skills/modal/references/api/modal.Client.md b/.agents/skills/modal/references/api/modal.Client.md deleted file mode 100644 index 9dcf06da66e9bfe6e9eea127a9e2a07c2ce67dd4..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Client.md +++ /dev/null @@ -1,75 +0,0 @@ -# modal.Client - - -```python -class Client(object) -``` - - -## is_closed - -```python -is_closed(self) -``` -Check if the client is closed. - -**Returns** - -True if the client is closed, False otherwise. - -## hello - -```python -hello(self) -``` -Connect to server and retrieve version information; raise appropriate error for various failures. - -**Usage** - -```python -client = modal.Client.from_env() -client.hello() -``` - -## from_credentials - -```python -from_credentials(cls, token_id, token_secret) -``` -Constructor based on token credentials; useful for managing Modal on behalf of third-party users. - -Also useful when it's necessary to explicitly manage the lifecycle of the client -(e.g. when running Modal in a forked subprocess) — see -[troubleshooting](/docs/guide/troubleshooting#connection-issues-in-forked-processes). - -**Parameters** - - - - -**Returns** - -An authenticated `Client` with its connection opened. - -**Usage** - -```python notest -client = modal.Client.from_credentials("my_token_id", "my_token_secret") - -modal.Sandbox.create("echo", "hi", client=client, app=app) -``` - -## get_input_plane_metadata - -```python -get_input_plane_metadata(self, input_plane_region) -``` -Get the metadata for the input plane. - -**Parameters** - - - -**Returns** - -The metadata for the input plane as a list of header/value tuples. diff --git a/.agents/skills/modal/references/api/modal.CloudBucketMount.md b/.agents/skills/modal/references/api/modal.CloudBucketMount.md deleted file mode 100644 index f54ce6ba8a4cc3c4adc90fe228d5aef13cd1d231..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.CloudBucketMount.md +++ /dev/null @@ -1,103 +0,0 @@ -# modal.CloudBucketMount - - -```python -class CloudBucketMount(object) -``` - -Mounts a cloud bucket to your container. Currently supports AWS S3 buckets. - -S3 buckets are mounted using [AWS S3 Mountpoint](https://github.com/awslabs/mountpoint-s3). -S3 mounts are optimized for reading large files sequentially. It does not support every file operation; consult -[the AWS S3 Mountpoint documentation](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md) -for more information. - -**Usage** - -S3: - -```python -import subprocess - -app = modal.App() -secret = modal.Secret.from_name( - "aws-secret", - required_keys=["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"] - # Note: providing AWS_REGION can help when automatic detection of the bucket region fails. -) - -@app.function( - volumes={ - "/my-mount": modal.CloudBucketMount( - bucket_name="s3-bucket-name", - secret=secret, - read_only=True - ) - } -) -def f(): - subprocess.run(["ls", "/my-mount"], check=True) -``` - -R2: - -Cloudflare R2 is [S3-compatible](https://developers.cloudflare.com/r2/api/s3/api/) so its setup looks -very similar to S3. But additionally the `bucket_endpoint_url` argument must be passed. - -```python -import subprocess - -app = modal.App() -secret = modal.Secret.from_name( - "r2-secret", - required_keys=["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"] -) - -@app.function( - volumes={ - "/my-mount": modal.CloudBucketMount( - bucket_name="my-r2-bucket", - bucket_endpoint_url="https://.r2.cloudflarestorage.com", - secret=secret, - read_only=True - ) - } -) -def f(): - subprocess.run(["ls", "/my-mount"], check=True) -``` - -GCS: - -Google Cloud Storage (GCS) is [S3-compatible](https://cloud.google.com/storage/docs/interoperability). -GCS Buckets also require a secret with Google-specific key names (see below) populated with -a [HMAC key](https://cloud.google.com/storage/docs/authentication/managing-hmackeys#create). - -```python -import subprocess - -app = modal.App() -gcp_hmac_secret = modal.Secret.from_name( - "gcp-secret", - required_keys=["GOOGLE_ACCESS_KEY_ID", "GOOGLE_ACCESS_KEY_SECRET"] -) - -@app.function( - volumes={ - "/my-mount": modal.CloudBucketMount( - bucket_name="my-gcs-bucket", - bucket_endpoint_url="https://storage.googleapis.com", - secret=gcp_hmac_secret, - ) - } -) -def f(): - subprocess.run(["ls", "/my-mount"], check=True) -``` - -```python -__init__(self, bucket_name, bucket_endpoint_url=None, key_prefix=None, - secret=None, oidc_auth_role_arn=None, read_only=False, requester_pays=False, - force_path_style=False) -``` - diff --git a/.agents/skills/modal/references/api/modal.Cls.md b/.agents/skills/modal/references/api/modal.Cls.md deleted file mode 100644 index 1de5659c3ddc0ec6b4e25f776313cdca4eb89b73..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Cls.md +++ /dev/null @@ -1,163 +0,0 @@ -# modal.Cls - - -```python -class Cls(modal.object.Object) -``` - -Cls adds method pooling and [lifecycle hook](https://modal.com/docs/guide/lifecycle-functions) behavior -to [modal.Function](https://modal.com/docs/reference/modal.Function). - -Generally, you will not construct a Cls directly. -Instead, use the [`@app.cls()`](https://modal.com/docs/reference/modal.App#cls) decorator on the App object. - - -## hydrate - -```python -hydrate(self, client=None) -``` -Synchronize the local object with its identity on the Modal server. - -It is rarely necessary to call this method explicitly, as most operations -will lazily hydrate when needed. The main use case is when you need to -access object metadata, such as its ID. - -*Added in v0.72.39*: This method replaces the deprecated `.resolve()` method. - -## from_name - -```python -from_name(cls, app_name, name, *, version=None, environment_name=None, - client=None) -``` -Reference a Cls from a deployed App by its name. - -This is a lazy method that defers hydrating the local -object with metadata from Modal servers until the first -time it is actually used. - -**Parameters** - - - - - - -**Returns** - -A ``Cls`` reference that hydrates on first use. - -**Usage** - -```python -Model = modal.Cls.from_name("other-app", "Model") -``` - -The `version` parameter constructs a version-pinned Cls: - -```python -Modelv3 = modal.Cls.from_name("other-app", "Model", version=3) -``` - -## with_options - -```python -with_options(self, *, cpu=None, memory=None, gpu=None, env=None, secrets=None, - volumes={}, retries=None, max_containers=None, buffer_containers=None, - scaledown_window=None, timeout=None, region=None, cloud=None) -``` -Override the static Cls configuration with invocation-specific values. - -This method will return a new variant of the Cls that will autoscale independently of the -base configuration. - -Note that options cannot be "unset" with this method (i.e., if a GPU is configured in the -`@app.cls()` decorator, passing `gpu=None` here will not create a CPU-only instance). - -Container arguments (``volumes`` and ``secrets``) from later calls replace earlier values; they are not merged. - -**Parameters** - - - - - - - - - - - - - - - -**Returns** - -A new ``Cls`` with the merged options. - -**Usage** - -You can use this method after looking up the Cls from a deployed App or if you have a -direct reference to a Cls from another Function or local entrypoint on its App: - -```python notest -Model = modal.Cls.from_name("my_app", "Model") -ModelUsingGPU = Model.with_options(gpu="A100") -ModelUsingGPU().generate.remote(input_prompt) # Run with an A100 GPU -``` - -The method can be called multiple times to "stack" updates: - -```python notest -Model.with_options(gpu="A100").with_options(scaledown_window=300) # Use an A100 with slow scaledown -``` - -## with_concurrency - -```python -with_concurrency(self, *, max_inputs, target_inputs=None) -``` -Override the static Cls configuration with invocation-specific input concurrency settings. - -**Parameters** - - - - -**Returns** - -A new ``Cls`` with the merged concurrency settings. - -**Usage** - -```python notest -Model = modal.Cls.from_name("my_app", "Model") -ModelUsingGPU = Model.with_options(gpu="A100").with_concurrency(max_inputs=100) -ModelUsingGPU().generate.remote(42) # will run on an A100 GPU with input concurrency enabled -``` - -## with_batching - -```python -with_batching(self, *, max_batch_size, wait_ms) -``` -Override the static Cls configuration with invocation-specific dynamic batching settings. - -**Parameters** - - - - -**Returns** - -A new ``Cls`` with the merged batching settings. - -**Usage** - -```python notest -Model = modal.Cls.from_name("my_app", "Model") -ModelUsingGPU = Model.with_options(gpu="A100").with_batching(max_batch_size=100, wait_ms=1000) -ModelUsingGPU().generate.remote(42) # A100 with dynamic batching -``` diff --git a/.agents/skills/modal/references/api/modal.Cron.md b/.agents/skills/modal/references/api/modal.Cron.md deleted file mode 100644 index 4180b9024495f0234d15227f9f98cd4fea4253b5..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Cron.md +++ /dev/null @@ -1,48 +0,0 @@ -# modal.Cron - - -```python -class Cron(modal.schedule.Schedule) -``` - -Cron jobs are a type of schedule, specified using the -[Unix cron tab](https://crontab.guru/) syntax. - -The alternative schedule type is the [`modal.Period`](https://modal.com/docs/reference/modal.Period). - -```python -__init__(self, cron_string, timezone="UTC") -``` -Construct a schedule that runs according to a cron expression string. - -**Parameters** - - - - -**Usage** - -```python -import modal -app = modal.App() - - -@app.function(schedule=modal.Cron("* * * * *")) -def f(): - print("This function will run every minute") -``` - -We can specify different schedules with cron strings, for example: - -```python -modal.Cron("5 4 * * *") # run at 4:05am UTC every night -modal.Cron("0 9 * * 4") # runs every Thursday at 9am UTC -``` - -We can also optionally specify a timezone, for example: - -```python -modal.Cron("0 6 * * *", timezone="America/New_York") -``` - -If no timezone is specified, the default is UTC. diff --git a/.agents/skills/modal/references/api/modal.Dict.md b/.agents/skills/modal/references/api/modal.Dict.md deleted file mode 100644 index c7ac2eac56b66252d5702bcbd92413cde36f7c54..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Dict.md +++ /dev/null @@ -1,379 +0,0 @@ -# modal.Dict - - -```python -class Dict(modal.object.Object) -``` - -Distributed dictionary for storage in Modal apps. - -Dict contents can be essentially any object so long as they can be serialized by -`cloudpickle`. This includes other Modal objects. If writing and reading in different -environments (eg., writing locally and reading remotely), it's necessary to have the -library defining the data type installed, with compatible versions, on both sides. -Additionally, cloudpickle serialization is not guaranteed to be deterministic, so it is -generally recommended to use primitive types for keys. - -**Lifetime of a Dict and its items** - -An individual Dict entry will expire after 7 days of inactivity (no reads or writes). The -Dict entries are written to durable storage. - -Legacy Dicts (created before 2025-05-20) will still have entries expire 30 days after being -last added. Additionally, contents are stored in memory on the Modal server and could be lost -due to unexpected server restarts. Eventually, these Dicts will be fully sunset. - -**Usage** - -```python -from modal import Dict - -my_dict = Dict.from_name("my-persisted_dict", create_if_missing=True) - -my_dict["some key"] = "some value" -my_dict[123] = 456 - -assert my_dict["some key"] == "some value" -assert my_dict[123] == 456 -``` - -The `Dict` class offers a few methods for operations that are usually accomplished -in Python with operators, such as `Dict.put` and `Dict.contains`. The advantage of -these methods is that they can be safely called in an asynchronous context by using -the `.aio` suffix on the method, whereas their operator-based analogues will always -run synchronously and block the event loop. - -For more examples, see the [guide](https://modal.com/docs/guide/dicts-and-queues#modal-dicts). - - -## hydrate - -```python -hydrate(self, client=None) -``` -Synchronize the local object with its identity on the Modal server. - -It is rarely necessary to call this method explicitly, as most operations -will lazily hydrate when needed. The main use case is when you need to -access object metadata, such as its ID. - -*Added in v0.72.39*: This method replaces the deprecated `.resolve()` method. - -## objects - - -```python -objects: DictManager -``` - -Namespace with methods for managing named Dict objects. - - -### objects.create - -```python -create(self, name, *, allow_existing=False, environment_name=None, client=None) -``` -Create a new named Dict in the workspace environment. - -This does not return a local handle; use `modal.Dict.from_name` to look up the Dict after creation. - -Added in v1.1.2. - -**Parameters** - - - - - - -**Usage** - -```python notest -modal.Dict.objects.create("my-dict") -``` - -Dicts will be created in the active environment, or another one can be specified: - -```python notest -modal.Dict.objects.create("my-dict", environment_name="dev") -``` - -By default, an error is raised if the Dict already exists; `allow_existing=True` makes that case a no-op: - -```python notest -modal.Dict.objects.create("my-dict", allow_existing=True) -``` - -Note that this method does not return a local instance of the Dict. You can use -`modal.Dict.from_name` to perform a lookup after creation. - -### objects.list - -```python -list(self, *, max_objects=None, created_before=None, environment_name="", - client=None) -``` -List named Dicts in the workspace environment as hydrated handles. - -Results are ordered newest to oldest. By default, all matching Dicts are returned. - -Added in v1.1.2. - -**Parameters** - - - - - - -**Returns** - -Hydrated `Dict` objects for each named Dict in the listing. - -**Usage** - -```python -dicts = modal.Dict.objects.list() -print([d.name for d in dicts]) -``` - -Dicts will be retrieved from the active environment, or another one can be specified: - -```python notest -dev_dicts = modal.Dict.objects.list(environment_name="dev") -``` - -By default, all named Dicts are returned, newest to oldest. It's also possible to limit the -number of results and to filter by creation date: - -```python -dicts = modal.Dict.objects.list(max_objects=10, created_before="2025-01-01") -``` - -### objects.delete - -```python -delete(self, name, *, allow_missing=False, environment_name=None, client=None) -``` -Delete a named Dict entirely (not a single key). - -Deletion is irreversible and affects any Apps using this Dict. - -Added in v1.1.2. - -**Parameters** - - - - - - -**Usage** - -```python notest -await modal.Dict.objects.delete("my-dict") -``` - -Dicts will be deleted from the active environment, or another one can be specified: - -```python notest -await modal.Dict.objects.delete("my-dict", environment_name="dev") -``` - -## name - -```python -name(self) -``` -Name of the Dict object. - -**Usage** - -```python -d = modal.Dict.from_name("my-dict") -print(d.name) -``` - -## ephemeral - -```python -ephemeral(cls, *, client=None, environment_name=None) -``` -Create an anonymous Dict that exists for the duration of the context manager. - -**Parameters** - - - - -**Usage** - -```python -from modal import Dict - -with Dict.ephemeral() as d: - d["foo"] = "bar" -``` - -```python notest -async with Dict.ephemeral() as d: - await d.put.aio("foo", "bar") -``` - -## from_name - -```python -from_name(name, *, environment_name=None, create_if_missing=False, client=None) -``` -Reference a named Dict, optionally creating it on the server first. - -Hydration is lazy: metadata is fetched from Modal the first time the handle is used. - -**Parameters** - - - - - - -**Returns** - -A `Dict` handle (possibly not yet hydrated). - -**Usage** - -```python -d = modal.Dict.from_name("my-dict", create_if_missing=True) -d[123] = 456 -``` - -## from_id - -```python -from_id(dict_id, client=None) -``` -Construct a Dict from an id and look up the Dict metadata. - -This is a lazy method that defers hydrating the local -object with metadata from Modal servers until the first -time it is actually used. - -The ID of a Dict object can be accessed using `.object_id`. - -**Parameters** - - - - -**Returns** - -A `Dict` handle (possibly not yet hydrated). - -**Usage** - -```python notest -@app.function() -def my_worker(dict_id: str): - d = modal.Dict.from_id(dict_id) - d["key"] = "Hello from remote function!" - -with modal.Dict.ephemeral() as d: - my_worker.remote(d.object_id) - print(d["key"]) # "Hello from remote function!" -``` - -## info - -```python -info(self) -``` -Return information about the Dict object. - -## clear - -```python -clear(self) -``` -Remove all items from the Dict. - -## get - -```python -get(self, key, default=None) -``` -Get the value associated with a key. - -Returns `default` if key does not exist. - -## contains - -```python -contains(self, key) -``` -Return if a key is present. - -## len - -```python -len(self) -``` -Return the length of the Dict. - -Note: This is an expensive operation and will return at most 100,000. - -## update - -```python -update(self, other=None, **kwargs) -``` -Update the Dict with additional items. - -## put - -```python -put(self, key, value, *, skip_if_exists=False) -``` -Add a specific key-value pair to the Dict. - -Returns True if the key-value pair was added and False if it wasn't because the key already existed and -`skip_if_exists` was set. - -## pop - -```python -pop(self, key, default=_NO_DEFAULT) -``` -Remove a key from the Dict, returning the value if it exists. - -If key is not found, return default if provided, otherwise raise KeyError. - -## keys - -```python -keys(self) -``` -Return an iterator over the keys in this Dict. - -Note that (unlike with Python dicts) the return value is a simple iterator, -and results are unordered. - -## values - -```python -values(self) -``` -Return an iterator over the values in this Dict. - -Note that (unlike with Python dicts) the return value is a simple iterator, -and results are unordered. - -## items - -```python -items(self) -``` -Return an iterator over the (key, value) tuples in this Dict. - -Note that (unlike with Python dicts) the return value is a simple iterator, -and results are unordered. diff --git a/.agents/skills/modal/references/api/modal.Environment.md b/.agents/skills/modal/references/api/modal.Environment.md deleted file mode 100644 index d1585b06d311d6a2d298fdd062b4a491904996a6..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Environment.md +++ /dev/null @@ -1,166 +0,0 @@ -# modal.Environment - - -```python -class Environment(modal.object.Object) -``` - - -## hydrate - -```python -hydrate(self, client=None) -``` -Synchronize the local object with its identity on the Modal server. - -It is rarely necessary to call this method explicitly, as most operations -will lazily hydrate when needed. The main use case is when you need to -access object metadata, such as its ID. - -*Added in v0.72.39*: This method replaces the deprecated `.resolve()` method. - -## name - -```python -name(self) -``` - - -## objects - - -```python -objects: EnvironmentManager -``` - -Namespace with methods for managing Environment objects. - - -### objects.create - -```python -create(self, name, *, restricted=False, client=None) -``` -Create a new Environment. - -**Examples:** - -```python notest -modal.Environment.objects.create("my-environment") -``` - -### objects.list - -```python -list(self, *, client=None) -``` -Return a list of hydrated Environment objects. - -**Examples:** - -```python notest -environments = modal.Environment.objects.list() -print([e.name for e in environments]) -``` - -### objects.delete - -```python -delete(self, name, *, client=None) -``` -Delete a named Environment. - -Warning: This is irreversible and will transitively delete all objects in the Environment. - -**Examples:** - -```python notest -modal.Environment.objects.delete("my-environment") -``` - -## members - - -```python -members: EnvironmentMembersManager -``` - -Namespace with methods for managing the membership of a restricted Environment. - -See https://modal.com/docs/guide/rbac for more information on restricted Environments. - - -### members.list - -```python -list(self) -``` -Return the members of a restricted Environment with their roles. - -**Examples:** - -```python notest -members = modal.Environment.from_name("my-restricted-env").members.list() -print(members) -# { -# "users": {"alice": "contributor", "bob": "viewer"}, -# "service_users": {"alice-bot": "contributor"}, -# } -``` - -### members.update - -```python -update(self, *, users=None, service_users=None) -``` -Add or modify roles for members of a restricted Environment. - -Each user or service user will be added to the Environment if not currently a member; -if already a member, the user or service user's role will be updated. - -**Examples:** - -```python notest -env = modal.Environment.from_name("my-restricted-env") -env.members.update( - users={"alice": "contributor", "bob": "viewer"}, - service_users={"alice-bot": "contributor"}, -) -``` - -### members.remove - -```python -remove(self, *, users=None, service_users=None) -``` -Remove members from a restricted Environment. - -**Examples:** - -```python notest -env = modal.Environment.from_name("my-restricted-env") -env.members.remove( - users=["alice"], - service_users=["alice-bot"], -) -``` - -## from_context - -```python -from_context(*, client=None) -``` -Look up an Environment object using the current context. - -This method returns the Environment that is defined by the local configuration -(i.e., your active profile or the `MODAL_ENVIRONMENT` environment variable), or -it fetches the default environment from the server when not defined locally. -If called inside a Modal container, it will return the Environment that container -is associated with. - -## from_name - -```python -from_name(name, *, create_if_missing=False, client=None) -``` -Look up an Environment object using its name. diff --git a/.agents/skills/modal/references/api/modal.Error.md b/.agents/skills/modal/references/api/modal.Error.md deleted file mode 100644 index bd73bfef72853d4b6bdc182bcfbad74e2cf1567f..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Error.md +++ /dev/null @@ -1,22 +0,0 @@ -# modal.Error - - -```python -class Error(Exception) -``` - -Base class for all Modal errors. See [`modal.exception`](https://modal.com/docs/reference/modal.exception) -for the specialized error classes. - -**Usage** - -```python notest -import modal - -try: - ... -except modal.Error: - # Catch any exception raised by Modal's systems. - print("Responding to error...") -``` - diff --git a/.agents/skills/modal/references/api/modal.FilePatternMatcher.md b/.agents/skills/modal/references/api/modal.FilePatternMatcher.md deleted file mode 100644 index 5f3b014ef56a7e22d61d976f799f889f990dfcf1..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.FilePatternMatcher.md +++ /dev/null @@ -1,69 +0,0 @@ -# modal.FilePatternMatcher - - -```python -class FilePatternMatcher(modal.file_pattern_matcher._AbstractPatternMatcher) -``` - -Allows matching file Path objects against a list of patterns. - -**Usage** - -```python -from pathlib import Path -from modal import FilePatternMatcher - -matcher = FilePatternMatcher("*.py") - -assert matcher(Path("foo.py")) - -# You can also negate the matcher. -negated_matcher = ~matcher - -assert not negated_matcher(Path("foo.py")) -``` - -```python -__init__(self, *pattern) -``` -Initialize a new FilePatternMatcher instance. - -**Parameters** - - - -**Raises** - -- `ValueError`: If an illegal exclusion pattern is provided. - -## can_prune_directories - -```python -can_prune_directories(self) -``` -Returns True if this pattern matcher allows safe early directory pruning. - -Directory pruning is safe when matching directories can be skipped entirely -without missing any files that should be included. This is for example not -safe when we have inverted/negated ignore patterns (e.g. "!**/*.py"). - -## from_file - -```python -from_file(cls, file_path) -``` -Initialize a new FilePatternMatcher instance from a file. - -The patterns in the file will be read lazily when the matcher is first used. - -**Parameters** - - - -**Usage** - -```python -from modal import FilePatternMatcher - -matcher = FilePatternMatcher.from_file("/path/to/ignorefile") -``` diff --git a/.agents/skills/modal/references/api/modal.Function.md b/.agents/skills/modal/references/api/modal.Function.md deleted file mode 100644 index ea9cb7cd0834aea9b8ae904cd70e74986ce1805d..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Function.md +++ /dev/null @@ -1,387 +0,0 @@ -# modal.Function - - -```python -class Function(typing.Generic, modal.object.Object) -``` - -Functions are the basic units of serverless execution on Modal. - -Generally, you will not construct a `Function` directly. Instead, use the -`App.function()` decorator to register your Python functions with your App. - - -## hydrate - -```python -hydrate(self, client=None) -``` -Synchronize the local object with its identity on the Modal server. - -It is rarely necessary to call this method explicitly, as most operations -will lazily hydrate when needed. The main use case is when you need to -access object metadata, such as its ID. - -*Added in v0.72.39*: This method replaces the deprecated `.resolve()` method. - -## update_autoscaler - -```python -update_autoscaler(self, *, min_containers=None, max_containers=None, - buffer_containers=None, scaledown_window=None) -``` -Override the current autoscaler behavior for this Function. - -Unspecified parameters will retain their current value, i.e. either the static value -from the function decorator, or an override value from a previous call to this method. - -Subsequent deployments of the App containing this Function will reset the autoscaler back to -its static configuration. - -**Parameters** - - - - - - -**Usage** - -```python notest - f = modal.Function.from_name("my-app", "function") - - # Always have at least 2 containers running, with an extra buffer when the Function is active - f.update_autoscaler(min_containers=2, buffer_containers=1) - - # Limit this Function to avoid spinning up more than 5 containers - f.update_autoscaler(max_containers=5) - - # Extend the scaledown window to increase the amount of time that idle containers stay alive - f.update_autoscaler(scaledown_window=300) -``` - -## from_name - -```python -from_name(cls, app_name, name, *, version=None, environment_name=None, - client=None) -``` -Reference a Function from a deployed App by its name. - -This is a lazy method that defers hydrating the local -object with metadata from Modal servers until the first -time it is actually used. - -**Parameters** - - - - - - -**Returns** - -A lazy `Function` handle. - -**Usage** - -```python -f = modal.Function.from_name("other-app", "function") -``` - -The `version` parameter allows you to invoke a version-pinned function: - -```python -f_v3 = modal.Function.from_name("other-app", "function", version=3) -``` - -## get_web_url - -```python -get_web_url(self) -``` -URL for addressing a Web Function via HTTP. - -**Returns** - -The HTTPS URL for the web endpoint, or `None` if this Function is not a web endpoint. - -## with_options - -```python -with_options(self, *, cpu=None, memory=None, gpu=None, env=None, secrets=None, - volumes={}, retries=None, max_containers=None, buffer_containers=None, - scaledown_window=None, timeout=None, region=None, cloud=None) -``` -Dynamically override the static Function configuration with invocation-specific values. - -This method returns a new Function instance with the dynamic configuration. Invocations of -the new Function will run in a distinct container pool and autoscale independently from the -base Function (and from other dynamic configurations). - -Note that options cannot be "unset" with this method (i.e., if a GPU is configured in the -`@app.cls()` decorator, passing `gpu=None` here will not create a CPU-only instance). -Additionally, container arguments like `volumes` and `secrets` will _replace_ the base -configuration or any previous use of this method rather than extending it. - -**Usage:** - -You can use this method after looking up a deployed Function: - -```python notest -fn = modal.Function.from_name("my_app", "fn").with_options(gpu="H100") -fn.remote() # will run on a H100 GPU -``` - -Or by referencing another Function defined in the same App: - -```python notest -@app.function() -def fn(): - ... - -# From a local entrypoint or another Function -fn.with_options(gpu="H100").remote() # Uses an H100 GPU -fn.remote() # Uses the static configuration with no GPU -``` - -## with_concurrency - -```python -with_concurrency(self, *, max_inputs, target_inputs=None) -``` -Override the static Function configuration with invocation-specific input concurrency. - -Returns a new Function instance that is dynamically configured to behave like a Function using -the `@modal.concurrent` decorator. This instance will autoscale independently from the base Function. - -## with_batching - -```python -with_batching(self, *, max_batch_size, wait_ms) -``` -Override the static Function configuration with invocation-specific dynamic batching. - -Returns a new Function instance that is dynamically configured to behave like a Function using -the `@modal.batched` decorator. This instance will autoscale independently from the base Function. - -## remote - -```python -remote(self, *args, **kwargs) -``` -Calls the function remotely, executing it with the given arguments and returning the execution's result. - -**Parameters** - - - - -**Returns** - -The value returned by the remote function. - -## remote_gen - -```python -remote_gen(self, *args, **kwargs) -``` -Calls the generator remotely, executing it with the given arguments. - -**Parameters** - - - - -**Returns** - -Values produced by the remote generator. - -## local - -```python -local(self, *args, **kwargs) -``` -Calls the function locally, executing it with the given arguments and returning the execution's result. - -The function will execute in the same environment as the caller, just like calling the underlying function -directly in Python. In particular, only secrets available in the caller environment will be available -through environment variables. - -**Parameters** - - - - -**Returns** - -The return value of the local call (or a coroutine for async functions). - -## spawn - -```python -spawn(self, *args, **kwargs) -``` -Calls the function with the given arguments, without waiting for the results. - -Conceptually similar to `multiprocessing.pool.apply_async`, or a Future/Promise in other contexts. - -**Parameters** - - - - -**Returns** - -A [`modal.FunctionCall`](https://modal.com/docs/reference/modal.FunctionCall) object -that can later be polled or waited for using -[`.get(timeout=...)`](https://modal.com/docs/reference/modal.FunctionCall#get). - -## get_raw_f - -```python -get_raw_f(self) -``` -Return the inner Python object wrapped by this Modal Function. - -**Returns** - -The original function object registered with Modal. - -## get_current_stats - -```python -get_current_stats(self) -``` -Return a `FunctionStats` object describing the current function's queue and runner counts. - -**Returns** - -Snapshot counts for backlog, runners, and running inputs. - -## map - -```python -map(self, *input_iterators, kwargs={}, order_outputs=True, - return_exceptions=False, wrap_returned_exceptions=None) -``` -Parallel map over a set of inputs. - -Pass one iterable per positional argument of the underlying function. Results are yielded as an -iterable (sync) or async iterator (``map.aio``). - -If applied to an ``@app.function``, ``map()`` returns one result per input and output order matches -input order by default. Set ``order_outputs=False`` to emit results in completion order. - -``return_exceptions`` can aggregate failures into the result stream instead of raising. - -**Parameters** - - - - - - - -**Usage** - -```python -@app.function() -def my_func(a): - return a ** 2 - - -@app.local_entrypoint() -def main(): - assert list(my_func.map([1, 2, 3, 4])) == [1, 4, 9, 16] -``` - -```python -@app.function() -def my_func(a): - if a == 2: - raise Exception("ohno") - return a ** 2 - - -@app.local_entrypoint() -def main(): - print(list(my_func.map(range(3), return_exceptions=True))) -``` - -## starmap - -```python -starmap(self, input_iterator, *, kwargs={}, order_outputs=True, - return_exceptions=False, wrap_returned_exceptions=None) -``` -Like ``map``, but each input item is unpacked into multiple positional arguments. - -Every element of ``input_iterator`` should be a sequence (for example a tuple) with length equal to the -arity of the function. - -**Parameters** - - - - - - - -**Usage** - -```python -@app.function() -def my_func(a, b): - return a + b - - -@app.local_entrypoint() -def main(): - assert list(my_func.starmap([(1, 2), (3, 4)])) == [3, 7] -``` - -## for_each - -```python -for_each(self, *input_iterators, kwargs={}, ignore_exceptions=False) -``` -Execute the function for all inputs and wait for completion, discarding return values. - -Like ``.map()`` but you do not need to iterate the result to drive work—Modal processes every input. - -**Parameters** - - - - - -## spawn_map - -```python -spawn_map(self, *input_iterators, kwargs={}) -``` -Spawn parallel execution over a set of inputs, exiting as soon as the inputs are created (without waiting -for the map to complete). - -Takes one iterator argument per argument in the function being mapped over. - -Programmatic retrieval of results will be supported in a future update. - -**Parameters** - - - - -**Usage** - -```python -@app.function() -def my_func(a): - return a ** 2 - - -@app.local_entrypoint() -def main(): - my_func.spawn_map([1, 2, 3, 4]) -``` diff --git a/.agents/skills/modal/references/api/modal.FunctionCall.md b/.agents/skills/modal/references/api/modal.FunctionCall.md deleted file mode 100644 index 84aac19d292290e2e663ba89fb006d168b2078ce..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.FunctionCall.md +++ /dev/null @@ -1,191 +0,0 @@ -# modal.FunctionCall - - -```python -class FunctionCall(typing.Generic, modal.object.Object) -``` - -A reference to an executed function call. - -Constructed using `.spawn(...)` on a Modal function with the same -arguments that a function normally takes. Acts as a reference to -an ongoing function call that can be passed around and used to -poll or fetch function results at some later time. - -Conceptually similar to a Future/Promise/AsyncResult in other contexts and languages. - - -## hydrate - -```python -hydrate(self, client=None) -``` -Synchronize the local object with its identity on the Modal server. - -It is rarely necessary to call this method explicitly, as most operations -will lazily hydrate when needed. The main use case is when you need to -access object metadata, such as its ID. - -*Added in v0.72.39*: This method replaces the deprecated `.resolve()` method. - -## num_inputs - -```python -num_inputs(self) -``` -Get the number of inputs in the function call. - -**Returns** - -How many inputs this function call includes (e.g. `1` for `.spawn()`, more for `.spawn_map()`). - -## get - -```python -get(self, timeout=None, *, index=0) -``` -Get the result of the index-th input of the function call. - -`.spawn()` calls have a single output, so only specifying `index=0` is valid. -A non-zero index is useful when your function has multiple outputs, like via `.spawn_map()`. - -This function waits indefinitely by default. It takes an optional -`timeout` argument that specifies the maximum number of seconds to wait, -which can be set to `0` to poll for an output immediately. - -The returned coroutine is not cancellation-safe. - -**Parameters** - - - - -**Returns** - -The deserialized return value from that input. - -## iter - -```python -iter(self, *, start=0, end=None) -``` -Iterate in-order over the results of the function call. - -Optionally, specify a range [start, end) to iterate over. - -If `end` is not provided, it will iterate over all results. - -**Parameters** - - - - -**Returns** - -Each result value in index order. - -**Usage** - -```python -@app.function() -def my_func(a): - return a ** 2 - - -@app.local_entrypoint() -def main(): - fc = my_func.spawn_map([1, 2, 3, 4]) - assert list(fc.iter()) == [1, 4, 9, 16] - assert list(fc.iter(start=1, end=3)) == [4, 9] -``` - -## get_call_graph - -```python -get_call_graph(self) -``` -Returns a structure representing the call graph from a given root -call ID, along with the status of execution for each node. - -See [`modal.call_graph`](https://modal.com/docs/reference/modal.call_graph) reference page -for documentation on the structure of the returned `InputInfo` items. - -**Returns** - -A list of `InputInfo` nodes describing the call graph. - -## cancel - -```python -cancel(self, terminate_containers=False) -``` -Cancels the function call, which will stop its execution and mark its inputs as -[`TERMINATED`](https://modal.com/docs/reference/modal.call_graph#modalcall_graphinputstatus). - -If `terminate_containers=True` - the containers running the cancelled inputs are all terminated -causing any non-cancelled inputs on those containers to be rescheduled in new containers. - -**Parameters** - - - -## from_id - -```python -from_id(cls, function_call_id, client=None) -``` -Instantiate a FunctionCall object from an existing ID. - -Note that it's only necessary to re-instantiate the `FunctionCall` with this method -if you no longer have access to the original object returned from `Function.spawn`. - -**Parameters** - - - - -**Returns** - -A `FunctionCall` handle for the given ID. - -**Usage** - -```python notest -# Spawn a FunctionCall and keep track of its object ID -fc = my_func.spawn() -fc_id = fc.object_id - -# Later, use the ID to re-instantiate the FunctionCall object -fc = FunctionCall.from_id(fc_id) -result = fc.get() -``` - -## gather - -```python -gather(*function_calls) -``` -Wait until all Modal FunctionCall objects have results before returning. - -Accepts a variable number of `FunctionCall` objects, as returned by `Function.spawn()`. - -Raises an exception from the first failing function call. - -*Added in v0.73.69*: This method replaces the deprecated `modal.functions.gather` function. - -**Parameters** - - - -**Returns** - -Results in the same order as `function_calls` (like `asyncio.gather`). - -**Usage** - -```python notest -fc1 = slow_func_1.spawn() -fc2 = slow_func_2.spawn() - -result_1, result_2 = modal.FunctionCall.gather(fc1, fc2) -``` diff --git a/.agents/skills/modal/references/api/modal.Image.md b/.agents/skills/modal/references/api/modal.Image.md deleted file mode 100644 index c64aec2e254cf3c31e0963a9dde0818125940350..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Image.md +++ /dev/null @@ -1,1181 +0,0 @@ -# modal.Image - - -```python -class Image(modal.object.Object) -``` - -Base class for container images to run functions in. - -Do not construct this class directly; instead use one of its static factory methods, -such as `modal.Image.debian_slim`, `modal.Image.from_registry`, or `modal.Image.micromamba`. - - -## add_local_file - -```python -add_local_file(self, local_path, remote_path, *, copy=False) -``` -Adds a local file to the image at `remote_path` within the container. - -By default (`copy=False`), the files are added to containers on startup and are not built into the actual Image, -which speeds up deployment. - -Set `copy=True` to copy the files into an Image layer at build time instead, similar to how -[`COPY`](https://docs.docker.com/engine/reference/builder/#copy) works in a `Dockerfile`. - -copy=True can slow down iteration since it requires a rebuild of the Image and any subsequent -build steps whenever the included files change, but it is required if you want to run additional -build steps after this one. - -*Added in v0.66.40*: This method replaces the deprecated `modal.Image.copy_local_file` method. - -**Parameters** - - - - - -**Returns** - -A new `Image` with the file layer or mount applied. - -## add_local_dir - -```python -add_local_dir(self, local_path, remote_path, *, copy=False, ignore=[]) -``` -Adds a local directory's content to the image at `remote_path` within the container. - -By default (`copy=False`), the files are added to containers on startup and are not built into the actual Image, -which speeds up deployment. - -Set `copy=True` to copy the files into an Image layer at build time instead, similar to how -[`COPY`](https://docs.docker.com/engine/reference/builder/#copy) works in a `Dockerfile`. - -copy=True can slow down iteration since it requires a rebuild of the Image and any subsequent -build steps whenever the included files change, but it is required if you want to run additional -build steps after this one. - -*Added in v0.66.40*: This method replaces the deprecated `modal.Image.copy_local_dir` method. - -**Parameters** - - - - - - -**Returns** - -A new `Image` with the directory layer or mount applied. - -**Usage** - -```python -from modal import FilePatternMatcher - -image = modal.Image.debian_slim().add_local_dir( - "~/assets", - remote_path="/assets", - ignore=["*.venv"], -) - -image = modal.Image.debian_slim().add_local_dir( - "~/assets", - remote_path="/assets", - ignore=lambda p: p.is_relative_to(".venv"), -) - -image = modal.Image.debian_slim().add_local_dir( - "~/assets", - remote_path="/assets", - ignore=FilePatternMatcher("**/*.txt"), -) - -# When including files is simpler than excluding them, you can use the `~` operator to invert the matcher. -image = modal.Image.debian_slim().add_local_dir( - "~/assets", - remote_path="/assets", - ignore=~FilePatternMatcher("**/*.py"), -) - -# You can also read ignore patterns from a file. -image = modal.Image.debian_slim().add_local_dir( - "~/assets", - remote_path="/assets", - ignore=FilePatternMatcher.from_file("/path/to/ignorefile"), -) -``` - -## add_local_python_source - -```python -add_local_python_source(self, *modules, copy=False, ignore=NON_PYTHON_FILES) -``` -Adds locally available Python packages/modules to containers. - -Adds all files from the specified Python package or module to containers running the Image. - -Packages are added to the `/root` directory of containers, which is on the `PYTHONPATH` -of any executed Modal Functions, enabling import of the module by that name. - -By default (`copy=False`), the files are added to containers on startup and are not built into the actual Image, -which speeds up deployment. - -Set `copy=True` to copy the files into an Image layer at build time instead. This can slow down iteration since -it requires a rebuild of the Image and any subsequent build steps whenever the included files change, but it is -required if you want to run additional build steps after this one. - -**Note:** This excludes all dot-prefixed subdirectories or files and all `.pyc`/`__pycache__` files. -To add full directories with finer control, use `.add_local_dir()` instead and specify `/root` as -the destination directory. - -By default only includes `.py`-files in the source modules. Set the `ignore` argument to a list of patterns -or a callable to override this behavior. - -*Added in v0.67.28*: This method replaces the deprecated `modal.Mount.from_local_python_packages` pattern. - -**Parameters** - - - - - -**Returns** - -A new `Image` with the Python source mount or layer applied. - -**Usage** - -```py -# includes everything except data.json -modal.Image.debian_slim().add_local_python_source("mymodule", ignore=["data.json"]) - -# exclude large files -modal.Image.debian_slim().add_local_python_source( - "mymodule", - ignore=lambda p: p.stat().st_size > 1e9 -) -``` - -## from_id - -```python -from_id(cls, image_id, client=None) -``` -Construct an Image from an id and look up the Image result. - -The ID of an Image object can be accessed using `.object_id`. - -**Parameters** - - - - -**Returns** - -A hydrated `Image` handle for the given ID. - -## build - -```python -build(self, app) -``` -Eagerly build an image. - -If your image was previously built, then this method will not rebuild your image -and your cached image is returned. - -For defining Modal functions, images are built automatically when deploying or running an App. -You do not need to build the image explicitly in that case. - -**Parameters** - - - -**Returns** - -This image after the build (and resolver load) completes. - -**Usage** - -```python -image = modal.Image.debian_slim().uv_pip_install("scipy", "numpy") - -app = modal.App.lookup("build-image", create_if_missing=True) -with modal.enable_output(): # To see logs in your local terminal - image.build(app) - -# Save the image id -my_image_id = image.object_id - -# Reference the image with the id or uses it another context. -built_image = modal.Image.from_id(my_image_id) -``` - -Alternatively, you can pre-build an image and use it in a sandbox: - -```python notest -app = modal.App.lookup("sandbox-example", create_if_missing=True) - -with modal.enable_output(): - image = modal.Image.debian_slim().uv_pip_install("scipy") - image.build(app) - -sb = modal.Sandbox.create("python", "-c", "import scipy; print(scipy)", app=app, image=image) -print(sb.stdout.read()) -sb.terminate() -``` - -```python notest -app = modal.App() -image = modal.Image.debian_slim() - -# No need to explicitly build the image for defining a function. -@app.function(image=image) -def f(): - ... -``` - -## pip_install - -```python -pip_install(self, *packages, find_links=None, index_url=None, - extra_index_url=None, pre=False, extra_options="", force_build=False, - env=None, secrets=None, gpu=None) -``` -Install a list of Python packages using pip. - -**Parameters** - - - - - - - - - - - - -**Returns** - -A new `Image` with the pip install layer applied. - -**Usage** - -Simple installation: - -```python -image = modal.Image.debian_slim().pip_install("click", "httpx~=0.23.3") -``` - -More complex installation: - -```python -image = ( - modal.Image.from_registry( - "nvidia/cuda:12.2.0-devel-ubuntu22.04", add_python="3.11" - ) - .pip_install( - "ninja", - "packaging", - "wheel", - "transformers==4.40.2", - ) - .pip_install( - "flash-attn==2.5.8", extra_options="--no-build-isolation" - ) -) -``` - -## pip_install_private_repos - -```python -pip_install_private_repos(self, *repositories, git_user, find_links=None, - index_url=None, extra_index_url=None, pre=False, extra_options="", gpu=None, - env=None, secrets=None, force_build=False) -``` -Install a list of Python packages from private git repositories using pip. - -This method currently supports Github and Gitlab only. - -- **Github:** Provide a `modal.Secret` that contains a `GITHUB_TOKEN` key-value pair -- **Gitlab:** Provide a `modal.Secret` that contains a `GITLAB_TOKEN` key-value pair - -These API tokens should have permissions to read the list of private repositories provided as arguments. - -We recommend using Github's ['fine-grained' access tokens](https://github.blog/2022-10-18-introducing-fine-grained-personal-access-tokens-for-github/). -These tokens are repo-scoped, and avoid granting read permission across all of a user's private repos. - -**Parameters** - - - - - - - - - - - - - -**Returns** - -A new `Image` with private repositories installed. - -**Usage** - -```python -image = ( - modal.Image - .debian_slim() - .pip_install_private_repos( - "github.com/ecorp/private-one@1.0.0", - "github.com/ecorp/private-two@main" - "github.com/ecorp/private-three@d4776502" - # install from 'inner' directory on default branch. - "github.com/ecorp/private-four#subdirectory=inner", - git_user="erikbern", - secrets=[modal.Secret.from_name("github-read-private")], - ) -) -``` - -## pip_install_from_requirements - -```python -pip_install_from_requirements(self, requirements_txt, find_links=None, *, - index_url=None, extra_index_url=None, pre=False, extra_options="", - force_build=False, env=None, secrets=None, gpu=None) -``` -Install a list of Python packages from a local `requirements.txt` file. - -**Parameters** - - - - - - - - - - - - -**Returns** - -A new `Image` with requirements installed. - -## pip_install_from_pyproject - -```python -pip_install_from_pyproject(self, pyproject_toml, optional_dependencies=[], *, - find_links=None, index_url=None, extra_index_url=None, pre=False, - extra_options="", force_build=False, env=None, secrets=None, gpu=None) -``` -Install dependencies specified by a local `pyproject.toml` file. - -`optional_dependencies` is a list of the keys of the -optional-dependencies section(s) of the `pyproject.toml` file -(e.g. test, doc, experiment, etc). When provided, -all of the packages in each listed section are installed as well. - -**Parameters** - - - - - - - - - - - - - -**Returns** - -A new `Image` with project dependencies installed. - -## uv_pip_install - -```python -uv_pip_install(self, *packages, requirements=None, find_links=None, - index_url=None, extra_index_url=None, pre=False, extra_options="", - force_build=False, uv_version=None, env=None, secrets=None, gpu=None) -``` -Install a list of Python packages using uv pip install. - -This method assumes that: -- Python is on the ``$PATH`` and dependencies are installed with the first Python on the ``$PATH``. -- The shell supports ``$()``-style substitution as used in the generated Dockerfile. -- The ``command`` builtin is available on the ``$PATH``. - -Added in v1.1.0. - -**Parameters** - - - - - - - - - - - - - - -**Returns** - -A new `Image` with packages installed via uv. - -**Usage** - -```python -image = modal.Image.debian_slim().uv_pip_install("torch==2.7.1", "numpy") -``` - -## poetry_install_from_file - -```python -poetry_install_from_file(self, poetry_pyproject_toml, poetry_lockfile=None, *, - ignore_lockfile=False, force_build=False, with_=[], without=[], only=[], - poetry_version="latest", old_installer=False, env=None, secrets=None, - gpu=None) -``` -Install poetry *dependencies* specified by a local `pyproject.toml` file. - -If not provided as argument the path to the lockfile is inferred. However, the -file has to exist, unless `ignore_lockfile` is set to `True`. - -Note that the root project of the poetry project is not installed, only the dependencies. -For including local python source files see `add_local_python_source` - -Poetry will be installed to the Image (using pip) unless `poetry_version` is set to None. -Note that the interpretation of `poetry_version="latest"` depends on the Modal Image Builder -version, with versions 2024.10 and earlier limiting poetry to 1.x. - -**Parameters** - - - - - - - - - - - - - - -**Returns** - -A new `Image` with Poetry dependencies installed. - -## uv_sync - -```python -uv_sync(self, uv_project_dir="./", *, force_build=False, groups=None, - extras=None, frozen=True, extra_options="", uv_version=None, env=None, - secrets=None, gpu=None) -``` -Creates a virtual environment with the dependencies in a uv managed project with `uv sync`. - -The `pyproject.toml` and `uv.lock` in `uv_project_dir` are automatically added to the build context. The -`uv_project_dir` is relative to the current working directory of where `modal` is called. - -NOTE: This does *not* install the project itself into the environment (this is equivalent to the -`--no-install-project` flag in the `uv sync` command) and you would be expected to add any local python source -files using `Image.add_local_python_source` or similar methods after this call. - -This ensures that updates to your project code wouldn't require reinstalling third-party dependencies -after every change. - -uv workspaces are currently not supported. - -Added in v1.1.0. - -**Parameters** - - - - - - - - - - - - -**Returns** - -A new `Image` with a uv-managed virtual environment. - -**Usage** - -```python -image = modal.Image.debian_slim().uv_sync() -``` - -## dockerfile_commands - -```python -dockerfile_commands(self, *dockerfile_commands, context_files={}, env=None, - secrets=None, gpu=None, context_dir=None, force_build=False, - ignore=AUTO_DOCKERIGNORE, build_args={}) -``` -Extend an image with arbitrary Dockerfile-like commands. - -**Parameters** - - - - - - - - - - - -**Returns** - -A new `Image` with the Dockerfile fragment applied. - -**Usage** - -```python -from modal import FilePatternMatcher - -# By default a .dockerignore file is used if present in the current working directory -image = modal.Image.debian_slim().dockerfile_commands( - ["COPY data /data"], -) - -image = modal.Image.debian_slim().dockerfile_commands( - ["COPY data /data"], - ignore=["*.venv"], -) - -image = modal.Image.debian_slim().dockerfile_commands( - ["COPY data /data"], - ignore=lambda p: p.is_relative_to(".venv"), -) - -image = modal.Image.debian_slim().dockerfile_commands( - ["COPY data /data"], - ignore=FilePatternMatcher("**/*.txt"), -) - -# When including files is simpler than excluding them, you can use the `~` operator to invert the matcher. -image = modal.Image.debian_slim().dockerfile_commands( - ["COPY data /data"], - ignore=~FilePatternMatcher("**/*.py"), -) - -# You can also read ignore patterns from a file. -image = modal.Image.debian_slim().dockerfile_commands( - ["COPY data /data"], - ignore=FilePatternMatcher.from_file("/path/to/dockerignore"), -) -``` - -## entrypoint - -```python -entrypoint(self, entrypoint_commands) -``` -Set the ENTRYPOINT for the image. - -**Parameters** - - - -**Returns** - -A new `Image` with the entrypoint Dockerfile directive applied. - -## shell - -```python -shell(self, shell_commands) -``` -Overwrite default shell for the image. - -**Parameters** - - - -**Returns** - -A new `Image` with the shell Dockerfile directive applied. - -## run_commands - -```python -run_commands(self, *commands, env=None, secrets=None, volumes=None, gpu=None, - force_build=False) -``` -Extend an image with a list of shell commands to run. - -**Parameters** - - - - - - - - -**Returns** - -A new `Image` with the commands executed as layers. - -## micromamba - -```python -micromamba(python_version=None, force_build=False) -``` -A Micromamba base image. Micromamba allows for fast building of small Conda-based containers. - -**Parameters** - - - - -**Returns** - -A Micromamba-based `Image`. - -## micromamba_install - -```python -micromamba_install(self, *packages, spec_file=None, channels=[], - force_build=False, env=None, secrets=None, gpu=None) -``` -Install a list of additional packages using micromamba. - -**Parameters** - - - - - - - - - -**Returns** - -A new `Image` with micromamba packages installed. - -## from_registry - -```python -from_registry(tag, secret=None, *, setup_dockerfile_commands=[], - force_build=False, add_python=None, **kwargs) -``` -Build a Modal Image from a public or private image registry, such as Docker Hub. - -The image must be built for the `linux/amd64` platform. - -If your image does not come with Python installed, you can use the `add_python` parameter -to specify a version of Python to add to the image. Otherwise, the image is expected to -have Python on PATH as `python`, along with `pip`. - -You may also use `setup_dockerfile_commands` to run Dockerfile commands before the -remaining commands run. This might be useful if you want a custom Python installation or to -set a `SHELL`. Prefer `run_commands()` when possible though. - -To authenticate against a private registry with static credentials, you must set the `secret` parameter to -a `modal.Secret` containing a username (`REGISTRY_USERNAME`) and -an access token or password (`REGISTRY_PASSWORD`). - -To authenticate against private registries with credentials from a cloud provider, -use `Image.from_gcp_artifact_registry()` or `Image.from_aws_ecr()`. - -**Parameters** - - - - - - - - -**Returns** - -An `Image` based on the registry tag. - -**Usage** - -```python -modal.Image.from_registry("python:3.11-slim-bookworm") -modal.Image.from_registry("ubuntu:22.04", add_python="3.11") -modal.Image.from_registry("nvcr.io/nvidia/pytorch:22.12-py3") -``` - -## from_gcp_artifact_registry - -```python -from_gcp_artifact_registry(tag, secret=None, *, setup_dockerfile_commands=[], - force_build=False, add_python=None, **kwargs) -``` -Build a Modal image from a private image in Google Cloud Platform (GCP) Artifact Registry. - -You will need to pass a `modal.Secret` containing [your GCP service account key data](https://cloud.google.com/iam/docs/keys-create-delete#creating) -as `SERVICE_ACCOUNT_JSON`. This can be done from the [Secrets](https://modal.com/secrets) page. -Your service account should be granted a specific role depending on the GCP registry used: - -- For Artifact Registry images (`pkg.dev` domains) use - the ["Artifact Registry Reader"](https://cloud.google.com/artifact-registry/docs/access-control#roles) role -- For Container Registry images (`gcr.io` domains) use - the ["Storage Object Viewer"](https://cloud.google.com/artifact-registry/docs/transition/setup-gcr-repo) role - -**Note:** This method does not use `GOOGLE_APPLICATION_CREDENTIALS` as that -variable accepts a path to a JSON file, not the actual JSON string. - -See `Image.from_registry()` for information about the other parameters. - -**Parameters** - - - - - - - - -**Returns** - -An `Image` based on the private GCP artifact. - -**Usage** - -```python -modal.Image.from_gcp_artifact_registry( - "us-east1-docker.pkg.dev/my-project-1234/my-repo/my-image:my-version", - secret=modal.Secret.from_name( - "my-gcp-secret", - required_keys=["SERVICE_ACCOUNT_JSON"], - ), - add_python="3.11", -) -``` - -## from_aws_ecr - -```python -from_aws_ecr(tag, secret=None, *, setup_dockerfile_commands=[], - force_build=False, add_python=None, **kwargs) -``` -Build a Modal image from a private image in AWS Elastic Container Registry (ECR). - -You will need to pass a `modal.Secret` containing either IAM user credentials or OIDC -configuration to access the target ECR registry. - -For IAM user authentication, set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_REGION`. - -For OIDC authentication, set `AWS_ROLE_ARN` and `AWS_REGION`. - -IAM configuration details can be found in the AWS documentation for -["Private repository policies"](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-policies.html). - -For more details on using an AWS role to access ECR, see the [OIDC integration guide](https://modal.com/docs/guide/oidc-integration). - -See `Image.from_registry()` for information about the other parameters. - -**Parameters** - - - - - - - - -**Returns** - -An `Image` based on the private ECR image. - -**Usage** - -```python -modal.Image.from_aws_ecr( - "000000000000.dkr.ecr.us-east-1.amazonaws.com/my-private-registry:my-version", - secret=modal.Secret.from_name( - "aws", - required_keys=["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_REGION"], - ), - add_python="3.11", -) -``` - -## from_dockerfile - -```python -from_dockerfile(path, *, force_build=False, context_dir=None, env=None, - secrets=None, gpu=None, add_python=None, build_args={}, - ignore=AUTO_DOCKERIGNORE) -``` -Build a Modal image from a local Dockerfile. - -If your Dockerfile does not have Python installed, you can use the `add_python` parameter -to specify a version of Python to add to the image. - -**Parameters** - - - - - - - - - - - -**Returns** - -An `Image` built from the Dockerfile plus Modal runtime dependencies. - -**Usage** - -```python -from modal import FilePatternMatcher - -# By default a .dockerignore file is used if present in the current working directory -image = modal.Image.from_dockerfile( - "./Dockerfile", - add_python="3.12", -) - -image = modal.Image.from_dockerfile( - "./Dockerfile", - add_python="3.12", - ignore=["*.venv"], -) - -image = modal.Image.from_dockerfile( - "./Dockerfile", - add_python="3.12", - ignore=lambda p: p.is_relative_to(".venv"), -) - -image = modal.Image.from_dockerfile( - "./Dockerfile", - add_python="3.12", - ignore=FilePatternMatcher("**/*.txt"), -) - -# When including files is simpler than excluding them, you can use the `~` operator to invert the matcher. -image = modal.Image.from_dockerfile( - "./Dockerfile", - add_python="3.12", - ignore=~FilePatternMatcher("**/*.py"), -) - -# You can also read ignore patterns from a file. -image = modal.Image.from_dockerfile( - "./Dockerfile", - add_python="3.12", - ignore=FilePatternMatcher.from_file("/path/to/dockerignore"), -) -``` - -## from_scratch - -```python -from_scratch(force_build=False) -``` -Create an empty Image, equivalent to `FROM scratch` in Docker. - -The resulting Image has no operating system, shell, or package manager. It is -primarily useful as a lightweight filesystem to mount into a Sandbox via -`Sandbox.mount_image`. - -Note that since this Image doesn't contain Python or other standard OS utilities, -higher-level Image build steps like `pip_install` cannot be chained onto it. It also -cannot be used for `modal.Function` execution, which requires a Python interpreter. - -**Parameters** - - - -**Returns** - -An empty `Image` suitable for minimal filesystem mounts. - -**Usage** - -```python notest -image = modal.Image.from_scratch().add_local_file(local_path, "/bin/my_binary", copy=True) -``` - -## debian_slim - -```python -debian_slim(python_version=None, force_build=False) -``` -Default image, based on the official `python` Docker images. - -**Parameters** - - - - -**Returns** - -The standard Debian slim Python `Image` used as Modal's default base. - -## apt_install - -```python -apt_install(self, *packages, force_build=False, env=None, secrets=None, - gpu=None) -``` -Install a list of Debian packages using `apt`. - -**Parameters** - - - - - - - -**Returns** - -A new `Image` with ``apt-get install`` layers applied. - -**Usage** - -```python -image = modal.Image.debian_slim().apt_install("git") -``` - -## run_function - -```python -run_function(self, raw_f, *, env=None, secrets=None, volumes={}, - network_file_systems={}, gpu=None, cpu=None, memory=None, timeout=60 * 60, - cloud=None, region=None, force_build=False, args=(), kwargs={}, - include_source=True) -``` -Run user-defined function `raw_f` as an image build step. - -The function runs like an ordinary Modal Function, accepting a resource configuration and integrating -with Modal features like Secrets and Volumes. Unlike ordinary Modal Functions, any changes to the -filesystem state will be captured on container exit and saved as a new Image. - -Only the source code of `raw_f`, the contents of `**kwargs`, and any referenced *global* variables -are used to determine whether the image has changed and needs to be rebuilt. -If this function references other functions or variables, the image will not be rebuilt if you -make changes to them. You can force a rebuild by changing the function's source code itself. - -**Parameters** - - - - - - - - - - - - - - - - - -**Returns** - -A new `Image` capturing the filesystem after `raw_f` completes. - -**Usage** - -```python notest - -def my_build_function(): - open("model.pt", "w").write("parameters!") - -image = ( - modal.Image - .debian_slim() - .pip_install("torch") - .run_function(my_build_function, secrets=[...], volumes={...}) -) -``` - -## env - -```python -env(self, vars) -``` -Sets the environment variables in an Image. - -**Parameters** - - - -**Returns** - -A new `Image` with ``ENV`` directives applied. - -**Usage** - -```python -image = ( - modal.Image.debian_slim() - .env({"HF_HUB_ENABLE_HF_TRANSFER": "1"}) -) -``` - -## workdir - -```python -workdir(self, path) -``` -Set the working directory for subsequent image build steps and function execution. - -**Parameters** - - - -**Returns** - -A new `Image` with ``WORKDIR`` applied. - -**Usage** - -```python -image = ( - modal.Image.debian_slim() - .run_commands("git clone https://xyz app") - .workdir("/app") - .run_commands("yarn install") -) -``` - -## cmd - -```python -cmd(self, cmd) -``` -Set the default command (`CMD`) to run when a container is started. - -Used with `modal.Sandbox`. Has no effect on `modal.Function`. - -**Parameters** - - - -**Returns** - -A new `Image` with ``CMD`` applied. - -**Usage** - -```python -image = ( - modal.Image.debian_slim().cmd(["python", "app.py"]) -) -``` - -## pipe - -```python -pipe(self, func, *args, **kwargs) -``` -Apply a local function to expand the Image recipe. - -This method can be useful for defining reusable Image build -recipes that compose well with the fluent Image builder interface. - -**Example** - -```python -def workspace_setup(image: modal.Image, repo: str) -> modal.Image: - return image.run_commands(f"git clone {repo}").uv_pip_install(".") - -image = ( - modal.Image.debian_slim() - .apt_install("git") - .pipe(workspace_setup, "https://github.com/example/repo.git") -) -``` - -## imports - -```python -imports(self) -``` -Used to import packages in global scope that are only available when running remotely. - -By using this context manager you can avoid an `ImportError` due to not having certain -packages installed locally. - -**Returns** - -Context manager that records import failures until the image is hydrated in the remote environment. - -**Usage** - -```python notest -with image.imports(): - import torch -``` - -## from_name - -```python -from_name(name, *, environment_name=None, client=None) -``` -Reference a named Image that was previously published with `.publish()`. - -Names can contain an optional `:tag` part - if no tag part is included `":latest"` is used, -matching Docker conventions. - -```python notest -image = modal.Image.from_name("my-image") # references my-image:latest -image_v1 = modal.Image.from_name("my-image:v1") - -@app.function(image=image) -def run(): - ... -``` - -## publish - -```python -publish(self, name, *, environment_name=None, client=None) -``` -Publish this image under the given name - -The Image must already be created (typically by calling `image.build()` or `sandbox.snapshot_filesystem()`). - -Image names can contain an explicit tag designation (using the `name:tag`). If no tag is included in the name, -`":latest"` is used, matching Docker conventions. To publish multiple tags, call `.publish()` once per tag. - -```python notest -image = modal.Image.debian_slim().pip_install("numpy") -image.build(app) -image.publish("my-image-with-numpy") # my-image-with-numpy:latest -image.publish("my-image-with-numpy:v1") -``` diff --git a/.agents/skills/modal/references/api/modal.NetworkFileSystem.md b/.agents/skills/modal/references/api/modal.NetworkFileSystem.md deleted file mode 100644 index 49ea1414310e94e9755e7368e771c7d656d6826c..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.NetworkFileSystem.md +++ /dev/null @@ -1,183 +0,0 @@ -# modal.NetworkFileSystem - - -```python -class NetworkFileSystem(modal.object.Object) -``` - -A shared, writable file system accessible by one or more Modal functions. - -By attaching this file system as a mount to one or more functions, they can -share and persist data with each other. - -**Note: `NetworkFileSystem` has been deprecated and will be removed.** - -**Usage** - -```python -import modal - -nfs = modal.NetworkFileSystem.from_name("my-nfs", create_if_missing=True) -app = modal.App() - -@app.function(network_file_systems={"/root/foo": nfs}) -def f(): - pass - -@app.function(network_file_systems={"/root/goo": nfs}) -def g(): - pass -``` - -Also see the CLI methods for accessing network file systems: - -``` -modal nfs --help -``` - -A `NetworkFileSystem` can also be useful for some local scripting scenarios, e.g.: - -```python notest -nfs = modal.NetworkFileSystem.from_name("my-network-file-system") -for chunk in nfs.read_file("my_db_dump.csv"): - ... -``` - - -## hydrate - -```python -hydrate(self, client=None) -``` -Synchronize the local object with its identity on the Modal server. - -It is rarely necessary to call this method explicitly, as most operations -will lazily hydrate when needed. The main use case is when you need to -access object metadata, such as its ID. - -*Added in v0.72.39*: This method replaces the deprecated `.resolve()` method. - -## from_name - -```python -from_name(name, *, environment_name=None, create_if_missing=False, client=None) -``` -Reference a NetworkFileSystem by name, optionally creating it on the server first. - -Hydration is lazy: metadata is fetched from Modal the first time the handle is used. - -**Parameters** - - - - - - -**Returns** - -A `NetworkFileSystem` handle (possibly not yet hydrated). - -**Usage** - -```python notest -nfs = NetworkFileSystem.from_name("my-nfs", create_if_missing=True) - -@app.function(network_file_systems={"/data": nfs}) -def f(): - pass -``` - -## ephemeral - -```python -ephemeral(cls, client=None, environment_name=None) -``` -Create an anonymous NetworkFileSystem that exists for the duration of the context manager. - -**Parameters** - - - - -**Usage** - -```python -with modal.NetworkFileSystem.ephemeral() as nfs: - assert nfs.listdir("/") == [] -``` - -```python notest -async with modal.NetworkFileSystem.ephemeral() as nfs: - assert await nfs.listdir("/") == [] -``` - -## delete - -```python -delete(name, client=None, environment_name=None) -``` - - -## write_file - -```python -write_file(self, remote_path, fp, progress_cb=None) -``` -Write from a file object to a path on the network file system, atomically. - -Will create any needed parent directories automatically. - -If remote_path ends with `/` it's assumed to be a directory and the -file will be uploaded with its current name to that directory. - -## read_file - -```python -read_file(self, path) -``` -Read a file from the network file system - -## iterdir - -```python -iterdir(self, path) -``` -Iterate over all files in a directory in the network file system. - -* Passing a directory path lists all files in the directory (names are relative to the directory) -* Passing a file path returns a list containing only that file's listing description -* Passing a glob path (including at least one * or ** sequence) returns all files matching -that glob path (using absolute paths) - -## add_local_file - -```python -add_local_file(self, local_path, remote_path=None, progress_cb=None) -``` - - -## add_local_dir - -```python -add_local_dir(self, local_path, remote_path=None, progress_cb=None) -``` - - -## listdir - -```python -listdir(self, path) -``` -List all files in a directory in the network file system. - -* Passing a directory path lists all files in the directory (names are relative to the directory) -* Passing a file path returns a list containing only that file's listing description -* Passing a glob path (including at least one * or ** sequence) returns all files matching -that glob path (using absolute paths) - -## remove_file - -```python -remove_file(self, path, recursive=False) -``` -Remove a file in a network file system. diff --git a/.agents/skills/modal/references/api/modal.Period.md b/.agents/skills/modal/references/api/modal.Period.md deleted file mode 100644 index c60789705f51e6bc1bd2ee130cbff51a1d8af5bd..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Period.md +++ /dev/null @@ -1,40 +0,0 @@ -# modal.Period - - -```python -class Period(modal.schedule.Schedule) -``` - -Create a schedule that runs every given time interval. - -Only `seconds` can be a float. All other arguments are integers. - -Note that `days=1` will trigger the function the same time every day. -This does not have the same behavior as `seconds=84000` since days have -different lengths due to daylight savings and leap seconds. Similarly, -using `months=1` will trigger the function on the same day each month. - -This behaves similar to the -[dateutil](https://dateutil.readthedocs.io/en/latest/relativedelta.html) -package. - -**Usage** - -```python -import modal -app = modal.App() - -@app.function(schedule=modal.Period(days=1)) -def f(): - print("This function will run every day") - -modal.Period(hours=4) # runs every 4 hours -modal.Period(minutes=15) # runs every 15 minutes -modal.Period(seconds=math.pi) # runs every 3.141592653589793 seconds -``` - -```python -__init__(self, *, years=0, months=0, weeks=0, days=0, hours=0, minutes=0, - seconds=0) -``` - diff --git a/.agents/skills/modal/references/api/modal.Probe.md b/.agents/skills/modal/references/api/modal.Probe.md deleted file mode 100644 index 50558630e15e40640197db61dc6718647d8599a8..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Probe.md +++ /dev/null @@ -1,47 +0,0 @@ -# modal.Probe - - -```python -class Probe(object) -``` - -Probe configuration for the Sandbox Readiness Probe. - -**Usage** - -```python notest -# Wait until a file exists. -readiness_probe = modal.Probe.with_exec( - "sh", "-c", "test -f /tmp/ready", -) - -# Wait until a TCP port is accepting connections. -readiness_probe = modal.Probe.with_tcp(8080) - -app = modal.App.lookup('sandbox-readiness-probe', create_if_missing=True) -sandbox = modal.Sandbox.create( - "python3", "-m", "http.server", "8080", - readiness_probe=readiness_probe, - app=app, -) -sandbox.wait_until_ready() -``` - -```python -__init__(self, tcp_port=None, exec_argv=None, interval_ms=100) -``` - - -## with_tcp - -```python -with_tcp(cls, port, *, interval_ms=100) -``` - - -## with_exec - -```python -with_exec(cls, *argv, interval_ms=100) -``` - diff --git a/.agents/skills/modal/references/api/modal.Proxy.md b/.agents/skills/modal/references/api/modal.Proxy.md deleted file mode 100644 index c0834f8cf55e8e781a2858b6461198506b4e4478..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Proxy.md +++ /dev/null @@ -1,45 +0,0 @@ -# modal.Proxy - - -```python -class Proxy(modal.object.Object) -``` - -Proxy objects give your Modal containers a static outbound IP address. - -This can be used for connecting to a remote address with network whitelist, for example -a database. See [the guide](https://modal.com/docs/guide/proxy-ips) for more information. - - -## hydrate - -```python -hydrate(self, client=None) -``` -Synchronize the local object with its identity on the Modal server. - -It is rarely necessary to call this method explicitly, as most operations -will lazily hydrate when needed. The main use case is when you need to -access object metadata, such as its ID. - -*Added in v0.72.39*: This method replaces the deprecated `.resolve()` method. - -## from_name - -```python -from_name(name, *, environment_name=None, client=None) -``` -Reference a Proxy by its name. - -In contrast to most other Modal objects, new Proxy objects must be -provisioned via the Dashboard and cannot be created on the fly from code. - -**Parameters** - - - - - -**Returns** - -A lazy `Proxy` handle. diff --git a/.agents/skills/modal/references/api/modal.Queue.md b/.agents/skills/modal/references/api/modal.Queue.md deleted file mode 100644 index 27f32da767fefef1837fe100d90afc0d44e95fb6..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Queue.md +++ /dev/null @@ -1,464 +0,0 @@ -# modal.Queue - - -```python -class Queue(modal.object.Object) -``` - -Distributed, FIFO queue for data flow in Modal apps. - -The queue can contain any object serializable by `cloudpickle`, including Modal objects. - -By default, the `Queue` object acts as a single FIFO queue which supports puts and gets (blocking and non-blocking). - -**Usage** - -```python -from modal import Queue - -# Create an ephemeral queue which is anonymous and garbage collected -with Queue.ephemeral() as my_queue: - # Putting values - my_queue.put("some value") - my_queue.put(123) - - # Getting values - assert my_queue.get() == "some value" - assert my_queue.get() == 123 - - # Using partitions - my_queue.put(0) - my_queue.put(1, partition="foo") - my_queue.put(2, partition="bar") - - # Default and "foo" partition are ignored by the get operation. - assert my_queue.get(partition="bar") == 2 - - # Set custom 10s expiration time on "foo" partition. - my_queue.put(3, partition="foo", partition_ttl=10) - - # Iterate through items in place (read immutably) - my_queue.put(1) - assert [v for v in my_queue.iterate()] == [0, 1] - -# You can also create persistent queues that can be used across apps -queue = Queue.from_name("my-persisted-queue", create_if_missing=True) -queue.put(42) -assert queue.get() == 42 -``` - -For more examples, see the [guide](https://modal.com/docs/guide/dicts-and-queues#modal-queues). - -**Queue partitions** - -Specifying partition keys gives access to other independent FIFO partitions within the same `Queue` object. -Across any two partitions, puts and gets are completely independent. -For example, a put in one partition does not affect a get in any other partition. - -When no partition key is specified (by default), puts and gets will operate on a default partition. -This default partition is also isolated from all other partitions. -Please see the Usage section below for an example using partitions. - -**Lifetime of a queue and its partitions** - -By default, each partition is cleared 24 hours after the last `put` operation. -A lower TTL can be specified by the `partition_ttl` argument in the `put` or `put_many` methods. -Each partition's expiry is handled independently. - -As such, `Queue`s are best used for communication between active functions and not relied on for persistent -storage. - -On app completion or after stopping an app any associated `Queue` objects are cleaned up. -All its partitions will be cleared. - -**Limits** - -A single `Queue` can contain up to 100,000 partitions, each with up to 5,000 items. Each item can be up to -1 MiB. - -Partition keys must be non-empty and must not exceed 64 bytes. - - -## hydrate - -```python -hydrate(self, client=None) -``` -Synchronize the local object with its identity on the Modal server. - -It is rarely necessary to call this method explicitly, as most operations -will lazily hydrate when needed. The main use case is when you need to -access object metadata, such as its ID. - -*Added in v0.72.39*: This method replaces the deprecated `.resolve()` method. - -## objects - - -```python -objects: QueueManager -``` - -Namespace with methods for managing named Queue objects. - - -### objects.create - -```python -create(self, name, *, allow_existing=False, environment_name=None, client=None) -``` -Create a new named Queue in the workspace environment. - -This does not return a local handle; use `modal.Queue.from_name` to look up the Queue after creation. - -Added in v1.1.2. - -**Parameters** - - - - - - -**Usage** - -```python notest -modal.Queue.objects.create("my-queue") -``` - -Queues will be created in the active environment, or another one can be specified: - -```python notest -modal.Queue.objects.create("my-queue", environment_name="dev") -``` - -By default, an error is raised if the Queue already exists; `allow_existing=True` makes that case a no-op: - -```python notest -modal.Queue.objects.create("my-queue", allow_existing=True) -``` - -Note that this method does not return a local instance of the Queue. You can use -`modal.Queue.from_name` to perform a lookup after creation. - -### objects.list - -```python -list(self, *, max_objects=None, created_before=None, environment_name="", - client=None) -``` -List named Queues in the workspace environment as hydrated handles. - -Results are ordered newest to oldest. By default, all matching Queues are returned. - -Added in v1.1.2. - -**Parameters** - - - - - - -**Returns** - -Hydrated `Queue` objects for each named Queue in the listing. - -**Usage** - -```python -queues = modal.Queue.objects.list() -print([q.name for q in queues]) -``` - -Queues will be retrieved from the active environment, or another one can be specified: - -```python notest -dev_queues = modal.Queue.objects.list(environment_name="dev") -``` - -By default, all named Queues are returned, newest to oldest. It's also possible to limit the -number of results and to filter by creation date: - -```python -queues = modal.Queue.objects.list(max_objects=10, created_before="2025-01-01") -``` - -### objects.delete - -```python -delete(self, name, *, allow_missing=False, environment_name=None, client=None) -``` -Delete a named Queue entirely (not a single message or partition). - -Deletion is irreversible and affects any Apps using this Queue. - -Added in v1.1.2. - -**Parameters** - - - - - - -**Usage** - -```python notest -await modal.Queue.objects.delete("my-queue") -``` - -Queues will be deleted from the active environment, or another one can be specified: - -```python notest -await modal.Queue.objects.delete("my-queue", environment_name="dev") -``` - -## name - -```python -name(self) -``` - - -## validate_partition_key - -```python -validate_partition_key(partition) -``` - - -## ephemeral - -```python -ephemeral(cls, client=None, environment_name=None) -``` -Create an anonymous Queue that exists for the duration of the context manager. - -**Parameters** - - - - -**Usage** - -```python -from modal import Queue - -with Queue.ephemeral() as q: - q.put(123) -``` - -```python notest -async with Queue.ephemeral() as q: - await q.put.aio(123) -``` - -## from_name - -```python -from_name(name, *, environment_name=None, create_if_missing=False, client=None) -``` -Reference a named Queue, optionally creating it on the server first. - -Hydration is lazy: metadata is fetched from Modal the first time the handle is used. - -**Parameters** - - - - - - -**Returns** - -A `Queue` handle (possibly not yet hydrated). - -**Usage** - -```python -q = modal.Queue.from_name("my-queue", create_if_missing=True) -q.put(123) -``` - -## from_id - -```python -from_id(queue_id, client=None) -``` -Construct a Queue from an id and look up the Queue metadata. - -This is a lazy method that defers hydrating the local -object with metadata from Modal servers until the first -time it is actually used. - -The ID of a Queue object can be accessed using `.object_id`. - -**Parameters** - - - - -**Returns** - -A `Queue` handle (possibly not yet hydrated). - -**Usage** - -```python notest -@app.function() -def my_consumer(queue_id: str): - queue = modal.Queue.from_id(queue_id) - queue.put("Hello from remote function!") - -with modal.Queue.ephemeral() as q: - my_consumer.remote(q.object_id) - print(q.get()) # "Hello from remote function!" -``` - -## info - -```python -info(self) -``` -Return information about the Queue object. - -## clear - -```python -clear(self, *, partition=None, all=False) -``` -Clear the contents of a single partition or all partitions. - -Warning: this is a destructive operation and will irrevocably delete data. - -**Parameters** - - - - -**Usage** - -```python -q = modal.Queue.from_name("my-queue", create_if_missing=True) -q.clear() -``` - -## get - -```python -get(self, block=True, timeout=None, *, partition=None) -``` -Remove and return the next object in the queue. - -If `block` is `True` (the default) and the queue is empty, `get` will wait indefinitely for -an object, or until `timeout` if specified. Raises a native `queue.Empty` exception -if the `timeout` is reached. - -If `block` is `False`, `get` returns `None` immediately if the queue is empty. The `timeout` is -ignored in this case. - -**Parameters** - - - - - -## get_many - -```python -get_many(self, n_values, block=True, timeout=None, *, partition=None) -``` -Remove and return up to `n_values` objects from the queue. - -If there are fewer than `n_values` items in the queue, return all of them. - -If `block` is `True` (the default) and the queue is empty, `get_many` waits until at least one -object is present, or until `timeout` if specified. Raises the stdlib's `queue.Empty` if the -timeout is reached before any item arrives. - -If `block` is `False`, this returns an empty list immediately when the queue is empty. The `timeout` -is ignored in that case. - -**Parameters** - - - - - - -## put - -```python -put(self, v, block=True, timeout=None, *, partition=None, partition_ttl=24 * - 3600) -``` -Add an object to the end of the queue. - -If `block` is `True` and the queue is full, this method will retry indefinitely or -until `timeout` if specified. Raises the stdlib's `queue.Full` exception if the `timeout` is reached. -If blocking it is not recommended to omit the `timeout`, as the operation could wait indefinitely. - -If `block` is `False`, this method raises `queue.Full` immediately if the queue is full. The `timeout` is -ignored in this case. - -**Parameters** - - - - - - - -## put_many - -```python -put_many(self, vs, block=True, timeout=None, *, partition=None, partition_ttl=24 - * 3600) -``` -Add several objects to the end of the queue. - -If `block` is `True` and the queue is full, this method will retry indefinitely or -until `timeout` if specified. Raises the stdlib's `queue.Full` exception if the `timeout` is reached. -If blocking it is not recommended to omit the `timeout`, as the operation could wait indefinitely. - -If `block` is `False`, this method raises `queue.Full` immediately if the queue is full. The `timeout` is -ignored in this case. - -**Parameters** - - - - - - - -## len - -```python -len(self, *, partition=None, total=False) -``` -Return the number of objects in the queue partition. - -**Parameters** - - - - -**Returns** - -Item count (capped by the server when very large). - -## iterate - -```python -iterate(self, *, partition=None, item_poll_timeout=0.0) -``` -Iterate through items in the queue without mutation. - -Specify `item_poll_timeout` to control how long the iterator should wait for the next time before giving up. - -**Parameters** - - - diff --git a/.agents/skills/modal/references/api/modal.Retries.md b/.agents/skills/modal/references/api/modal.Retries.md deleted file mode 100644 index 66d3047b1ba846fca8b191a08c74321f77014518..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Retries.md +++ /dev/null @@ -1,58 +0,0 @@ -# modal.Retries - - -```python -class Retries(object) -``` - -Adds a retry policy to a Modal function. - -**Usage** - -```python -import modal -app = modal.App() - -# Basic configuration. -# This sets a policy of max 4 retries with 1-second delay between failures. -@app.function(retries=4) -def f(): - pass - - -# Fixed-interval retries with 3-second delay between failures. -@app.function( - retries=modal.Retries( - max_retries=2, - backoff_coefficient=1.0, - initial_delay=3.0, - ) -) -def g(): - pass - - -# Exponential backoff, with retry delay doubling after each failure. -@app.function( - retries=modal.Retries( - max_retries=4, - backoff_coefficient=2.0, - initial_delay=1.0, - ) -) -def h(): - pass -``` - -```python -__init__(self, *, max_retries, backoff_coefficient=2.0, initial_delay=1.0, - max_delay=60.0) -``` -Construct a new retries policy, supporting exponential and fixed-interval delays via a backoff coefficient. - -**Parameters** - - - - - diff --git a/.agents/skills/modal/references/api/modal.Sandbox.md b/.agents/skills/modal/references/api/modal.Sandbox.md deleted file mode 100644 index c4e9ba48399df670b73ed47e1f006ee39e5dbbdc..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Sandbox.md +++ /dev/null @@ -1,950 +0,0 @@ -# modal.Sandbox - - -```python -class Sandbox(modal.object.Object) -``` - -A `Sandbox` object lets you interact with a running sandbox. This API is similar to Python's -[asyncio.subprocess.Process](https://docs.python.org/3/library/asyncio-subprocess.html#asyncio.subprocess.Process). - -Refer to the [guide](https://modal.com/docs/guide/sandbox) on how to spawn and use sandboxes. - - -## hydrate - -```python -hydrate(self, client=None) -``` -Synchronize the local object with its identity on the Modal server. - -It is rarely necessary to call this method explicitly, as most operations -will lazily hydrate when needed. The main use case is when you need to -access object metadata, such as its ID. - -*Added in v0.72.39*: This method replaces the deprecated `.resolve()` method. - -## create - -```python -create(*args, app=None, name=None, tags=None, image=None, env=None, - secrets=None, network_file_systems={}, timeout=300, idle_timeout=None, - workdir=None, gpu=None, cloud=None, region=None, cpu=None, memory=None, - block_network=False, outbound_cidr_allowlist=None, - outbound_domain_allowlist=None, inbound_cidr_allowlist=None, volumes={}, - pty=False, encrypted_ports=[], h2_ports=[], unencrypted_ports=[], - custom_domain=None, proxy=None, include_oidc_identity_token=False, - readiness_probe=None, verbose=False, experimental_options=None, - _experimental_enable_snapshot=False, client=None, environment_name=None, - pty_info=None, cidr_allowlist=None) -``` -Create a new Sandbox to run untrusted, arbitrary code. - -The Sandbox's corresponding container will be created asynchronously. - -**Parameters** - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -**Returns** - -A `Sandbox` object representing the created sandbox which can be used to interact with the sandbox. - -**Raises** - -- `AlreadyExistsError`: If a sandbox with the same name already exists. - -**Usage** - -```python -app = modal.App.lookup('sandbox-hello-world', create_if_missing=True) -sandbox = modal.Sandbox.create("echo", "hello world", app=app) -print(sandbox.stdout.read()) -sandbox.wait() -``` - -## detach - -```python -detach(self) -``` -Disconnects your client from the sandbox and cleans up resources assoicated with the connection. - -Be sure to only call `detach` when you are done interacting with the sandbox. After calling `detach`, -any operation using the Sandbox object is not guaranteed to work anymore. If you want to continue interacting -with a running sandbox, use `Sandbox.from_id` to get a new Sandbox object. - -## from_name - -```python -from_name(app_name, name, *, environment_name=None, client=None) -``` -Get a running Sandbox by name from a deployed App. - -A Sandbox's name is the `name` argument passed to `Sandbox.create`. - -**Parameters** - - - - - - -**Returns** - -A `Sandbox` handle for the running sandbox. - -**Raises** - -- `NotFoundError`: If no running sandbox exists with the given name. - -## from_id - -```python -from_id(sandbox_id, client=None) -``` -Construct a Sandbox from an id and look up the Sandbox result. - -The ID of a Sandbox object can be accessed using `.object_id`. - -**Parameters** - - - - -**Returns** - -A `Sandbox` handle with any available result metadata populated from the server. - -## get_tags - -```python -get_tags(self) -``` -Fetches any tags (key-value pairs) currently attached to this Sandbox from the server. - -**Returns** - -Tags as a map from tag name to tag value. - -## set_tags - -```python -set_tags(self, tags, *, client=None) -``` -Set tags (key-value pairs) on the Sandbox. Tags can be used to filter results in `Sandbox.list`. - -**Parameters** - - - - -## snapshot_filesystem - -```python -snapshot_filesystem(self, timeout=55, *, ttl=30 * 24 * 3600) -``` -Snapshot the filesystem of the Sandbox. - -**Parameters** - - - - -**Returns** - -An [`Image`](https://modal.com/docs/reference/modal.Image) object which can be used to spawn a new -Sandbox with the same filesystem. - -## mount_image - -```python -mount_image(self, path, image, *, _experimental_encryption_key=None) -``` -Mount an Image at a specified path in a running Sandbox. - -`path` should be a directory that is **not** the root path (`/`). If the path doesn't exist -it will be created. If it exists and contains data, the previous directory will be replaced -by the mount. - -The `image` argument supports any Image that has an object ID, including: -- Images built using `image.build()` -- Images referenced by ID, e.g. `Image.from_id(...)` -- Filesystem/directory snapshots, e.g. created by `.snapshot_directory()` or `.snapshot_filesystem()` -- Empty images created with `Image.from_scratch()` - -**Parameters** - - - - -**Usage** - -```py notest -user_project_snapshot: Image = sandbox_session_1.snapshot_directory("/user_project") - -# You can later mount this snapshot to another Sandbox: -sandbox_session_2 = modal.Sandbox.create(...) -sandbox_session_2.mount_image("/user_project", user_project_snapshot) -sandbox_session_2.filesystem.list_files("/user_project") -``` - -## unmount_image - -```python -unmount_image(self, path) -``` -Unmount a previously mounted Image from a running Sandbox. - -`path` must be the exact mount point that was passed to `.mount_image()`. -After unmounting, the underlying Sandbox filesystem at that path becomes -visible again. - -**Parameters** - - - -## snapshot_directory - -```python -snapshot_directory(self, path, *, timeout=55, ttl=30 * 24 * 3600, - _experimental_encryption_key=None) -``` -Snapshot a directory in a running Sandbox, creating a new Image with its content. - -`timeout` If the snapshot does not return within that window, the call is cancelled -and `modal.exception.TimeoutError` is raised. - -`ttl` The resulting Image is retained for `ttl` seconds (default: 30 days) -Pass `ttl=None` to retain the image indefinitely. - -**Parameters** - - - -**Returns** - -An `Image` containing the directory contents. - -**Usage** - -```py notest -user_project_snapshot: Image = sandbox_session_1.snapshot_directory("/user_project") - -# You can later mount this snapshot to another Sandbox: -sandbox_session_2 = modal.Sandbox.create(...) -sandbox_session_2.mount_image("/user_project", user_project_snapshot) -sandbox_session_2.filesystem.list_files("/user_project") -``` - -## wait - -```python -wait(self, raise_on_termination=True) -``` -Wait for the Sandbox to finish running. - -**Parameters** - - - -## wait_until_ready - -```python -wait_until_ready(self, *, timeout=300) -``` -Wait for the Sandbox readiness probe to report that the Sandbox is ready. - -The Sandbox must be configured with a `readiness_probe` in order to use this method. - -**Parameters** - - - -**Usage** - -```py notest -app = modal.App.lookup('sandbox-wait-until-ready', create_if_missing=True) -sandbox = modal.Sandbox.create( - "python3", "-m", "http.server", "8080", - readiness_probe=modal.Probe.with_tcp(8080), - app=app, -) -sandbox.wait_until_ready() -``` - -## tunnels - -```python -tunnels(self, timeout=50) -``` -Get Tunnel metadata for the sandbox. - -NOTE: Previous to client [v0.64.153](https://modal.com/docs/reference/changelog#064153-2024-09-30), this -returned a list of `TunnelData` objects. - -**Parameters** - - - -**Returns** - -A dictionary mapping container port to `Tunnel` metadata. - -**Raises** - -- `SandboxTimeoutError`: If the tunnels are not available after the timeout. - -## create_connect_token - -```python -create_connect_token(self, user_metadata=None) -``` -Create a token for making HTTP connections to the Sandbox. - -Also accepts an optional user_metadata string or dict to associate with the token. This metadata -will be added to the headers by the proxy when forwarding requests to the Sandbox. - -**Parameters** - - - -**Returns** - -URL and token credentials for connecting to the sandbox over HTTP. - -## reload_volumes - -```python -reload_volumes(self) -``` -Reload all Volumes mounted in the Sandbox. - -Added in v1.1.0. - -## terminate - -```python -terminate(self, *, wait=False) -``` -Terminate Sandbox execution. - -This is a no-op if the Sandbox has already finished running. - -**Parameters** - - - -**Returns** - -The sandbox exit code when `wait` is True; otherwise None. - -## poll - -```python -poll(self) -``` -Check if the Sandbox has finished running. - -**Returns** - -`None` if the Sandbox is still running, otherwise the exit code. - -## exec - -```python -exec(self, *args, stdout=StreamType.PIPE, stderr=StreamType.PIPE, timeout=None, - workdir=None, env=None, secrets=None, text=True, bufsize=-1, pty=False, - _pty_info=None, pty_info=None) -``` -Execute a command in the Sandbox and return a ContainerProcess handle. - -See the [`ContainerProcess`](https://modal.com/docs/reference/modal.container_process#modalcontainer_processcontainerprocess) -docs for more information. - -**Parameters** - - - - - - - - - - - - - - -**Returns** - -A `ContainerProcess` handle for the running command (text or bytes depending on `text`). - -**Usage** - -```python fixture:sandbox -process = sandbox.exec("bash", "-c", "for i in $(seq 1 3); do echo foo $i; sleep 0.1; done") -for line in process.stdout: - print(line) -``` - -## filesystem - - -```python -filesystem: SandboxFilesystem -``` - -Namespace for Sandbox filesystem APIs. - - -### filesystem.copy_from_local - -```python -copy_from_local(self, local_path, remote_path) -``` -Copy a local file into the Sandbox. - -`remote_path` must be an absolute path to a file in the Sandbox. -Parent directories for `remote_path` are created if needed. -The remote file is overwritten if it already exists. - -**Parameters** - - - - -**Raises** - -- `SandboxFilesystemNotADirectoryError`: A parent path component of ``remote_path`` is not a directory. -- `SandboxFilesystemIsADirectoryError`: ``remote_path`` points to a directory. -- `SandboxFilesystemPermissionError`: Write permission is denied in the Sandbox. -- `SandboxFilesystemError`: The command fails for any other reason. -- `FileNotFoundError`: ``local_path`` does not exist. -- `IsADirectoryError`: ``local_path`` is a directory. -- `PermissionError`: Reading ``local_path`` is not permitted. - -**Usage** - -```python fixture:sandbox fixture:tmpdir -import tempfile -from pathlib import Path - -local_path = Path(tempfile.mktemp()) -local_path.write_text("Hello, world!\n") -sandbox.filesystem.copy_from_local(local_path, "/tmp/hello.txt") -``` - -### filesystem.copy_to_local - -```python -copy_to_local(self, remote_path, local_path) -``` -Copy a file from the Sandbox to a local path. - -`remote_path` must be an absolute path to a file in the Sandbox. -Parent directories for `local_path` are created if needed. -The local file is overwritten if it already exists. - -**Raises** - -- `SandboxFilesystemNotFoundError`: the remote path does not exist. -- `SandboxFilesystemIsADirectoryError`: the remote path points to a directory. -- `SandboxFilesystemPermissionError`: read permission is denied in the Sandbox. -- `SandboxFilesystemError`: the command fails for any other reason. -- `IsADirectoryError`: `local_path` points to a directory. -- `NotADirectoryError`: a component of the `local_path` parent is not a directory. -- `PermissionError`: writing `local_path` is not permitted. - -**Usage** - -```python fixture:sandbox fixture:tmpdir -sandbox.filesystem.write_text("Hello, world!\n", "/tmp/hello.txt") -sandbox.filesystem.copy_to_local("/tmp/hello.txt", "/tmp/local-hello.txt") -``` - -### filesystem.list_files - -```python -list_files(self, remote_path) -``` -List files and directories in a Sandbox directory. - -**Parameters** - - - -**Returns** - -A list of `FileInfo` objects describing each entry. - -**Raises** - -- `SandboxFilesystemNotFoundError`: The path does not exist. -- `SandboxFilesystemNotADirectoryError`: The path is not a directory. -- `SandboxFilesystemPermissionError`: Read permission is denied. -- `SandboxFilesystemError`: The command fails for any other reason. - -**Usage** - -```python fixture:sandbox -entries = sandbox.filesystem.list_files("/tmp") -for entry in entries: - print(entry.name, entry.type, entry.size) -``` - -### filesystem.make_directory - -```python -make_directory(self, remote_path, *, create_parents=True) -``` -Create a new directory in the Sandbox. - -`remote_path` must be an absolute path in the Sandbox. - -When `create_parents` is `True` (the default), any missing parent directories are created and the call is -idempotent (succeeds silently if the directory already exists). When `create_parents` is `False`, the -immediate parent directory must already exist and the path must not already exist. - -**Parameters** - - - - -**Raises** - -- `SandboxFilesystemNotFoundError`: The parent directory does not exist and ``create_parents`` is false. -- `SandboxFilesystemPathAlreadyExistsError`: The path already exists. -- `SandboxFilesystemNotADirectoryError`: A path component is not a directory. -- `SandboxFilesystemPermissionError`: Creation is not permitted. -- `InvalidError`: The operation is not supported by the mount. -- `SandboxFilesystemError`: The command fails for any other reason. - -**Usage** - -```python fixture:sandbox -sandbox.filesystem.make_directory("/tmp/a/b/c") -``` - -### filesystem.read_bytes - -```python -read_bytes(self, remote_path) -``` -Read a file from the Sandbox and return its contents as bytes. - -`remote_path` must be an absolute path to a file in the Sandbox. - -**Parameters** - - - -**Returns** - -Raw bytes read from the file. - -**Raises** - -- `SandboxFilesystemNotFoundError`: The path does not exist. -- `SandboxFilesystemIsADirectoryError`: The path points to a directory. -- `SandboxFilesystemPermissionError`: Read permission is denied. -- `SandboxFilesystemError`: The command fails for any other reason. - -**Usage** - -```python fixture:sandbox -sandbox.filesystem.write_bytes(b"Hello, world!\n", "/tmp/hello.bin") -contents = sandbox.filesystem.read_bytes("/tmp/hello.bin") -print(contents.decode("utf-8")) -``` - -### filesystem.read_text - -```python -read_text(self, remote_path) -``` -Read a file from the Sandbox and return its contents as a UTF-8 string. - -`remote_path` must be an absolute path to a file in the Sandbox. - -**Parameters** - - - -**Returns** - -File contents decoded as UTF-8. - -**Raises** - -- `SandboxFilesystemNotFoundError`: The path does not exist. -- `SandboxFilesystemIsADirectoryError`: The path points to a directory. -- `SandboxFilesystemPermissionError`: Read permission is denied. -- `SandboxFilesystemError`: The command fails for any other reason. - -**Usage** - -```python fixture:sandbox -sandbox.filesystem.write_text("Hello, world!\n", "/tmp/hello.txt") -contents = sandbox.filesystem.read_text("/tmp/hello.txt") -print(contents) -``` - -### filesystem.remove - -```python -remove(self, remote_path, *, recursive=False) -``` -Remove a file or directory in the Sandbox. - -When `remote_path` is a directory and `recursive` is `False` (the -default), removes it only if it is empty. When `recursive` is `True`, -removes the directory and all its contents. - -Recursive directory removal is not supported on all mounts. -In particular, `CloudBucketMount` does not support it. An -`InvalidError` is raised in that case. - -**Parameters** - - - - -**Raises** - -- `SandboxFilesystemNotFoundError`: The remote path does not exist. -- `SandboxFilesystemDirectoryNotEmptyError`: `recursive` is `False` and the directory is not empty. -- `SandboxFilesystemPermissionError`: Read permission is denied in the Sandbox. -- `InvalidError`: The operation is not supported by the mount. -- `SandboxFilesystemError`: The command fails for any other reason. - -**Usage** - -To remove a file: - -```python fixture:sandbox -sandbox.filesystem.write_bytes(b"Hello, world!\n", "/tmp/hello.bin") -sandbox.filesystem.remove("/tmp/hello.bin") -``` - -To remove a directory and all its contents: - -```python fixture:sandbox -sandbox.filesystem.make_directory("/tmp/mydir/subdir") -sandbox.filesystem.remove("/tmp/mydir", recursive=True) -``` - -### filesystem.stat - -```python -stat(self, remote_path) -``` -Return metadata for a single file, directory, or symlink in the Sandbox. - -`remote_path` must be an absolute path in the Sandbox. If `remote_path` is a symlink, the returned -`FileInfo` object describes the symlink, not the target it points to. - -**Raises** - -- `SandboxFilesystemNotFoundError`: the path does not exist. -- `SandboxFilesystemNotADirectoryError`: a non-leaf component of the path is not a directory. -- `SandboxFilesystemPermissionError`: a component of the path is not searchable. -- `SandboxFilesystemError`: the command fails for any other reason. - -**Usage** - -```python fixture:sandbox -sandbox.filesystem.write_text("Hello, world!\n", "/tmp/hello.txt") -info = sandbox.filesystem.stat("/tmp/hello.txt") -print(info.size, info.permissions, info.modified_time) -``` - -### filesystem.watch - -```python -watch(self, remote_path, *, filter=None, recursive=False, timeout=None) -``` -Watch a path in the Sandbox for filesystem changes. - -`remote_path` must be an absolute path in the Sandbox. If it points -to a file, events for that file are reported. If it points to a -directory, events for entries directly inside it are reported. Set -`recursive=True` to also receive events for all nested subdirectories. -If `remote_path` is a symlink, it is followed and events reference -paths under the resolved target. - -Yields `FileWatchEvent` objects as changes occur, until either -`timeout` seconds elapse, the iterator is closed, or the Sandbox -is terminated. - -Optionally restrict the kinds of events emitted to those included -in `filter`. The default filter `None` permits all event types. - -`timeout` is in seconds. `None` means watch indefinitely. When -`timeout` elapses, the iterator stops without raising an exception. - -**Raises** - -- `SandboxFilesystemNotFoundError`: `remote_path` does not exist. -- `SandboxFilesystemPermissionError`: watch access is denied. -- `InvalidError`: the filesystem at `remote_path` does not support - watching. -- `SandboxFilesystemError`: the command fails for any other reason. - -**Usage** - -```python notest -for event in sandbox.filesystem.watch( - "/tmp/foo", - recursive=True, - filter=[FileWatchEventType.Create], - timeout=60, -): - if any(p.endswith(".done") for p in event.paths): - break -``` - -### filesystem.write_bytes - -```python -write_bytes(self, data, remote_path) -``` -Write binary content to a file in the Sandbox. - -`remote_path` must be an absolute path to a file in the Sandbox. -Parent directories for `remote_path` are created if needed. -The remote file is overwritten if it already exists. - -**Parameters** - - - - -**Raises** - -- `TypeError`: ``data`` is not bytes-like. -- `SandboxFilesystemNotADirectoryError`: A parent path component is not a directory. -- `SandboxFilesystemIsADirectoryError`: ``remote_path`` points to a directory. -- `SandboxFilesystemPermissionError`: Write permission is denied. -- `SandboxFilesystemError`: The command fails for any other reason. - -**Usage** - -```python fixture:sandbox -sandbox.filesystem.write_bytes(b"Hello, world!\n", "/tmp/hello.bin") -``` - -### filesystem.write_text - -```python -write_text(self, data, remote_path) -``` -Write UTF-8 text to a file in the Sandbox. - -`remote_path` must be an absolute path to a file in the Sandbox. -Parent directories for `remote_path` are created if needed. -The remote file is overwritten if it already exists. - -**Parameters** - - - - -**Raises** - -- `TypeError`: ``data`` is not a string. -- `SandboxFilesystemNotADirectoryError`: A parent path component is not a directory. -- `SandboxFilesystemIsADirectoryError`: ``remote_path`` points to a directory. -- `SandboxFilesystemPermissionError`: Write permission is denied. -- `SandboxFilesystemError`: The command fails for any other reason. - -**Usage** - -```python fixture:sandbox -sandbox.filesystem.write_text("Hello, world!\n", "/tmp/hello.txt") -``` - -## open - -```python -open(self, path, mode="r") -``` -[Alpha] Open a file in the Sandbox and return a FileIO handle. - -**Deprecated (2026-03-09):** Use the `Sandbox.filesystem` APIs instead for improved reliability. - -See the [`FileIO`](https://modal.com/docs/reference/modal.file_io#modalfile_iofileio) docs for more information. - -**Parameters** - - - - -**Returns** - -A `FileIO` handle for reading or writing the remote file. - -**Usage** - -```python notest -sb = modal.Sandbox.create(app=sb_app) -f = sb.open("/test.txt", "w") -f.write("hello") -f.close() -``` - -## ls - -```python -ls(self, path) -``` -[Alpha] List the contents of a directory in the Sandbox. - -**Deprecated (2026-04-15):** Use `Sandbox.filesystem.list_files()` instead for improved reliability. - -**Parameters** - - - -**Returns** - -Entry names in the directory as a list of strings. - -## mkdir - -```python -mkdir(self, path, parents=False) -``` -[Alpha] Create a new directory in the Sandbox. - -**Deprecated (2026-04-15):** Use `Sandbox.filesystem.make_directory()` instead for improved reliability. - -## rm - -```python -rm(self, path, recursive=False) -``` -[Alpha] Remove a file or directory in the Sandbox. - -**Deprecated (2026-04-15):** Use `Sandbox.filesystem.remove()` instead for improved reliability. - -## watch - -```python -watch(self, path, filter=None, recursive=None, timeout=None) -``` -[Alpha] Watch a file or directory in the Sandbox for changes. - -**Deprecated (2026-05-08):** Use `Sandbox.filesystem.watch()` instead for improved reliability. - -**Parameters** - - - - - - -**Returns** - -An async iterator of `FileWatchEvent` values. - -## stdout - -```python -stdout(self) -``` -[`StreamReader`](https://modal.com/docs/reference/modal.io_streams#modalio_streamsstreamreader) -for the sandbox's stdout stream. - -**Returns** - -Stream reader for sandbox stdout. - -## stderr - -```python -stderr(self) -``` -[`StreamReader`](https://modal.com/docs/reference/modal.io_streams#modalio_streamsstreamreader) -for the Sandbox's stderr stream. - -**Returns** - -Stream reader for sandbox stderr. - -## stdin - -```python -stdin(self) -``` -[`StreamWriter`](https://modal.com/docs/reference/modal.io_streams#modalio_streamsstreamwriter) -for the Sandbox's stdin stream. - -**Returns** - -Stream writer for sandbox stdin. - -## returncode - -```python -returncode(self) -``` -Return code of the Sandbox process if it has finished running, else `None`. - -**Returns** - -Exit code when the sandbox process has completed, otherwise None. - -## list - -```python -list(*, app_id=None, tags=None, client=None) -``` -List all Sandboxes for the current Environment or App ID (if specified). If tags are specified, only -Sandboxes that have at least those tags are returned. - -**Parameters** - - - - - -**Returns** - -An async generator yielding `Sandbox` objects. diff --git a/.agents/skills/modal/references/api/modal.SandboxSnapshot.md b/.agents/skills/modal/references/api/modal.SandboxSnapshot.md deleted file mode 100644 index 722072245d18f3d46043a3560b3b9c6380392098..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.SandboxSnapshot.md +++ /dev/null @@ -1,42 +0,0 @@ -# modal.SandboxSnapshot - - -```python -class SandboxSnapshot(modal.object.Object) -``` - -> Sandbox memory snapshots are in **early preview**. - -A `SandboxSnapshot` object lets you interact with a stored Sandbox snapshot that was created by calling -`._experimental_snapshot()` on a Sandbox instance. This includes both the filesystem and memory state of -the original Sandbox at the time the snapshot was taken. - - -## hydrate - -```python -hydrate(self, client=None) -``` -Synchronize the local object with its identity on the Modal server. - -It is rarely necessary to call this method explicitly, as most operations -will lazily hydrate when needed. The main use case is when you need to -access object metadata, such as its ID. - -*Added in v0.72.39*: This method replaces the deprecated `.resolve()` method. - -## from_id - -```python -from_id(cls, sandbox_snapshot_id, client=None) -``` -Construct a `SandboxSnapshot` for an existing snapshot ID. - -**Parameters** - - - - -**Returns** - -A `SandboxSnapshot` handle (hydration validates the ID when used). diff --git a/.agents/skills/modal/references/api/modal.Secret.md b/.agents/skills/modal/references/api/modal.Secret.md deleted file mode 100644 index c6bf8f1587ae4534a2ba7c94050b7909cb570cb1..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Secret.md +++ /dev/null @@ -1,285 +0,0 @@ -# modal.Secret - - -```python -class Secret(modal.object.Object) -``` - -Secrets provide a dictionary of environment variables for images. - -Secrets are a secure way to add credentials and other sensitive information -to the containers your functions run in. You can create and edit secrets on -[the dashboard](https://modal.com/secrets), or programmatically from Python code. - -See [the secrets guide page](https://modal.com/docs/guide/secrets) for more information. - - -## hydrate - -```python -hydrate(self, client=None) -``` -Synchronize the local object with its identity on the Modal server. - -It is rarely necessary to call this method explicitly, as most operations -will lazily hydrate when needed. The main use case is when you need to -access object metadata, such as its ID. - -*Added in v0.72.39*: This method replaces the deprecated `.resolve()` method. - -## objects - - -```python -objects: SecretManager -``` - -Namespace with methods for managing named Secret objects. - - -### objects.create - -```python -create(self, name, env_dict, *, allow_existing=False, environment_name=None, - client=None) -``` -Create a new named Secret in the workspace environment. - -This does not return a local handle; use `modal.Secret.from_name` to look up the Secret after creation. - -Added in v1.1.2. - -**Parameters** - - - - - - - -**Usage** - -```python notest -contents = {"MY_KEY": "my-value", "MY_OTHER_KEY": "my-other-value"} -modal.Secret.objects.create("my-secret", contents) -``` - -Secrets will be created in the active environment, or another one can be specified: - -```python notest -modal.Secret.objects.create("my-secret", contents, environment_name="dev") -``` - -By default, an error will be raised if the Secret already exists, but passing -`allow_existing=True` will make the creation attempt a no-op in this case. -If the `env_dict` data differs from the existing Secret, it will be ignored. - -```python notest -modal.Secret.objects.create("my-secret", contents, allow_existing=True) -``` - -Note that this method does not return a local instance of the Secret. You can use -`modal.Secret.from_name` to perform a lookup after creation. - -### objects.list - -```python -list(self, *, max_objects=None, created_before=None, environment_name="", - client=None) -``` -List named Secrets in the workspace environment as hydrated handles. - -Results are ordered newest to oldest. By default, all matching Secrets are returned. - -Added in v1.1.2. - -**Parameters** - - - - - - -**Returns** - -Hydrated `Secret` objects for each named Secret in the listing. - -**Usage** - -```python -secrets = modal.Secret.objects.list() -print([s.name for s in secrets]) -``` - -Secrets will be retrieved from the active environment, or another one can be specified: - -```python notest -dev_secrets = modal.Secret.objects.list(environment_name="dev") -``` - -By default, all named Secrets are returned, newest to oldest. It's also possible to limit the -number of results and to filter by creation date: - -```python -secrets = modal.Secret.objects.list(max_objects=10, created_before="2025-01-01") -``` - -### objects.delete - -```python -delete(self, name, *, allow_missing=False, environment_name=None, client=None) -``` -Delete a named Secret entirely. - -Deletion is irreversible and affects any Apps using this Secret. - -Added in v1.1.2. - -**Parameters** - - - - - - -**Usage** - -```python notest -await modal.Secret.objects.delete("my-secret") -``` - -Secrets will be deleted from the active environment, or another one can be specified: - -```python notest -await modal.Secret.objects.delete("my-secret", environment_name="dev") -``` - -## name - -```python -name(self) -``` - - -## from_dict - -```python -from_dict(env_dict={}) -``` -Create a Secret from a dictionary of environment variable names to string values. - -Values may be ``None``; those keys are omitted from the Secret. - -**Parameters** - - - -**Returns** - -A lazy `Secret` handle backed by the given key-value pairs. - -**Usage** - -```python -@app.function(secrets=[modal.Secret.from_dict({"FOO": "bar"})]) -def run(): - print(os.environ["FOO"]) -``` - -## from_local_environ - -```python -from_local_environ(env_keys) -``` -Build a Secret from the current process environment (local runs only). - -In remote execution, returns an empty Secret. - -**Parameters** - - - -**Returns** - -A `Secret` containing the resolved variables (or empty when not local). - -## from_dotenv - -```python -from_dotenv(path=None, *, filename=".env", client=None) -``` -Load environment variables from a `.env` file into a Secret. - -With no `path`, searches from the current working directory (not the caller's file path). -With `path` set, walks upward from that file or directory to find `filename`. - -**Parameters** - - - - - -**Returns** - -A lazy `Secret` handle whose values are loaded from the resolved `.env` file. - -**Usage** - -```python -@app.function(secrets=[modal.Secret.from_dotenv(__file__)]) -def run(): - print(os.environ["USERNAME"]) # Assumes USERNAME is defined in your .env file -``` - -```python -@app.function(secrets=[modal.Secret.from_dotenv(filename=".env-dev")]) -def run(): - ... -``` - -## from_name - -```python -from_name(name, *, environment_name=None, required_keys=[], client=None) -``` -Reference a deployed Secret by name. - -Hydration is lazy until the Secret is used. - -**Parameters** - - - - - - -**Returns** - -A `Secret` handle (possibly not yet hydrated). - -**Usage** - -```python -secret = modal.Secret.from_name("my-secret") - -@app.function(secrets=[secret]) -def run(): - ... -``` - -## info - -```python -info(self) -``` -Return information about the Secret object. - -## update - -```python -update(self, env_dict) -``` -Update this Secret, adding or overwriting key-value pairs. - -Like dict.update(), this merges `env_dict` into the existing Secret. -Keys not mentioned in `env_dict` are left unchanged. diff --git a/.agents/skills/modal/references/api/modal.Tunnel.md b/.agents/skills/modal/references/api/modal.Tunnel.md deleted file mode 100644 index 63ebadc87bfbe60eefde3979f1c13a569500377d..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Tunnel.md +++ /dev/null @@ -1,36 +0,0 @@ -# modal.Tunnel - - -```python -class Tunnel(object) -``` - -A port forwarded from within a running Modal container. Created by `modal.forward()`. - -**Important:** This is an experimental API which may change in the future. - -```python -__init__(self, host, port, unencrypted_host, unencrypted_port) -``` - - -## url - -```python -url(self) -``` -Get the public HTTPS URL of the forwarded port. - -## tls_socket - -```python -tls_socket(self) -``` -Get the public TLS socket as a (host, port) tuple. - -## tcp_socket - -```python -tcp_socket(self) -``` -Get the public TCP socket as a (host, port) tuple. diff --git a/.agents/skills/modal/references/api/modal.Volume.md b/.agents/skills/modal/references/api/modal.Volume.md deleted file mode 100644 index bd3b2dc51e94ed906463b6d7ba532d8f9ad069da..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Volume.md +++ /dev/null @@ -1,480 +0,0 @@ -# modal.Volume - - -```python -class Volume(modal.object.Object) -``` - -A writeable volume that can be used to share files between one or more Modal functions. - -The contents of a volume is exposed as a filesystem. You can use it to share data between different functions, or -to persist durable state across several instances of the same function. - -Unlike a networked filesystem, you need to explicitly reload the volume to see changes made since it was mounted. -Similarly, you need to explicitly commit any changes you make to the volume for the changes to become visible -outside the current container. - -Concurrent modification is supported, but concurrent modifications of the same files should be avoided! Last write -wins in case of concurrent modification of the same file - any data the last writer didn't have when committing -changes will be lost! - -As a result, volumes are typically not a good fit for use cases where you need to make concurrent modifications to -the same file (nor is distributed file locking supported). - -Volumes can only be reloaded if there are no open files for the volume - attempting to reload with open files -will result in an error. - -**Usage** - -```python -import modal - -app = modal.App() -volume = modal.Volume.from_name("my-persisted-volume", create_if_missing=True) - -@app.function(volumes={"/root/foo": volume}) -def f(): - with open("/root/foo/bar.txt", "w") as f: - f.write("hello") - volume.commit() # Persist changes - -@app.function(volumes={"/root/foo": volume}) -def g(): - volume.reload() # Fetch latest changes - with open("/root/foo/bar.txt", "r") as f: - print(f.read()) -``` - - -## hydrate - -```python -hydrate(self, client=None) -``` -Synchronize the local object with its identity on the Modal server. - -It is rarely necessary to call this method explicitly, as most operations -will lazily hydrate when needed. The main use case is when you need to -access object metadata, such as its ID. - -*Added in v0.72.39*: This method replaces the deprecated `.resolve()` method. - -## objects - - -```python -objects: VolumeManager -``` - -Namespace with methods for managing named Volume objects. - - -### objects.create - -```python -create(self, name, *, version=None, allow_existing=False, environment_name=None, - client=None) -``` -Create a new named Volume in the workspace environment. - -This does not return a local handle; use `modal.Volume.from_name` to look up the Volume after creation. - -Added in v1.1.2. - -**Parameters** - - - - - - - -**Usage** - -```python notest -modal.Volume.objects.create("my-volume") -``` - -Volumes will be created in the active environment, or another one can be specified: - -```python notest -modal.Volume.objects.create("my-volume", environment_name="dev") -``` - -By default, an error is raised if the Volume already exists; `allow_existing=True` makes that case a no-op: - -```python notest -modal.Volume.objects.create("my-volume", allow_existing=True) -``` - -Note that this method does not return a local instance of the Volume. You can use -`modal.Volume.from_name` to perform a lookup after creation. - -### objects.list - -```python -list(self, *, max_objects=None, created_before=None, environment_name="", - client=None) -``` -List named Volumes in the workspace environment as hydrated handles. - -Results are ordered newest to oldest. By default, all matching Volumes are returned. - -Added in v1.1.2. - -**Parameters** - - - - - - -**Returns** - -Hydrated `Volume` objects for each named Volume in the listing. - -**Usage** - -```python -volumes = modal.Volume.objects.list() -print([v.name for v in volumes]) -``` - -Volumes will be retrieved from the active environment, or another one can be specified: - -```python notest -dev_volumes = modal.Volume.objects.list(environment_name="dev") -``` - -By default, all named Volumes are returned, newest to oldest. It's also possible to limit the -number of results and to filter by creation date: - -```python -volumes = modal.Volume.objects.list(max_objects=10, created_before="2025-01-01") -``` - -### objects.delete - -```python -delete(self, name, *, allow_missing=False, environment_name=None, client=None) -``` -Delete a named Volume entirely (not individual files). - -Deletion is irreversible and affects any Apps using this Volume. - -Added in v1.1.2. - -**Parameters** - - - - - - -**Usage** - -```python notest -await modal.Volume.objects.delete("my-volume") -``` - -Volumes will be deleted from the active environment, or another one can be specified: - -```python notest -await modal.Volume.objects.delete("my-volume", environment_name="dev") -``` - -## name - -```python -name(self) -``` - - -## with_mount_options - -```python -with_mount_options(self, *, read_only=None, sub_path=None) -``` -Configure options used when mounting this Volume. - -Note that these options are not properties stored with the Volume itself - they can be individually configured -for each Volume - container association. - -**Parameters** - - - - -**Returns** - -A `Volume` handle with the mount options applied. - -**Usage** - -To mount a volume in read-only mode: - -```python -import modal - -volume = modal.Volume.from_name("my-volume") - -@app.function(volumes={"/mnt": volume.with_mount_options(read_only=True)}) -def f(): - return os.mkdir("/mnt/foo") # not possible! -``` - -To mount only part of a Volume using sub_path: - -```python -import modal - -volume = modal.Volume.from_name("my-volume") - -@app.function(volumes={"/user_data": volume.with_mount_options(sub_path="/users/my_user")}) -def f(): - return os.listdir("/user_data") # lists data from /users/my_user -``` - -## from_name - -```python -from_name(name, *, environment_name=None, create_if_missing=False, version=None, - client=None) -``` -Reference a Volume by name, optionally creating it on the server first. - -Hydration is lazy: metadata is fetched from Modal the first time the handle is used. - -**Parameters** - - - - - - - -**Returns** - -A `Volume` handle (possibly not yet hydrated). - -**Usage** - -```python -vol = modal.Volume.from_name("my-volume", create_if_missing=True) - -app = modal.App() - -@app.function(volumes={"/data": vol}) -def f(): - pass -``` - -## from_id - -```python -from_id(volume_id, client=None) -``` -Construct a Volume from an id and look up the Volume metadata. - -This is a lazy method that defers hydrating the local -object with metadata from Modal servers until the first -time it is actually used. - -The ID of a Volume object can be accessed using `.object_id`. - -**Parameters** - - - - -**Returns** - -A `Volume` handle (possibly not yet hydrated). - -**Usage** - -```python notest -@app.function() -def my_worker(volume_id: str): - vol = modal.Volume.from_id(volume_id) - for entry in vol.listdir("/"): - print(entry.path) - -with modal.Volume.ephemeral() as vol: - my_worker.remote(vol.object_id) -``` - -## ephemeral - -```python -ephemeral(cls, client=None, environment_name=None, version=None) -``` -Create an anonymous Volume that exists for the duration of the context manager. - -**Parameters** - - - - - -**Usage** - -```python -import modal - -with modal.Volume.ephemeral() as vol: - assert vol.listdir("/") == [] -``` - -```python notest -async with modal.Volume.ephemeral() as vol: - assert await vol.listdir("/") == [] -``` - -## info - -```python -info(self) -``` -Return information about the Volume object. - -## commit - -```python -commit(self) -``` -Commit changes to a mounted volume. - -If successful, the changes made are now persisted in durable storage and available to other containers accessing -the volume. - -## reload - -```python -reload(self) -``` -Make latest committed state of volume available in the running container. - -Any uncommitted changes to the volume, such as new or modified files, may implicitly be committed when -reloading. - -Reloading will fail if there are open files for the volume. - -## iterdir - -```python -iterdir(self, path, *, recursive=True) -``` -Iterate over all files in a directory in the volume. - -Passing a directory path lists all files in the directory. For a file path, return only that -file's description. If `recursive` is set to True, list all files and folders under the path -recursively. - -## listdir - -```python -listdir(self, path, *, recursive=False) -``` -List all files under a path prefix in the modal.Volume. - -Passing a directory path lists all files in the directory. For a file path, return only that -file's description. If `recursive` is set to True, list all files and folders under the path -recursively. - -## read_file - -```python -read_file(self, path) -``` -Read a file from the modal.Volume. - -Note - this function is primarily intended to be used outside of a Modal App. -For more information on downloading files from a Modal Volume, see -[the guide](https://modal.com/docs/guide/volumes). - -**Parameters** - - - -**Usage** - -```python notest -vol = modal.Volume.from_name("my-modal-volume") -data = b"" -for chunk in vol.read_file("1mb.csv"): - data += chunk -print(len(data)) # == 1024 * 1024 -``` - -## remove_file - -```python -remove_file(self, path, recursive=False) -``` -Remove a file or directory from a volume. - -## copy_files - -```python -copy_files(self, src_paths, dst_path, recursive=False) -``` -Copy files within the volume from src_paths to dst_path. -The semantics of the copy operation follow those of the UNIX cp command. - -The `src_paths` parameter is a list. If you want to copy a single file, you should pass a list with a -single element. - -`src_paths` and `dst_path` should refer to the desired location *inside* the volume. You do not need to prepend -the volume mount path. - -Note that if the volume is already mounted on the Modal function, you should use normal filesystem operations -like `os.rename()` and then `commit()` the volume. The `copy_files()` method is useful when you don't have -the volume mounted as a filesystem, e.g. when running a script on your local computer. - -**Parameters** - - - - - -**Usage** - -```python notest -vol = modal.Volume.from_name("my-modal-volume") - -vol.copy_files(["bar/example.txt"], "bar2") -vol.copy_files(["bar/example.txt"], "bar/example2.txt") -``` - -## batch_upload - -```python -batch_upload(self, force=False) -``` -Initiate a batched upload to a volume. - -To allow overwriting existing files, set `force` to `True` (you cannot overwrite existing directories with -uploaded files regardless). - -**Parameters** - - - -**Usage** - -```python notest -vol = modal.Volume.from_name("my-modal-volume") - -with vol.batch_upload() as batch: - batch.put_file("local-path.txt", "/remote-path.txt") - batch.put_directory("/local/directory/", "/remote/directory") - batch.put_file(io.BytesIO(b"some data"), "/foobar") -``` - -## rename - -```python -rename(old_name, new_name, *, client=None, environment_name=None) -``` - diff --git a/.agents/skills/modal/references/api/modal.Workspace.md b/.agents/skills/modal/references/api/modal.Workspace.md deleted file mode 100644 index 65a4d776babf7784f76d07ec0c97fc3ef038b1f1..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.Workspace.md +++ /dev/null @@ -1,63 +0,0 @@ -# modal.Workspace - - -```python -class Workspace(modal.object.Object) -``` - - -## hydrate - -```python -hydrate(self, client=None) -``` -Synchronize the local object with its identity on the Modal server. - -It is rarely necessary to call this method explicitly, as most operations -will lazily hydrate when needed. The main use case is when you need to -access object metadata, such as its ID. - -*Added in v0.72.39*: This method replaces the deprecated `.resolve()` method. - -## name - -```python -name(self) -``` - - -## members - - -```python -members: WorkspaceMembersManager -``` - -Namespace with methods for managing the membership of a Workspace. - - -### members.list - -```python -list(self) -``` -Return the members of the Workspace. - -**Examples:** - -```python notest -members = modal.Workspace.from_context().members.list() -print([m.name for m in members]) -``` - -## from_context - -```python -from_context(*, client=None) -``` -Look up the Workspace associated with the current context. - -This returns the Workspace that the active Modal credentials authenticate against -(i.e., your active profile or the `MODAL_TOKEN_ID` / `MODAL_TOKEN_SECRET` environment -variables). If called inside a Modal container, it returns the Workspace that the -container is running in. diff --git a/.agents/skills/modal/references/api/modal.asgi_app.md b/.agents/skills/modal/references/api/modal.asgi_app.md deleted file mode 100644 index 13d312fa798e3d2e29408384f431175683144c37..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.asgi_app.md +++ /dev/null @@ -1,23 +0,0 @@ -# modal.asgi_app - -```python -asgi_app(*, label=None, custom_domains=None, requires_proxy_auth=False) -``` -Decorator for registering an ASGI app as a Web Function. - -Asynchronous Server Gateway Interface (ASGI) is a standard for Python -web apps, supported by all popular Python web libraries. - -To learn how to use Modal with popular web frameworks, see the -[guide on Web Functions](https://modal.com/docs/guide/webhooks). - -**Usage** - -```python -from typing import Callable - -@app.function() -@modal.asgi_app() -def create_asgi() -> Callable: - ... -``` diff --git a/.agents/skills/modal/references/api/modal.batched.md b/.agents/skills/modal/references/api/modal.batched.md deleted file mode 100644 index e7d7ce5f798689a232c66722c6b9597c9f07b8e9..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.batched.md +++ /dev/null @@ -1,28 +0,0 @@ -# modal.batched - -```python -batched(*, max_batch_size, wait_ms) -``` -Decorator for functions or class methods that should be batched. - -See the [dynamic batching guide](https://modal.com/docs/guide/dynamic-batching) for more information. - -**Usage** - -```python -# Stack the decorator under `@app.function()` to enable dynamic batching -@app.function() -@modal.batched(max_batch_size=4, wait_ms=1000) -async def batched_multiply(xs: list[int], ys: list[int]) -> list[int]: - return [x * y for x, y in zip(xs, ys)] - -# call batched_multiply with individual inputs -# batched_multiply.remote.aio(2, 100) - -# With `@app.cls()`, apply the decorator to a method (this may change in the future) -@app.cls() -class BatchedClass: - @modal.batched(max_batch_size=4, wait_ms=1000) - def batched_multiply(self, xs: list[int], ys: list[int]) -> list[int]: - return [x * y for x, y in zip(xs, ys)] -``` diff --git a/.agents/skills/modal/references/api/modal.billing.md b/.agents/skills/modal/references/api/modal.billing.md deleted file mode 100644 index 71a2d1ed4624d24c4a90cadda15db1014ab992d9..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.billing.md +++ /dev/null @@ -1,46 +0,0 @@ -# modal.billing - -## modal.billing.WorkspaceBillingReportItem - -```python -class WorkspaceBillingReportItem(TypedDict): - object_id: str - description: str - environment_name: str - interval_start: datetime.datetime - cost: decimal.Decimal - tags: dict[str, str] -``` - -## modal.billing.workspace_billing_report - -```python -workspace_billing_report(*, start, end=None, resolution="d", tag_names=None, - client=None) -``` -Generate a tabular report of workspace usage by object and time. - -The result will be a list of dictionaries for each interval (determined by `resolution`) -between the `start` and `end` limits. The dictionary represents a single Modal object -that billing can be attributed to (e.g., an App) along with metadata (including user-defined -tags) for identifying that object. - -The `start` and `end` parameters are required to either have a UTC timezone or to be -timezone-naive (which will be interpreted as UTC times). The timestamps in the result will -be in UTC. Cost will be reported for full intervals, even if the provided `start` or `end` -parameters are partial: `start` will be rounded to the beginning of its interval, while -partial `end` intervals will be excluded. - -Additional user-provided metadata can be included in the report if the objects have tags -and `tag_names` (i.e., keys) are specified in the request. Alternatively, pass `tag_names=["*"]` -to include all tags in the report. Note that tags will be attributed to the entire interval even -if they were added or removed at some point within it. If the tag name was not in use during an -interval, it will be absent from the tags dictionary in that output row. - -In most cases, billing data will be available in the database that this API queries within -minutes, although there may be collection delays. If completeness is important for your use -case, we recommend leaving a buffer after the end of the query interval. - -It's also possible to generate reports using the -[`modal billing report`](https://modal.com/docs/reference/cli/billing) CLI command. The CLI -has a few convenience features for generating reports across relative time ranges. diff --git a/.agents/skills/modal/references/api/modal.call_graph.md b/.agents/skills/modal/references/api/modal.call_graph.md deleted file mode 100644 index 393d2b1b96285a4128d5bbc568ada32cca718874..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.call_graph.md +++ /dev/null @@ -1,33 +0,0 @@ -# modal.call_graph - -## modal.call_graph.InputInfo - - -```python -class InputInfo(object) -``` - -Simple data structure storing information about a function input. - -```python -__init__(self, input_id, function_call_id, task_id, status, function_name, - module_name, children) -``` - -## modal.call_graph.InputStatus - - -```python -class InputStatus(enum.IntEnum) -``` - -Enum representing status of a function input. - -The possible values are: - -* `PENDING` -* `SUCCESS` -* `FAILURE` -* `INIT_FAILURE` -* `TERMINATED` -* `TIMEOUT` diff --git a/.agents/skills/modal/references/api/modal.concurrent.md b/.agents/skills/modal/references/api/modal.concurrent.md deleted file mode 100644 index f422eacbff9c3e1a87f90c502e03a9067351d50c..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.concurrent.md +++ /dev/null @@ -1,45 +0,0 @@ -# modal.concurrent - -```python -concurrent(*, max_inputs=None, target_inputs=None) -``` -Decorator that allows individual containers to handle multiple inputs concurrently. - -The concurrency mechanism depends on whether the function is async or not: -- Async functions will run inputs on a single thread as asyncio tasks. -- Synchronous functions will use multi-threading. The code must be thread-safe. - -Input concurrency will be most useful for workflows that are IO-bound -(e.g., making network requests) or when running an inference server that supports -dynamic batching. - -When `target_inputs` is set, Modal's autoscaler will try to provision resources -such that each container is running that many inputs concurrently, rather than -autoscaling based on `max_inputs`. Containers may burst up to up to `max_inputs` -if resources are insufficient to remain at the target concurrency, e.g. when the -arrival rate of inputs increases. This can trade-off a small increase in average -latency to avoid larger tail latencies from input queuing. - -*Added in v0.73.148:* This decorator replaces the `allow_concurrent_inputs` parameter -in `@app.function()` and `@app.cls()`. - -**Usage** - -```python -# Stack the decorator under `@app.function()` to enable input concurrency -@app.function() -@modal.concurrent(max_inputs=100) -async def f(data): - # Async function; will be scheduled as asyncio task - ... - -# With `@app.cls()`, apply the decorator at the class level, not on individual methods -@app.cls() -@modal.concurrent(max_inputs=100, target_inputs=80) -class C: - @modal.method() - def f(self, data): - # Sync function; must be thread-safe - ... - -``` diff --git a/.agents/skills/modal/references/api/modal.config.md b/.agents/skills/modal/references/api/modal.config.md deleted file mode 100644 index 8a24e8404c66a64f679d9bf5198fcb45e8ec5d42..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.config.md +++ /dev/null @@ -1,163 +0,0 @@ -# modal.config - -Modal intentionally keeps configurability to a minimum. - -The main configuration options are the API tokens: the token id and the token secret. -These can be configured in two ways: - -1. By running the `modal token set` command. - This writes the tokens to `.modal.toml` file in your home directory. -2. By setting the environment variables `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`. - This takes precedence over the previous method. - -.modal.toml ---------------- - -The `.modal.toml` file is generally stored in your home directory. -It should look like this:: - -```toml -[default] -token_id = "ak-12345..." -token_secret = "as-12345..." -``` - -You can create this file manually, or you can run the `modal token set ...` -command (see below). - -Setting tokens using the CLI ----------------------------- - -You can set a token by running the command:: - -``` -modal token set \ - --token-id \ - --token-secret -``` - -This will write the token id and secret to `.modal.toml`. - -If the token id or secret is provided as the string `-` (a single dash), -then it will be read in a secret way from stdin instead. - -Other configuration options ---------------------------- - -Other possible configuration options are: - -* `loglevel` (in the .toml file) / `MODAL_LOGLEVEL` (as an env var). - Defaults to `WARNING`. Set this to `DEBUG` to see internal messages. -* `logs_timeout` (in the .toml file) / `MODAL_LOGS_TIMEOUT` (as an env var). - Defaults to 10. - Number of seconds to wait for logs to drain when closing the session, - before giving up. -* `max_throttle_wait` (in the .toml file) / `MODAL_MAX_THROTTLE_WAIT` (as an env var). - Defaults to None (no limit). - Maximum number of seconds to wait when requests are being throttled (i.e., due - to rate limiting or other cases that can normally be resolved through backoff). -* `force_build` (in the .toml file) / `MODAL_FORCE_BUILD` (as an env var). - Defaults to False. - When set, ignores the Image cache and builds all Image layers. Note that this - will break the cache for all images based on the rebuilt layers, so other images - may rebuild on subsequent runs / deploys even if the config is reverted. -* `ignore_cache` (in the .toml file) / `MODAL_IGNORE_CACHE` (as an env var). - Defaults to False. - When set, ignores the Image cache and builds all Image layers. Unlike `force_build`, - this will not overwrite the cache for other images that have the same recipe. - Subsequent runs that do not use this option will pull the *previous* Image from - the cache, if one exists. It can be useful for testing an App's robustness to - Image rebuilds without clobbering Images used by other Apps. -* `traceback` (in the .toml file) / `MODAL_TRACEBACK` (as an env var). - Defaults to False. Enables printing full tracebacks on unexpected CLI - errors, which can be useful for debugging client issues. -* `log_pattern` (in the .toml file) / `MODAL_LOG_PATTERN` (as an env var). - Defaults to `"[modal-client] %(asctime)s %(message)s"` - The log formatting pattern that will be used by the modal client itself. - See https://docs.python.org/3/library/logging.html#logrecord-attributes for available - log attributes. -* `dev_suffix` (in the .toml file) / `MODAL_DEV_SUFFIX` (as an env var). - Overrides the default `-dev` suffix added to URLs generated for Web Functions - when the App is ephemeral (i.e., created via `modal serve`). Must be a short - alphanumeric string. - -Meta-configuration ------------------- - -Some "meta-options" are set using environment variables only: - -* `MODAL_CONFIG_PATH` lets you override the location of the .toml file, - by default `~/.modal.toml`. -* `MODAL_PROFILE` lets you use multiple sections in the .toml file - and switch between them. It defaults to "default". - -## modal.config.Config - - -```python -class Config(object) -``` - -Singleton that holds configuration used by Modal internally. - -```python -__init__(self) -``` - - -### get - -```python -get(self, key, profile=None, use_env=True) -``` -Look up a configuration value. - -Resolution order (highest priority first): - -1. Environment variable ``MODAL_`` (underscore-separated, uppercased), when ``use_env`` is True. -2. The named profile in ``.modal.toml``. -3. The built-in default for that setting. - -**Parameters** - - - - - -**Returns** - -The transformed configuration value (type depends on the setting). - -### override_locally - -```python -override_locally(self, key, value) -``` - - -### to_dict - -```python -to_dict(self) -``` - -## modal.config.config_profiles - -```python -config_profiles() -``` -List the available Modal profiles in the ``.modal.toml`` file. - -**Returns** - -Profile section names present in the configuration file. -## modal.config.config_set_active_profile - -```python -config_set_active_profile(profile) -``` -Set the user's active Modal profile by writing it to the ``.modal.toml`` file. - -**Parameters** - - diff --git a/.agents/skills/modal/references/api/modal.container_process.md b/.agents/skills/modal/references/api/modal.container_process.md deleted file mode 100644 index 22498a139732398263656f4e7045bdf4d442efb0..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.container_process.md +++ /dev/null @@ -1,64 +0,0 @@ -# modal.container_process - -## modal.container_process.ContainerProcess - - -```python -class ContainerProcess(typing.Generic) -``` - -Represents a running process in a container. - -Container processes communicate via direct communication with -the Modal worker where the container is running. - -```python -__init__(self, process_id, task_id, client, command_router_client, - stdout=StreamType.PIPE, stderr=StreamType.PIPE, exec_deadline=None, - text=True, by_line=False) -``` - - -### stdout - -```python -stdout(self) -``` -StreamReader for the container process's stdout stream. - -### stderr - -```python -stderr(self) -``` -StreamReader for the container process's stderr stream. - -### stdin - -```python -stdin(self) -``` -StreamWriter for the container process's stdin stream. - -### returncode - -```python -returncode(self) -``` - - -### poll - -```python -poll(self) -``` -Check if the container process has finished running. - -Returns `None` if the process is still running, else returns the exit code. - -### wait - -```python -wait(self) -``` -Wait for the container process to finish running. Returns the exit code. diff --git a/.agents/skills/modal/references/api/modal.current_function_call_id.md b/.agents/skills/modal/references/api/modal.current_function_call_id.md deleted file mode 100644 index 15b2ee6f6202117ab6eff2a82b4a1c1e9c9915f6..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.current_function_call_id.md +++ /dev/null @@ -1,16 +0,0 @@ -# modal.current_function_call_id - -```python -current_function_call_id() -``` -Returns the function call ID for the current input. - -Can only be called from Modal function (i.e. in a container context). - -```python -from modal import current_function_call_id - -@app.function() -def process_stuff(): - print(f"Starting to process input from {current_function_call_id()}") -``` diff --git a/.agents/skills/modal/references/api/modal.current_input_id.md b/.agents/skills/modal/references/api/modal.current_input_id.md deleted file mode 100644 index f939a0b93ee622e70f1882e2c0f9587892ff78a9..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.current_input_id.md +++ /dev/null @@ -1,16 +0,0 @@ -# modal.current_input_id - -```python -current_input_id() -``` -Returns the input ID for the current input. - -Can only be called from Modal function (i.e. in a container context). - -```python -from modal import current_input_id - -@app.function() -def process_stuff(): - print(f"Starting to process {current_input_id()}") -``` diff --git a/.agents/skills/modal/references/api/modal.enable_output.md b/.agents/skills/modal/references/api/modal.enable_output.md deleted file mode 100644 index 389eabca1d65f4760816781ee08349656a75a054..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.enable_output.md +++ /dev/null @@ -1,20 +0,0 @@ -# modal.enable_output - -```python -enable_output() -``` -Context manager that enable output when using the Python SDK. - -This will print to stdout and stderr things such as -1. Logs from running functions -2. Status of creating objects -3. Map progress - -**Usage** - -```python -app = modal.App() -with modal.enable_output(): - with app.run(): - ... -``` diff --git a/.agents/skills/modal/references/api/modal.enter.md b/.agents/skills/modal/references/api/modal.enter.md deleted file mode 100644 index f8dd3f491591dbd8d9188299a5e38f3054bfd541..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.enter.md +++ /dev/null @@ -1,8 +0,0 @@ -# modal.enter - -```python -enter(*, snap=False) -``` -Decorator for methods which should be executed when a new container is started. - -See the [lifeycle function guide](https://modal.com/docs/guide/lifecycle-functions#enter) for more information. diff --git a/.agents/skills/modal/references/api/modal.exception.md b/.agents/skills/modal/references/api/modal.exception.md deleted file mode 100644 index b3f1c5413a3af24a3e44796bf198ac6bd63c45e5..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.exception.md +++ /dev/null @@ -1,766 +0,0 @@ -# modal.exception - -Modal-specific exception types. - -## Notes on `grpclib.GRPCError` migration - -Historically, the Modal SDK could propagate `grpclib.GRPCError` exceptions out -to user code. As of v1.3, we are in the process of gracefully migrating to -always raising a Modal exception type in these cases. To avoid breaking user -code that relies on catching `grpclib.GRPCError`, a subset of Modal exception -types temporarily inherit from `grpclib.GRPCError`. - -We encourage users to migrate any code that currently catches `grpclib.GRPCError` -to instead catch the appropriate Modal exception type. The following mapping -between GRPCError status codes and Modal exception types is currently in use: - -``` -CANCELLED -> ServiceError -UNKNOWN -> ServiceError -INVALID_ARGUMENT -> InvalidError -DEADLINE_EXCEEDED -> ServiceError -NOT_FOUND -> NotFoundError -ALREADY_EXISTS -> AlreadyExistsError -PERMISSION_DENIED -> PermissionDeniedError -RESOURCE_EXHAUSTED -> ResourceExhaustedError -FAILED_PRECONDITION -> ConflictError -ABORTED -> ConflictError -OUT_OF_RANGE -> InvalidError -UNIMPLEMENTED -> UnimplementedError -INTERNAL -> InternalError -UNAVAILABLE -> ServiceError -DATA_LOSS -> DataLossError -UNAUTHENTICATED -> AuthError -``` - -## modal.exception.AlreadyExistsError - - -```python -class AlreadyExistsError(modal.exception.Error, modal.exception._GRPCErrorWrapper) -``` - -Raised when a resource creation conflicts with an existing resource. - -```python -__init__(self, message=None) -``` - - -### message - -```python -message(self) -``` - - -### status - -```python -status(self) -``` - - -### details - -```python -details(self) -``` - -## modal.exception.AsyncUsageWarning - - -```python -class AsyncUsageWarning(UserWarning) -``` - -Warning emitted when a blocking Modal interface is used in an async context. - -## modal.exception.AuthError - - -```python -class AuthError(modal.exception.Error, modal.exception._GRPCErrorWrapper) -``` - -Raised when a client has missing or invalid authentication. - -```python -__init__(self, message=None) -``` - - -### message - -```python -message(self) -``` - - -### status - -```python -status(self) -``` - - -### details - -```python -details(self) -``` - -## modal.exception.ClientClosed - - -```python -class ClientClosed(modal.exception.Error) -``` - -## modal.exception.ConflictError - - -```python -class ConflictError(modal.exception.InvalidError, modal.exception._GRPCErrorWrapper) -``` - -Raised when a resource conflict occurs between the request and current system state. - -```python -__init__(self, message=None) -``` - - -### message - -```python -message(self) -``` - - -### status - -```python -status(self) -``` - - -### details - -```python -details(self) -``` - -## modal.exception.ConnectionError - - -```python -class ConnectionError(modal.exception.Error) -``` - -Raised when an issue occurs while connecting to the Modal servers. - -## modal.exception.DataLossError - - -```python -class DataLossError(modal.exception.Error, modal.exception._GRPCErrorWrapper) -``` - -Raised when data is lost or corrupted. - -```python -__init__(self, message=None) -``` - - -### message - -```python -message(self) -``` - - -### status - -```python -status(self) -``` - - -### details - -```python -details(self) -``` - -## modal.exception.DeprecationError - - -```python -class DeprecationError(UserWarning) -``` - -UserWarning category emitted when a deprecated Modal feature or API is used. - -## modal.exception.DeserializationError - - -```python -class DeserializationError(modal.exception.Error) -``` - -Raised to provide more context when an error is encountered during deserialization. - -## modal.exception.Error - - -```python -class Error(Exception) -``` - -Base class for all Modal errors. See [`modal.exception`](https://modal.com/docs/reference/modal.exception) -for the specialized error classes. - -**Usage** - -```python notest -import modal - -try: - ... -except modal.Error: - # Catch any exception raised by Modal's systems. - print("Responding to error...") -``` - -## modal.exception.ExecTimeoutError - - -```python -class ExecTimeoutError(modal.exception.TimeoutError) -``` - -Raised when a container process exceeds its execution duration limit and times out. - -## modal.exception.ExecutionError - - -```python -class ExecutionError(modal.exception.Error) -``` - -Raised when something unexpected happened during runtime. - -## modal.exception.FilesystemExecutionError - - -```python -class FilesystemExecutionError(modal.exception.Error) -``` - -Raised when an unknown error is thrown during a container filesystem operation. - -## modal.exception.FunctionTimeoutError - - -```python -class FunctionTimeoutError(modal.exception.TimeoutError) -``` - -Raised when a Function exceeds its execution duration limit and times out. - -## modal.exception.InputCancellation - - -```python -class InputCancellation(BaseException) -``` - -Raised when the current input is cancelled by the task - -Intentionally a BaseException instead of an Exception, so it won't get -caught by unspecified user exception clauses that might be used for retries and -other control flow. - -## modal.exception.InteractiveTimeoutError - - -```python -class InteractiveTimeoutError(modal.exception.TimeoutError) -``` - -Raised when interactive frontends time out while trying to connect to a container. - -## modal.exception.InternalError - - -```python -class InternalError(modal.exception.Error, modal.exception._GRPCErrorWrapper) -``` - -Raised when an internal error occurs in the Modal system. - -```python -__init__(self, message=None) -``` - - -### message - -```python -message(self) -``` - - -### status - -```python -status(self) -``` - - -### details - -```python -details(self) -``` - -## modal.exception.InternalFailure - - -```python -class InternalFailure(modal.exception.Error) -``` - -Retriable internal error. - -## modal.exception.InvalidError - - -```python -class InvalidError(modal.exception.Error, modal.exception._GRPCErrorWrapper) -``` - -Raised when user does something invalid. - -```python -__init__(self, message=None) -``` - - -### message - -```python -message(self) -``` - - -### status - -```python -status(self) -``` - - -### details - -```python -details(self) -``` - -## modal.exception.LogsFetchError - - -```python -class LogsFetchError(modal.exception.Error) -``` - -Raised when trying to fetch too many logs. - -## modal.exception.ModuleNotMountable - - -```python -class ModuleNotMountable(Exception) -``` - -## modal.exception.MountUploadTimeoutError - - -```python -class MountUploadTimeoutError(modal.exception.TimeoutError) -``` - -Raised when a Mount upload times out. - -## modal.exception.NotFoundError - - -```python -class NotFoundError(modal.exception.Error, modal.exception._GRPCErrorWrapper) -``` - -Raised when a requested resource was not found. - -```python -__init__(self, message=None) -``` - - -### message - -```python -message(self) -``` - - -### status - -```python -status(self) -``` - - -### details - -```python -details(self) -``` - -## modal.exception.OutputExpiredError - - -```python -class OutputExpiredError(modal.exception.TimeoutError) -``` - -Raised when the Output exceeds expiration and times out. - -## modal.exception.PermissionDeniedError - - -```python -class PermissionDeniedError(modal.exception.Error, modal.exception._GRPCErrorWrapper) -``` - -Raised when a user does not have permission to perform the requested operation. - -```python -__init__(self, message=None) -``` - - -### message - -```python -message(self) -``` - - -### status - -```python -status(self) -``` - - -### details - -```python -details(self) -``` - -## modal.exception.RemoteError - - -```python -class RemoteError(modal.exception.Error) -``` - -Raised when an error occurs on the Modal server. - -## modal.exception.RequestSizeError - - -```python -class RequestSizeError(modal.exception.Error) -``` - -Raised when an operation produces a gRPC request that is rejected by the server for being too large. - -## modal.exception.ResourceExhaustedError - - -```python -class ResourceExhaustedError(modal.exception.Error, modal.exception._GRPCErrorWrapper) -``` - -Raised when a server-side resource has been exhausted, e.g. a quota or rate limit. - -```python -__init__(self, message=None) -``` - - -### message - -```python -message(self) -``` - - -### status - -```python -status(self) -``` - - -### details - -```python -details(self) -``` - -## modal.exception.SandboxFilesystemDirectoryNotEmptyError - - -```python -class SandboxFilesystemDirectoryNotEmptyError(modal.exception.SandboxFilesystemError) -``` - -Raised when a directory is not empty. - -## modal.exception.SandboxFilesystemError - - -```python -class SandboxFilesystemError(modal.exception.Error) -``` - -Base class for sandbox filesystem errors. - -## modal.exception.SandboxFilesystemFileTooLargeError - - -```python -class SandboxFilesystemFileTooLargeError(modal.exception.SandboxFilesystemError) -``` - -Raised when a file exceeds the maximum allowed size for a read operation in the sandbox. - -## modal.exception.SandboxFilesystemIsADirectoryError - - -```python -class SandboxFilesystemIsADirectoryError(modal.exception.SandboxFilesystemError) -``` - -Raised when a file operation in the sandbox targets a directory when it should target a non-directory file. - -## modal.exception.SandboxFilesystemNotADirectoryError - - -```python -class SandboxFilesystemNotADirectoryError(modal.exception.SandboxFilesystemError) -``` - -Raised when a path component in the sandbox is not a directory. - -## modal.exception.SandboxFilesystemNotFoundError - - -```python -class SandboxFilesystemNotFoundError(modal.exception.SandboxFilesystemError) -``` - -Raised when a file or directory is not found in the sandbox. - -## modal.exception.SandboxFilesystemPathAlreadyExistsError - - -```python -class SandboxFilesystemPathAlreadyExistsError(modal.exception.SandboxFilesystemError) -``` - -Raised when a path already exists and the operation requires it to be absent. - -## modal.exception.SandboxFilesystemPermissionError - - -```python -class SandboxFilesystemPermissionError(modal.exception.SandboxFilesystemError) -``` - -Raised when permission is denied for a file operation in the sandbox. - -## modal.exception.SandboxTerminatedError - - -```python -class SandboxTerminatedError(modal.exception.Error) -``` - -Raised when a Sandbox is terminated for an internal reason. - -## modal.exception.SandboxTimeoutError - - -```python -class SandboxTimeoutError(modal.exception.TimeoutError) -``` - -Raised when a Sandbox exceeds its execution duration limit and times out. - -## modal.exception.SerializationError - - -```python -class SerializationError(modal.exception.Error) -``` - -Raised to provide more context when an error is encountered during serialization. - -## modal.exception.ServerWarning - - -```python -class ServerWarning(UserWarning) -``` - -Warning originating from the Modal server and re-issued in client code. - -## modal.exception.ServiceError - - -```python -class ServiceError(modal.exception.Error, modal.exception._GRPCErrorWrapper) -``` - -Raised when an error occurs in basic client/server communication. - -```python -__init__(self, message=None) -``` - - -### message - -```python -message(self) -``` - - -### status - -```python -status(self) -``` - - -### details - -```python -details(self) -``` - -## modal.exception.TimeoutError - - -```python -class TimeoutError(modal.exception.Error) -``` - -Base class for Modal timeouts. - -## modal.exception.UnimplementedError - - -```python -class UnimplementedError(modal.exception.Error, modal.exception._GRPCErrorWrapper) -``` - -Raised when a requested operation is not implemented or not supported. - -```python -__init__(self, message=None) -``` - - -### message - -```python -message(self) -``` - - -### status - -```python -status(self) -``` - - -### details - -```python -details(self) -``` - -## modal.exception.VersionError - - -```python -class VersionError(modal.exception.Error) -``` - -Raised when the current client version of Modal is unsupported. - -## modal.exception.VolumeUploadTimeoutError - - -```python -class VolumeUploadTimeoutError(modal.exception.TimeoutError) -``` - -Raised when a Volume upload times out. - -## modal.exception.WorkspaceManagementError - - -```python -class WorkspaceManagementError(modal.exception.Error) -``` - -Raised when an error occurs while managing a workspace. - -## modal.exception.simulate_preemption - -```python -simulate_preemption(wait_seconds, jitter_seconds=0) -``` -Utility for simulating a preemption interrupt after `wait_seconds` seconds. -The first interrupt is the SIGINT signal. After 30 seconds, a second -interrupt will trigger. - -This second interrupt simulates SIGKILL, and should not be caught. -Optionally add between zero and `jitter_seconds` seconds of additional waiting before first interrupt. - -**Usage** - -```python notest -import time -from modal.exception import simulate_preemption - -simulate_preemption(3) - -try: - time.sleep(4) -except KeyboardInterrupt: - print("got preempted") # Handle interrupt - raise -``` - -See https://modal.com/docs/guide/preemption for more details on preemption -handling. diff --git a/.agents/skills/modal/references/api/modal.exit.md b/.agents/skills/modal/references/api/modal.exit.md deleted file mode 100644 index 44b4fd93cefc70a1fb3069ba9c07e953cb0c4feb..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.exit.md +++ /dev/null @@ -1,8 +0,0 @@ -# modal.exit - -```python -exit(_warn_parentheses_missing=None) -``` -Decorator for methods which should be executed when a container is about to exit. - -See the [lifeycle function guide](https://modal.com/docs/guide/lifecycle-functions#exit) for more information. diff --git a/.agents/skills/modal/references/api/modal.fastapi_endpoint.md b/.agents/skills/modal/references/api/modal.fastapi_endpoint.md deleted file mode 100644 index 162d4ceccda7c8fa3b4ac422e5d5477f9f927322..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.fastapi_endpoint.md +++ /dev/null @@ -1,20 +0,0 @@ -# modal.fastapi_endpoint - -```python -fastapi_endpoint(*, method="GET", label=None, custom_domains=None, docs=False, - requires_proxy_auth=False) -``` -Create a Web Function that can be addressed via HTTP at a public URL. - -Modal will internally use [FastAPI](https://fastapi.tiangolo.com/) to expose a -simple, single request handler. If you are defining your own `FastAPI` application -(e.g. if you want to define multiple routes), use `@modal.asgi_app` instead. - -The Web Function created with this decorator will automatically have -[CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) enabled -and can leverage many of FastAPI's features. - -For more information on using Modal with popular web frameworks, see our -[guide on Web Functions](https://modal.com/docs/guide/webhooks). - -*Added in v0.73.82*: This function replaces the deprecated `@web_endpoint` decorator. diff --git a/.agents/skills/modal/references/api/modal.file_io.md b/.agents/skills/modal/references/api/modal.file_io.md deleted file mode 100644 index 805384a96bd9fe29c057c7f7beaf3d6aed271c20..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.file_io.md +++ /dev/null @@ -1,152 +0,0 @@ -# modal.file_io - -## modal.file_io.FileIO - - -```python -class FileIO(typing.Generic) -``` - -[Alpha] FileIO handle, used in the Sandbox filesystem API. - -Deprecated on 2026-03-09. Use the `Sandbox.filesystem` APIs instead. - -The API is designed to mimic Python's io.FileIO. - -Currently this API is in Alpha and is subject to change. File I/O operations -may be limited in size to 100 MiB, and the throughput of requests is -restricted in the current implementation. For our recommendations on large file transfers -see the Sandbox [filesystem access guide](https://modal.com/docs/guide/sandbox-files). - -**Usage** - -```python notest -import modal - -app = modal.App.lookup("my-app", create_if_missing=True) - -sb = modal.Sandbox.create(app=app) -f = sb.open("/tmp/foo.txt", "w") -f.write("hello") -f.close() -``` - -```python -__init__(self, client, task_id) -``` - - -### create - -```python -create(cls, path, mode, client, task_id) -``` -Create a new FileIO handle. - -### read - -```python -read(self, n=None) -``` -Read n bytes from the current position, or the entire remaining file if n is None. - -### readline - -```python -readline(self) -``` -Read a single line from the current position. - -### readlines - -```python -readlines(self) -``` -Read all lines from the current position. - -### write - -```python -write(self, data) -``` -Write data to the current position. - -Writes may not appear until the entire buffer is flushed, which -can be done manually with `flush()` or automatically when the file is -closed. - -### flush - -```python -flush(self) -``` -Flush the buffer to disk. - -### seek - -```python -seek(self, offset, whence=0) -``` -Move to a new position in the file. - -`whence` defaults to 0 (absolute file positioning); other values are 1 -(relative to the current position) and 2 (relative to the file's end). - -### ls - -```python -ls(cls, path, client, task_id) -``` -List the contents of the provided directory. - -### mkdir - -```python -mkdir(cls, path, client, task_id, parents=False) -``` -Create a new directory. - -### rm - -```python -rm(cls, path, client, task_id, recursive=False) -``` -Remove a file or directory in the Sandbox. - -### watch - -```python -watch(cls, path, client, task_id, filter=None, recursive=False, timeout=None) -``` - - -### close - -```python -close(self) -``` -Flush the buffer and close the file. -## modal.file_io.ls - -```python -ls(path, client, task_id) -``` -List the contents of the provided directory. -## modal.file_io.mkdir - -```python -mkdir(path, client, task_id, parents=False) -``` -Create a new directory. -## modal.file_io.rm - -```python -rm(path, client, task_id, recursive=False) -``` -Remove a file or directory in the Sandbox. -## modal.file_io.watch - -```python -watch(path, client, task_id, filter=None, recursive=False, timeout=None) -``` -Watch a file or directory for changes. diff --git a/.agents/skills/modal/references/api/modal.forward.md b/.agents/skills/modal/references/api/modal.forward.md deleted file mode 100644 index 664875c386ce0a66de8402ab88abd369ee89f30a..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.forward.md +++ /dev/null @@ -1,117 +0,0 @@ -# modal.forward - -```python -forward(port, *, unencrypted=False, h2_enabled=False, client=None) -``` -Expose a port publicly from inside a running Modal container, with TLS. - -If `unencrypted` is set, this also exposes the TCP socket without encryption on a random port -number. This can be used to SSH into a container (see example below). Note that it is on the public Internet, so -make sure you are using a secure protocol over TCP. - -If `h2_enabled` is set, the TLS server will advertise support for HTTP/2. - -**Important:** This is an experimental API which may change in the future. - -**Usage** - -```python notest -import modal -from flask import Flask - -app = modal.App(image=modal.Image.debian_slim().pip_install("Flask")) -flask_app = Flask(__name__) - - -@flask_app.route("/") -def hello_world(): - return "Hello, World!" - - -@app.function() -def run_app(): - # Start a web server inside the container at port 8000. `modal.forward(8000)` lets us - # expose that port to the world at a random HTTPS URL. - with modal.forward(8000) as tunnel: - print("Server listening at", tunnel.url) - flask_app.run("0.0.0.0", 8000) - - # When the context manager exits, the port is no longer exposed. -``` - -**Raw TCP usage:** - -```python -import socket -import threading - -import modal - - -def run_echo_server(port: int): - """Run a TCP echo server listening on the given port.""" - sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - sock.bind(("0.0.0.0", port)) - sock.listen(1) - - while True: - conn, addr = sock.accept() - print("Connection from:", addr) - - # Start a new thread to handle the connection - def handle(conn): - with conn: - while True: - data = conn.recv(1024) - if not data: - break - conn.sendall(data) - - threading.Thread(target=handle, args=(conn,)).start() - - -app = modal.App() - - -@app.function() -def tcp_tunnel(): - # This exposes port 8000 to public Internet traffic over TCP. - with modal.forward(8000, unencrypted=True) as tunnel: - # You can connect to this TCP socket from outside the container, for example, using `nc`: - # nc - print("TCP tunnel listening at:", tunnel.tcp_socket) - run_echo_server(8000) -``` - -**SSH example:** -This assumes you have a rsa keypair in `~/.ssh/id_rsa{.pub}`, this is a bare-bones example -letting you SSH into a Modal container. - -```python -import subprocess -import time - -import modal - -app = modal.App() -image = ( - modal.Image.debian_slim() - .apt_install("openssh-server") - .run_commands("mkdir /run/sshd") - .add_local_file("~/.ssh/id_rsa.pub", "/root/.ssh/authorized_keys", copy=True) -) - - -@app.function(image=image, timeout=3600) -def some_function(): - subprocess.Popen(["/usr/sbin/sshd", "-D", "-e"]) - with modal.forward(port=22, unencrypted=True) as tunnel: - hostname, port = tunnel.tcp_socket - connection_cmd = f'ssh -p {port} root@{hostname}' - print(f"ssh into container using: {connection_cmd}") - time.sleep(3600) # keep alive for 1 hour or until killed -``` - -If you intend to use this more generally, a suggestion is to put the subprocess and port -forwarding code in an `@enter` lifecycle method of an @app.cls, to only make a single -ssh server and port for each container (and not one for each input to the function). diff --git a/.agents/skills/modal/references/api/modal.interact.md b/.agents/skills/modal/references/api/modal.interact.md deleted file mode 100644 index 4cdf276c39a8ee4755500830954ac679c2f006b2..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.interact.md +++ /dev/null @@ -1,9 +0,0 @@ -# modal.interact - -```python -interact() -``` -Enable interactivity with user input inside a Modal container. - -See the [interactivity guide](https://modal.com/docs/guide/developing-debugging#interactivity) -for more information on how to use this function. diff --git a/.agents/skills/modal/references/api/modal.io_streams.md b/.agents/skills/modal/references/api/modal.io_streams.md deleted file mode 100644 index 7aac4d6c61b36bd6a0cc540ad333dde0e0c15148..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.io_streams.md +++ /dev/null @@ -1,96 +0,0 @@ -# modal.io_streams - -## modal.io_streams.StreamReader - - -```python -class StreamReader(typing.Generic) -``` - -Retrieve logs from a stream (`stdout` or `stderr`). - -As an asynchronous iterable, the object supports the `for` and `async for` -statements. Just loop over the object to read in chunks. - - -### file_descriptor - -```python -file_descriptor(self) -``` -Possible values are `1` for stdout and `2` for stderr. - -### read - -```python -read(self) -``` -Fetch the entire contents of the stream until EOF. -## modal.io_streams.StreamWriter - - -```python -class StreamWriter(object) -``` - -Provides an interface to buffer and write logs to a sandbox or container process stream (`stdin`). - - -### write - -```python -write(self, data) -``` -Write data to the stream but does not send it immediately. - -This is non-blocking and queues the data to an internal buffer. Must be -used along with the `drain()` method, which flushes the buffer. - -**Usage** - -```python fixture:sandbox -proc = sandbox.exec( - "bash", - "-c", - "while read line; do echo $line; done", -) -proc.stdin.write(b"foo\n") -proc.stdin.write(b"bar\n") -proc.stdin.write_eof() -proc.stdin.drain() -``` - -### write_eof - -```python -write_eof(self) -``` -Close the write end of the stream after the buffered data is drained. - -If the process was blocked on input, it will become unblocked after -`write_eof()`. This method needs to be used along with the `drain()` -method, which flushes the EOF to the process. - -### drain - -```python -drain(self) -``` -Flush the write buffer and send data to the running process. - -This is a flow control method that blocks until data is sent. It returns -when it is appropriate to continue writing data to the stream. - -**Usage** - -```python notest -writer.write(data) -writer.drain() -``` - -Async usage: - -```python notest -writer.write(data) # not a blocking operation -await writer.drain.aio() -``` diff --git a/.agents/skills/modal/references/api/modal.is_local.md b/.agents/skills/modal/references/api/modal.is_local.md deleted file mode 100644 index 92055d40c8934dba462a3eef0457c7a5d3ce8295..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.is_local.md +++ /dev/null @@ -1,11 +0,0 @@ -# modal.is_local - -```python -is_local() -``` -Indicate the execution context of the current process. - -Note: this function specifically returns False when the current process is -running a Modal Function and True in all other cases. It will return True -when called from a child process of a Function or inside a Modal Sandbox, -even though those processes are running on Modal hardware. diff --git a/.agents/skills/modal/references/api/modal.method.md b/.agents/skills/modal/references/api/modal.method.md deleted file mode 100644 index bcfae37aead46b83d7d69933a0c27cf75c6d3006..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.method.md +++ /dev/null @@ -1,17 +0,0 @@ -# modal.method - -```python -method(*, is_generator=None) -``` -Decorator for methods that should be transformed into a Modal Function registered against this class's App. - -**Usage** - -```python -@app.cls(cpu=8) -class MyCls: - - @modal.method() - def f(self): - ... -``` diff --git a/.agents/skills/modal/references/api/modal.parameter.md b/.agents/skills/modal/references/api/modal.parameter.md deleted file mode 100644 index b628f750ddf92ba2286652a02eb7157ebe423464..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.parameter.md +++ /dev/null @@ -1,17 +0,0 @@ -# modal.parameter - -```python -parameter(*, default=_no_default, init=True) -``` -Used to specify options for modal.cls parameters, similar to dataclass.field for dataclasses -``` -class A: - a: str = modal.parameter() - -``` - -If `init=False` is specified, the field is not considered a parameter for the -Modal class and not used in the synthesized constructor. This can be used to -optionally annotate the type of a field that's used internally, for example values -being set by @enter lifecycle methods, without breaking type checkers, but it has -no runtime effect on the class. diff --git a/.agents/skills/modal/references/api/modal.web_server.md b/.agents/skills/modal/references/api/modal.web_server.md deleted file mode 100644 index 9fa4701894068600787ba2b6761ca7289edb215b..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.web_server.md +++ /dev/null @@ -1,31 +0,0 @@ -# modal.web_server - -```python -web_server(port, *, startup_timeout=5.0, label=None, custom_domains=None, - requires_proxy_auth=False) -``` -Decorator that registers an HTTP web server inside the container. - -This is similar to `@modal.asgi_app` and `@modal.wsgi_app`, but it allows you to expose a full -HTTP server listening on a container port. This is useful for servers written in other languages -like Rust, as well as integrating with non-ASGI frameworks like aiohttp and Tornado. - -The above example starts a simple file server, displaying the contents of the root directory. -Here, requests to the URL will go to external port 8000 on the container. The -`http.server` module is included with Python, but you could run anything here. - -Internally, the web server is transparently converted into a Web Function by Modal, so it has -the same serverless autoscaling behavior as other Web Functions. - -For more info, see the [guide on Web Functions](https://modal.com/docs/guide/webhooks). - -**Usage** - -```python -import subprocess - -@app.function() -@modal.web_server(8000) -def my_file_server(): - subprocess.Popen("python -m http.server -d / 8000", shell=True) -``` diff --git a/.agents/skills/modal/references/api/modal.wsgi_app.md b/.agents/skills/modal/references/api/modal.wsgi_app.md deleted file mode 100644 index 8920fd614c529ead7d2c5ff0feaa26b822aeb665..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/api/modal.wsgi_app.md +++ /dev/null @@ -1,25 +0,0 @@ -# modal.wsgi_app - -```python -wsgi_app(*, label=None, custom_domains=None, requires_proxy_auth=False) -``` -Decorator for registering a WSGI app with a Modal function. - -Web Server Gateway Interface (WSGI) is a standard for synchronous Python web apps. -It has been [succeeded by the ASGI interface](https://asgi.readthedocs.io/en/latest/introduction.html#wsgi-compatibility) -which is compatible with ASGI and supports additional functionality such as web sockets. -Modal supports ASGI via [`asgi_app`](https://modal.com/docs/reference/modal.asgi_app). - -To learn how to use this decorator with popular web frameworks, see the -[guide on Web Functions](https://modal.com/docs/guide/webhooks). - -**Usage** - -```python -from typing import Callable - -@app.function() -@modal.wsgi_app() -def create_wsgi() -> Callable: - ... -``` diff --git a/.agents/skills/modal/references/examples/agent.md b/.agents/skills/modal/references/examples/agent.md deleted file mode 100644 index baa3055edbbd07e8e93ef5d9eea25e120d2067da..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/agent.md +++ /dev/null @@ -1,219 +0,0 @@ -# Build a coding agent with Modal Sandboxes and LangGraph - -This example demonstrates how to build an LLM coding "agent" that can generate and execute Python code, using -documentation from the web to inform its approach. - -Naturally, we use the agent to generate code that runs language models. - -The agent is built with [LangGraph](https://github.com/langchain-ai/langgraph), a library for building -directed graphs of computation popular with AI agent developers, -and uses models from the OpenAI API. - -## Setup - -```python -import modal - -from .src import edges, nodes, retrieval -from .src.common import COLOR, PYTHON_VERSION, image - -``` - -You will need two [Modal Secrets](https://modal.com/docs/guide/secrets) to run this example: -one to access the OpenAI API and another to access the LangSmith API for logging the agent's behavior. - -To create them, head to the [Secrets dashboard](https://modal.com/secrets), select "Create new secret", -and use the provided templates for OpenAI and LangSmith. - -```python -app = modal.App( - "example-agent", - image=image, - secrets=[ - modal.Secret.from_name("openai-secret", required_keys=["OPENAI_API_KEY"]), - modal.Secret.from_name("langsmith-secret", required_keys=["LANGCHAIN_API_KEY"]), - ], -) - -``` - -## Creating a Sandbox - -We execute the agent's code in a Modal [Sandbox](https://modal.com/docs/guide/sandbox), which allows us to -run arbitrary code in a safe environment. In this example, we will use the [`transformers`](https://huggingface.co/docs/transformers/index) -library to generate text with a pre-trained model. Let's create a Sandbox with the necessary dependencies. - -```python -def create_sandbox(app) -> modal.Sandbox: - # Change this image (and the retrieval logic in the retrieval module) - # if you want the agent to give coding advice on other libraries! - agent_image = modal.Image.debian_slim(python_version=PYTHON_VERSION).uv_pip_install( - "torch==2.5.0", - "transformers==4.46.0", - ) - - return modal.Sandbox.create( - image=agent_image, - timeout=60 * 10, # 10 minutes - app=app, - # Modal sandboxes support GPUs! - gpu="T4", - # you can also pass secrets here -- note that the main app's secrets are not shared - ) - - -``` - -We also need a way to run our code in the sandbox. For this, we'll write a simple wrapper -around the Modal Sandbox `exec` method. We use `exec` because it allows us to run code without spinning up a -new container. And we can reuse the same container for multiple runs, preserving state. - -```python -def run(code: str, sb: modal.Sandbox) -> tuple[str, str]: - print( - f"{COLOR['HEADER']}📦: Running in sandbox{COLOR['ENDC']}", - f"{COLOR['GREEN']}{code}{COLOR['ENDC']}", - sep="\n", - ) - - exc = sb.exec("python", "-c", code) - exc.wait() - - stdout = exc.stdout.read() - stderr = exc.stderr.read() - - if exc.returncode != 0: - print( - f"{COLOR['HEADER']}📦: Failed with exitcode {sb.returncode}{COLOR['ENDC']}" - ) - - return stdout, stderr - - -``` - -## Constructing the agent's graph - -Now that we have the sandbox to execute code in, we can construct our agent's graph. Our graph is -defined in the `edges` and `nodes` modules -[associated with this example](https://github.com/modal-labs/modal-examples/tree/main/13_sandboxes/codelangchain). -Nodes are actions that change the state. Edges are transitions between nodes. - -The idea is simple: we start at the node `generate`, which invokes the LLM to generate code based off documentation. -The generated code is executed (in the sandbox) as part of an edge called `check_code_execution` -and then the outputs are passed to the LLM for evaluation (the `evaluate_execution` node). -If the LLM determines that the code has executed correctly -- which might mean that the code raised an exception! -- -we pass along the `decide_to_finish` edge and finish. - -```python -def construct_graph(sandbox: modal.Sandbox, debug: bool = False): - from langgraph.graph import StateGraph - - from .src.common import GraphState - - # Crawl the transformers documentation to inform our code generation - context = retrieval.retrieve_docs(debug=debug) - - graph = StateGraph(GraphState) - - # Attach our nodes to the graph - graph_nodes = nodes.Nodes(context, sandbox, run, debug=debug) - for key, value in graph_nodes.node_map.items(): - graph.add_node(key, value) - - # Construct the graph by adding edges - graph = edges.enrich(graph) - - # Set the starting and ending nodes of the graph - graph.set_entry_point(key="generate") - graph.set_finish_point(key="finish") - - return graph - - -``` - -We now set up the graph and compile it. See the `src` module for details -on the content of the graph and the nodes we've defined. - -```python -DEFAULT_QUESTION = "How do I generate Python code using a pre-trained model from the transformers library?" - - -@app.function() -def go( - question: str = DEFAULT_QUESTION, - debug: bool = False, -): - """Compiles the Python code generation agent graph and runs it, returning the result.""" - sb = create_sandbox(app) - - graph = construct_graph(sb, debug=debug) - runnable = graph.compile() - result = runnable.invoke( - {"keys": {"question": question, "iterations": 0}}, - config={"recursion_limit": 50}, - ) - - sb.terminate() - - return result["keys"]["response"] - - -``` - -## Running the Graph - -Now let's call the agent from the command line! - -We define a `local_entrypoint` that runs locally and triggers execution on Modal. - -You can invoke it by executing following command from a folder that contains the `codelangchain` directory -[from our examples repo](https://github.com/modal-labs/modal-examples/tree/main/13_sandboxes/codelangchain): - -```bash -modal run -m codelangchain.agent --question "How do I run a pre-trained model from the transformers library?" -``` - -```python -@app.local_entrypoint() -def main( - question: str = DEFAULT_QUESTION, - debug: bool = False, -): - """Sends a question to the Python code generation agent. - - Switch to debug mode for shorter context and smaller model.""" - if debug: - if question == DEFAULT_QUESTION: - question = "hi there, how are you?" - - print(go.remote(question, debug=debug)) - - -``` - -If things are working properly, you should see output like the following: - -```bash -$ modal run -m codelangchain.agent --question "generate some cool output with transformers" ----DECISION: FINISH--- ----FINISHING--- -To generate some cool output using transformers, we can use a pre-trained language model from the Hugging Face Transformers library. In this example, we'll use the GPT-2 model to generate text based on a given prompt. The GPT-2 model is a popular choice for text generation tasks due to its ability to produce coherent and contextually relevant text. We'll use the pipeline API from the Transformers library, which simplifies the process of using pre-trained models for various tasks, including text generation. - -from transformers import pipeline -# Initialize the text generation pipeline with the GPT-2 model -generator = pipeline('text-generation', model='gpt2') - -# Define a prompt for the model to generate text from -prompt = "Once upon a time in a land far, far away" - -# Generate text using the model -output = generator(prompt, max_length=50, num_return_sequences=1) - -# Print the generated text -print(output[0]['generated_text']) - -Result of code execution: -Once upon a time in a land far, far away, and still inhabited even after all the human race, there would be one God: a perfect universal God who has always been and will ever be worshipped. All His acts and deeds are immutable, -``` diff --git a/.agents/skills/modal/references/examples/algolia_indexer.md b/.agents/skills/modal/references/examples/algolia_indexer.md deleted file mode 100644 index 6db11ea53f9660b723b707c9cf55507c753b26a5..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/algolia_indexer.md +++ /dev/null @@ -1,179 +0,0 @@ -# Algolia docsearch crawler - -This tutorial shows you how to use Modal to run the [Algolia docsearch -crawler](https://docsearch.algolia.com/docs/legacy/run-your-own/) to index your -website and make it searchable. This is not just example code - we run the same -code in production to power search on this page (`Ctrl+K` to try it out!). - -## Basic setup - -Let's get the imports out of the way. - -```python -import json -import os -import subprocess - -import modal - -``` - -Modal lets you [use and extend existing Docker images](https://modal.com/docs/guide/custom-container#use-an-existing-container-image-with-from_registry), -as long as they have `python` and `pip` available. We'll use the official crawler image built by Algolia, with a small -adjustment: since this image has `python` symlinked to `python3.6` and Modal is not compatible with Python 3.6, we -install Python 3.11 and symlink that as the `python` executable instead. - -```python -algolia_image = modal.Image.from_registry( - "algolia/docsearch-scraper:v1.16.0", - add_python="3.11", - setup_dockerfile_commands=["ENTRYPOINT []"], -) - -app = modal.App("example-algolia-indexer") - -``` - -## Configure the crawler - -Now, let's configure the crawler with the website we want to index, and which -CSS selectors we want to scrape. Complete documentation for crawler configuration is available -[here](https://docsearch.algolia.com/docs/legacy/config-file). - -```python -CONFIG = { - "index_name": "modal_docs", - "custom_settings": { - "separatorsToIndex": "._", - "synonyms": [["cls", "class"]], - }, - "stop_urls": [ - "https://modal.com/docs/reference/modal.Stub", - "https://modal.com/gpu-glossary", - "https://modal.com/docs/reference/changelog", - ], - "start_urls": [ - { - "url": "https://modal.com/docs/guide", - "selectors_key": "default", - "page_rank": 2, - }, - { - "url": "https://modal.com/docs/examples", - "selectors_key": "examples", - "page_rank": 1, - }, - { - "url": "https://modal.com/docs/reference", - "selectors_key": "reference", - "page_rank": 1, - }, - ], - "selectors": { - "default": { - "lvl0": { - "selector": "header .navlink-active", - "global": True, - }, - "lvl1": "article h1", - "lvl2": "article h2", - "lvl3": "article h3", - "text": "article p,article ol,article ul", - }, - "examples": { - "lvl0": { - "selector": "header .navlink-active", - "global": True, - }, - "lvl1": "article h1", - "text": "article p,article ol,article ul", - }, - "reference": { - "lvl0": { - "selector": "//div[contains(@class, 'sidebar')]//a[contains(@class, 'active')]//preceding::a[contains(@class, 'header')][1]", - "type": "xpath", - "global": True, - "default_value": "", - "skip": {"when": {"value": ""}}, - }, - "lvl1": "article h1", - "lvl2": "article h2", - "lvl3": "article h3", - "text": "article p,article ol,article ul", - }, - }, -} - -``` - -## Create an API key - -If you don't already have one, sign up for an account on [Algolia](https://www.algolia.com/). Set up -a project and create an API key with `write` access to your index, and with the ACL permissions -`addObject`, `editSettings` and `deleteIndex`. Now, create a Secret on the Modal [Secrets](https://modal.com/secrets) -page with the `API_KEY` and `APPLICATION_ID` you just created. You can name this anything you want, -but we named it `algolia-secret` and so that's what the code below expects. - -## The actual function - -We want to trigger our crawler from our CI/CD pipeline, so we're serving it as a -[Web Function](https://modal.com/docs/guide/webhooks) that can be triggered by a `GET` request during deploy. -You could also consider running the crawler on a [schedule](https://modal.com/docs/guide/cron). - -The Algolia crawler is written for Python 3.6 and needs to run in the `pipenv` created for it, -so we're invoking it using a subprocess. - -```python -@app.function( - image=algolia_image, - secrets=[modal.Secret.from_name("algolia-secret")], -) -def crawl(): - # Installed with a 3.6 venv; Python 3.6 is unsupported by Modal, so use a subprocess instead. - subprocess.run( - ["pipenv", "run", "python", "-m", "src.index"], - env={**os.environ, "CONFIG": json.dumps(CONFIG)}, - ) - - -``` - -We want to be able to trigger this function through a webhook. - -```python -@app.function(image=modal.Image.debian_slim().uv_pip_install("fastapi[standard]")) -@modal.fastapi_endpoint() -def crawl_webhook(): - crawl.remote() - return "Finished indexing docs" - - -``` - -## Deploy the indexer - -That's all the code we need! To deploy your application, run - -```shell -modal deploy algolia_indexer.py -``` - -If successful, this will print a URL for your new webhook, that you can hit using -`curl` or a browser. Logs from webhook invocations can be found from the [apps](https://modal.com/apps) -page. - -The indexed contents can be found at https://www.algolia.com/apps/APP\_ID/explorer/browse/, for your -APP\_ID. Once you're happy with the results, you can [set up the `docsearch` package with your -website](https://docsearch.algolia.com/docs/docsearch-v3/), and create a search component that uses this index. - -## Entrypoint for development - -To make it easier to test this, we also have an entrypoint for when you run -`modal run algolia_indexer.py` - -```python -@app.local_entrypoint() -def run(): - crawl.remote() - -``` diff --git a/.agents/skills/modal/references/examples/amazon_embeddings.md b/.agents/skills/modal/references/examples/amazon_embeddings.md deleted file mode 100644 index 3719f1563985cce2615ee3c191936627e99b92fb..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/amazon_embeddings.md +++ /dev/null @@ -1,318 +0,0 @@ -# Embed 30 million Amazon reviews at 575k tokens per second with Qwen2-7B - -This example demonstrates how to create embeddings for a large text dataset. This is -often necessary to enable semantic search, translation, and other language -processing tasks. Modal makes it easy to deploy large, capable embedding models and handles -all of the scaling to process very large datasets in parallel on many cloud GPUs. - -We create a Modal Function that will handle all of the data loading and submit inputs to an -inference Cls that will automatically scale up to handle hundreds of large -batches in parallel. - -Between the time a batch is submitted and the time it is fetched, it is stored via -Modal's `spawn` system, which can hold onto up to one million inputs for up to a week. - -```python -import json -import subprocess -from pathlib import Path - -import modal - -app = modal.App(name="example-amazon-embeddings") -MINUTES = 60 # seconds -HOURS = 60 * MINUTES - -``` - -We define our `main` function as a `local_entrypoint`. This is what we'll call locally -to start the job on Modal. - -You can run it with the command - -```bash -modal run --detach amazon_embeddings.py -``` - -By default we `down-scale` to 1/100th of the data for demonstration purposes. -To launch the full job, set the `--down-scale` parameter to `1`. -But note that this will cost you! - -The entrypoint starts the job and gets back a `f`unction `c`all ID for each batch. -We can use these IDs to retrieve the embeddings once the job is finished. -Modal will keep the results around for up to 7 days after completion. Take a look at our -[job processing guide](https://modal.com/docs/guide/job-queue) -for more details. - -```python -@app.local_entrypoint() -def main( - dataset_name: str = "McAuley-Lab/Amazon-Reviews-2023", - dataset_subset: str = "raw_review_Books", - down_scale: float = 0.001, -): - out_path = Path("/tmp") / "embeddings-example-fc-ids.json" - function_ids = launch_job.remote( - dataset_name=dataset_name, dataset_subset=dataset_subset, down_scale=down_scale - ) - out_path.write_text(json.dumps(function_ids, indent=2) + "\n") - print(f"output handles saved to {out_path}") - - -``` - -## Load the data and start the inference job - -Next we define the Function that will do the data loading and feed it to our embedding model. -We define a container [Image](https://modal.com/docs/guide/images) -with the data loading dependencies. - -In it, we download the data we need and cache it to the container's local disk, -which will disappear when the job is finished. We will be saving the review data -along with the embeddings, so we don't need to keep the dataset around. - -Embedding a large dataset like this can take some time, but we don't need to wait -around for it to finish. We use `spawn` to invoke our embedding Function -and get back a handle with an ID that we can use to get the results later. -This can bottleneck on just sending data over the network for processing, so -we speed things up by using `ThreadPoolExecutor` to submit batches using multiple threads. - -Once all of the batches have been sent for inference, we can return the function IDs -to the local client to save. - -```python -@app.function( - image=modal.Image.debian_slim().uv_pip_install("datasets==3.5.1"), timeout=2 * HOURS -) -def launch_job(dataset_name: str, dataset_subset: str, down_scale: float): - import time - from concurrent.futures import ThreadPoolExecutor, as_completed - - from datasets import load_dataset - from tqdm import tqdm - - print("Loading dataset...") - dataset = load_dataset( - dataset_name, - dataset_subset, - split="full", - trust_remote_code=True, - ) - - data_subset = dataset.select(range(int(len(dataset) * down_scale))) - - tei = TextEmbeddingsInference() - batches = generate_batches_of_chunks(data_subset) - - start = time.perf_counter() - with ThreadPoolExecutor() as executor: - futures = [executor.submit(tei.embed.spawn, batch) for batch in tqdm(batches)] - function_ids = [] - for future in tqdm(as_completed(futures), total=len(futures)): - function_ids.append(future.result().object_id) - - print(f"Finished submitting job: {time.perf_counter() - start:.2f}s") - - return function_ids - - -``` - -## Massively scaling up and scaling out embedding inference on many beefy GPUs - -We're going to spin up many containers to run inference, and we don't want each -one to have to download the embedding model from Hugging Face. We can download and save it to a -Modal [Volume](https://modal.com/docs/guide/volumes) -during the image build step using `run_function`. - -We'll use the -[GTE-Qwen2-7B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct) -model from Alibaba, which performs well on the -[Massive Text Embedding Benchmark](https://huggingface.co/spaces/mteb/leaderboard). - -```python -MODEL_ID = "Alibaba-NLP/gte-Qwen2-7B-instruct" -MODEL_DIR = "/model" -MODEL_CACHE_VOLUME = modal.Volume.from_name( - "embeddings-example-model-cache", create_if_missing=True -) - - -def download_model(): - from huggingface_hub import snapshot_download - - snapshot_download(MODEL_ID, cache_dir=MODEL_DIR) - - -``` - -For inference, we will use Hugging Face's -[Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference) -framework for embedding model deployment. - -Running lots of separate machines is "scaling out". But we can also "scale up" -by running on large, high-performance machines. - -We'll use L40S GPUs for a good balance between cost and performance. Hugging Face has -prebuilt Docker images we can use as a base for our Modal Image. -We'll use the one built for the L40S's -[SM89/Ada Lovelace architecture](https://modal.com/gpu-glossary/device-hardware/streaming-multiprocessor-architecture) -and install the rest of our dependencies on top. - -```python -tei_image = "ghcr.io/huggingface/text-embeddings-inference:89-1.7" - -inference_image = ( - modal.Image.from_registry(tei_image, add_python="3.12") - .dockerfile_commands("ENTRYPOINT []") - .uv_pip_install( - "httpx==0.28.1", - "huggingface-hub==0.36.0", - "numpy==2.2.5", - "tqdm==4.67.1", - ) - .env({"HF_XET_HIGH_PERFORMANCE": "1", "HF_HOME": MODEL_DIR}) - .run_function(download_model, volumes={MODEL_DIR: MODEL_CACHE_VOLUME}) -) - - -``` - -Next we define our inference class. Modal will auto-scale the number of -containers ready to handle inputs based on the parameters we set in the `@app.cls` -and `@modal.concurrent` decorators. Here we limit the total number of containers to -100 and the maximum number of concurrent inputs to 10, which caps us at 1000 concurrent batches. -On Modal's Starter (free) and Team plans, the maximum number of concurrent GPUs is lower, -reducing the total number of concurrent batches and so the throughput. - -Customers on Modal's Enterprise Plan regularly scale up another order of magnitude above this. -If you're interested in running on thousands of GPUs, -[get in touch](https://form.fillout.com/t/onUBuQZ5vCus). - -Here we also specify the GPU type and attach the Modal Volume where we saved the -embedding model. - -This class will spawn a local Text Embeddings Inference server when the container -starts, and process each batch by receiving the text data over HTTP, returning a list of -tuples with the batch text data and embeddings. - -```python -@app.cls( - image=inference_image, - gpu="L40S", - volumes={MODEL_DIR: MODEL_CACHE_VOLUME}, - max_containers=100, - scaledown_window=5 * MINUTES, # idle for 5 min without inputs before scaling down - retries=3, # handle transient failures and storms in the cloud - timeout=2 * HOURS, # run for at most 2 hours -) -@modal.concurrent(max_inputs=10) -class TextEmbeddingsInference: - @modal.enter() - def open_connection(self): - from httpx import AsyncClient - - print("Starting text embedding inference server...") - self.process = spawn_server() - self.client = AsyncClient(base_url="http://127.0.0.1:8000", timeout=30) - - @modal.exit() - def terminate_connection(self): - self.process.terminate() - - @modal.method() - async def embed(self, batch): - texts = [chunk[-1] for chunk in batch] - res = await self.client.post("/embed", json={"inputs": texts}) - return [chunk + (embedding,) for chunk, embedding in zip(batch, res.json())] - - -``` - -## Helper Functions - -The book review dataset contains ~30M reviews with ~12B total characters, -indicating an average review length of ~500 characters. Some are much longer. -Embedding models have a limit on the number of tokens they can process in a single -input. We will need to split each review into chunks that are under this limit. - -The proper way to split text data is to use a tokenizer to ensure that any -single request is under the models token limit, and to overlap chunks to provide -semantic context and preserve information. For the sake of this example, we're going -just to split by a set character length (`CHUNK_SIZE`). - -While the embedding model has a limit on the number of input tokens for a single -embedding, the number of chunks that we can process in a single batch is limited by -the VRAM of the GPU. We set the `BATCH_SIZE` accordingly. - -```python -BATCH_SIZE = 256 -CHUNK_SIZE = 512 - - -def generate_batches_of_chunks( - dataset, chunk_size: int = CHUNK_SIZE, batch_size: int = BATCH_SIZE -): - """Creates batches of chunks by naively slicing strings according to CHUNK_SIZE.""" - batch = [] - for entry_index, data in enumerate(dataset): - product_id = data["asin"] - user_id = data["user_id"] - timestamp = data["timestamp"] - title = data["title"] - text = data["text"] - for chunk_index, chunk_start in enumerate(range(0, len(text), chunk_size)): - batch.append( - ( - entry_index, - chunk_index, - product_id, - user_id, - timestamp, - title, - text[chunk_start : chunk_start + chunk_size], - ) - ) - if len(batch) == batch_size: - yield batch - batch = [] - if batch: - yield batch - - -def spawn_server( - model_id: str = MODEL_ID, - port: int = 8000, - max_client_batch_size: int = BATCH_SIZE, - max_batch_tokens: int = BATCH_SIZE * CHUNK_SIZE, - huggingface_hub_cache: str = MODEL_DIR, -): - """Starts a text embedding inference server in a subprocess.""" - import socket - - LAUNCH_FLAGS = [ - "--model-id", - model_id, - "--port", - str(port), - "--max-client-batch-size", - str(max_client_batch_size), - "--max-batch-tokens", - str(max_batch_tokens), - "--huggingface-hub-cache", - huggingface_hub_cache, - ] - - process = subprocess.Popen(["text-embeddings-router"] + LAUNCH_FLAGS) - # Poll until webserver at 127.0.0.1:8000 accepts connections before running inputs. - while True: - try: - socket.create_connection(("127.0.0.1", port), timeout=1).close() - print("Inference server ready!") - return process - except (socket.timeout, ConnectionRefusedError): - retcode = process.poll() # Check if the process has terminated. - if retcode is not None: - raise RuntimeError(f"Launcher exited unexpectedly with code {retcode}") - -``` diff --git a/.agents/skills/modal/references/examples/anthropic_computer_use.md b/.agents/skills/modal/references/examples/anthropic_computer_use.md deleted file mode 100644 index d6fad25be10143a4d76c703b580ffa9bdc280845..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/anthropic_computer_use.md +++ /dev/null @@ -1,115 +0,0 @@ -# Run Anthropic's computer use demo in a Modal Sandbox - -This example demonstrates how to run Anthropic's [Computer Use demo](https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo) -in a Modal [Sandbox](https://modal.com/docs/guide/sandbox). - -## Sandbox Setup - -All Sandboxes are associated with an App. - -We start by looking up an existing App by name, or creating one if it doesn't exist. - -```python -import time -import urllib.request - -import modal - -app = modal.App.lookup("example-anthropic-computer-use", create_if_missing=True) - -``` - -The Computer Use [quickstart](https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo) -provides a prebuilt Docker image. We use this hosted image to create our sandbox environment. - -```python -sandbox_image = ( - modal.Image.from_registry( - "ghcr.io/anthropics/anthropic-quickstarts:computer-use-demo-latest", - ) - .env({"WIDTH": "1920", "HEIGHT": "1080"}) - .workdir("/home/computeruse") - .entrypoint([]) -) - -``` - -We'll provide the Anthropic API key via a Modal [Secret](https://modal.com/docs/guide/secrets) -which the sandbox can access at runtime. - -```python -secret = modal.Secret.from_name("anthropic-secret", required_keys=["ANTHROPIC_API_KEY"]) - -``` - -Now, we can start our Sandbox. -We use `modal.enable_output()` to print the Sandbox's image build logs to the console. -We'll also expose the ports required for the demo's interfaces: - -* Port 8501 serves the Streamlit UI for interacting with the agent loop -* Port 6080 serves the VNC desktop view via a browser-based noVNC client - -```python -with modal.enable_output(): - sandbox = modal.Sandbox.create( - "sudo", - "--preserve-env=ANTHROPIC_API_KEY,DISPLAY_NUM,WIDTH,HEIGHT,PATH", - "-u", - "computeruse", - "./entrypoint.sh", - app=app, - image=sandbox_image, - secrets=[secret], - encrypted_ports=[8501, 6080], - timeout=60 * 60, # stay alive for one hour, maximum one day - ) - -print(f"🏖️ Sandbox ID: {sandbox.object_id}") - -``` - -After starting the sandbox, we retrieve the public URLs for the exposed ports. - -```python -tunnels = sandbox.tunnels() -for port, tunnel in tunnels.items(): - print(f"Waiting for service on port {port} to start at {tunnel.url}") - -``` - -We can check on each server's status by making an HTTP request to the server's URL -and verifying that it responds with a 200 status code. - -```python -def is_server_up(url): - try: - response = urllib.request.urlopen(url) - return response.getcode() == 200 - except Exception: - return False - - -timeout = 60 # seconds -start_time = time.time() -up_ports = set() -while time.time() - start_time < timeout: - for port, tunnel in tunnels.items(): - if port not in up_ports and is_server_up(tunnel.url): - print(f"🏖️ Server is up and running on port {port}!") - up_ports.add(port) - if len(up_ports) == len(tunnels): - break - time.sleep(1) -else: - print("🏖️ Timed out waiting for server to start.") - - -``` - -You can now open the URLs in your browser to interact with the demo! -Note: The sandbox logs may mention `localhost:8080`. -Ignore this and use the printed tunnel URLs instead. - -When finished, you can terminate the sandbox from your [Modal dashboard](https://modal.com/containers) -or by running `Sandbox.from_id(sandbox.object_id).terminate()`. -The Sandbox will also spin down after one hour. diff --git a/.agents/skills/modal/references/examples/basic_web.md b/.agents/skills/modal/references/examples/basic_web.md deleted file mode 100644 index 32a68cddd6565c33bde9aee3f3975faca65a508d..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/basic_web.md +++ /dev/null @@ -1,225 +0,0 @@ -# Hello world wide web! - -Modal makes it easy to turn your Python functions into serverless web services: -access them via a browser or call them from any client that speaks HTTP, all -without having to worry about setting up servers or managing infrastructure. - -This tutorial shows the path with the shortest ["time to 200"](https://shkspr.mobi/blog/2021/05/whats-your-apis-time-to-200/): -[`modal.fastapi_endpoint`](https://modal.com/docs/reference/modal.fastapi_endpoint). - -On Modal, Web Functions have all the superpowers of Modal Functions: -they can be [accelerated with GPUs](https://modal.com/docs/guide/gpu), -they can access [Secrets](https://modal.com/docs/guide/secrets) or [Volumes](https://modal.com/docs/guide/volumes), -and they [automatically scale](https://modal.com/docs/guide/cold-start) to handle more traffic. - -Under the hood, we use the [FastAPI library](https://fastapi.tiangolo.com/), -which has [high-quality documentation](https://fastapi.tiangolo.com/tutorial/), -linked throughout this tutorial. - -## Turn a Modal Function into an API endpoint with a single decorator - -Modal Functions are already accessible remotely -- when you add the `@app.function` decorator to a Python function -and run `modal deploy`, you make it possible for your [other Python functions to call it](https://modal.com/docs/guide/trigger-deployed-functions). - -That's great, but it's not much help if you want to share what you've written with someone running code in a different language -- -or not running code at all! - -And that's where most of the power of the Internet comes from: sharing information and functionality across different computer systems. - -So we provide the `fastapi_endpoint` decorator to wrap your Modal Functions in the lingua franca of the web: HTTP. -Here's what that looks like: - -```python -import modal - -image = modal.Image.debian_slim().uv_pip_install("fastapi[standard]") -app = modal.App(name="example-basic-web", image=image) - - -@app.function() -@modal.fastapi_endpoint( - docs=True # adds interactive documentation in the browser -) -def hello(): - return "Hello world!" - - -``` - -You can expose this Web Function to the internet by running `modal serve basic_web.py`. -In the output, you should see a URL that ends with `hello-dev.modal.run`. -If you navigate to this URL, you should see the `"Hello world!"` message appear in your browser. - -You can also find interactive documentation, powered by OpenAPI and Swagger, -if you add `/docs` to the end of the URL. -From this documentation, you can interact with your endpoint, sending HTTP requests and receiving HTTP responses. -For more details, see the [FastAPI documentation](https://fastapi.tiangolo.com/features/#automatic-docs). - -By running the App with `modal serve`, you created a temporary endpoint that will disappear if you interrupt your terminal. -These temporary endpoints are great for debugging -- when you save a change to any of your dependent files, the endpoint will redeploy. -Try changing the message to something else, hitting save, and then hitting refresh in your browser or re-sending -the request from `/docs` or the command line. You should see the new message, along with logs in your terminal showing the redeploy and the request. - -When you're ready to deploy the Web Function persistently, run `modal deploy basic_web.py`. -Now, your Function will be available even when you've closed your terminal or turned off your computer. - -## Send data to a Web Function - -The function above was a bit silly: it always returns the same message. - -Most functions need an input to be useful. There are two ways to send data to a Web Function: - -* in the URL as a [query parameter](#sending-data-in-query-parameters) -* in the [body of the request](#sending-data-in-the-request-body) as JSON - -### Sending data in query parameters - -By default, your function's arguments are treated as query parameters: -they are extracted from the end of the URL, where they should be added in the form -`?arg1=foo&arg2=bar`. - -From the Python side, there's hardly anything to do: - -```python -@app.function() -@modal.fastapi_endpoint(docs=True) -def greet(user: str) -> str: - return f"Hello {user}!" - - -``` - -If you are already running `modal serve basic_web.py`, this Function will be available at a URL, printed in your terminal, that ends with `greet-dev.modal.run`. - -We provide Python type-hints to get type information in the docs and -[automatic validation](https://fastapi.tiangolo.com/tutorial/query-params-str-validations/). -For example, if you navigate directly to the URL for `greet`, you will get a detailed error message -indicating that the `user` parameter is missing. Navigate instead to `/docs` to see how to structure the request. - -You can read more about query parameters in the [FastAPI documentation](https://fastapi.tiangolo.com/tutorial/query-params/). - -### Sending data in the request body - -For larger and more complex data, it is generally preferable to send data in the body of the HTTP request. -This body is formatted as [JSON](https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Objects/JSON), -the most common data interchange format on the web. - -To set up a Web Function that accepts JSON data, add an argument with a `dict` type-hint to your function. -This argument will be populated with the data sent in the request body. - -```python -@app.function() -@modal.fastapi_endpoint(method="POST", docs=True) -def goodbye(data: dict) -> str: - name = data.get("name") or "world" - return f"Goodbye {name}!" - - -``` - -Note that we gave a value of `"POST"` for the `method` argument here. -This argument defines the HTTP request method that the function will respond to, -and it defaults to `"GET"`. -If you head to the URL for the `goodbye` function in your browser, -you will get a 405 Method Not Allowed error, because browsers only send GET requests by default. -While this is technically a separate concern from query parameters versus request bodies -and you can define a function that accepts GET requests and uses data from the body, -it is [considered bad form](https://stackoverflow.com/a/983458). - -Navigate to `/docs` for more on how to call the function. -You will need to send a POST request with a JSON body containing a `name` key. -To get the same typing and validation benefits as with query parameters, -use a [Pydantic model](https://fastapi.tiangolo.com/tutorial/body/) -for this argument. - -You can read more about request bodies in the [FastAPI documentation](https://fastapi.tiangolo.com/tutorial/body/). - -## Handle expensive startup with `modal.Cls` - -Sometimes your function needs to do something before it can handle its first request, -like get a value from a database or set the value of a variable. -If that step is expensive, like [loading a large ML model](https://modal.com/docs/guide/model-weights), -it'd be a shame to have to do it every time a request comes in! - -Web Functions can be methods on a [`modal.Cls`](https://modal.com/docs/guide/lifecycle-functions#container-lifecycle-functions-and-parameters), -which allows you to manage the container's lifecycle independently from processing individual requests. - -This example will only set the `start_time` instance variable once, on container startup. - -```python -@app.cls() -class WebApp: - @modal.enter() - def startup(self): - from datetime import datetime, timezone - - print("🏁 Starting up!") - self.start_time = datetime.now(timezone.utc) - - @modal.fastapi_endpoint(docs=True) - def web(self): - from datetime import datetime, timezone - - current_time = datetime.now(timezone.utc) - return {"start_time": self.start_time, "current_time": current_time} - - -``` - -## Protect Web Functions with proxy authentication - -Sharing your Python functions on the web is great, but it's not always a good idea -to make those functions available to just anyone. - -For example, you might have a function like the one below that -is more expensive to run than to call (and so might be abused by your enemies) -or reveals information that you would rather keep secret. - -To protect your Modal Web Functions so that they can't be triggered except -by members of your [Modal workspace](https://modal.com/docs/guide/workspaces), -add the `requires_proxy_auth=True` flag to the `fastapi_endpoint` decorator. - -```python -@app.function(gpu="h100") -@modal.fastapi_endpoint(requires_proxy_auth=True, docs=False) -def expensive_secret(): - return "I didn't care for 'The Godfather'. It insists upon itself." - - -``` - -The `expensive-secret` endpoint URL will still be printed to the output when you `modal serve` or `modal deploy`, -along with a "🔑" emoji indicating that it is secured with proxy authentication. -If you head to that URL via the browser, you will get a -[`401 Unauthorized`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401) error code in response. -You should also check the dashboard page for this app (at the URL printed at the very top of the `modal` command output) -so you can see that no containers were spun up to handle the request -- this authorization is handled entirely inside Modal's infrastructure. - -You can trigger the Web Function by [creating a Proxy Auth Token](https://modal.com/settings/proxy-auth-tokens) -and then including the token ID and secret in the `Modal-Key` and `Modal-Secret` headers. - -From the command line, that might look like - -```shell -export TOKEN_ID=wk-1234abcd -export TOKEN_SECRET=ws-1234abcd -curl -H "Modal-Key: $TOKEN_ID" \ - -H "Modal-Secret: $TOKEN_SECRET" \ - https://your-workspace-name--expensive-secret.modal.run -``` - -For more details, see the -[guide to proxy authentication](https://modal.com/docs/guide/webhook-proxy-auth). - -## What next? - -Modal's `fastapi_endpoint` decorator is opinionated and designed for relatively simple web applications -- -one or a few independent Python functions that you want to expose to the web. - -Three additional decorators allow you to serve more complex web applications with greater control: - -* [`asgi_app`](https://modal.com/docs/guide/webhooks#asgi) to serve applications compliant with the ASGI standard, - like [FastAPI](https://fastapi.tiangolo.com/) -* [`wsgi_app`](https://modal.com/docs/guide/webhooks#wsgi) to serve applications compliant with the WSGI standard, - like [Flask](https://flask.palletsprojects.com/) -* [`web_server`](https://modal.com/docs/guide/webhooks#non-asgi-web-servers) to serve any application that listens on a port diff --git a/.agents/skills/modal/references/examples/batched_whisper.md b/.agents/skills/modal/references/examples/batched_whisper.md deleted file mode 100644 index fd394c9c82fce7f12dd889af8918bf902d94eea0..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/batched_whisper.md +++ /dev/null @@ -1,187 +0,0 @@ -# Fast Whisper inference using dynamic batching - -In this example, we demonstrate how to run [dynamically batched inference](https://modal.com/docs/guide/dynamic-batching) -for OpenAI's speech recognition model, [Whisper](https://openai.com/index/whisper/), on Modal. -Batching multiple audio samples together or batching chunks of a single audio sample can help to achieve a 2.8x increase -in inference throughput on an A10G! - -We will be running the [Whisper Large V3](https://huggingface.co/openai/whisper-large-v3) model. -To run [any of the other HuggingFace Whisper models](https://huggingface.co/models?search=openai/whisper), -simply replace the `MODEL_NAME` and `MODEL_REVISION` variables. - -## Setup - -Let's start by importing the Modal client and defining the model that we want to serve. - -```python -from typing import Optional - -import modal - -MODEL_DIR = "/model" -MODEL_NAME = "openai/whisper-large-v3" -MODEL_REVISION = "afda370583db9c5359511ed5d989400a6199dfe1" - - -``` - -## Define a container image - -We’ll start with Modal's baseline `debian_slim` image and install the relevant libraries. - -```python -image = ( - modal.Image.debian_slim(python_version="3.11") - .uv_pip_install( - "torch==2.5.1", - "transformers==4.47.1", - "huggingface-hub==0.36.0", - "librosa==0.10.2", - "soundfile==0.12.1", - "accelerate==1.2.1", - "datasets==3.2.0", - ) - .env({"HF_XET_HIGH_PERFORMANCE": "1", "HF_HUB_CACHE": MODEL_DIR}) -) - -model_cache = modal.Volume.from_name("hf-hub-cache", create_if_missing=True) -app = modal.App( - "example-batched-whisper", - image=image, - volumes={MODEL_DIR: model_cache}, -) - -``` - -## Caching the model weights - -We'll define a function to download the model and cache it in a volume. -You can `modal run` against this function prior to deploying the App. - -```python -@app.function() -def download_model(): - from huggingface_hub import snapshot_download - from transformers.utils import move_cache - - snapshot_download( - MODEL_NAME, - ignore_patterns=["*.pt", "*.bin"], # Using safetensors - revision=MODEL_REVISION, - ) - move_cache() - - -``` - -## The model class - -The inference function is best represented using Modal's [class syntax](https://modal.com/docs/guide/lifecycle-functions). - -We define a `@modal.enter` method to load the model when the container starts, before it picks up any inputs. -The weights will be loaded from the Hugging Face cache volume so that we don't need to download them when -we start a new container. For more on storing model weights on Modal, see -[this guide](https://modal.com/docs/guide/model-weights). - -We also define a `transcribe` method that uses the `@modal.batched` decorator to enable dynamic batching. -This allows us to invoke the function with individual audio samples, and the function will automatically batch them -together before running inference. Batching is critical for making good use of the GPU, since GPUs are designed -for running parallel operations at high throughput. - -The `max_batch_size` parameter limits the maximum number of audio samples combined into a single batch. -We used a `max_batch_size` of `64`, the largest power-of-2 batch size that can be accommodated by the 24 A10G GPU memory. -This number will vary depending on the model and the GPU you are using. - -The `wait_ms` parameter sets the maximum time to wait for more inputs before running the batched transcription. -To tune this parameter, you can set it to the target latency of your application minus the execution time of an inference batch. -This allows the latency of any request to stay within your target latency. - -```python -@app.cls( - gpu="a10g", # Try using an A100 or H100 if you've got a large model or need big batches! - max_containers=10, # default max GPUs for Modal's free tier -) -class Model: - @modal.enter() - def load_model(self): - import torch - from transformers import ( - AutoModelForSpeechSeq2Seq, - AutoProcessor, - pipeline, - ) - - self.processor = AutoProcessor.from_pretrained(MODEL_NAME) - self.model = AutoModelForSpeechSeq2Seq.from_pretrained( - MODEL_NAME, - torch_dtype=torch.float16, - low_cpu_mem_usage=True, - use_safetensors=True, - ).to("cuda") - - self.model.generation_config.language = "<|en|>" - - # Create a pipeline for preprocessing and transcribing speech data - self.pipeline = pipeline( - "automatic-speech-recognition", - model=self.model, - tokenizer=self.processor.tokenizer, - feature_extractor=self.processor.feature_extractor, - torch_dtype=torch.float16, - device="cuda", - ) - - @modal.batched(max_batch_size=64, wait_ms=1000) - def transcribe(self, audio_samples): - import time - - start = time.monotonic_ns() - print(f"Transcribing {len(audio_samples)} audio samples") - transcriptions = self.pipeline(audio_samples, batch_size=len(audio_samples)) - end = time.monotonic_ns() - print( - f"Transcribed {len(audio_samples)} samples in {round((end - start) / 1e9, 2)}s" - ) - return transcriptions - - -``` - -## Transcribe a dataset - -In this example, we use the [librispeech\_asr\_dummy dataset](https://huggingface.co/datasets/hf-internal-testing/librispeech_asr_dummy) -from Hugging Face's Datasets library to test the model. - -We use [`map.aio`](https://modal.com/docs/reference/modal.Function#map) to asynchronously map over the audio files. -This allows us to invoke the batched transcription method on each audio sample in parallel. - -```python -@app.function() -async def transcribe_hf_dataset(dataset_name): - from datasets import load_dataset - - print("📂 Loading dataset", dataset_name) - ds = load_dataset(dataset_name, "clean", split="validation") - print("📂 Dataset loaded") - batched_whisper = Model() - print("📣 Sending data for transcription") - async for transcription in batched_whisper.transcribe.map.aio(ds["audio"]): - yield transcription - - -``` - -## Run the model - -We define a [`local_entrypoint`](https://modal.com/docs/guide/apps#entrypoints-for-ephemeral-apps) -to run the transcription. You can run this locally with `modal run batched_whisper.py`. - -```python -@app.local_entrypoint() -async def main(dataset_name: Optional[str] = None): - if dataset_name is None: - dataset_name = "hf-internal-testing/librispeech_asr_dummy" - async for result in transcribe_hf_dataset.remote_gen.aio(dataset_name): - print(result["text"]) - -``` diff --git a/.agents/skills/modal/references/examples/blender_video.md b/.agents/skills/modal/references/examples/blender_video.md deleted file mode 100644 index dcffe9516e96a5196b6329e4f9b276ac9642a607..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/blender_video.md +++ /dev/null @@ -1,211 +0,0 @@ -# Render a video with Blender on many GPUs or CPUs in parallel - -This example shows how you can render an animated 3D scene using -[Blender](https://www.blender.org/)'s Python interface. - -You can run it on CPUs to scale out on one hundred containers -or run it on GPUs to get higher throughput per node. -Even for this simple scene, GPUs render >10x faster than CPUs. - -The final render looks something like this: - -
- -
- -## Defining a Modal app - -```python -from pathlib import Path - -import modal - -``` - -Modal runs your Python functions for you in the cloud. -You organize your code into apps, collections of functions that work together. - -```python -app = modal.App("example-blender-video") - -``` - -We need to define the environment each function runs in -- its container image. -The block below defines a container image, starting from a basic Debian Linux image -adding Blender's system-level dependencies -and then installing the `bpy` package, which is Blender's Python API. - -```python -rendering_image = ( - modal.Image.debian_slim(python_version="3.11") - .apt_install("xorg", "libxkbcommon0") # X11 (Unix GUI) dependencies - .uv_pip_install("bpy==4.5.0") # Blender as a Python package -) - -``` - -## Rendering a single frame - -We define a function that renders a single frame. We'll scale this function out on Modal later. - -Functions in Modal are defined along with their hardware and their dependencies. -This function can be run with GPU acceleration or without it, and we'll use a global flag in the code to switch between the two. - -```python -WITH_GPU = ( - True # try changing this to False to run rendering massively in parallel on CPUs! -) - -``` - -We decorate the function with `@app.function` to define it as a Modal function. -Note that in addition to defining the hardware requirements of the function, -we also specify the container image that the function runs in (the one we defined above). - -The details of the scene aren't too important for this example, but we'll load -a .blend file that we created earlier. This scene contains a rotating -Modal logo made of a transmissive ice-like material, with a generated displacement map. The -animation keyframes were defined in Blender. - -```python -@app.function( - gpu="L40S" if WITH_GPU else None, - # default limits on Modal free tier - max_containers=10 if WITH_GPU else 100, - image=rendering_image, -) -def render(blend_file: bytes, frame_number: int = 0) -> bytes: - """Renders the n-th frame of a Blender file as a PNG.""" - import bpy - - input_path = "/tmp/input.blend" - output_path = f"/tmp/output-{frame_number}.png" - - # Blender requires input as a file. - Path(input_path).write_bytes(blend_file) - - bpy.ops.wm.open_mainfile(filepath=input_path) - bpy.context.scene.frame_set(frame_number) - bpy.context.scene.render.filepath = output_path - configure_rendering(bpy.context, with_gpu=WITH_GPU) - bpy.ops.render.render(write_still=True) - - # Blender renders image outputs to a file as well. - return Path(output_path).read_bytes() - - -``` - -### Rendering with acceleration - -We can configure the rendering process to use GPU acceleration with NVIDIA CUDA. -We select the [Cycles rendering engine](https://www.cycles-renderer.org/), which is compatible with CUDA, -and then activate the GPU. - -```python -def configure_rendering(ctx, with_gpu: bool): - # configure the rendering process - ctx.scene.render.engine = "CYCLES" - ctx.scene.render.resolution_x = 3000 - ctx.scene.render.resolution_y = 2000 - ctx.scene.render.resolution_percentage = 50 - ctx.scene.cycles.samples = 128 - - cycles = ctx.preferences.addons["cycles"] - - # Use GPU acceleration if available. - if with_gpu: - cycles.preferences.compute_device_type = "CUDA" - ctx.scene.cycles.device = "GPU" - - # reload the devices to update the configuration - cycles.preferences.get_devices() - for device in cycles.preferences.devices: - device.use = True - - else: - ctx.scene.cycles.device = "CPU" - - # report rendering devices -- a nice snippet for debugging and ensuring the accelerators are being used - for dev in cycles.preferences.devices: - print(f"ID:{dev['id']} Name:{dev['name']} Type:{dev['type']} Use:{dev['use']}") - - -``` - -## Combining frames into a video - -Rendering 3D images is fun, and GPUs can make it faster, but rendering 3D videos is better! -We add another function to our app, running on a different, simpler container image -and different hardware, to combine the frames into a video. - -```python -combination_image = modal.Image.debian_slim(python_version="3.11").apt_install("ffmpeg") - -``` - -The function to combine the frames into a video takes a sequence of byte sequences, one for each rendered frame, -and converts them into a single sequence of bytes, the MP4 file. - -```python -@app.function(image=combination_image) -def combine(frames_bytes: list[bytes], fps: int = 60) -> bytes: - import subprocess - import tempfile - - with tempfile.TemporaryDirectory() as tmpdir: - for i, frame_bytes in enumerate(frames_bytes): - frame_path = Path(tmpdir) / f"frame_{i:05}.png" - frame_path.write_bytes(frame_bytes) - out_path = Path(tmpdir) / "output.mp4" - subprocess.run( - f"ffmpeg -framerate {fps} -pattern_type glob -i '{tmpdir}/*.png' -c:v libx264 -pix_fmt yuv420p {out_path}", - shell=True, - ) - return out_path.read_bytes() - - -``` - -## Rendering in parallel in the cloud from the comfort of the command line - -With these two functions defined, we need only a few more lines to run our rendering at scale on Modal. - -First, we need a function that coordinates our functions to `render` frames and `combine` them. -We decorate that function with `@app.local_entrypoint` so that we can run it with `modal run blender_video.py`. - -In that function, we use `render.map` to map the `render` function over the range of frames. - -We give the `local_entrypoint` two parameters to control the render -- the number of frames to render and how many frames to skip. -These demonstrate a basic pattern for controlling Functions on Modal from a local client. - -We collect the bytes from each frame into a `list` locally and then send it to `combine` with `.remote`. - -The bytes for the video come back to our local machine, and we write them to a file. - -The whole rendering process (for four seconds of 1080p 60 FPS video) takes about three minutes to run on 10 L40S GPUs, -with a per-frame latency of about six seconds, and about five minutes to run on 100 CPUs, with a per-frame latency of about one minute. - -```python -@app.local_entrypoint() -def main(frame_count: int = 250, frame_skip: int = 1): - output_directory = Path("/tmp") / "render" - output_directory.mkdir(parents=True, exist_ok=True) - - input_path = Path(__file__).parent / "IceModal.blend" - blend_bytes = input_path.read_bytes() - args = [(blend_bytes, frame) for frame in range(1, frame_count + 1, frame_skip)] - images = list(render.starmap(args)) - for i, image in enumerate(images): - frame_path = output_directory / f"frame_{i + 1}.png" - frame_path.write_bytes(image) - print(f"Frame saved to {frame_path}") - - video_path = output_directory / "output.mp4" - video_bytes = combine.remote(images) - video_path.write_bytes(video_bytes) - print(f"Video saved to {video_path}") - -``` diff --git a/.agents/skills/modal/references/examples/boltz_predict.md b/.agents/skills/modal/references/examples/boltz_predict.md deleted file mode 100644 index 06b061ba4911ed25b45f06f031a9cf462bc88d44..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/boltz_predict.md +++ /dev/null @@ -1,216 +0,0 @@ -# Fold proteins with Boltz-2 - -
-Boltz-2 -
Example of Boltz-2 protein structure prediction -of a protein-ligand complex
-
- -Boltz-2 is an open source molecular structure prediction model. -In contrast to previous models like Boltz-1, [Chai-1](https://modal.com/docs/examples/chai1), and AlphaFold-3, it not only predicts protein structures but also the [binding affinities](https://en.wikipedia.org/wiki/Ligand_\(biochemistry\)#Receptor/ligand_binding_affinity) between proteins and [ligands](https://en.wikipedia.org/wiki/Ligand_\(biochemistry\)). -It was created by the [MIT Jameel Clinic](https://jclinic.mit.edu/boltz-2/). -For details, see [their technical report](https://jeremywohlwend.com/assets/boltz2.pdf). - -Here, we demonstrate how to run Boltz-2 on Modal. - -## Setup - -```python -from pathlib import Path -from typing import Optional - -import modal - -here = Path(__file__).parent # the directory of this file - -MINUTES = 60 # seconds - -app = modal.App(name="example-boltz-predict") - -``` - -## Fold a protein from the command line - -The logic for running Boltz-2 is encapsulated in the function below, -which you can trigger from the command line by running - -```shell -modal run boltz_predict.py -``` - -This will set up the environment for running Boltz-2 inference in Modal's cloud, -run it, and then save the results locally as a [tarball](https://computing.help.inf.ed.ac.uk/FAQ/whats-tarball-or-how-do-i-unpack-or-create-tgz-or-targz-file). -That tarball archive contains, among other things, the predicted structure as a -[Crystallographic Information File](https://en.wikipedia.org/wiki/Crystallographic_Information_File), -which you can render with the online [Molstar Viewer](https://molstar.org/viewer). - -You can pass any options for the [`boltz predict` command line tool](https://github.com/jwohlwend/boltz/blob/main/docs/prediction.md) -as a string, like - -```shell -modal run boltz_predict.py --args "--sampling_steps 10" -``` - -To see more options, run the command with the `--help` flag. - -To learn how it works, read on! - -```python -@app.local_entrypoint() -def main( - force_download: bool = False, input_yaml_path: Optional[str] = None, args: str = "" -): - print("🧬 loading model remotely") - download_model.remote(force_download) - - if input_yaml_path is None: - input_yaml_path = here / "data" / "boltz_affinity.yaml" - else: - input_yaml_path = Path(input_yaml_path) - input_yaml = input_yaml_path.read_text() - - print(f"🧬 running boltz with input from {input_yaml_path}") - output = boltz_inference.remote(input_yaml) - - output_path = Path("/tmp") / "boltz" / "boltz_result.tar.gz" - output_path.parent.mkdir(exist_ok=True, parents=True) - print(f"🧬 writing output to {output_path}") - output_path.write_bytes(output) - - -``` - -## Installing Boltz-2 Python dependencies on Modal - -Code running on Modal runs inside containers built from [container images](https://modal.com/docs/guide/images) -that include that code's dependencies. - -Because Modal images include [GPU drivers](https://modal.com/docs/guide/cuda) by default, -installation of higher-level packages like `boltz` that require GPUs is painless. - -Here, we do it in a few lines, using the `uv` package manager for extra speed. - -```python -image = modal.Image.debian_slim(python_version="3.12").uv_pip_install("boltz==2.1.1") - -``` - -## Storing Boltz-2 model weights on Modal with Volumes - -Not all "dependencies" belong in a container image. Boltz-2, for example, depends on -the weights of the model and a [Chemical Component Dictionary](https://www.wwpdb.org/data/ccd) (CCD) file. - -Rather than loading them dynamically at run-time (which would add several minutes of GPU time to each inference), -or installing them into the image (which would require they be re-downloaded any time the other dependencies changed), -we load them onto a [Modal Volume](https://modal.com/docs/guide/volumes). -A Modal Volume is a file system that all of your code running on Modal (or elsewhere!) can access. -For more on storing model weights on Modal, see [this guide](https://modal.com/docs/guide/model-weights). -For details on how we download the weights in this case, see the [Addenda](#addenda). - -```python -boltz_model_volume = modal.Volume.from_name("boltz-models", create_if_missing=True) -models_dir = Path("/models/boltz") - -``` - -## Running Boltz-2 on Modal - -To run inference on Modal we wrap our function in a decorator, `@app.function`. -We provide that decorator with some arguments that describe the infrastructure our code needs to run: -the Volume we created, the Image we defined, and of course a fast GPU! - -Note that the `boltz` command-line tool we use takes the path to a -[specially-formatted YAML file](https://github.com/jwohlwend/boltz/blob/main/docs/prediction.md#yaml-format) -that includes definitions of molecules to predict the structures of and optionally paths to -[Multiple Sequence Alignment](https://en.wikipedia.org/wiki/Multiple_sequence_alignment) (MSA) files -for any protein molecules. We pass the [--use\_msa\_server](https://github.com/jwohlwend/boltz/blob/main/docs/prediction.md) flag to auto-generate the MSA using the mmseqs2 server. - -```python -@app.function( - image=image, - volumes={models_dir: boltz_model_volume}, - timeout=10 * MINUTES, - gpu="H100", -) -def boltz_inference(boltz_input_yaml: str, args="") -> bytes: - import shlex - import subprocess - - input_path = Path("input.yaml") - input_path.write_text(boltz_input_yaml) - - args = shlex.split(args) - - print(f"🧬 predicting structure using boltz model from {models_dir}") - subprocess.run( - ["boltz", "predict", input_path, "--use_msa_server", "--cache", str(models_dir)] - + args, - check=True, - ) - - print("🧬 packaging up outputs") - output_bytes = package_outputs(f"boltz_results_{input_path.with_suffix('').name}") - - return output_bytes - - -``` - -## Addenda - -Above, we glossed over just how we got hold of the model weights -- -the `local_entrypoint` just called a function named `download_model`. - -Here's the implementation of that function. For details, see our -[guide to storing model weights on Modal](https://modal.com/docs/guide/model-weights). - -```python -download_image = ( - modal.Image.debian_slim() - .uv_pip_install("huggingface-hub==0.36.0") - .env({"HF_XET_HIGH_PERFORMANCE": "1"}) -) - - -@app.function( - volumes={models_dir: boltz_model_volume}, - timeout=20 * MINUTES, - image=download_image, -) -def download_model( - force_download: bool = False, - revision: str = "6fdef46d763fee7fbb83ca5501ccceff43b85607", -): - from huggingface_hub import snapshot_download - - snapshot_download( - repo_id="boltz-community/boltz-2", - revision=revision, - local_dir=models_dir, - force_download=force_download, - ) - boltz_model_volume.commit() - - print(f"🧬 model downloaded to {models_dir}") - - -``` - -We package the outputs into a tarball which contains the predicted structure as a -[Crystallographic Information File](https://en.wikipedia.org/wiki/Crystallographic_Information_File) -and the binding affinity as a JSON file. -You can render the structure with the online [Molstar Viewer](https://molstar.org/viewer). - -```python -def package_outputs(output_dir: str) -> bytes: - import io - import tarfile - - tar_buffer = io.BytesIO() - - with tarfile.open(fileobj=tar_buffer, mode="w:gz") as tar: - tar.add(output_dir, arcname=output_dir) - - return tar_buffer.getvalue() - -``` diff --git a/.agents/skills/modal/references/examples/chai1.md b/.agents/skills/modal/references/examples/chai1.md deleted file mode 100644 index 57233657068721dba4704c1e37190f12056af472..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/chai1.md +++ /dev/null @@ -1,309 +0,0 @@ -# Fold proteins with Chai-1 - -In biology, function follows form quite literally: -the physical shapes of proteins dictate their behavior. -Measuring those shapes directly is difficult -and first-principles physical simulation prohibitively expensive. - -And so predicting protein shape from content -- -determining how the one-dimensional chain of amino acids encoded by DNA *folds* into a 3D object -- -has emerged as a key application for machine learning and neural networks in biology. - -In this example, we demonstrate how to run the open source [Chai-1](https://github.com/chaidiscovery/chai-lab/) -protein structure prediction model on Modal's flexible serverless infrastructure. -For details on how the Chai-1 model works and what it can be used for, -see the authors' [technical report on bioRxiv](https://www.biorxiv.org/content/10.1101/2024.10.10.615955). - -This simple script is meant as a starting point showing how to handle fiddly bits -like installing dependencies, loading weights, and formatting outputs so that you can get on with the fun stuff. -To experience the full power of Modal, try scaling inference up and running on hundreds or thousands of structures! - -
- -
- -## Setup - -```python -import hashlib -import json -from pathlib import Path -from typing import Optional -from uuid import uuid4 - -import modal - -here = Path(__file__).parent # the directory of this file - -MINUTES = 60 # seconds - -app = modal.App(name="example-chai1") - -``` - -## Fold a protein from the command line - -The logic for running Chai-1 is encapsulated in the function below, -which you can trigger from the command line by running - -```shell -modal run chai1 -``` - -This will set up the environment for running Chai-1 inference in Modal's cloud, -run it, and then save the results remotely and locally. The results are returned in the -[Crystallographic Information File](https://en.wikipedia.org/wiki/Crystallographic_Information_File) format, -which you can render with the online [Molstar Viewer](https://molstar.org/). - -To see more options, run the command with the `--help` flag. - -To learn how it works, read on! - -```python -@app.local_entrypoint() -def main( - force_redownload: bool = False, - fasta_file: Optional[str] = None, - inference_config_file: Optional[str] = None, - output_dir: Optional[str] = None, - run_id: Optional[str] = None, -): - print("🧬 checking inference dependencies") - download_inference_dependencies.remote(force=force_redownload) - - if fasta_file is None: - fasta_file = here / "data" / "chai1_default_input.fasta" - print(f"🧬 running Chai inference on {fasta_file}") - fasta_content = Path(fasta_file).read_text() - - if inference_config_file is None: - inference_config_file = here / "data" / "chai1_default_inference.json" - print(f"🧬 loading Chai inference config from {inference_config_file}") - inference_config = json.loads(Path(inference_config_file).read_text()) - - if run_id is None: - run_id = hashlib.sha256(uuid4().bytes).hexdigest()[:8] # short id - print(f"🧬 running inference with {run_id=}") - - results = chai1_inference.remote(fasta_content, inference_config, run_id) - - if output_dir is None: - output_dir = Path("/tmp/chai1") - output_dir.mkdir(parents=True, exist_ok=True) - - print(f"🧬 saving results to disk locally in {output_dir}") - for ii, (scores, cif) in enumerate(results): - (Path(output_dir) / f"{run_id}-scores.model_idx_{ii}.npz").write_bytes(scores) - (Path(output_dir) / f"{run_id}-preds.model_idx_{ii}.cif").write_text(cif) - - -``` - -## Installing Chai-1 Python dependencies on Modal - -Code running on Modal runs inside containers built from [container images](https://modal.com/docs/guide/images) -that include that code's dependencies. - -Because Modal images include [GPU drivers](https://modal.com/docs/guide/cuda) by default, -installation of higher-level packages like `chai_lab` that require GPUs is painless. - -Here, we do it with one line, using the `uv` package manager for extra speed. - -```python -image = ( - modal.Image.debian_slim(python_version="3.12") - .uv_pip_install( - "chai_lab==0.5.0", - "huggingface-hub==0.36.0", - ) - .uv_pip_install( - "torch==2.7.1", - index_url="https://download.pytorch.org/whl/cu128", - ) -) - -``` - -## Storing Chai-1 model weights on Modal with Volumes - -Not all "dependencies" belong in a container image. Chai-1, for example, depends on -the weights of several models. - -Rather than loading them dynamically at run-time (which would add several minutes of GPU time to each inference), -or installing them into the image (which would require they be re-downloaded any time the other dependencies changed), -we load them onto a [Modal Volume](https://modal.com/docs/guide/volumes). -A Modal Volume is a file system that all of your code running on Modal (or elsewhere!) can access. -For more on storing model weights on Modal, see [this guide](https://modal.com/docs/guide/model-weights). - -```python -chai_model_volume = ( - modal.Volume.from_name( # create distributed filesystem for model weights - "chai1-models", - create_if_missing=True, - ) -) -models_dir = Path("/models/chai1") - -``` - -The details of how we handle the download here (e.g. running concurrently for extra speed) -are in the [Addenda](#addenda). - -```python -image = image.env( # update the environment variables in the image to... - { - "CHAI_DOWNLOADS_DIR": str(models_dir), # point the chai code to it - "HF_XET_HIGH_PERFORMANCE": "1", # speed up downloads - } -) - -``` - -## Storing Chai-1 outputs on Modal Volumes - -Chai-1 produces its outputs by writing to disk -- -the model's scores for the structure and the structure itself along with rich metadata. - -But Modal is a *serverless* platform, and the filesystem your Modal Functions write to -is not persistent. Any file can be converted into bytes and sent back from a Modal Function -\-- and we mean any! You can send files that are gigabytes in size that way. -So we do that below. - -But for larger jobs, like folding every protein in the PDB, storing bytes on a local client -like a laptop won't cut it. - -So we again lean on Modal Volumes, which can store thousands of files each. -We attach a Volume to a Modal Function that runs Chai-1 and the inference code -saves the results to distributed storage, without any fuss or source code changes. - -```python -chai_preds_volume = modal.Volume.from_name("chai1-preds", create_if_missing=True) -preds_dir = Path("/preds") - -``` - -## Running Chai-1 on Modal - -Now we're ready to define a Modal Function that runs Chai-1. - -We put our function on Modal by wrapping it in a decorator, `@app.function`. -We provide that decorator with some arguments that describe the infrastructure our code needs to run: -the Volumes we created, the Image we defined, and of course a fast GPU! - -Note that Chai-1 takes a file path as input -- -specifically, a path to a file in the [FASTA format](https://en.wikipedia.org/wiki/FASTA_format). -We pass the file contents to the function as a string and save them to disk so they can be picked up by the inference code. - -Because Modal is serverless, we don't need to worry about cleaning up these resources: -the disk is ephemeral and the GPU only costs you money when you're using it. - -```python -@app.function( - timeout=15 * MINUTES, - gpu="H100", - volumes={models_dir: chai_model_volume, preds_dir: chai_preds_volume}, - image=image, -) -def chai1_inference( - fasta_content: str, inference_config: dict, run_id: str -) -> list[(bytes, str)]: - from pathlib import Path - - import torch - from chai_lab import chai1 - - N_DIFFUSION_SAMPLES = 5 # hard-coded in chai-1 - - fasta_file = Path("/tmp/inputs.fasta") - fasta_file.write_text(fasta_content.strip()) - - output_dir = Path("/preds") / run_id - - chai1.run_inference( - fasta_file=fasta_file, - output_dir=output_dir, - device=torch.device("cuda"), - **inference_config, - ) - - print( - f"🧬 done, results written to /{output_dir.relative_to('/preds')} on remote volume" - ) - - results = [] - for ii in range(N_DIFFUSION_SAMPLES): - scores = (output_dir / f"scores.model_idx_{ii}.npz").read_bytes() - cif = (output_dir / f"pred.model_idx_{ii}.cif").read_text() - - results.append((scores, cif)) - - return results - - -``` - -## Addenda - -Above, we glossed over just how we got hold of the model weights -- -the `local_entrypoint` just called a function named `download_inference_dependencies`. - -Here's that function's implementation. - -A few highlights: - -* This Modal Function can access the model weights Volume, like the inference Function, - but it can't access the model predictions Volume. - -* This Modal Function has a different Image (the default!) and doesn't use a GPU. Modal helps you - separate the concerns, and the costs, of your infrastructure's components. - -* We use the `async` keyword here so that we can run the download for each model file - as a separate task, concurrently. We don't need to worry about this use of `async` - spreading to the rest of our code -- Modal launches just this Function in an async runtime. - -```python -@app.function(volumes={models_dir: chai_model_volume}) -async def download_inference_dependencies(force=False): - import asyncio - - import aiohttp - - base_url = "https://chaiassets.com/chai1-inference-depencencies/" # sic - inference_dependencies = [ - "conformers_v1.apkl", - "models_v2/trunk.pt", - "models_v2/token_embedder.pt", - "models_v2/feature_embedding.pt", - "models_v2/diffusion_module.pt", - "models_v2/confidence_head.pt", - ] - - headers = { - "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3" - } - - # launch downloads concurrently - async with aiohttp.ClientSession(headers=headers) as session: - tasks = [] - for dep in inference_dependencies: - local_path = models_dir / dep - if force or not local_path.exists(): - url = base_url + dep - print(f"🧬 downloading {dep}") - tasks.append(download_file(session, url, local_path)) - - # run all of the downloads and await their completion - await asyncio.gather(*tasks) - - chai_model_volume.commit() # ensures models are visible on remote filesystem before exiting, otherwise takes a few seconds, racing with inference - - -async def download_file(session, url: str, local_path: Path): - async with session.get(url) as response: - response.raise_for_status() - local_path.parent.mkdir(parents=True, exist_ok=True) - with open(local_path, "wb") as f: - while chunk := await response.content.read(8192): - f.write(chunk) - -``` diff --git a/.agents/skills/modal/references/examples/chat_with_pdf_vision.md b/.agents/skills/modal/references/examples/chat_with_pdf_vision.md deleted file mode 100644 index dbf151f71e1e9c3658d88a9a276fb80a731aa3e0..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/chat_with_pdf_vision.md +++ /dev/null @@ -1,524 +0,0 @@ -# Chat with PDF: RAG with ColQwen2 - -In this example, we demonstrate how to use the the [ColQwen2](https://huggingface.co/vidore/colqwen2-v0.1) model to build a simple -"Chat with PDF" retrieval-augmented generation (RAG) app. -The ColQwen2 model is based on [ColPali](https://huggingface.co/blog/manu/colpali) but uses the -[Qwen2-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) vision-language model. -ColPali is in turn based on the late-interaction embedding approach pioneered in [ColBERT](https://dl.acm.org/doi/pdf/10.1145/3397271.3401075). - -Vision-language models with high-quality embeddings obviate the need for complex pre-processing pipelines. -See [this blog post from Jo Bergum of Vespa](https://blog.vespa.ai/announcing-colbert-embedder-in-vespa/) for more. - -## Setup - -First, we’ll import the libraries we need locally and define some constants. - -```python -from pathlib import Path -from typing import Optional -from urllib.request import urlopen -from uuid import uuid4 - -import modal - -MINUTES = 60 # seconds - -app = modal.App("example-chat-with-pdf-vision") - -``` - -## Setting up dependencies - -In Modal, we define [container images](https://modal.com/docs/guide/custom-container) that run our serverless workloads. -We install the packages required for our application in those images. - -```python -CACHE_DIR = "/hf-cache" - -model_image = ( - modal.Image.debian_slim(python_version="3.12") - .apt_install("git") - .uv_pip_install( - [ - "colpali-engine==0.3.5", - "transformers>=4.45.0", - "torch>=2.0.0", - "huggingface-hub==0.36.0", - "qwen-vl-utils==0.0.8", - "torchvision==0.19.1", - ] - ) - .env({"HF_XET_HIGH_PERFORMANCE": "1", "HF_HUB_CACHE": CACHE_DIR}) -) - -``` - -These dependencies are only installed remotely, so we can't import them locally. -Use the `.imports` context manager to import them only on Modal instead. - -```python -with model_image.imports(): - import torch - from colpali_engine.models import ColQwen2, ColQwen2Processor - from qwen_vl_utils import process_vision_info - from transformers import AutoProcessor, Qwen2VLForConditionalGeneration - -``` - -## Specifying the ColQwen2 model - -Vision-language models (VLMs) for embedding and generation add another layer of simplification -to RAG apps based on vector search: we only need one model. - -```python -MODEL_NAME = "Qwen/Qwen2-VL-2B-Instruct" -MODEL_REVISION = "aca78372505e6cb469c4fa6a35c60265b00ff5a4" - -``` - -## Managing state with Modal Volumes and Dicts - -Chat services are stateful: -the response to an incoming user message depends on past user messages in a session. - -RAG apps add even more state: -the documents being retrieved from and the index over those documents, -e.g. the embeddings. - -Modal Functions are stateless in and of themselves. -They don't retain information from input to input. -That's what enables Modal Functions to automatically scale up and down -[based on the number of incoming requests](https://modal.com/docs/guide/cold-start). - -### Managing chat sessions with Modal Dicts - -In this example, we use a [`modal.Dict`](https://modal.com/docs/guide/dicts-and-queues) -to store state information between Function calls. - -Modal Dicts behave similarly to Python dictionaries, -but they are backed by remote storage and accessible to all of your Modal Functions. -They can contain any Python object -that can be serialized using [`cloudpickle`](https://github.com/cloudpipe/cloudpickle). - -A Dict can hold a few gigabytes across keys of size up to 100 MiB, -so it works well for our chat session state, which is a few KiB per session, -and for our embeddings, which are a few hundred KiB per PDF page, -up to about 100,000 pages of PDFs. - -At a larger scale, we'd need to replace this with a database, like Postgres, -or push more state to the client. - -```python -sessions = modal.Dict.from_name("colqwen-chat-sessions", create_if_missing=True) - - -class Session: - def __init__(self): - self.images = None - self.messages = [] - self.pdf_embeddings = None - - -``` - -### Storing PDFs on a Modal Volume - -Images extracted from PDFs are larger than our session state or embeddings -\-- low tens of MiB per page. - -So we store them on a [Modal Volume](https://modal.com/docs/guide/volumes), -which can store terabytes (or more!) of data across tens of thousands of files. - -Volumes behave like a remote file system: -we read and write from them much like a local file system. - -```python -pdf_volume = modal.Volume.from_name("colqwen-chat-pdfs", create_if_missing=True) -PDF_ROOT = Path("/vol/pdfs/") - -``` - -### Caching the model weights - -We'll also use a Volume to cache the model weights. - -```python -cache_volume = modal.Volume.from_name("hf-hub-cache", create_if_missing=True) - - -``` - -Running this function will download the model weights to the cache volume. -Otherwise, the model weights will be downloaded on the first query. For more on storing model weights on Modal, see -[this guide](https://modal.com/docs/guide/model-weights). - -```python -@app.function( - image=model_image, volumes={CACHE_DIR: cache_volume}, timeout=20 * MINUTES -) -def download_model(): - from huggingface_hub import snapshot_download - - result = snapshot_download( - MODEL_NAME, - revision=MODEL_REVISION, - ignore_patterns=["*.pt", "*.bin"], # using safetensors - ) - print(f"Downloaded model weights to {result}") - - -``` - -## Defining a Chat with PDF service - -To deploy an autoscaling "Chat with PDF" vision-language model service on Modal, -we just need to wrap our Python logic in a [Modal App](https://modal.com/docs/guide/apps): - -It uses [Modal `@app.cls`](https://modal.com/docs/guide/lifecycle-functions) decorators -to organize the "lifecycle" of the app: -loading the model on container start (`@modal.enter`) and running inference on request (`@modal.method`). - -We include in the arguments to the `@app.cls` decorator -all the information about this service's infrastructure: -the container image, the remote storage, and the GPU requirements. - -```python -@app.cls( - image=model_image, - gpu="A100-80GB", - scaledown_window=10 * MINUTES, # spin down when inactive - volumes={"/vol/pdfs/": pdf_volume, CACHE_DIR: cache_volume}, -) -class Model: - @modal.enter() - def load_models(self): - self.colqwen2_model = ColQwen2.from_pretrained( - "vidore/colqwen2-v0.1", - torch_dtype=torch.bfloat16, - device_map="cuda:0", - ) - self.colqwen2_processor = ColQwen2Processor.from_pretrained( - "vidore/colqwen2-v0.1" - ) - self.qwen2_vl_model = Qwen2VLForConditionalGeneration.from_pretrained( - MODEL_NAME, - revision=MODEL_REVISION, - torch_dtype=torch.bfloat16, - ) - self.qwen2_vl_model.to("cuda:0") - self.qwen2_vl_processor = AutoProcessor.from_pretrained( - "Qwen/Qwen2-VL-2B-Instruct", trust_remote_code=True - ) - - @modal.method() - def index_pdf(self, session_id, target: bytes | list): - # We store concurrent user chat sessions in a modal.Dict - - # For simplicity, we assume that each user only runs one session at a time - - session = sessions.get(session_id) - if session is None: - session = Session() - - if isinstance(target, bytes): - images = convert_pdf_to_images.remote(target) - else: - images = target - - # Store images on a Volume for later retrieval - session_dir = PDF_ROOT / f"{session_id}" - session_dir.mkdir(exist_ok=True, parents=True) - for ii, image in enumerate(images): - filename = session_dir / f"{str(ii).zfill(3)}.jpg" - image.save(filename) - - # Generated embeddings from the image(s) - BATCH_SZ = 4 - pdf_embeddings = [] - batches = [images[i : i + BATCH_SZ] for i in range(0, len(images), BATCH_SZ)] - for batch in batches: - batch_images = self.colqwen2_processor.process_images(batch).to( - self.colqwen2_model.device - ) - pdf_embeddings += list(self.colqwen2_model(**batch_images).to("cpu")) - - # Store the image embeddings in the session, for later retrieval - session.pdf_embeddings = pdf_embeddings - - # Write embeddings back to the modal.Dict - sessions[session_id] = session - - @modal.method() - def respond_to_message(self, session_id, message): - session = sessions.get(session_id) - if session is None: - session = Session() - - pdf_volume.reload() # make sure we have the latest data - - images = (PDF_ROOT / str(session_id)).glob("*.jpg") - images = list(sorted(images, key=lambda p: int(p.stem))) - - # Nothing to chat about without a PDF! - if not images: - return "Please upload a PDF first" - elif session.pdf_embeddings is None: - return "Indexing PDF..." - - # RAG, Retrieval-Augmented Generation, is two steps: - - # _Retrieval_ of the most relevant data to answer the user's query - relevant_image = self.get_relevant_image(message, session, images) - - # _Generation_ based on the retrieved data - output_text = self.generate_response(message, session, relevant_image) - - # Update session state for future chats - append_to_messages(message, session, user_type="user") - append_to_messages(output_text, session, user_type="assistant") - sessions[session_id] = session - - return output_text - - # Retrieve the most relevant image from the PDF for the input query - def get_relevant_image(self, message, session, images): - import PIL - - batch_queries = self.colqwen2_processor.process_queries([message]).to( - self.colqwen2_model.device - ) - query_embeddings = self.colqwen2_model(**batch_queries) - - # This scores our query embedding against the image embeddings from index_pdf - scores = self.colqwen2_processor.score_multi_vector( - query_embeddings, session.pdf_embeddings - )[0] - - # Select the best matching image - max_index = max(range(len(scores)), key=lambda index: scores[index]) - return PIL.Image.open(images[max_index]) - - # Pass the query and retrieved image along with conversation history into the VLM for a response - def generate_response(self, message, session, image): - chatbot_message = get_chatbot_message_with_image(message, image) - query = self.qwen2_vl_processor.apply_chat_template( - [*session.messages, chatbot_message], - tokenize=False, - add_generation_prompt=True, - ) - image_inputs, _ = process_vision_info([chatbot_message]) - inputs = self.qwen2_vl_processor( - text=[query], - images=image_inputs, - padding=True, - return_tensors="pt", - ) - inputs = inputs.to("cuda:0") - - generated_ids = self.qwen2_vl_model.generate(**inputs, max_new_tokens=512) - generated_ids_trimmed = [ - out_ids[len(in_ids) :] - for in_ids, out_ids in zip(inputs.input_ids, generated_ids) - ] - output_text = self.qwen2_vl_processor.batch_decode( - generated_ids_trimmed, - skip_special_tokens=True, - clean_up_tokenization_spaces=False, - )[0] - return output_text - - -``` - -## Loading PDFs as images - -Vision-Language Models operate on images, not PDFs directly, -so we need to convert our PDFs into images first. - -We separate this from our indexing and chatting logic -- -we run on a different container with different dependencies. - -```python -pdf_image = ( - modal.Image.debian_slim(python_version="3.12") - .apt_install("poppler-utils") - .uv_pip_install("pdf2image==1.17.0", "pillow==10.4.0") -) - - -@app.function(image=pdf_image) -def convert_pdf_to_images(pdf_bytes): - from pdf2image import convert_from_bytes - - images = convert_from_bytes(pdf_bytes, fmt="jpeg") - return images - - -``` - -## Chatting with a PDF from the terminal - -Before deploying in a UI, we can test our service from the terminal. - -Just run - -```bash -modal run chat_with_pdf_vision.py -``` - -and optionally pass in a path to or URL of a PDF with the `--pdf-path` argument -and specify a question with the `--question` argument. - -Continue a previous chat by passing the session ID printed to the terminal at start -with the `--session-id` argument. - -```python -@app.local_entrypoint() -def main( - question: Optional[str] = None, - pdf_path: Optional[str] = None, - session_id: Optional[str] = None, -): - model = Model() - if session_id is None: - session_id = str(uuid4()) - print("Starting a new session with id", session_id) - - if pdf_path is None: - pdf_path = "https://arxiv.org/pdf/1706.03762" # all you need - - if pdf_path.startswith("http"): - pdf_bytes = urlopen(pdf_path).read() - else: - pdf_bytes = Path(pdf_path).read_bytes() - - print("Indexing PDF from", pdf_path) - model.index_pdf.remote(session_id, pdf_bytes) - else: - if pdf_path is not None: - raise ValueError("Start a new session to chat with a new PDF") - print("Resuming session with id", session_id) - - if question is None: - question = "What is this document about?" - - print("QUESTION:", question) - print(model.respond_to_message.remote(session_id, question)) - - -``` - -## A hosted Gradio interface - -With the [Gradio](https://gradio.app) library, we can create a simple web interface around our class in Python, -then use Modal to host it for anyone to try out. - -To deploy your own, run - -```bash -modal deploy chat_with_pdf_vision.py -``` - -and navigate to the URL that appears in your teriminal. -If you’re editing the code, use `modal serve` instead to see changes hot-reload. - -```python -web_image = pdf_image.uv_pip_install( - "fastapi[standard]==0.115.4", - "pydantic==2.9.2", - "starlette==0.41.2", - "gradio==4.44.1", - "pillow==10.4.0", - "gradio-pdf==0.0.15", - "pdf2image==1.17.0", -) - - -@app.function( - image=web_image, - # gradio requires sticky sessions - # so we limit the number of concurrent containers to 1 - # and allow it to scale to 1000 concurrent inputs - max_containers=1, -) -@modal.concurrent(max_inputs=100) -@modal.asgi_app() -def ui(): - import uuid - - import gradio as gr - from fastapi import FastAPI - from gradio.routes import mount_gradio_app - from gradio_pdf import PDF - from pdf2image import convert_from_path - - web_app = FastAPI() - - # Since this Gradio app is running from its own container, - # allowing us to run the inference service via .remote() methods. - model = Model() - - def upload_pdf(path, session_id): - if session_id == "" or session_id is None: - # Generate session id if new client - session_id = str(uuid.uuid4()) - - images = convert_from_path(path) - # Call to our remote inference service to index the PDF - model.index_pdf.remote(session_id, images) - - return session_id - - def respond_to_message(message, _, session_id): - # Call to our remote inference service to run RAG - return model.respond_to_message.remote(session_id, message) - - with gr.Blocks(theme="soft") as demo: - session_id = gr.State("") - - gr.Markdown("# Chat with PDF") - with gr.Row(): - with gr.Column(scale=1): - gr.ChatInterface( - fn=respond_to_message, - additional_inputs=[session_id], - retry_btn=None, - undo_btn=None, - clear_btn=None, - ) - with gr.Column(scale=1): - pdf = PDF( - label="Upload a PDF", - ) - pdf.upload(upload_pdf, [pdf, session_id], session_id) - - return mount_gradio_app(app=web_app, blocks=demo, path="/") - - -``` - -## Addenda - -The remainder of this code consists of utility functions and boiler plate used in the -main code above. - -```python -def get_chatbot_message_with_image(message, image): - return { - "role": "user", - "content": [ - {"type": "image", "image": image}, - {"type": "text", "text": message}, - ], - } - - -def append_to_messages(message, session, user_type="user"): - session.messages.append( - { - "role": user_type, - "content": {"type": "text", "text": message}, - } - ) - -``` diff --git a/.agents/skills/modal/references/examples/chatterbox_tts.md b/.agents/skills/modal/references/examples/chatterbox_tts.md deleted file mode 100644 index a1bfc8dcc22582f11027be3cc33b6e361c4a0b47..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/chatterbox_tts.md +++ /dev/null @@ -1,158 +0,0 @@ -# Create a Chatterbox TTS API on Modal - -This example demonstrates how to deploy a text-to-speech (TTS) API using the open source model Chatterbox Turbo on Modal. - -Chatterbox Turbo is a state-of-the-art TTS model that can generate natural, expressive speech that rivals proprietary models. -Prompts can include paralinguistic tags like `[chuckle]`, `[sigh]`, and `[gasp]`. Chatterbox also support voice cloning by passing -a short (about 10 seconds) audio prompt of the target voice. - -Check out [Resemble AI's website](https://www.resemble.ai/) or -the [Chatterbox Github](https://github.com/resemble-ai/chatterbox) repo for more details. - -## Setup - -Import `modal`, the only required local dependency. - -```python -import modal - -``` - -## Define a container image - -We start with Modal's baseline `debian_slim` image and install the required packages. - -* `chatterbox-tts`: The TTS model library -* `fastapi`: Web framework for creating the API endpoint -* "peft": Required for properly loading the model - -```python -image = modal.Image.debian_slim(python_version="3.10").uv_pip_install( - "chatterbox-tts==0.1.6", - "fastapi[standard]==0.124.4", - "peft==0.18.0", -) - -``` - -We'll also use Chatterbox's provided set of voice prompts which you can download [here](https://modal-cdn.com/blog/audio/chatterbox-tts-voices.zip). -Unzip the file and upload it to a `modal.Volume` called `chatterbox-tts-voices` with the following CLI commands: - -```shell -modal volume create chatterbox-tts-voices -modal volume put chatterbox-tts-voices -``` - -Now we can instantiate the volume and use it with our app. - -```python -chatterbox_tts_voices_vol = modal.Volume.from_name("chatterbox-tts-voices") -VOICE_PROMPTS_DIR = "/chatterbox-tts/prompts" - -app = modal.App("example-chatterbox-tts", image=image) - -``` - -Import the required libraries within the image context to ensure they're available -when the container runs. This includes audio processing modules and the Chatterbox TTS module itself. - -```python -with image.imports(): - import io - - import torchaudio as ta - from chatterbox.tts_turbo import ChatterboxTurboTTS - from fastapi.responses import StreamingResponse - -``` - -## The TTS model class - -The TTS service is implemented using Modal's class syntax with GPU acceleration. -We configure the class to use an A10G GPU with additional parameters: - -* `scaledown_window=60 * 5`: Keep containers alive for 5 minutes after last request -* `@modal.concurrent(max_inputs=10)`: Allow up to 10 concurrent requests per container - -We'll also need to provide a Hugging Face token using a `modal.Secret` to access the model weights, -and attach the `chatterbox-tts-voices` volume to the container. - -```python -@app.cls( - gpu="a10g", - scaledown_window=60 * 5, - secrets=[modal.Secret.from_name("hf-token")], - volumes={VOICE_PROMPTS_DIR: chatterbox_tts_voices_vol}, -) -@modal.concurrent(max_inputs=10) -class Chatterbox: - @modal.enter() - def load(self): - self.model = ChatterboxTurboTTS.from_pretrained(device="cuda") - - @modal.fastapi_endpoint(docs=True, method="POST") - def api_endpoint(self, prompt: str): - # Get the audio bytes from the generate method - audio_bytes = self.generate.local(prompt) - - # Return the audio as a streaming response with appropriate MIME type. - # This allows for browsers to playback audio directly. - return StreamingResponse( - io.BytesIO(audio_bytes), - media_type="audio/wav", - ) - - @modal.method() - def generate(self, prompt: str) -> bytes: - # Generate audio waveform from the input text - wav = self.model.generate( - prompt, - audio_prompt_path=VOICE_PROMPTS_DIR - + "/chatterbox-tts-voices" - + "/prompts" - + "/Lucy.wav", - ) - - # Convert the waveform to bytes - buffer = io.BytesIO() - ta.save(buffer, wav, self.model.sr, format="wav") - buffer.seek(0) - return buffer.read() - - -@app.local_entrypoint() -def test( - prompt: str = "Chatterbox running on Modal [chuckle].", - output_path: str = "/tmp/chatterbox-tts/output.wav", -): - chatterbox = Chatterbox() - audio_bytes = chatterbox.generate.remote(prompt=prompt) - - # Save the audio bytes to a file - import pathlib - - output_path = pathlib.Path(output_path) - output_path.parent.mkdir(parents=True, exist_ok=True) - output_path.write_bytes(audio_bytes) - print(f"Audio saved to {output_path}") - - -``` - -Now deploy the Chatterbox API from this file's directory: - -```shell -modal deploy -m 06_gpu_and_ml.text-to-audio.chatterbox_tts -``` - -And query the endpoint with: - -```shell -mkdir -p /tmp/chatterbox-tts # create tmp directory - -curl -X POST --get "" \ - --data-urlencode "prompt=Chatterbox running on Modal [chuckle]." \ - --output /tmp/chatterbox-tts/output.wav -``` - -You'll receive a WAV file named `/tmp/chatterbox-tts/output.wav` containing the generated audio. diff --git a/.agents/skills/modal/references/examples/claude-slack-gif-creator.md b/.agents/skills/modal/references/examples/claude-slack-gif-creator.md deleted file mode 100644 index ea09605f2912ae4efb06f3123bf0496a8d15c088..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/claude-slack-gif-creator.md +++ /dev/null @@ -1,198 +0,0 @@ -# Claude Slack GIF Creator - -

- GIF of a pelican riding a bicycle - GIF of an AGI party - GIF of Gongy shipping -

- -[This repo](https://github.com/modal-projects/claude-slack-gif-creator) -shows how to build -a bot powered by Claude that creates custom Slackmoji-ready GIFs. - -Or, in GIF form: - -![A bot powered by Claude that creates custom Slackmoji-ready GIFs](https://modal-cdn.com/claude-slack-gif-creator/claude-gif-gif.gif) - -The bot runs on [Modal](https://modal.com/) and uses the [Claude Agent SDK](https://platform.claude.com/docs/en/agent-sdk/overview) -with the [`slack-gif-creator` skill from Anthropic](https://github.com/anthropics/skills/). - -## Features - -* **Natural Language GIF Generation**: Describe what you want and Claude will create a 128x128 emoji-optimized GIF -* **Persistent Threads**: Each Slack thread creates a conversation context, persisted on Modal -* **Image Upload Support**: Upload images to the bot to incorporate them into your GIFs -* **Background Removal**: Backgrounds removed using the `rembg` tool, so you can make GIFs of your friends -* **Real-time Tool Logging**: See Claude's tool usage in the Slack thread as it works - -## Architecture - -The bot consists of three main components: -a Slack Bot Server, -a Claude Agent Sandbox, -and an Anthropic API Proxy. - -### Slack Bot Server - -This component handles Slack events (mentions and thread replies) and manages [Modal Sandboxes](https://modal.com/docs/guide/sandbox). -It's a simple [FastAPI ASGI app](https://modal.com/docs/guide/webhooks) hosted on Modal. - -### Claude Agent Sandbox - -This component runs a Claude client and executes Claude skills, -like Bash execution and GIF creation. - -Because these skills are tantamount to giving the agent total control over the computing environment -and we are going to allow anyone who can access the bot to prompt the agent, -we need to isolate and secure this component. -To that tend, it runs inside a Modal [Sandbox](https://modal.com/docs/guide/sandbox). -Modal can readily scale to [hundreds or thousands of Sandboxes](https://modal.com/blog/modal-vibe). - -Each Slack thread gets its own persistent [Modal Sandbox](https://modal.com/docs/guide/sandbox) with a dedicated [Volume](https://modal.com/docs/guide/volumes) for storing generated GIFs and session data. - -### Anthropic API Proxy - -This component proxies requests to the Anthropic API. - -The proxy keeps the API key out of the Sandbox. -It's included so that Claude can't leak your API key when -a naughty prompt hacker asks for a GIF containing it, -as in the (mock) example below. - -![Fake API keys revealed in a GIF](https://modal-cdn.com/claude-slack-gif-creator/mocked-pwn.gif) - -## Prerequisites - -* Python 3.10 or higher -* A [Modal](https://modal.com/) account -* A Slack workspace -* An Anthropic API key - -## Setup - -### 1. Install Dependencies - -```bash -pip install modal -``` - -That's it! - -If you've never used Modal before on this machine, also run - -```bash -modal setup -``` - -### 2. Configure Slack App - -[Create a new Slack app](https://api.slack.com/apps) in your workspace. - -Your Slack app needs: - -[**OAuth Scopes**](https://api.slack.com/scopes) - -* `app_mentions:read` -* `chat:write` -* `files:read` -* `files:write` -* `channels:history` -* `groups:history` -* `im:history` -* `mpim:history` - -[**Event Subscriptions**](https://api.slack.com/apis/connections/events-api): - -* `app_mention` -* `message.channels` -* `message.groups` -* `message.im` -* `message.mpim` - -### 3. Configure Modal Secrets - -Create two Modal [Secrets](https://modal.com/docs/guide/secrets): - -**anthropic-secret** with: - -* `ANTHROPIC_API_KEY`: Your Anthropic API key - -**claude-code-slackbot-secret** with: - -* `SLACK_BOT_TOKEN`: Your [Slack bot token](https://api.slack.com/authentication/token-types#bot) (starts with `xoxb-`) -* `SLACK_SIGNING_SECRET`: Your Slack app's [signing secret](https://api.slack.com/authentication/verifying-requests-from-slack#about) - -### 4. Deploy to Modal - -```bash -modal deploy src/main.py -``` - -After deployment, Modal will provide a webhook URL. Add this URL to your Slack app's [Event Subscriptions Request URL](https://api.slack.com/apis/connections/events-api#the-events-api__subscribing-to-event-types__events-api-request-urls). - -Finally, [install the app to your workspace](https://api.slack.com/start/quickstart#installing) and invite the bot to the channels where you want to use it. - -## Usage - -### Mention the Bot - -Mention the bot in any channel with a description of the GIF you want: - -> @GIFBot create a GIF of a pelican riding a bicycle - -![Pelican riding a bicycle](https://modal-cdn.com/claude-slack-gif-creator/claude-pelican-bicycle.gif) - -### Upload Images - -Attach images to your message for the bot to incorporate: - -> @GIFBot make a party GIF of this entity that flashes the letters "AGI" - -> \[attach image] - -![Are you feeling the AGI?](https://modal-cdn.com/claude-slack-gif-creator/agi-party.gif) - -### Background Removal - -Request background removal for transparent GIFs: - -> @GIFBot make a GIF of this guy riding on a boat - -> \[attach image with background] - -![Gongy ships](https://modal-cdn.com/claude-slack-gif-creator/gongy-ships.gif) - -### Thread Replies - -Reply to the bot's messages in a thread to continue the conversation: - -> @GIFBot make a GIF showing "A bot powered by Claude that creates custom Slackmoji-ready GIFs." on a screen - -> the text runs off the screen, fix the wrapping - -![A bot powered by Claude that creates custom Slackmoji-ready GIFs](https://modal-cdn.com/claude-slack-gif-creator/claude-gif-gif.gif) - -## How It Works - -1. User mentions the bot or replies in a thread -2. Slack sends an event to the Modal webhook -3. The bot creates or resumes a Modal sandbox for that thread -4. Images attached to the message are downloaded and uploaded to the sandbox -5. Claude Agent SDK runs inside the sandbox with the user's message -6. Claude uses the `slack-gif-creator` skill to generate the GIF -7. The generated GIF is uploaded back to the Slack thread -8. The sandbox remains alive for 20 minutes for follow-up requests - -## Debug Mode - -Set `DEBUG_TOOL_USE = True` in `src/main.py` to enable real-time tool logging in Slack threads. - -## Resources - -* [Modal Documentation](https://modal.com/docs) -* [Modal Sandboxes](https://modal.com/products/sandboxes) -* [Claude Agent SDK](https://github.com/anthropics/anthropic-sdk-python) -* [Slack API Documentation](https://api.slack.com/) -* [Slack Bolt Framework](https://slack.dev/bolt-python/) -* [Building Slack Apps](https://api.slack.com/start) -* [`slack-gif-creator` Skill](https://github.com/anthropics/skills/) diff --git a/.agents/skills/modal/references/examples/cloud_bucket_mount_loras.md b/.agents/skills/modal/references/examples/cloud_bucket_mount_loras.md deleted file mode 100644 index f437795c1956b4fdfae9bed9ba1d4778d0b9ea02..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/cloud_bucket_mount_loras.md +++ /dev/null @@ -1,368 +0,0 @@ -# LoRAs Galore: Create a LoRA Playground with Modal, Gradio, and S3 - -This example shows how to mount an S3 bucket in a Modal app using [`CloudBucketMount`](https://modal.com/docs/reference/modal.CloudBucketMount). -We will download a bunch of LoRA adapters from the [HuggingFace Hub](https://huggingface.co/models) into our S3 bucket -then read from that bucket, on the fly, when doing inference. - -By default, we use the [IKEA instructions LoRA](https://huggingface.co/ostris/ikea-instructions-lora-sdxl) as an example, -which produces the following image when prompted to generate "IKEA instructions for building a GPU rig for deep learning": - -![IKEA instructions for building a GPU rig for deep learning](./ikea-instructions-for-building-a-gpu-rig-for-deep-learning.png) - -By the end of this example, we've deployed a "playground" app where anyone with a browser can try -out these custom models. That's the power of Modal: custom, autoscaling AI applications, deployed in seconds. -You can try out our deployment [here](https://modal-labs-examples--example-cloud-bucket-mount-loras-ui.modal.run). - -## Basic setup - -```python -import io -import os -from pathlib import Path -from typing import Optional - -import modal - -``` - -You will need to have an S3 bucket and AWS credentials to run this example. Refer to the documentation -for the detailed [IAM permissions](https://modal.com/docs/guide/cloud-bucket-mounts#iam-permissions) those credentials will need. - -After you are done creating a bucket and configuring IAM settings, -you now need to create a [Modal Secret](https://modal.com/docs/guide/secrets). Navigate to the "Secrets" tab and -click on the AWS card, then fill in the fields with the AWS key and secret created -previously. Name the Secret `s3-bucket-secret`. - -```python -bucket_secret = modal.Secret.from_name( - "s3-bucket-secret", - required_keys=["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"], -) - -MOUNT_PATH: Path = Path("/mnt/bucket") -LORAS_PATH: Path = MOUNT_PATH / "loras/v5" - -BASE_MODEL = "stabilityai/stable-diffusion-xl-base-1.0" -CACHE_DIR = "/hf-cache" - -``` - -Modal runs serverless functions inside containers. -The environments those functions run in are defined by -the container `Image`. The line below constructs an image -with the dependencies we need -- no need to install them locally. - -```python -image = ( - modal.Image.debian_slim(python_version="3.12") - .uv_pip_install( - "huggingface_hub==0.21.4", - "transformers==4.38.2", - "diffusers==0.26.3", - "peft==0.9.0", - "accelerate==0.27.2", - ) - .env({"HF_HUB_CACHE": CACHE_DIR}) -) - -with image.imports(): - # we import these dependencies only inside the container - import diffusers - import huggingface_hub - import torch - -``` - -We attach the S3 bucket to all the Modal functions in this app by mounting it on the filesystem they see, -passing a `CloudBucketMount` to the `volumes` dictionary argument. We can read and write to this mounted bucket -(almost) as if it were a local directory. - -```python -app = modal.App( - "example-cloud-bucket-mount-loras", - image=image, - volumes={ - MOUNT_PATH: modal.CloudBucketMount( - "modal-s3mount-test-bucket", - secret=bucket_secret, - ) - }, -) - - -``` - -For the base model, we'll use a modal.Volume to store the Hugging Face cache. - -```python -cache_volume = modal.Volume.from_name("hf-hub-cache", create_if_missing=True) - - -@app.function(image=image, volumes={CACHE_DIR: cache_volume}) -def download_model(): - loc = huggingface_hub.snapshot_download(repo_id=BASE_MODEL) - print(f"Saved model to {loc}") - - -``` - -## Acquiring LoRA weights - -`search_loras()` will use the Hub API to search for LoRAs. We limit LoRAs -to a maximum size to avoid downloading very large model weights. -We went with 800 MiB, but feel free to adapt to what works best for you. - -```python -@app.function(secrets=[bucket_secret]) -def search_loras(limit: int, max_model_size: int = 1024 * 1024 * 1024): - api = huggingface_hub.HfApi() - - model_ids: list[str] = [] - for model in api.list_models( - tags=["lora", f"base_model:{BASE_MODEL}"], - library="diffusers", - sort="downloads", # sort by most downloaded - ): - try: - model_size = 0 - for file in api.list_files_info(model.id): - model_size += file.size - - except huggingface_hub.utils.GatedRepoError: - print(f"gated model ({model.id}); skipping") - continue - - # Skip models that are larger than file limit. - if model_size > max_model_size: - print(f"model {model.id} is too large; skipping") - continue - - model_ids.append(model.id) - if len(model_ids) >= limit: - return model_ids - - return model_ids - - -``` - -We want to take the LoRA weights we found and move them from Hugging Face onto S3, -where they'll be accessible, at short latency and high throughput, for our Modal functions. -Downloading files in this mount will automatically upload files to S3. -To speed things up, we will run this function in parallel using Modal's -[`map`](https://modal.com/docs/reference/modal.Function#map). - -```python -@app.function() -def download_lora(repository_id: str) -> Optional[str]: - os.environ["HF_HUB_DISABLE_SYMLINKS_WARNING"] = "1" - - # CloudBucketMounts will report 0 bytes of available space leading to many - # unnecessary warnings, so we patch the method that emits those warnings. - from huggingface_hub import file_download - - file_download._check_disk_space = lambda x, y: False - - repository_path = LORAS_PATH / repository_id - try: - # skip models we've already downloaded - if not repository_path.exists(): - huggingface_hub.snapshot_download( - repository_id, - local_dir=repository_path.as_posix().replace(".", "_"), - allow_patterns=["*.safetensors"], - ) - downloaded_lora = len(list(repository_path.rglob("*.safetensors"))) > 0 - except OSError: - downloaded_lora = False - except FileNotFoundError: - downloaded_lora = False - if downloaded_lora: - return repository_id - else: - return None - - -``` - -## Inference with LoRAs - -We define a `StableDiffusionLoRA` class to organize our inference code. -We load Stable Diffusion XL 1.0 as a base model, then, when doing inference, -we load whichever LoRA the user specifies from the S3 bucket. -For more on the decorators we use on the methods below to speed up building and booting, -check out the [container lifecycle hooks guide](https://modal.com/docs/guide/lifecycle-functions). - -```python -@app.cls( - gpu="a10g", # A10G GPUs are great for inference - volumes={CACHE_DIR: cache_volume}, # We cache the base model -) -class StableDiffusionLoRA: - @modal.enter() # when a new container starts, we load the base model into the GPU - def load(self): - self.pipe = diffusers.DiffusionPipeline.from_pretrained( - BASE_MODEL, torch_dtype=torch.float16 - ).to("cuda") - - @modal.method() # at inference time, we pull in the LoRA weights and pass the final model the prompt - def run_inference_with_lora( - self, lora_id: str, prompt: str, seed: int = 8888 - ) -> bytes: - for file in (LORAS_PATH / lora_id).rglob("*.safetensors"): - self.pipe.load_lora_weights(lora_id, weight_name=file.name) - break - - lora_scale = 0.9 - image = self.pipe( - prompt, - num_inference_steps=10, - cross_attention_kwargs={"scale": lora_scale}, - generator=torch.manual_seed(seed), - ).images[0] - - buffer = io.BytesIO() - image.save(buffer, format="PNG") - - return buffer.getvalue() - - -``` - -## Try it locally! - -To use our inference code from our local command line, we add a `local_entrypoint` to our `app`. -Run it using `modal run cloud_bucket_mount_loras.py`, and pass `--help` -to see the available options. - -The inference code will run on our machines, but the results will be available on yours. - -```python -@app.local_entrypoint() -def main( - limit: int = 100, - example_lora: str = "ostris/ikea-instructions-lora-sdxl", - prompt: str = "IKEA instructions for building a GPU rig for deep learning", - seed: int = 8888, -): - # Download LoRAs in parallel. - lora_model_ids = [example_lora] - lora_model_ids += search_loras.remote(limit) - - downloaded_loras = [] - for model in download_lora.map(lora_model_ids): - if model: - downloaded_loras.append(model) - - print(f"downloaded {len(downloaded_loras)} loras => {downloaded_loras}") - - # Run inference using one of the downloaded LoRAs. - byte_stream = StableDiffusionLoRA().run_inference_with_lora.remote( - example_lora, prompt, seed - ) - dir = Path("/tmp/stable-diffusion-xl") - if not dir.exists(): - dir.mkdir(exist_ok=True, parents=True) - - output_path = dir / f"{as_slug(prompt.lower())}.png" - print(f"Saving it to {output_path}") - with open(output_path, "wb") as f: - f.write(byte_stream) - - -``` - -## LoRA Exploradora: A hosted Gradio interface - -Command line tools are cool, but we can do better! -With the Gradio library by Hugging Face, we can create a simple web interface -around our Python inference function, then use Modal to host it for anyone to try out. - -To set up your own, run `modal deploy cloud_bucket_mount_loras.py` and navigate to the URL it prints out. -If you're playing with the code, use `modal serve` instead to see changes live. - -```python -web_image = modal.Image.debian_slim(python_version="3.12").uv_pip_install( - "fastapi[standard]==0.115.4", - "gradio~=5.7.1", - "pillow~=10.2.0", -) - - -@app.function( - image=web_image, - scaledown_window=60 * 20, - # gradio requires sticky sessions - # so we limit the number of concurrent containers to 1 - # and allow it to scale to 100 concurrent inputs - max_containers=1, -) -@modal.concurrent(max_inputs=100) -@modal.asgi_app() -def ui(): - """A simple Gradio interface around our LoRA inference.""" - import io - - import gradio as gr - from fastapi import FastAPI - from gradio.routes import mount_gradio_app - from PIL import Image - - # determine which loras are available - lora_ids = [ - f"{lora_dir.parent.stem}/{lora_dir.stem}" for lora_dir in LORAS_PATH.glob("*/*") - ] - - # pick one to be default, set a default prompt - default_lora_id = ( - "ostris/ikea-instructions-lora-sdxl" - if "ostris/ikea-instructions-lora-sdxl" in lora_ids - else lora_ids[0] - ) - default_prompt = ( - "IKEA instructions for building a GPU rig for deep learning" - if default_lora_id == "ostris/ikea-instructions-lora-sdxl" - else "text" - ) - - # the simple path to making an app on Gradio is an Interface: a UI wrapped around a function. - def go(lora_id: str, prompt: str, seed: int) -> Image: - return Image.open( - io.BytesIO( - StableDiffusionLoRA().run_inference_with_lora.remote( - lora_id, prompt, seed - ) - ), - ) - - iface = gr.Interface( - go, - inputs=[ # the inputs to go/our inference function - gr.Dropdown(choices=lora_ids, value=default_lora_id, label="👉 LoRA ID"), - gr.Textbox(default_prompt, label="🎨 Prompt"), - gr.Number(value=8888, label="🎲 Random Seed"), - ], - outputs=gr.Image(label="Generated Image"), - # some extra bits to make it look nicer - title="LoRAs Galore", - description="# Try out some of the top custom SDXL models!" - "\n\nPick a LoRA finetune of SDXL from the dropdown, then prompt it to generate an image." - "\n\nCheck out [the code on GitHub](https://github.com/modal-labs/modal-examples/blob/main/10_integrations/cloud_bucket_mount_loras.py)" - " if you want to create your own version or just see how it works." - "\n\nPowered by [Modal](https://modal.com) 🚀", - theme="soft", - allow_flagging="never", - ) - - return mount_gradio_app(app=FastAPI(), blocks=iface, path="/") - - -def as_slug(name): - """Converts a string, e.g. a prompt, into something we can use as a filename.""" - import re - - s = str(name).strip().replace(" ", "-") - s = re.sub(r"(?u)[^-\w.]", "", s) - return s - -``` diff --git a/.agents/skills/modal/references/examples/cron_datasette.md b/.agents/skills/modal/references/examples/cron_datasette.md deleted file mode 100644 index a3b7675972598ec9db00fbe19dcde25b064c569d..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/cron_datasette.md +++ /dev/null @@ -1,424 +0,0 @@ -# Publish interactive datasets with Datasette - -![Datasette user interface](https://modal-cdn.com/cdnbot/imdb_datasetteqzaj3q9d_a83d82fd.webp) - -Build and deploy an interactive movie database that automatically updates daily with the latest IMDb data. -This example shows how to serve a Datasette application on Modal with millions of movie and TV show records. - -Try it out for yourself [here](https://modal-labs-examples--example-cron-datasette-ui.modal.run). - -Along the way, we will learn how to use the following Modal features: - -* [Volumes](https://modal.com/docs/guide/volumes): a persisted volume lets us store and grow the published dataset over time. - -* [Scheduled functions](https://modal.com/docs/guide/cron): the underlying dataset is refreshed daily, so we schedule a function to run daily. - -* [Web Functions](https://modal.com/docs/guide/webhooks): exposes the Datasette application for web browser interaction and API requests. - -## Basic setup - -Let's get started writing code. -For the Modal container image we need a few Python packages. - -```python -import asyncio -import gzip -import pathlib -import shutil -import tempfile -from datetime import datetime -from urllib.request import urlretrieve - -import modal - -app = modal.App("example-cron-datasette") -cron_image = modal.Image.debian_slim(python_version="3.12").uv_pip_install( - "datasette==0.65.1", "sqlite-utils==3.38", "tqdm~=4.67.1", "setuptools<80" -) - -``` - -## Persistent dataset storage - -To separate database creation and maintenance from serving, we'll need the underlying -database file to be stored persistently. To achieve this we use a -[Volume](https://modal.com/docs/guide/volumes). - -```python -volume = modal.Volume.from_name( - "example-cron-datasette-cache-vol", create_if_missing=True -) -DB_FILENAME = "imdb.db" -VOLUME_DIR = "/cache-vol" -DATA_DIR = pathlib.Path(VOLUME_DIR, "imdb-data") -DB_PATH = pathlib.Path(VOLUME_DIR, DB_FILENAME) - -``` - -## Getting a dataset - -[IMDb Datasets](https://datasets.imdbws.com/) are available publicly and are updated daily. -We will download the title.basics.tsv.gz file which contains basic information about all titles (movies, TV shows, etc.). -Since we are serving an interactive database which updates daily, we will download the files into a temporary directory and then move them to the volume to prevent downtime. - -```python -BASE_URL = "https://datasets.imdbws.com/" -IMDB_FILES = [ - "title.basics.tsv.gz", -] - - -@app.function( - image=cron_image, - volumes={VOLUME_DIR: volume}, - retries=2, - timeout=1800, -) -def download_dataset(force_refresh=False): - """Download IMDb dataset files.""" - if DATA_DIR.exists() and not force_refresh: - print( - f"Dataset already present and force_refresh={force_refresh}. Skipping download." - ) - return - - TEMP_DATA_DIR = pathlib.Path(VOLUME_DIR, "imdb-data-temp") - if TEMP_DATA_DIR.exists(): - shutil.rmtree(TEMP_DATA_DIR) - - TEMP_DATA_DIR.mkdir(parents=True, exist_ok=True) - - print("Downloading IMDb dataset...") - - try: - for filename in IMDB_FILES: - print(f"Downloading {filename}...") - url = BASE_URL + filename - output_path = TEMP_DATA_DIR / filename - - urlretrieve(url, output_path) - print(f"Successfully downloaded {filename}") - - if DATA_DIR.exists(): - # move the current data to a backup location - OLD_DATA_DIR = pathlib.Path(VOLUME_DIR, "imdb-data-old") - if OLD_DATA_DIR.exists(): - shutil.rmtree(OLD_DATA_DIR) - shutil.move(DATA_DIR, OLD_DATA_DIR) - - # move the new data into place - shutil.move(TEMP_DATA_DIR, DATA_DIR) - - # clean up the old data - shutil.rmtree(OLD_DATA_DIR) - else: - shutil.move(TEMP_DATA_DIR, DATA_DIR) - - volume.commit() - print("Finished downloading dataset.") - - except Exception as e: - print(f"Error during download: {e}") - if TEMP_DATA_DIR.exists(): - shutil.rmtree(TEMP_DATA_DIR) - raise - - -``` - -## Data processing - -This dataset is no swamp, but a bit of data cleaning is still in order. -The following function reads a .tsv file, cleans the data and yields batches of records. - -```python -def parse_tsv_file(filepath, batch_size=50000, filter_year=None): - """Parse a gzipped TSV file and yield batches of records.""" - import csv - - with gzip.open(filepath, "rt", encoding="utf-8") as gz_file: - reader = csv.DictReader(gz_file, delimiter="\t") - batch = [] - total_processed = 0 - - for row in reader: - # map missing values to None - row = {k: (None if v == "\\N" else v) for k, v in row.items()} - - # remove nsfw data - if row.get("isAdult") == "1": - continue - - if filter_year: - start_year = int(row.get("startYear", 0) or 0) - if start_year < filter_year: - continue - - batch.append(row) - total_processed += 1 - - if len(batch) >= batch_size: - yield batch - batch = [] - - # Yield any remaining records - if batch: - yield batch - - print(f"Finished processing {total_processed:,} titles.") - - -``` - -## Inserting into SQLite - -With the TSV processing out of the way, we’re ready to create a SQLite database and feed data into it. - -Importantly, the `prep_db` function mounts the same volume used by `download_dataset`, and rows are batch inserted with progress logged after each batch, -as the full IMDb dataset has millions of rows and does take some time to be fully inserted. - -A more sophisticated implementation would only load new data instead of performing a full refresh, -but we’re keeping things simple for this example! -We will also create indexes for the titles table to speed up queries. - -```python -@app.function( - image=cron_image, - volumes={VOLUME_DIR: volume}, - timeout=900, -) -def prep_db(filter_year=None): - """Process IMDb data files and create SQLite database.""" - import sqlite_utils - import tqdm - - volume.reload() - - # Create database in a temporary directory first - with tempfile.TemporaryDirectory() as tmpdir: - tmpdir_path = pathlib.Path(tmpdir) - tmp_db_path = tmpdir_path / DB_FILENAME - - db = sqlite_utils.Database(tmp_db_path) - - # Process title.basics.tsv.gz - titles_file = DATA_DIR / "title.basics.tsv.gz" - - if titles_file.exists(): - titles_table = db["titles"] - batch_count = 0 - total_processed = 0 - - with tqdm.tqdm(desc="Processing titles", unit="batch", leave=True) as pbar: - for i, batch in enumerate( - parse_tsv_file( - titles_file, batch_size=50000, filter_year=filter_year - ) - ): - titles_table.insert_all(batch, batch_size=50000, truncate=(i == 0)) - batch_count += len(batch) - total_processed += len(batch) - pbar.update(1) - pbar.set_postfix({"titles": f"{total_processed:,}"}) - - print(f"Total titles in database: {batch_count:,}") - - # Create indexes for titles so we can query the database faster - print("Creating indexes...") - titles_table.create_index(["tconst"], if_not_exists=True, unique=True) - titles_table.create_index(["primaryTitle"], if_not_exists=True) - titles_table.create_index(["titleType"], if_not_exists=True) - titles_table.create_index(["startYear"], if_not_exists=True) - titles_table.create_index(["genres"], if_not_exists=True) - print("Created indexes for titles table") - - db.close() - - # Copy the database to the volume - DB_PATH.parent.mkdir(parents=True, exist_ok=True) - shutil.copyfile(tmp_db_path, DB_PATH) - - print("Syncing DB with volume.") - volume.commit() - print("Volume changes committed.") - - -``` - -## Keep it fresh - -IMDb updates their data daily, so we set up -a [scheduled](https://modal.com/docs/guide/cron) function to automatically refresh the database -every 24 hours. - -```python -@app.function(schedule=modal.Period(hours=24), timeout=4000) -def refresh_db(): - """Scheduled function to refresh the database daily.""" - print(f"Running scheduled refresh at {datetime.now()}") - download_dataset.remote(force_refresh=True) - prep_db.remote() - - -``` - -## Web Function - -Hooking up the SQLite database to a Modal Function is as simple as it gets. -The Modal `@asgi_app` decorator wraps a few lines of code: one `import` and a couple more -that instantiate the `Datasette` instance and return its app server. - -First, let's define a metadata object for the database. -This will be used to configure Datasette to display a custom UI with some pre-defined queries. - -```python -columns = { - "tconst": "Unique identifier", - "titleType": "Type (movie, tvSeries, short, etc.)", - "primaryTitle": "Main title", - "originalTitle": "Original language title", - "startYear": "Release year", - "endYear": "End year (for TV series)", - "runtimeMinutes": "Runtime in minutes", - "genres": "Comma-separated genres", -} - -queries = { - "movies_2024": { - "sql": """ - SELECT - primaryTitle as title, - genres, - runtimeMinutes as runtime - FROM titles - WHERE titleType = 'movie' - AND startYear = 2024 - ORDER BY primaryTitle - LIMIT 100 - """, - "title": "Movies Released in 2024", - }, - "longest_movies": { - "sql": """ - SELECT - primaryTitle as title, - startYear as year, - runtimeMinutes as runtime, - genres - FROM titles - WHERE titleType = 'movie' - AND runtimeMinutes IS NOT NULL - AND runtimeMinutes > 180 - ORDER BY runtimeMinutes DESC - LIMIT 50 - """, - "title": "Longest Movies (3+ hours)", - }, - "genre_breakdown": { - "sql": """ - SELECT - genres, - COUNT(*) as count - FROM titles - WHERE titleType = 'movie' - AND genres IS NOT NULL - GROUP BY genres - ORDER BY count DESC - LIMIT 25 - """, - "title": "Popular Genres", - }, -} - - -metadata = { - "title": "IMDb Database Explorer", - "description": "Explore IMDb movie and TV show data", - "databases": { - "imdb": { - "tables": { - "titles": { - "description": "Basic information about all titles (movies, TV shows, etc.)", - "columns": columns, - } - }, - "queries": { - "movies_2024": queries["movies_2024"], - "longest_movies": queries["longest_movies"], - "genre_breakdown": queries["genre_breakdown"], - }, - } - }, -} - -``` - -Now we can define the Web Function that will serve the Datasette application - -```python -@app.function( - image=cron_image, - volumes={VOLUME_DIR: volume}, -) -@modal.concurrent(max_inputs=16) -@modal.asgi_app() -def ui(): - """Web Function backing the Datasette UI.""" - from datasette.app import Datasette - - ds = Datasette( - files=[DB_PATH], - settings={ - "sql_time_limit_ms": 60000, - "max_returned_rows": 10000, - "allow_download": True, - "facet_time_limit_ms": 5000, - "allow_facet": True, - }, - metadata=metadata, - ) - asyncio.run(ds.invoke_startup()) - return ds.app() - - -``` - -## Publishing to the web - -Run this script using `modal run cron_datasette.py` and it will create the database under 5 minutes! - -If you would like to force a refresh of the dataset, you can use: - -`modal run cron_datasette.py --force-refresh` - -If you would like to filter the data to be after a specific year, you can use: - -`modal run cron_datasette.py --filter-year year` - -You can then use `modal serve cron_datasette.py` to create a short-lived web URL -that exists until you terminate the script. - -When publishing the interactive Datasette app you'll want to create a persistent URL. -Just run `modal deploy cron_datasette.py` and your app will be deployed in seconds! - -```python -@app.local_entrypoint() -def run(force_refresh: bool = False, filter_year: int = None): - if force_refresh: - print("Force refreshing the dataset...") - - if filter_year: - print(f"Filtering data to be after {filter_year}") - - print("Downloading IMDb dataset...") - download_dataset.remote(force_refresh=force_refresh) - print("Processing data and creating SQLite DB...") - prep_db.remote(filter_year=filter_year) - print("\nDatabase ready! You can now run:") - print(" modal serve cron_datasette.py # For development") - print(" modal deploy cron_datasette.py # For production deployment") - - -``` - -You can explore the data at the [deployed Web Function](https://modal-labs-examples--example-cron-datasette-ui.modal.run). diff --git a/.agents/skills/modal/references/examples/db_to_sheet.md b/.agents/skills/modal/references/examples/db_to_sheet.md deleted file mode 100644 index 48c48bcce77616920d81cd53498eb425947f7b6c..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/db_to_sheet.md +++ /dev/null @@ -1,281 +0,0 @@ -# Write to Google Sheets from Postgres - -In this tutorial, we'll show how to use Modal to schedule a daily report in a spreadsheet on Google Sheets -that combines data from a PostgreSQL database with data from an external API. - -In particular, we'll extract the city of each user from the database, look up the current weather in that city, -and then build a count/histogram of how many users are experiencing each type of weather. - -## Entering credentials - -We begin by setting up some credentials that we'll need in order to access our database and output -spreadsheet. To do that in a secure manner, we log in to our Modal account on the web and go to -the [Secrets](https://modal.com/secrets) section. - -### Database - -First we will enter our database credentials. The easiest way to do this is to click **New -secret** and select the **Postgres compatible** Secret preset and fill in the requested -information. Then we press **Next** and name our Secret `postgres-secret` and click **Create**. - -### Google Sheets/GCP - -We'll now add another Secret for Google Sheets access through Google Cloud Platform. Click **New -secret** and select the Google Sheets preset. - -In order to access the Google Sheets API, we'll need to create a *Service Account* in Google Cloud -Platform. You can skip this step if you already have a Service Account json file. - -1. Sign up to Google Cloud Platform or log in if you haven't - (). - -2. Go to . - -3. In the navigation pane on the left, go to **IAM & Admin** > **Service Accounts**. - -4. Click the **+ CREATE SERVICE ACCOUNT** button. - -5. Give the service account a suitable name, like "sheet-access-bot". Click **Done**. You don't - have to grant it any specific access privileges at this time. - -6. Click your new service account in the list view that appears and navigate to the **Keys** - section. - -7. Click **Add key** and choose **Create new key**. Use the **JSON** key type and confirm by - clicking **Create**. - -8. A json key file should be downloaded to your computer at this point. Copy the contents of that - file and use it as the value for the `SERVICE_ACCOUNT_JSON` field in your new secret. - -We'll name this other Secret `"gsheets-secret"`. - -Now you can access the values of your Secrets from Modal Functions that you annotate with the -corresponding `modal.Secret`s, e.g.: - -```python -import os - -import modal - -app = modal.App("example-db-to-sheet") - - -@app.function(secrets=[modal.Secret.from_name("postgres-secret")]) -def show_host(): - # automatically filled from the specified secret - print("Host is " + os.environ["PGHOST"]) - - -``` - -Because these Secrets are Python objects, you can construct and manipulate them in your code. -We'll do that below by defining a variable to hold our Secret for accessing Postgres - -You can additionally specify - -```python -pg_secret = modal.Secret.from_name( - "postgres-secret", - required_keys=["PGHOST", "PGPORT", "PGDATABASE", "PGUSER", "PGPASSWORD"], -) - - -``` - -In order to connect to the database, we'll use the `psycopg2` Python package. To make it available -to your Modal Function you need to supply it with an `image` argument that tells Modal how to -build the container image that contains that package. We'll base it off of the `Image.debian_slim` base -image that's built into Modal, and make sure to install the required binary packages as well as -the `psycopg2` package itself: - -```python -pg_image = ( - modal.Image.debian_slim(python_version="3.11") - .apt_install("libpq-dev") - .uv_pip_install("psycopg2~=2.9.9") -) - -``` - -Since the default keynames for a **Postgres compatible** secret correspond to the environment -variables that `psycopg2` looks for, we can now easily connect to the database even without -explicit credentials in your code. We'll create a simple function that queries the city for each -user in the `users` table. - -```python -@app.function(image=pg_image, secrets=[pg_secret]) -def get_db_rows(verbose=True): - import psycopg2 - - conn = psycopg2.connect() # no explicit credentials needed - cur = conn.cursor() - cur.execute("SELECT city FROM users") - results = [row[0] for row in cur.fetchall()] - if verbose: - print(results) - return results - - -``` - -Note that we import `psycopg2` inside our function instead of the global scope. This allows us to -run this Modal Function even from an environment where `psycopg2` is not installed. We can test run -this function using the `modal run` shell command: `modal run db_to_sheet.py::app.get_db_rows`. - -To run this function, make sure there is a table called `users` in your database with a column called `city`. -You can populate the table with some example data using the following SQL commands: - -```sql -CREATE TABLE users (city TEXT); -INSERT INTO users VALUES ('Stockholm,,Sweden'); -INSERT INTO users VALUES ('New York,NY,USA'); -INSERT INTO users VALUES ('Tokyo,,Japan'); -``` - -## Applying Python logic - -For each row in our source data we'll run an online lookup of the current weather using the - API. To do this, we'll add the API key to -another Modal Secret. We'll use a custom secret called "weather-secret" with the key -`OPENWEATHER_API_KEY` containing our API key for OpenWeatherMap. - -```python -requests_image = modal.Image.debian_slim(python_version="3.11").uv_pip_install( - "requests~=2.31.0" -) - - -@app.function( - image=requests_image, - secrets=[ - modal.Secret.from_name("weather-secret", required_keys=["OPENWEATHER_API_KEY"]) - ], -) -def city_weather(city): - import requests - - url = "https://api.openweathermap.org/data/2.5/weather" - params = {"q": city, "appid": os.environ["OPENWEATHER_API_KEY"]} - response = requests.get(url, params=params) - weather_label = response.json()["weather"][0]["main"] - return weather_label - - -``` - -We'll make use of Modal's built-in `function.map` method to create our report. `function.map` -makes it really easy to parallelize work by executing a Function on every element in a sequence of -data. For this example we'll just do a simple count of rows per weather type -- -answering the question "how many of our users are experiencing each type of weather?". - -```python -from collections import Counter - - -@app.function() -def create_report(cities): - # run city_weather for each city in parallel - user_weather = city_weather.map(cities) - count_users_by_weather = Counter(user_weather).items() - return count_users_by_weather - - -``` - -Let's try to run this! To make it simple to trigger the function with some -predefined input data, we create a "local entrypoint" that can be -run from the command line with - -```bash -modal run db_to_sheet.py -``` - -```python -@app.local_entrypoint() -def main(): - cities = [ - "Stockholm,,Sweden", - "New York,NY,USA", - "Tokyo,,Japan", - ] - print(create_report.remote(cities)) - - -``` - -Running the local entrypoint using `modal run db_to_sheet.py` should print something like: -`dict_items([('Clouds', 3)])`. -Note that since this file only has a single app, and the app has only one local entrypoint -we only have to specify the file to run it - the function/entrypoint is inferred. - -In this case the logic is quite simple, but in a real world context you could have applied a -machine learning model or any other tool you could build into a container to transform the data. - -## Sending output to a Google Sheet - -We'll set up a new Google Sheet to send our report to. Using the "Sharing" dialog in Google -Sheets, share the document to the service account's email address (the value of the `client_email` field in the json file) -and make the service account an editor of the document. - -You may also need to enable the Google Sheets API for your project in the Google Cloud Platform console. -If so, the URL will be printed inside the message of a 403 Forbidden error when you run the function. -It begins with https://console.developers.google.com/apis/api/sheets.googleapis.com/overview. - -Lastly, we need to point our code to the correct Google Sheet. We'll need the *key* of the document. -You can find the key in the URL of the Google Sheet. It appears after the `/d/` in the URL, like: -`https://docs.google.com/spreadsheets/d/1wOktal......IJR77jD8Do`. - -We'll make use of the `pygsheets` python package to authenticate with -Google Sheets and then update the spreadsheet with information from the report we just created: - -```python -pygsheets_image = modal.Image.debian_slim(python_version="3.11").uv_pip_install( - "pygsheets~=2.0.6" -) - - -@app.function( - image=pygsheets_image, - secrets=[ - modal.Secret.from_name("gsheets-secret", required_keys=["SERVICE_ACCOUNT_JSON"]) - ], -) -def update_sheet_report(rows): - import pygsheets - - gc = pygsheets.authorize(service_account_env_var="SERVICE_ACCOUNT_JSON") - document_key = "1JxhGsht4wltyPFFOd2hP0eIv6lxZ5pVxJN_ZwNT-l3c" - sh = gc.open_by_key(document_key) - worksheet = sh.sheet1 - worksheet.clear("A2") - - worksheet.update_values("A2", [list(row) for row in rows]) - - -``` - -At this point, we have everything we need in order to run the full program. We can put it all together in -another Modal Function, and add a [`schedule`](https://modal.com/docs/guide/cron) argument so it runs every day automatically: - -```python -@app.function(schedule=modal.Period(days=1)) -def db_to_sheet(): - rows = get_db_rows.remote() - report = create_report.remote(rows) - update_sheet_report.remote(report) - print("Updated sheet with new weather distribution") - for weather, count in report: - print(f"{weather}: {count}") - - -``` - -This entire app can now be deployed using `modal deploy db_to_sheet.py`. The [apps page](https://modal.com/apps) -shows our cron job's execution history and lets you navigate to each invocation's logs. -To trigger a manual run from your local code during development, you can also trigger this function using the cli: -`modal run db_to_sheet.py::db_to_sheet` - -Note that all of the `@app.function()` annotated functions above run remotely in isolated containers that are specified per -function, but they are called as seamlessly as if we were using regular Python functions. This is a simple -showcase of how you can mix and match Modal Functions that use different environments and have them feed -into each other or even call each other as if they were all functions in the same local program. diff --git a/.agents/skills/modal/references/examples/dbt_duckdb.md b/.agents/skills/modal/references/examples/dbt_duckdb.md deleted file mode 100644 index 6d2f1133b4f5af249c5494459a1af0736a1d6d5a..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/dbt_duckdb.md +++ /dev/null @@ -1,300 +0,0 @@ -# Build your own data warehouse with DuckDB, DBT, and Modal - -This example contains a minimal but capable [data warehouse](https://en.wikipedia.org/wiki/Data_warehouse). -It's comprised of the following: - -* [DuckDB](https://duckdb.org) as the warehouse's [OLAP](https://en.wikipedia.org/wiki/Online_analytical_processing) database engine - -* [AWS S3](https://aws.amazon.com/s3/) as the data storage provider - -* [DBT](https://docs.getdbt.com/docs/introduction) as the data transformation tool - -Meet your new serverless cloud data warehouse, powered by Modal! - -## Configure Modal, S3, and DBT - -The only thing in the source code that you must update is the S3 bucket name. -AWS S3 bucket names are globally unique, and the one in this source is used by us to host this example. - -Update the `BUCKET_NAME` variable below and also any references to the original value -within `sample_proj_duckdb_s3/models/`. The AWS IAM policy below also includes the bucket -name and that must be updated. - -```python -from pathlib import Path - -import modal - -BUCKET_NAME = "modal-example-dbt-duckdb-s3" -LOCAL_DBT_PROJECT = ( # local path - Path(__file__).parent / "sample_proj_duckdb_s3" -) -PROJ_PATH = "/root/dbt" # remote paths -PROFILES_PATH = "/root/dbt_profile" -TARGET_PATH = "/root/target" -``` - -Most of the DBT code and configuration is taken directly from the classic -[Jaffle Shop](https://github.com/dbt-labs/jaffle_shop) demo and modified to support -using `dbt-duckdb` with an S3 bucket. - -The DBT `profiles.yml` configuration is taken from -[the `dbt-duckdb` docs](https://github.com/jwills/dbt-duckdb#configuring-your-profile). - -We also define the environment our application will run in -- -a container image, as in Docker. -See [this guide](https://modal.com/docs/guide/custom-container) for details. - -```python -dbt_image = ( # start from a slim Linux image - modal.Image.debian_slim(python_version="3.11") - .uv_pip_install( # install python packages - "boto3~=1.34", # aws client sdk - "dbt-duckdb~=1.8.1", # dbt and duckdb and a connector - "pandas~=2.2.2", # dataframes - "pyarrow~=16.1.0", # columnar data lib - "fastapi[standard]~=0.115.4", # web app - ) - .env( # configure DBT environment variables - { - "DBT_PROJECT_DIR": PROJ_PATH, - "DBT_PROFILES_DIR": PROFILES_PATH, - "DBT_TARGET_PATH": TARGET_PATH, - } - ) - # Here we add all local code and configuration into the Modal Image - # so that it will be available when we run DBT on Modal. - .add_local_dir(LOCAL_DBT_PROJECT, remote_path=PROJ_PATH) - .add_local_file( - LOCAL_DBT_PROJECT / "profiles.yml", - remote_path=f"{PROFILES_PATH}/profiles.yml", - ) -) - -app = modal.App(name="example-dbt-duckdb", image=dbt_image) - -dbt_target = modal.Volume.from_name("dbt-target-vol", create_if_missing=True) - -``` - -We'll also need to authenticate with AWS to store data in S3. - -```python -s3_secret = modal.Secret.from_name( - "modal-examples-aws-user", - required_keys=["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_REGION"], -) - -``` - -Create this Secret using the "AWS" template from the [Secrets dashboard](https://modal.com/secrets). -Below we will use the provided credentials in a Modal Function to create an S3 bucket and -populate it with `.parquet` data, so be sure to provide credentials for a user -with permission to create S3 buckets and read & write data from them. - -The policy required for this example is the following. -Not that you *must* update the bucket name listed in the policy to your -own bucket name. - -```json -{ - "Statement": [ - { - "Action": "s3:*", - "Effect": "Allow", - "Resource": [ - "arn:aws:s3:::modal-example-dbt-duckdb-s3/*", - "arn:aws:s3:::modal-example-dbt-duckdb-s3" - ], - "Sid": "duckdbs3access" - } - ], - "Version": "2012-10-17" -} -``` - -## Upload seed data - -In order to provide source data for DBT to ingest and transform, -we have the below `create_source_data` function which creates an AWS S3 bucket and -populates it with Parquet files based off the CSV data in the `seeds/` directory. - -You can kick it off by running this script on Modal: - -```bash -modal run dbt_duckdb.py -``` - -This script also runs the full data warehouse setup, and the whole process takes a minute or two. -We'll walk through the rest of the steps below. See the `app.local_entrypoint` -below for details. - -Note that this is not the typical way that `seeds/` data is used, but it's useful for this -demonstration. See [the DBT docs](https://docs.getdbt.com/docs/build/seeds) for more info. - -```python -@app.function( - secrets=[s3_secret], -) -def create_source_data(): - import boto3 - import pandas as pd - from botocore.exceptions import ClientError - - s3_client = boto3.client("s3") - s3_client.create_bucket(Bucket=BUCKET_NAME) - - for seed_csv_path in Path(PROJ_PATH, "seeds").glob("*.csv"): - print(f"Found seed file {seed_csv_path}") - name = seed_csv_path.stem - parquet_filename = f"{name}.parquet" - object_key = f"sources/{parquet_filename}" - try: - s3_client.head_object(Bucket=BUCKET_NAME, Key=object_key) - print( - f"File '{object_key}' already exists in bucket '{BUCKET_NAME}'. Skipping." - ) - except ClientError: - df = pd.read_csv(seed_csv_path) - df.to_parquet(parquet_filename) - print(f"Uploading '{object_key}' to S3 bucket '{BUCKET_NAME}'") - s3_client.upload_file(parquet_filename, BUCKET_NAME, object_key) - print(f"File '{object_key}' uploaded successfully.") - - -``` - -## Run DBT on the cloud with Modal - -Modal makes it easy to run Python code in the cloud. -And DBT is a Python tool, so it's easy to run DBT with Modal: -below, we import the `dbt` library's `dbtRunner` to pass commands from our -Python code, running on Modal, the same way we'd pass commands on a command line. - -Note that this Modal Function has access to our AWS S3 Secret, -the local files associated with our DBT project and profiles, -and a remote Modal Volume that acts as a distributed file system. - -```python -@app.function( - secrets=[s3_secret], - volumes={TARGET_PATH: dbt_target}, -) -def run(command: str) -> None: - from dbt.cli.main import dbtRunner - - res = dbtRunner().invoke(command.split(" ")) - if res.exception: - print(res.exception) - - -``` - -You can run this Modal Function from the command line with - -`modal run dbt_duckdb.py::run --command run` - -A successful run will log something like the following: - -``` -03:41:04 Running with dbt=1.5.0 -03:41:05 Found 5 models, 8 tests, 0 snapshots, 0 analyses, 313 macros, 0 operations, 3 seed files, 3 sources, 0 exposures, 0 metrics, 0 groups -03:41:05 -03:41:06 Concurrency: 1 threads (target='modal') -03:41:06 -03:41:06 1 of 5 START sql table model main.stg_customers ................................ [RUN] -03:41:06 1 of 5 OK created sql table model main.stg_customers ........................... [OK in 0.45s] -03:41:06 2 of 5 START sql table model main.stg_orders ................................... [RUN] -03:41:06 2 of 5 OK created sql table model main.stg_orders .............................. [OK in 0.34s] -03:41:06 3 of 5 START sql table model main.stg_payments ................................. [RUN] -03:41:07 3 of 5 OK created sql table model main.stg_payments ............................ [OK in 0.36s] -03:41:07 4 of 5 START sql external model main.customers ................................. [RUN] -03:41:07 4 of 5 OK created sql external model main.customers ............................ [OK in 0.72s] -03:41:07 5 of 5 START sql table model main.orders ....................................... [RUN] -03:41:08 5 of 5 OK created sql table model main.orders .................................. [OK in 0.22s] -03:41:08 -03:41:08 Finished running 4 table models, 1 external model in 0 hours 0 minutes and 3.15 seconds (3.15s). -03:41:08 Completed successfully -03:41:08 -03:41:08 Done. PASS=5 WARN=0 ERROR=0 SKIP=0 TOTAL=5 -``` - -Look for the `'materialized='external'` DBT config in the SQL templates -to see how `dbt-duckdb` is able to write back the transformed data to AWS S3! - -After running the `run` command and seeing it succeed, check what's contained -under the bucket's `out/` key prefix. You'll see that DBT has run the transformations -defined in `sample_proj_duckdb_s3/models/` and produced output `.parquet` files. - -## Serve fresh data documentation with FastAPI and Modal - -DBT also automatically generates [rich, interactive data docs](https://docs.getdbt.com/docs/collaborate/explore-projects). -You can serve these docs on Modal. -Just define a simple [FastAPI](https://fastapi.tiangolo.com/) app: - -```python -@app.function(volumes={TARGET_PATH: dbt_target}) -@modal.concurrent(max_inputs=100) -@modal.asgi_app() # wrap a function that returns a FastAPI app in this decorator to host on Modal -def serve_dbt_docs(): - import fastapi - from fastapi.staticfiles import StaticFiles - - web_app = fastapi.FastAPI() - web_app.mount( - "/", - StaticFiles( # dbt docs are automatically generated and sitting in the Volume - directory=TARGET_PATH, html=True - ), - name="static", - ) - - return web_app - - -``` - -And deploy that app to Modal with - -```bash -modal deploy dbt_duckdb.py -# ... -# Created web function serve_dbt_docs => -``` - -If you navigate to the output URL, you should see something like -[![example dbt docs](./dbt_docs.png)](https://modal-labs-examples--example-dbt-duckdb-serve-dbt-docs.modal.run) - -You can also check out our instance of the docs [here](https://modal-labs-examples--example-dbt-duckdb-serve-dbt-docs.modal.run). -The app will be served "serverlessly" -- it will automatically scale up or down -during periods of increased or decreased usage, and you won't be charged at all -when it has scaled to zero. - -## Schedule daily updates - -The following `daily_build` function [runs on a schedule](https://modal.com/docs/guide/cron) -to keep the DuckDB data warehouse up-to-date. It is also deployed by the same `modal deploy` command for the docs app. - -The source data for this warehouse is static, -so the daily executions don't really "update" anything, just re-build. But this example could be extended -to have sources which continually provide new data across time. -It will also generate the DBT docs daily to keep them fresh. - -```python -@app.function( - schedule=modal.Period(days=1), - secrets=[s3_secret], - volumes={TARGET_PATH: dbt_target}, -) -def daily_build() -> None: - run.remote("build") - run.remote("docs generate") - - -@app.local_entrypoint() -def main(): - create_source_data.remote() - run.remote("run") - daily_build.remote() - -``` diff --git a/.agents/skills/modal/references/examples/diffusers_lora_finetune.md b/.agents/skills/modal/references/examples/diffusers_lora_finetune.md deleted file mode 100644 index cb3453b2761ca326c0d783aa67ed1a61a8049306..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/diffusers_lora_finetune.md +++ /dev/null @@ -1,576 +0,0 @@ -# Fine-tune Flux on your pet using LoRA - -This example finetunes the [Flux.1-dev model](https://huggingface.co/black-forest-labs/FLUX.1-dev) -on images of a pet (by default, a puppy named Qwerty) -using a technique called textual inversion from [the "Dreambooth" paper](https://dreambooth.github.io/). -Effectively, it teaches a general image generation model a new "proper noun", -allowing for the personalized generation of art and photos. -We supplement textual inversion with low-rank adaptation (LoRA) -for increased efficiency during training. - -It then makes the model shareable with others -- without costing $25/day for a GPU server-- -by hosting a [Gradio app](https://gradio.app/) on Modal. - -It demonstrates a simple, productive, and cost-effective pathway -to building on large pretrained models using Modal's building blocks, like -[GPU-accelerated](https://modal.com/docs/guide/gpu) Modal Functions for compute-intensive work, -[Volumes](https://modal.com/docs/guide/volumes) for storage, -and [Web Functions](https://modal.com/docs/guide/webhooks) for serving. - -And with some light customization, you can use it to generate images of your pet! - -![Gradio.app image generation interface](./gradio-image-generate.png) - -You can find a video walkthrough of this example on the Modal YouTube channel -[here](https://www.youtube.com/watch?v=df-8fiByXMI). - -## Imports and setup - -We start by importing the necessary libraries and setting up the environment. - -```python -from dataclasses import dataclass -from pathlib import Path - -import modal - -``` - -## Building up the environment - -Machine learning environments are complex, and the dependencies can be hard to manage. -Modal makes creating and working with environments easy via -[containers and container images](https://modal.com/docs/guide/custom-container). - -We start from a base image and specify all of our dependencies. -We'll call out the interesting ones as they come up below. -Note that these dependencies are not installed locally -\-- they are only installed in the remote environment where our Modal App runs. - -```python -app = modal.App(name="example-diffusers-lora-finetune") - -image = modal.Image.debian_slim(python_version="3.10").uv_pip_install( - "accelerate==0.31.0", - "datasets~=2.13.0", - "fastapi[standard]==0.115.4", - "ftfy~=6.1.0", - "gradio~=5.5.0", - "huggingface-hub==0.36.0", - "numpy<2", - "peft==0.11.1", - "pydantic==2.9.2", - "sentencepiece>=0.1.91,!=0.1.92", - "smart_open~=6.4.0", - "starlette==0.41.2", - "transformers~=4.41.2", - "torch~=2.2.0", - "torchvision~=0.16", - "triton~=2.2.0", - "wandb==0.17.6", -) - -``` - -### Downloading scripts and installing a git repo with `run_commands` - -We'll use an example script from the `diffusers` library to train the model. -We acquire it from GitHub and install it in our environment with a series of commands. -The container environments Modal Functions run in are highly flexible -- -see [the docs](https://modal.com/docs/guide/custom-container) for more details. - -```python -GIT_SHA = "e649678bf55aeaa4b60bd1f68b1ee726278c0304" # specify the commit to fetch - -image = ( - image.apt_install("git") - # Perform a shallow fetch of just the target `diffusers` commit, checking out - # the commit in the container's home directory, /root. Then install `diffusers` - .run_commands( - "cd /root && git init .", - "cd /root && git remote add origin https://github.com/huggingface/diffusers", - f"cd /root && git fetch --depth=1 origin {GIT_SHA} && git checkout {GIT_SHA}", - "cd /root && pip install -e .", - ) -) - -``` - -### Configuration with `dataclass`es - -Machine learning apps often have a lot of configuration information. -We collect up all of our configuration into dataclasses to avoid scattering special/magic values throughout code. - -```python -@dataclass -class SharedConfig: - """Configuration information shared across project components.""" - - # The instance name is the "proper noun" we're teaching the model - instance_name: str = "Qwerty" - # That proper noun is usually a member of some class (person, bird), - # and sharing that information with the model helps it generalize better. - class_name: str = "Golden Retriever" - # identifier for pretrained models on Hugging Face - model_name: str = "black-forest-labs/FLUX.1-dev" - - -``` - -### Storing data created by our app with `modal.Volume` - -The tools we've used so far work well for fetching external information, -which defines the environment our app runs in, -but what about data that we create or modify during the app's execution? -A persisted [`modal.Volume`](https://modal.com/docs/guide/volumes) can store and share data across Modal Apps and Functions. - -We'll use one to store both the original and fine-tuned weights we create during training -and then load them back in for inference. For more on storing model weights on Modal, see -[this guide](https://modal.com/docs/guide/model-weights). - -```python -volume = modal.Volume.from_name( - "dreambooth-finetuning-volume-flux", create_if_missing=True -) -MODEL_DIR = "/model" - -``` - -Note that access to the Flux.1-dev model on Hugging Face is -[gated by a license agreement](https://huggingface.co/docs/hub/en/models-gated) which -you must agree to [here](https://huggingface.co/black-forest-labs/FLUX.1-dev). -After you have accepted the license, [create a Modal Secret](https://modal.com/secrets) -with the name `huggingface-secret` following the instructions in the template. - -```python -huggingface_secret = modal.Secret.from_name( - "huggingface-secret", required_keys=["HF_TOKEN"] -) - -image = image.env( - {"HF_XET_HIGH_PERFORMANCE": "1"} # turn on faster downloads from HF -) - - -@app.function( - volumes={MODEL_DIR: volume}, - image=image, - secrets=[huggingface_secret], - timeout=600, # 10 minutes -) -def download_models(config): - import torch - from diffusers import DiffusionPipeline - from huggingface_hub import snapshot_download - - snapshot_download( - config.model_name, - local_dir=MODEL_DIR, - ignore_patterns=["*.pt", "*.bin"], # using safetensors - ) - - DiffusionPipeline.from_pretrained(MODEL_DIR, torch_dtype=torch.bfloat16) - - -``` - -### Load fine-tuning dataset - -Part of the magic of the low-rank fine-tuning is that we only need 3-10 images for fine-tuning. -So we can fetch just a few images, stored on consumer platforms like Imgur or Google Drive, -whenever we need them -- no need for expensive, hard-to-maintain data pipelines. - -```python -def load_images(image_urls: list[str]) -> Path: - import PIL.Image - from smart_open import open - - img_path = Path("/img") - - img_path.mkdir(parents=True, exist_ok=True) - for ii, url in enumerate(image_urls): - with open(url, "rb") as f: - image = PIL.Image.open(f) - image.save(img_path / f"{ii}.png") - print(f"{ii + 1} images loaded") - - return img_path - - -``` - -## Low-Rank Adaptation (LoRA) fine-tuning for a text-to-image model - -The base model we start from is trained to do a sort of "reverse [ekphrasis](https://en.wikipedia.org/wiki/Ekphrasis)": -it attempts to recreate a visual work of art or image from only its description. - -We can use the model to synthesize wholly new images -by combining the concepts it has learned from the training data. - -We use a pretrained model, the Flux model from Black Forest Labs. -In this example, we "finetune" Flux, making only small adjustments to the weights. -Furthermore, we don't change all the weights in the model. -Instead, using a technique called [*low-rank adaptation*](https://arxiv.org/abs/2106.09685), -we change a much smaller matrix that works "alongside" the existing weights, nudging the model in the direction we want. - -We can get away with such a small and simple training process because we're just teach the model the meaning of a single new word: the name of our pet. - -The result is a model that can generate novel images of our pet: -as an astronaut in space, as painted by Van Gogh or Bastiat, etc. - -### Finetuning with Hugging Face 🧨 Diffusers and Accelerate - -The model weights, training libraries, and training script are all provided by [🤗 Hugging Face](https://huggingface.co). - -You can kick off a training job with the command `modal run dreambooth_app.py::app.train`. -It should take about ten minutes. - -Training machine learning models takes time and produces a lot of metadata -- -metrics for performance and resource utilization, -metrics for model quality and training stability, -and model inputs and outputs like images and text. -This is especially important if you're fiddling around with the configuration parameters. - -This example can optionally use [Weights & Biases](https://wandb.ai) to track all of this training information. -Just sign up for an account, switch the flag below, and add your API key as a [Modal Secret](https://modal.com/secrets). - -```python -USE_WANDB = False - -``` - -You can see an example W\&B dashboard [here](https://wandb.ai/cfrye59/dreambooth-lora-sd-xl). -Check out [this run](https://wandb.ai/cfrye59/dreambooth-lora-sd-xl/runs/ca3v1lsh?workspace=user-cfrye59), -which [despite having high GPU utilization](https://wandb.ai/cfrye59/dreambooth-lora-sd-xl/runs/ca3v1lsh/system) -suffered from numerical instability during training and produced only black images -- hard to debug without experiment management logs! - -You can read more about how the values in `TrainConfig` are chosen and adjusted [in this blog post on Hugging Face](https://huggingface.co/blog/dreambooth). -To run training on images of your own pet, upload the images to separate URLs and edit the contents of the file at `TrainConfig.instance_example_urls_file` to point to them. - -Tip: if the results you're seeing don't match the prompt too well, and instead produce an image -of your subject without taking the prompt into account, the model has likely overfit. In this case, repeat training with a lower -value of `max_train_steps`. If you used W\&B, look back at results earlier in training to determine where to stop. -On the other hand, if the results don't look like your subject, you might need to increase `max_train_steps`. - -```python -@dataclass -class TrainConfig(SharedConfig): - """Configuration for the finetuning step.""" - - # training prompt looks like `{PREFIX} {INSTANCE_NAME} the {CLASS_NAME} {POSTFIX}` - prefix: str = "a photo of" - postfix: str = "" - - # locator for plaintext file with urls for images of target instance - instance_example_urls_file: str = str( - Path(__file__).parent / "instance_example_urls.txt" - ) - - # Hyperparameters/constants from the huggingface training example - resolution: int = 512 - train_batch_size: int = 3 - rank: int = 16 # lora rank - gradient_accumulation_steps: int = 1 - learning_rate: float = 4e-4 - lr_scheduler: str = "constant" - lr_warmup_steps: int = 0 - max_train_steps: int = 500 - checkpointing_steps: int = 1000 - seed: int = 117 - - -@app.function( - image=image, - gpu="A100-80GB", # fine-tuning is VRAM-heavy and requires a high-VRAM GPU - volumes={MODEL_DIR: volume}, # stores fine-tuned model - timeout=1800, # 30 minutes - secrets=[huggingface_secret] - + ( - [modal.Secret.from_name("wandb-secret", required_keys=["WANDB_API_KEY"])] - if USE_WANDB - else [] - ), -) -def train(instance_example_urls, config): - import subprocess - - from accelerate.utils import write_basic_config - - # load data locally - img_path = load_images(instance_example_urls) - - # set up hugging face accelerate library for fast training - write_basic_config(mixed_precision="bf16") - - # define the training prompt - instance_phrase = f"{config.instance_name} the {config.class_name}" - prompt = f"{config.prefix} {instance_phrase} {config.postfix}".strip() - - # the model training is packaged as a script, so we have to execute it as a subprocess, which adds some boilerplate - def _exec_subprocess(cmd: list[str]): - """Executes subprocess and prints log to terminal while subprocess is running.""" - process = subprocess.Popen( - cmd, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT, - ) - with process.stdout as pipe: - for line in iter(pipe.readline, b""): - line_str = line.decode() - print(f"{line_str}", end="") - - if exitcode := process.wait() != 0: - raise subprocess.CalledProcessError(exitcode, "\n".join(cmd)) - - # run training -- see huggingface accelerate docs for details - print("launching dreambooth training script") - _exec_subprocess( - [ - "accelerate", - "launch", - "examples/dreambooth/train_dreambooth_lora_flux.py", - "--mixed_precision=bf16", # half-precision floats most of the time for faster training - f"--pretrained_model_name_or_path={MODEL_DIR}", - f"--instance_data_dir={img_path}", - f"--output_dir={MODEL_DIR}", - f"--instance_prompt={prompt}", - f"--resolution={config.resolution}", - f"--train_batch_size={config.train_batch_size}", - f"--gradient_accumulation_steps={config.gradient_accumulation_steps}", - f"--learning_rate={config.learning_rate}", - f"--lr_scheduler={config.lr_scheduler}", - f"--lr_warmup_steps={config.lr_warmup_steps}", - f"--max_train_steps={config.max_train_steps}", - f"--checkpointing_steps={config.checkpointing_steps}", - f"--seed={config.seed}", # increased reproducibility by seeding the RNG - ] - + ( - [ - "--report_to=wandb", - # validation output tracking is useful, but currently broken for Flux LoRA training - # f"--validation_prompt={prompt} in space", # simple test prompt - # f"--validation_epochs={config.max_train_steps // 5}", - ] - if USE_WANDB - else [] - ), - ) - # The trained model information has been output to the volume mounted at `MODEL_DIR`. - # To persist this data for use in our web app, we 'commit' the changes - # to the volume. - volume.commit() - - -``` - -## Running our model - -To generate images from prompts using our fine-tuned model, we define a Modal Function called `inference`. - -Naively, this would seem to be a bad fit for the flexible, serverless infrastructure of Modal: -wouldn't you need to include the steps to load the model and spin it up in every function call? - -In order to initialize the model just once on container startup, -we use Modal's [container lifecycle](https://modal.com/docs/guide/lifecycle-functions) features, which require the function to be part -of a class. Note that the `modal.Volume` we saved the model to is mounted here as well, -so that the fine-tuned model created by `train` is available to us. - -```python -@app.cls(image=image, gpu="A100", volumes={MODEL_DIR: volume}) -class Model: - @modal.enter() - def load_model(self): - import torch - from diffusers import DiffusionPipeline - - # Reload the modal.Volume to ensure the latest state is accessible. - volume.reload() - - # set up a hugging face inference pipeline using our model - pipe = DiffusionPipeline.from_pretrained( - MODEL_DIR, - torch_dtype=torch.bfloat16, - ).to("cuda") - pipe.load_lora_weights(MODEL_DIR) - self.pipe = pipe - - @modal.method() - def inference(self, text, config): - image = self.pipe( - text, - num_inference_steps=config.num_inference_steps, - guidance_scale=config.guidance_scale, - ).images[0] - - return image - - -``` - -## Wrap the trained model in a Gradio web UI - -[Gradio](https://gradio.app) makes it super easy to expose a model's functionality -in an easy-to-use, responsive web interface. - -This model is a text-to-image generator, -so we set up an interface that includes a user-entry text box -and a frame for displaying images. - -We also provide some example text inputs to help -guide users and to kick-start their creative juices. - -And we couldn't resist adding some Modal style to it as well! - -You can deploy the app on Modal with the command -`modal deploy dreambooth_app.py`. -You'll be able to come back days, weeks, or months later and find it still ready to go, -even though you don't have to pay for a server to run while you're not using it. - -```python -@dataclass -class AppConfig(SharedConfig): - """Configuration information for inference.""" - - num_inference_steps: int = 50 - guidance_scale: float = 6 - - -web_image = image.add_local_dir( - # Add local web assets to the image - Path(__file__).parent / "assets", - remote_path="/assets", -) - - -@app.function( - image=web_image, - max_containers=1, -) -@modal.concurrent(max_inputs=100) -@modal.asgi_app() -def fastapi_app(): - import gradio as gr - from fastapi import FastAPI - from fastapi.responses import FileResponse - from gradio.routes import mount_gradio_app - - web_app = FastAPI() - - # Call out to the inference in a separate Modal environment with a GPU - def go(text=""): - if not text: - text = example_prompts[0] - return Model().inference.remote(text, config) - - # set up AppConfig - config = AppConfig() - - instance_phrase = f"{config.instance_name} the {config.class_name}" - - example_prompts = [ - f"{instance_phrase}", - f"a painting of {instance_phrase.title()} With A Pearl Earring, by Vermeer", - f"oil painting of {instance_phrase} flying through space as an astronaut", - f"a painting of {instance_phrase} in cyberpunk city. character design by cory loftis. volumetric light, detailed, rendered in octane", - f"drawing of {instance_phrase} high quality, cartoon, path traced, by studio ghibli and don bluth", - ] - - modal_docs_url = "https://modal.com/docs" - modal_example_url = f"{modal_docs_url}/examples/dreambooth_app" - - description = f"""Describe what they are doing or how a particular artist or style would depict them. Be fantastical! Try the examples below for inspiration. - -### Learn how to make a "Dreambooth" for your own pet [here]({modal_example_url}). - """ - - # custom styles: an icon, a background, and a theme - @web_app.get("/favicon.ico", include_in_schema=False) - async def favicon(): - return FileResponse("/assets/favicon.svg") - - @web_app.get("/assets/background.svg", include_in_schema=False) - async def background(): - return FileResponse("/assets/background.svg") - - with open("/assets/index.css") as f: - css = f.read() - - theme = gr.themes.Default( - primary_hue="green", secondary_hue="emerald", neutral_hue="neutral" - ) - - # add a gradio UI around inference - with gr.Blocks( - theme=theme, - css=css, - title=f"Generate images of {config.instance_name} on Modal", - ) as interface: - gr.Markdown( - f"# Generate images of {instance_phrase}.\n\n{description}", - ) - with gr.Row(): - inp = gr.Textbox( # input text component - label="", - placeholder=f"Describe the version of {instance_phrase} you'd like to see", - lines=10, - ) - out = gr.Image( # output image component - height=512, width=512, label="", min_width=512, elem_id="output" - ) - with gr.Row(): - btn = gr.Button("Dream", variant="primary", scale=2) - btn.click( - fn=go, inputs=inp, outputs=out - ) # connect inputs and outputs with inference function - - gr.Button( # shameless plug - "⚡️ Powered by Modal", - variant="secondary", - link="https://modal.com", - ) - - with gr.Column(variant="compact"): - # add in a few examples to inspire users - for ii, prompt in enumerate(example_prompts): - btn = gr.Button(prompt, variant="secondary") - btn.click(fn=lambda idx=ii: example_prompts[idx], outputs=inp) - - # mount for execution on Modal - return mount_gradio_app( - app=web_app, - blocks=interface, - path="/", - ) - - -``` - -## Running your fine-tuned model from the command line - -You can use the `modal` command-line interface to set up, customize, and deploy this app: - -* `modal run diffusers_lora_finetune.py` will train the model. Change the `instance_example_urls_file` to point to your own pet's images. -* `modal serve diffusers_lora_finetune.py` will [serve](https://modal.com/docs/guide/webhooks#developing-with-modal-serve) the Gradio interface at a temporary location. Great for iterating on code! -* `modal shell diffusers_lora_finetune.py` is a convenient helper to open a bash [shell](https://modal.com/docs/guide/developing-debugging#interactive-shell) in our image. Great for debugging environment issues. - -Remember, once you've trained your own fine-tuned model, you can deploy it permanently -- for no cost when it is not being used! -- -using `modal deploy diffusers_lora_finetune.py`. - -If you just want to try the app out, you can find our deployment [here](https://modal-labs--example-diffusers-lora-finetune-fastapi-app.modal.run). - -```python -@app.local_entrypoint() -def run( # add more config params here to make training configurable - max_train_steps: int = 250, -): - print("🎨 loading model") - download_models.remote(SharedConfig()) - print("🎨 setting up training") - config = TrainConfig(max_train_steps=max_train_steps) - instance_example_urls = ( - Path(TrainConfig.instance_example_urls_file).read_text().splitlines() - ) - train.remote(instance_example_urls, config) - print("🎨 training finished") - -``` diff --git a/.agents/skills/modal/references/examples/discord_bot.md b/.agents/skills/modal/references/examples/discord_bot.md deleted file mode 100644 index 5f93c3538f6e6b9e339cd2406b296dd594ed8e72..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/discord_bot.md +++ /dev/null @@ -1,417 +0,0 @@ -# Serve a Discord Bot on Modal - -In this example we will demonstrate how to use Modal to build and serve a Discord bot that uses -[slash commands](https://discord.com/developers/docs/interactions/application-commands). - -Slash commands send information from Discord server members to a service at a URL. -Here, we set up a simple [FastAPI app](https://fastapi.tiangolo.com/) -to run that service and deploy it easily Modal’s -[`@asgi_app`](https://modal.com/docs/guide/webhooks#serving-asgi-and-wsgi-apps) decorator. - -As our example service, we hit a simple free API: -the [Free Public APIs API](https://www.freepublicapis.com/api), -a directory of free public APIs. - -[Try it out on Discord](https://discord.gg/PmG7P47EPQ)! - -## Set up our App and its Image - -First, we define the [container image](https://modal.com/docs/guide/images) -that all the pieces of our bot will run in. - -We set that as the default image for a Modal [App](https://modal.com/docs/guide/apps). -The App is where we'll attach all the components of our bot. - -```python -import json -from enum import Enum - -import modal - -image = modal.Image.debian_slim(python_version="3.11").uv_pip_install( - "fastapi[standard]==0.115.4", "pynacl~=1.5.0", "requests~=2.32.3" -) - -app = modal.App("example-discord-bot", image=image) - -``` - -## Hit the Free Public APIs API - -We start by defining the core service that our bot will provide. - -In a real application, this might be [music generation](https://modal.com/docs/examples/musicgen), -a [chatbot](https://modal.com/docs/examples/chat_with_pdf_vision), -or [interacting with a database](https://modal.com/docs/examples/cron_datasette). - -Here, we just hit a simple free public API: -the [Free Public APIs](https://www.freepublicapis.com) API, -an "API of APIs" that returns information about free public APIs, -like the [Global Shark Attack API](https://www.freepublicapis.com/global-shark-attack-api) -and the [Corporate Bullshit Generator](https://www.freepublicapis.com/corporate-bullshit-generator). -We convert the response into a Markdown-formatted message. - -We turn our Python function into a Modal Function by attaching the `app.function` decorator. -We make the function `async` and add `@modal.concurrent()` with a large `max_inputs` value, because -communicating with an external API is a classic case for better performance from asynchronous execution. -Modal handles things like the async event loop for us. - -```python -@app.function() -@modal.concurrent(max_inputs=100) -async def fetch_api() -> str: - import aiohttp - - url = "https://www.freepublicapis.com/api/random" - - async with aiohttp.ClientSession() as session: - try: - async with session.get(url) as response: - response.raise_for_status() - data = await response.json() - message = ( - f"# {data.get('emoji') or '🤖'} [{data['title']}]({data['source']})" - ) - message += f"\n _{''.join(data['description'].splitlines())}_" - except Exception as e: - message = f"# 🤖: Oops! {e}" - - return message - - -``` - -This core component has nothing to do with Discord, -and it's nice to be able to interact with and test it in isolation. - -For that, we add a `local_entrypoint` that calls the Modal Function. -Notice that we add `.remote` to the function's name. - -Later, when you replace this component of the app with something more interesting, -test it by triggering this entrypoint with `modal run discord_bot.py`. - -```python -@app.local_entrypoint() -def test_fetch_api(): - result = fetch_api.remote() - if result.startswith("# 🤖: Oops! "): - raise Exception(result) - else: - print(result) - - -``` - -## Integrate our Modal Function with Discord Interactions - -Now we need to map this function onto Discord's interface -- -in particular the [Interactions API](https://discord.com/developers/docs/interactions/overview). - -Reviewing the documentation, we see that we need to send a JSON payload -to a specific API URL that will include an `app_id` that identifies our bot -and a `token` that identifies the interaction (loosely, message) that we're participating in. - -So let's write that out. This function doesn't need to live on Modal, -since it's just encapsulating some logic -- we don't want to turn it into a service or an API on its own. -That means we don't need any Modal decorators. - -```python -async def send_to_discord(payload: dict, app_id: str, interaction_token: str): - import aiohttp - - interaction_url = f"https://discord.com/api/v10/webhooks/{app_id}/{interaction_token}/messages/@original" - - async with aiohttp.ClientSession() as session: - async with session.patch(interaction_url, json=payload) as resp: - print("🤖 Discord response: " + await resp.text()) - - -``` - -Other parts of our application might want to both hit the Free Public APIs API and send the result to Discord, -so we both write a Python function for this and we promote it to a Modal Function with a decorator. - -Notice that we use the `.local` suffix to call our `fetch_api` Function. That means we run -the Function the same way we run all the other Python functions, rather than treating it as a special -Modal Function. This reduces a bit of extra latency, but couples these two Functions more tightly. - -```python -@app.function() -@modal.concurrent(max_inputs=100) -async def reply(app_id: str, interaction_token: str): - message = await fetch_api.local() - await send_to_discord({"content": message}, app_id, interaction_token) - - -``` - -## Set up a Discord app - -Now, we need to actually connect to Discord. -We start by creating an application on the Discord Developer Portal. - -1. Go to the - [Discord Developer Portal](https://discord.com/developers/applications) and - log in with your Discord account. -2. On the portal, go to **Applications** and create a new application by - clicking **New Application** in the top right next to your profile picture. -3. [Create a custom Modal Secret](https://modal.com/docs/guide/secrets) for your Discord bot. - On Modal's Secret creation page, select 'Discord'. Copy your Discord application’s - **Public Key** and **Application ID** (from the **General Information** tab in the Discord Developer Portal) - and paste them as the value of `DISCORD_PUBLIC_KEY` and `DISCORD_CLIENT_ID`. - Additionally, head to the **Bot** tab and use the **Reset Token** button to create a new bot token. - Paste this in the value of an additional key in the Secret, `DISCORD_BOT_TOKEN`. - Name this Secret `discord-secret`. - -We access that Secret in code like so: - -```python -discord_secret = modal.Secret.from_name( - "discord-secret", - required_keys=[ # included so we get nice error messages if we forgot a key - "DISCORD_BOT_TOKEN", - "DISCORD_CLIENT_ID", - "DISCORD_PUBLIC_KEY", - ], -) - -``` - -## Register a Slash Command - -Next, we’re going to register a [Slash Command](https://discord.com/developers/docs/interactions/application-commands#slash-commands) -for our Discord app. Slash Commands are triggered by users in servers typing `/` and the name of the command. - -The Modal Function below will register a Slash Command for your bot named `bored`. -More information about Slash Commands can be found in the Discord docs -[here](https://discord.com/developers/docs/interactions/application-commands). - -You can run this Function with - -```bash -modal run discord_bot::create_slash_command -``` - -```python -@app.function(secrets=[discord_secret], image=image) -def create_slash_command(force: bool = False): - """Registers the slash command with Discord. Pass the force flag to re-register.""" - import os - - import requests - - BOT_TOKEN = os.getenv("DISCORD_BOT_TOKEN") - CLIENT_ID = os.getenv("DISCORD_CLIENT_ID") - - headers = { - "Content-Type": "application/json", - "Authorization": f"Bot {BOT_TOKEN}", - } - url = f"https://discord.com/api/v10/applications/{CLIENT_ID}/commands" - - command_description = { - "name": "api", - "description": "Information about a random free, public API", - } - - # first, check if the command already exists - response = requests.get(url, headers=headers) - try: - response.raise_for_status() - except Exception as e: - raise Exception("Failed to create slash command") from e - - commands = response.json() - command_exists = any( - command.get("name") == command_description["name"] for command in commands - ) - - # and only recreate it if the force flag is set - if command_exists and not force: - print(f"🤖: command {command_description['name']} exists") - return - - response = requests.post(url, headers=headers, json=command_description) - try: - response.raise_for_status() - except Exception as e: - raise Exception("Failed to create slash command") from e - print(f"🤖: command {command_description['name']} created") - - -``` - -## Host a Discord Interactions endpoint on Modal - -If you look carefully at the definition of the Slash Command above, -you'll notice that it doesn't know anything about our bot besides an ID. - -To hook the Slash Commands in the Discord UI up to our logic for hitting the Bored API, -we need to set up a service that listens at some URL and follows a specific protocol, -described [here](https://discord.com/developers/docs/interactions/overview#configuring-an-interactions-endpoint-url). - -Here are some of the most important facets: - -1. We'll need to respond within five seconds or Discord will assume we are dead. - Modal's fast-booting serverless containers usually start faster than that, - but it's not guaranteed. So we'll add the `min_containers` parameter to our - Function so that there's at least one live copy ready to respond quickly at any time. - Modal charges a minimum of about 2¢ an hour for live containers (pricing details [here](https://modal.com/pricing)). - Note that that still fits within Modal's $30/month of credits on the free tier. - -2. We have to respond to Discord that quickly, but we don't have to respond to the user that quickly. - We instead send an acknowledgement so that they know we're alive and they can close their connection to us. - We also trigger our `reply` Modal Function, which will respond to the user via Discord's Interactions API, - but we don't wait for the result, we just `spawn` the call. - -3. The protocol includes some authentication logic that is mandatory - and checked by Discord. We'll explain in more detail in the next section. - -We can set up our interaction endpoint by deploying a FastAPI app on Modal. -This is as easy as creating a Python Function that returns a FastAPI app -and adding the `modal.asgi_app` decorator. -For more details on serving Python web apps on Modal, see -[this guide](https://modal.com/docs/guide/webhooks). - -```python -@app.function(secrets=[discord_secret], min_containers=1) -@modal.concurrent(max_inputs=100) -@modal.asgi_app() -def web_app(): - from fastapi import FastAPI, HTTPException, Request - from fastapi.middleware.cors import CORSMiddleware - - web_app = FastAPI() - - # must allow requests from other domains, e.g. from Discord's servers - web_app.add_middleware( - CORSMiddleware, - allow_origins=["*"], - allow_credentials=True, - allow_methods=["*"], - allow_headers=["*"], - ) - - @web_app.post("/api") - async def get_api(request: Request): - body = await request.body() - - # confirm this is a request from Discord - authenticate(request.headers, body) - - print("🤖: parsing request") - data = json.loads(body.decode()) - if data.get("type") == DiscordInteractionType.PING.value: - print("🤖: acking PING from Discord during auth check") - return {"type": DiscordResponseType.PONG.value} - - if data.get("type") == DiscordInteractionType.APPLICATION_COMMAND.value: - print("🤖: handling slash command") - app_id = data["application_id"] - interaction_token = data["token"] - - # kick off request asynchronously, will respond when ready - reply.spawn(app_id, interaction_token) - - # respond immediately with defer message - return { - "type": DiscordResponseType.DEFERRED_CHANNEL_MESSAGE_WITH_SOURCE.value - } - - print(f"🤖: unable to parse request with type {data.get('type')}") - raise HTTPException(status_code=400, detail="Bad request") - - return web_app - - -``` - -The authentication for Discord is a bit involved and there aren't, -to our knowledge, any good Python libraries for it. - -So we have to implement the protocol "by hand". - -Essentially, Discord sends a header in their request -that we can use to verify the request comes from them. -For that, we use the `DISCORD_PUBLIC_KEY` from -our Application Information page. - -The details aren't super important, but they appear in the `authenticate` function below -(which defers the real cryptography work to [PyNaCl](https://pypi.org/project/PyNaCl/), -a Python wrapper for [`libsodium`](https://github.com/jedisct1/libsodium)). - -Discord will also check that we reject unauthorized requests, -so we have to be sure to get this right! - -```python -def authenticate(headers, body): - import os - - from fastapi.exceptions import HTTPException - from nacl.exceptions import BadSignatureError - from nacl.signing import VerifyKey - - print("🤖: authenticating request") - # verify the request is from Discord using their public key - public_key = os.getenv("DISCORD_PUBLIC_KEY") - verify_key = VerifyKey(bytes.fromhex(public_key)) - - signature = headers.get("X-Signature-Ed25519") - timestamp = headers.get("X-Signature-Timestamp") - - message = timestamp.encode() + body - - try: - verify_key.verify(message, bytes.fromhex(signature)) - except BadSignatureError: - # either an unauthorized request or Discord's "negative control" check - raise HTTPException(status_code=401, detail="Invalid request") - - -``` - -The code above used a few enums to abstract bits of the Discord protocol. -Now that we've walked through all of it, -we're in a position to understand what those are -and so the code for them appears below. - -```python -class DiscordInteractionType(Enum): - PING = 1 # hello from Discord during auth check - APPLICATION_COMMAND = 2 # an actual command - - -class DiscordResponseType(Enum): - PONG = 1 # hello back during auth check - DEFERRED_CHANNEL_MESSAGE_WITH_SOURCE = 5 # we'll send a message later - - -``` - -## Deploy on Modal - -You can deploy this app on Modal by running the following commands: - -```shell -modal run discord_bot.py # checks the API wrapper, little test -modal run discord_bot.py::create_slash_command # creates the slash command, if missing -modal deploy discord_bot.py # deploys the web app and the API wrapper -``` - -Copy the Modal URL that is printed in the output and go back to the **General Information** section on the -[Discord Developer Portal](https://discord.com/developers/applications). -Paste the URL, making sure to append the path of your `POST` route (here, `/api`), in the -**Interactions Endpoint URL** field, then click **Save Changes**. If your -endpoint URL is incorrect or if authentication is incorrectly implemented, -Discord will refuse to save the URL. Once it saves, you can start -handling interactions! - -## Finish setting up Discord bot - -To start using the Slash Command you just set up, you need to invite the bot to -a Discord server. To do so, go to your application's **Installation** section on the -[Discord Developer Portal](https://discord.com/developers/applications). -Copy the **Discored Provided Link** and visit it to invite the bot to your bot to the server. - -Now you can open your Discord server and type `/api` in a channel to trigger the bot. -You can see a working version [in our test Discord server](https://discord.gg/PmG7P47EPQ). diff --git a/.agents/skills/modal/references/examples/doc_ocr_jobs.md b/.agents/skills/modal/references/examples/doc_ocr_jobs.md deleted file mode 100644 index 3f8ad98a6afc62be85676c3aaae86eea9433493d..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/doc_ocr_jobs.md +++ /dev/null @@ -1,219 +0,0 @@ -# Run a job queue that turns documents into structured data with Datalab Marker - -This tutorial shows you how to use Modal as an infinitely scalable job queue -that can service async tasks from a web app. - -Our job queue will handle a single task: converting images/PDFs into structured data. -We'll use [Marker](https://github.com/datalab-to/marker) from [Datalab](https://www.datalab.to), -which can convert images of documents or PDFs to Markdown, JSON, and HTML. Marker is an open-weights model; -to learn more about commercial usage, see [here](https://github.com/datalab-to/marker?tab=readme-ov-file#commercial-usage). - -For the purpose of this tutorial, we've also built a [React + FastAPI web app on Modal](https://modal.com/docs/examples/doc_ocr_webapp) -that works together with it, but note that you don't need a web app running on Modal -to use this pattern. You can submit async tasks to Modal from any Python -application (for example, a regular Django app running on Kubernetes). - -Try it out for yourself [here](https://modal-labs-examples--example-doc-ocr-webapp-wrapper.modal.run/). - -## Define an App - -Let's first import `modal` and define an [`App`](https://modal.com/docs/reference/modal.App). -Later, we'll use the name provided for our job queue App to find it from our web app and submit tasks to it. - -```python -from typing import Optional - -import modal -from typing_extensions import Literal - -app = modal.App("example-doc-ocr-jobs") - -``` - -We also define the dependencies we need by specifying an -[Image](https://modal.com/docs/guide/images). - -```python -inference_image = modal.Image.debian_slim(python_version="3.12").uv_pip_install( - "marker-pdf[full]==1.9.3", "torch==2.8.0" -) - -``` - -## Cache the pre-trained model on a Modal Volume - -We can obtain the pre-trained model we want to run from Datalab -by using the Marker library. - -```python -def load_models(): - import marker.models - - print("loading models") - - return marker.models.create_model_dict() - - -``` - -The `create_model_dict` function downloads model weights from Datalab's -cloud storage (S3 bucket) if they aren't already present in the filesystem. -However, in Modal's serverless environment, filesystems are ephemeral, -so using this code alone would mean that models need to be downloaded -many times (every time a new instance of our Function spins up). - -So instead, we create a Modal [Volume](https://modal.com/docs/guide/volumes) -to store the models. Each Modal Volume is a durable filesystem that any Modal Function can access. -You can read more about storing model weights on Modal in [our guide](https://modal.com/docs/guide/model-weights). - -```python -marker_cache_path = "/root/.cache/datalab/" -marker_cache_volume = modal.Volume.from_name( - "marker-models-modal-demo", create_if_missing=True -) -marker_cache = {marker_cache_path: marker_cache_volume} - -``` - -## Run Datalab Marker on Modal - -Now let's set up the actual inference. - -Using the [`@app.function`](https://modal.com/docs/reference/modal.App#function) -decorator, we set up a Modal [Function](https://modal.com/docs/reference/modal.Function). -We provide arguments to that decorator to customize the hardware, scaling, and other features -of the Function. - -Here, we say that this Function should use NVIDIA L40S [GPUs](https://modal.com/docs/guide/gpu), -automatically [retry](https://modal.com/docs/guide/retries#function-retries) failures up to 3 times, -and have access to our [shared model cache](https://modal.com/docs/guide/volumes). - -Inside the Function, we write out our inference logic, -which mostly involves configuring components provided by the `marker` library. - -```python -@app.function(gpu="l40s", retries=3, volumes=marker_cache, image=inference_image) -def parse_document( - document: bytes, - page_range: str | None = None, - force_ocr: bool = False, - paginate_output: bool = False, - output_format: Literal["markdown", "html", "chunks", "json"] = "markdown", - use_llm: bool = False, -) -> str | dict: - """ - Args: - document: Document data (PDF, JPG, PNG) as bytes. - page_range: Specify which pages to process. Accepts comma-separated page numbers and ranges. - force_ocr: Force OCR processing on the entire document, even for pages that might contain extractable text. - This will also format inline math properly. - paginate_output: Paginates the output, using \n\n{PAGE_NUMBER} followed by - * 48, then \n\n - output_format: Output format. Can be markdown, JSON, HTML, or chunks. - use_llm: use an llm to improve the marker results. - """ - from tempfile import NamedTemporaryFile - - import marker.config.parser - import marker.converters.pdf - import marker.output - - models = load_models() - - # Set up document "converter" - config = { - "page_range": page_range, - "force_ocr": force_ocr, - "paginate_output": paginate_output, - "output_format": output_format, - "use_llm": use_llm, - } - - config_parser = marker.config.parser.ConfigParser(config) - config_dict = config_parser.generate_config_dict() - config_dict["pdftext_workers"] = 1 - - converter = marker.converters.pdf.PdfConverter( - config=config_dict, - artifact_dict=models, - processor_list=config_parser.get_processors(), - renderer=config_parser.get_renderer(), - llm_service=config_parser.get_llm_service() if use_llm else None, - ) - - # Run the converter on our document - with NamedTemporaryFile(delete=False, mode="wb+") as temp_path: - temp_path.write(document) - rendered_output = converter(temp_path.name) - - # Format the output and return it - if output_format == "json": - result = rendered_output.model_dump_json() - else: - text, _, images = marker.output.text_from_rendered(rendered_output) - - result = text - - return result - - -``` - -## Testing and debugging remote code - -To make sure this code works, we want a way to kick the tires and debug it. - -We can run it on Modal, with no need to set up separate local testing, -by adding a [`local_entrypoint`](https://modal.com/docs/reference/modal.App#local_entrypoint) -that invokes the Function `.remote`ly. - -```python -@app.local_entrypoint() -def main(document_filename: Optional[str] = None): - import urllib.request - from pathlib import Path - - if document_filename is None: - document_filename = Path(__file__).parent / "receipt.png" - else: - document_filename = Path(document_filename) - - if document_filename.exists(): - image = document_filename.read_bytes() - print(f"running OCR on {document_filename}") - else: - document_url = "https://modal-cdn.com/cdnbot/Brandys-walmart-receipt-8g68_a_hk_f9c25fce.webp" - print(f"running OCR on sample from URL {document_url}") - request = urllib.request.Request(document_url) - with urllib.request.urlopen(request) as response: - image = response.read() - print(parse_document.remote(image, output_format="html")) - - -``` - -You can then run this from the command line with: - -```shell -modal run doc_ocr_jobs.py -``` - -## Deploying the document conversion service - -Now that we have a Function, we can publish it by deploying the App: - -```shell -modal deploy doc_ocr_jobs.py -``` - -Once it's published, we can [look up](https://modal.com/docs/guide/trigger-deployed-functions) this Function -from another Python process and submit tasks to it: - -```python -fn = modal.Function.from_name("example-doc-ocr-jobs", "parse_document") -fn.spawn(my_document) -``` - -Modal will auto-scale to handle all the tasks queued, and -then scale back down to 0 when there's no work left. To see how you could use this from a Python web -app, take a look at the [receipt parser frontend](https://modal.com/docs/examples/doc_ocr_webapp) -tutorial. diff --git a/.agents/skills/modal/references/examples/doc_ocr_webapp.md b/.agents/skills/modal/references/examples/doc_ocr_webapp.md deleted file mode 100644 index 5965eaed2670d152f4ed15908b4715d8ac18a1cd..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/doc_ocr_webapp.md +++ /dev/null @@ -1,159 +0,0 @@ -# Serve a receipt parsing web app - -This tutorial shows you how to use Modal to deploy a fully serverless -[React](https://reactjs.org/) + [FastAPI](https://fastapi.tiangolo.com/) application. - -We're going to build a simple "Receipt Parser" web app that submits document parsing -tasks to a separate Modal app defined in [another example](https://modal.com/docs/examples/doc_ocr_jobs), -polls until the task is completed, and displays -the results. Try it out for yourself -[here](https://modal-labs-examples--example-doc-ocr-webapp-wrapper.modal.run/). - -It should look something like this: - -[![Webapp frontend](https://modal-cdn.com/doc_ocr_frontend.jpg)](https://modal-labs-examples--example-doc-ocr-webapp-wrapper.modal.run/) - -## Basic setup - -Let's get the imports out of the way and define an [`App`](https://modal.com/docs/reference/modal.App). - -```python -from pathlib import Path - -import fastapi -import fastapi.staticfiles -import modal - -app = modal.App("example-doc-ocr-webapp") - -``` - -Modal works with any [ASGI](https://modal.com/docs/guide/webhooks#serving-asgi-and-wsgi-apps) or -[WSGI](https://modal.com/docs/guide/webhooks#wsgi) web framework. Here, we choose to use [FastAPI](https://fastapi.tiangolo.com/). - -```python -web_app = fastapi.FastAPI() - -``` - -## Define endpoints - -We need two endpoints: one to accept an image and submit it to the Modal job queue, -and another to poll for the results of the job. - -In `parse`, we're going to submit tasks to the Function defined in the [Job -Queue tutorial](https://modal.com/docs/examples/doc_ocr_jobs), so we import it first using -[`Function.lookup`](https://modal.com/docs/reference/modal.Function#lookup). - -We call [`.spawn()`](https://modal.com/docs/reference/modal.Function#spawn) on the Function handle -we imported above to kick off our Function without blocking on the results. `spawn` returns -a unique ID for the function call, which we then use -to poll for its result. - -```python -@web_app.post("/parse") -async def parse(request: fastapi.Request): - parse_receipt = modal.Function.from_name("example-doc-ocr-jobs", "parse_document") - - form = await request.form() - receipt = await form["receipt"].read() # type: ignore - call = parse_receipt.spawn(receipt) - return {"call_id": call.object_id} - - -``` - -`/result` uses the provided `call_id` to instantiate a `modal.FunctionCall` object, and attempt -to get its result. If the call hasn't finished yet, we return a `202` status code, which indicates -that the server is still working on the job. - -```python -@web_app.get("/result/{call_id}") -async def poll_results(call_id: str): - function_call = modal.functions.FunctionCall.from_id(call_id) - try: - result = await function_call.get.aio(timeout=0) - except TimeoutError: - return fastapi.responses.JSONResponse(content="", status_code=202) - - return result - - -``` - -Now that we've defined our endpoints, we're ready to host them on Modal. -First, we specify our dependencies -- here, a basic Debian Linux -environment with FastAPI installed. - -```python -image = modal.Image.debian_slim(python_version="3.12").uv_pip_install( - "fastapi[standard]==0.115.4" -) - -``` - -Then, we add the static files for our front-end. We've made [a simple React -app](https://github.com/modal-labs/modal-examples/tree/main/09_job_queues/doc_ocr_frontend) -that hits the two endpoints defined above. To package these files with our app, we use -`add_local_dir` with the local directory of the assets, and specify that we want them -in the `/assets` directory inside our container (the `remote_path`). Then, we instruct FastAPI to [serve -this static file directory](https://fastapi.tiangolo.com/tutorial/static-files/) at our root path. - -```python -local_assets_path = Path(__file__).parent / "doc_ocr_frontend" -image = image.add_local_dir(local_assets_path, remote_path="/assets") - -``` - -We serve them from our FastAPI app as `StaticFiles`. - -To put our FastAPI app on Modal, we need to return it from a Python function -that is wrapped with some extra decorators: - -* [`modal.asgi_app`](https://modal.com/docs/reference/modal.asgi_app) - to ensure the Modal system knows to route web traffic to it (and in what format) -* [`modal.concurrent`](https://modal.com/docs/reference/modal.concurrent) - to allow more than one request (e.g. for stylesheet and for HTML) to be served concurrently -* [`app.function`](https://modal.com/docs/reference/modal.App#function) - to turn our Python function into a Modal Function and define the infrastructure it needs - (here, just the dependencies). - -```python -@app.function(image=image) -@modal.concurrent(max_inputs=100) -@modal.asgi_app() -def wrapper(): - web_app.mount("/", fastapi.staticfiles.StaticFiles(directory="/assets", html=True)) - return web_app - - -``` - -## Running - -While developing, you can run this as an ephemeral app by executing the command - -```shell -modal serve doc_ocr_webapp.py -``` - -If successful, this will print a URL for your app that you can navigate to in -your browser 🎉 . - -The result should look something like this: - -[![Webapp frontend](https://modal-cdn.com/doc_ocr_frontend.jpg)](https://modal-labs-examples--example-doc-ocr-webapp-wrapper.modal.run/) - -Modal watches all the mounted files and updates the app if anything changes. -See [these docs](https://modal.com/docs/guide/webhooks#developing-with-modal-serve) -for more details. - -## Deploy - -To deploy your application, run - -```shell -modal deploy doc_ocr_webapp.py -``` - -That's all! diff --git a/.agents/skills/modal/references/examples/esm3.md b/.agents/skills/modal/references/examples/esm3.md deleted file mode 100644 index 35382722ff95c479fdbedc3c500599e20c084cb9..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/esm3.md +++ /dev/null @@ -1,488 +0,0 @@ -# Build a protein folding dashboard with ESM3, Molstar, and Gradio - -![Image of dashboard UI for ESM3 protein folding](https://modal-cdn.com/example-esm3-ui.png) - -There are perhaps a quadrillion distinct proteins on the planet Earth, -each one a marvel of nanotechnology discovered by painstaking evolution. -We know the amino acid sequence of nearly a billion but we only -know the three-dimensional structure of a few hundred thousand, -gathered by slow, difficult observational methods like X-ray crystallography. -Built upon this data are machine learning models like -EvolutionaryScale's [ESM3](https://www.evolutionaryscale.ai/blog/esm3-release) -that can predict the structure of any sequence in seconds. - -In this example, we'll show how you can use Modal to not -just run the latest protein-folding model but also build tools around it for -you and your team of scientists to understand and analyze the results. - -## Basic Setup - -```python -import base64 -import io -from pathlib import Path -from typing import Optional - -import modal - -MINUTES = 60 # seconds - -app = modal.App("example-esm3") - -``` - -### Create a Volume to store ESM3 model weights and Entrez sequence data - -To minimize cold start times, we'll store the ESM3 model weights on a Modal -[Volume](https://modal.com/docs/guide/volumes). -For patterns and best practices for storing model weights on Modal, see -[this guide](https://modal.com/docs/guide/model-weights). -We'll use this same distributed storage primitive to store sequence data. - -```python -volume = modal.Volume.from_name("example-esm3", create_if_missing=True) -VOLUME_PATH = Path("/vol") -MODELS_PATH = VOLUME_PATH / "models" -DATA_PATH = VOLUME_PATH / "data" - -``` - -### Define dependencies in container images - -The container image for structure inference is based on Modal's default slim Debian -Linux image with `esm` for loading and running the model, `gemmi` for -managing protein structure file conversions, and setting an environment variable -for faster downloading of the model weights from Hugging Face. - -```python -esm3_image = ( - modal.Image.debian_slim(python_version="3.11") - .uv_pip_install( - "esm==3.1.1", - "torch==2.4.1", - "gemmi==0.7.0", - "huggingface-hub==0.36.0", - ) - .env({"HF_XET_HIGH_PERFORMANCE": "1", "HF_HOME": str(MODELS_PATH)}) -) - -``` - -We'll also define a separate image, with different dependencies, -for the part of our app that hosts the dashboard. -This helps reduce the complexity of Python dependency management -by "walling off" the different parts, e.g. separating -functions that depend on finicky ML packages -from those that depend on pedantic web packages. -Dependencies include `gradio` for building a web UI in Python and -`biotite` for extracting sequences from UniProt accession numbers. - -You can read more about how to configure container images on Modal in -[this guide](https://modal.com/docs/guide/images). - -```python -web_app_image = ( - modal.Image.debian_slim(python_version="3.11") - .uv_pip_install("gradio~=4.44.0", "biotite==0.41.2", "fastapi[standard]==0.115.4") - .add_local_dir(Path(__file__).parent / "frontend", remote_path="/assets") -) - - -``` - -Here we "pre-import" libraries that will be used by the functions we run -on Modal in a given image using the `with image.imports` context manager. - -```python -with esm3_image.imports(): - import tempfile - - import gemmi - import torch - from esm.models.esm3 import ESM3 - from esm.sdk.api import ESMProtein, GenerationConfig - -with web_app_image.imports(): - import biotite.database.entrez as entrez - import biotite.sequence.io.fasta as fasta - from fastapi import FastAPI - -``` - -## Define a `Model` inference class for ESM3 - -Next, we map the model's setup and inference code onto Modal. - -1. For setup code that only needs to run once, we put it in a method - decorated with `@enter`, which runs on container start. For details, - see [this guide](https://modal.com/docs/guide/cold-start). -2. The rest of the inference code goes in a method decorated with `@method`. -3. We accelerate the compute-intensive inference with a GPU, specifically an A10G. - For more on using GPUs on Modal, see [this guide](https://modal.com/docs/guide/gpu). - -```python -@app.cls( - image=esm3_image, - volumes={VOLUME_PATH: volume}, - secrets=[modal.Secret.from_name("huggingface-secret")], - gpu="A10G", - timeout=20 * MINUTES, -) -class Model: - @modal.enter() - def enter(self): - self.model = ESM3.from_pretrained("esm3_sm_open_v1") - self.model.to("cuda") - - print("using half precision and tensor cores for fast ESM3 inference") - self.model = self.model.half() - torch.backends.cuda.matmul.allow_tf32 = True - - self.max_steps = 250 - print(f"setting max ESM steps to: {self.max_steps}") - - def convert_protein_to_MMCIF(self, esm_protein, output_path): - structure = gemmi.read_pdb_string(esm_protein.to_pdb_string()) - doc = structure.make_mmcif_document() - doc.write_file(str(output_path), gemmi.cif.WriteOptions()) - - def get_generation_config(self, num_steps): - return GenerationConfig(track="structure", num_steps=num_steps) - - @modal.method() - def inference(self, sequence: str): - num_steps = min(len(sequence), self.max_steps) - - print(f"running ESM3 inference with num_steps={num_steps}") - esm_protein = self.model.generate( - ESMProtein(sequence=sequence), self.get_generation_config(num_steps) - ) - - print("checking for errors in output") - if hasattr(esm_protein, "error_msg"): - raise ValueError(esm_protein.error_msg) - - print("converting ESMProtein into MMCIF file") - save_path = Path(tempfile.mktemp() + ".mmcif") - self.convert_protein_to_MMCIF(esm_protein, save_path) - - print("returning MMCIF bytes") - return io.BytesIO(save_path.read_bytes()) - - -``` - -## Serve a dashboard as an `asgi_app` - -In this section we'll create a web interface around the ESM3 model -that can help scientists and stakeholders understand and interrogate the results of the model. - -You can deploy this UI, along with the backing inference endpoint, -with the following command: - -```bash -modal deploy esm3.py -``` - -### Integrating Modal Functions - -The integration between our dashboard and our inference backend -is made simple by the Modal SDK: -because the definition of the `Model` class is available in the same Python -context as the defintion of the web UI, -we can instantiate an instance and call its methods with `.remote`. - -The inference runs in a GPU-accelerated container with all of ESM3's -dependencies, while this code executes in a CPU-only container -with only our web dependencies. - -```python -def run_esm(sequence: str) -> str: - sequence = sequence.strip() - - print("running ESM") - mmcif_buffer = Model().inference.remote(sequence) - - print("converting mmCIF bytes to base64 for compatibility with HTML") - mmcif_content = mmcif_buffer.read().decode() - mmcif_base64 = base64.b64encode(mmcif_content.encode()).decode() - - return get_molstar_html(mmcif_base64) - - -``` - -### Building a UI in Python with Gradio - -We'll visualize the results using [Mol\* ](https://molstar.org/). -Mol\* (pronounced "molstar") is an open-source toolkit for -visualizing and analyzing large-scale molecular data, including secondary structures -and residue-specific positions of proteins. - -Second, we'll create links to lookup the metadata and structure of known -proteins using the [Universal Protein Resource](https://www.uniprot.org/) -database from the UniProt consortium which is supported by the European -Bioinformatics Institute, the National Human Genome Research -Institute, and the Swiss Institute of Bioinformatics. UniProt -is also a hub that links to many other databases, like the RCSB Protein -Data Bank. - -To pull sequence data, we'll use the [Biotite](https://www.biotite-python.org/) -library to pull [FASTA](https://en.wikipedia.org/wiki/FASTA_format) files from -UniProt which contain labelled sequences. - -You should see the URL for this UI in the output of `modal deploy` -or on your [Modal app dashboard](https://modal.com/apps) for this app. - -```python -@app.function( - image=web_app_image, - volumes={VOLUME_PATH: volume}, - max_containers=1, # Gradio requires sticky sessions -) -@modal.concurrent(max_inputs=100) # Gradio can handle many async inputs -@modal.asgi_app() -def ui(): - import gradio as gr - from fastapi.responses import FileResponse - from gradio.routes import mount_gradio_app - - web_app = FastAPI() - - # custom styles: an icon, a background, and some CSS - @web_app.get("/favicon.ico", include_in_schema=False) - async def favicon(): - return FileResponse("/assets/favicon.svg") - - @web_app.get("/assets/background.svg", include_in_schema=False) - async def background(): - return FileResponse("/assets/background.svg") - - css = Path("/assets/index.css").read_text() - - theme = gr.themes.Default( - primary_hue="green", secondary_hue="emerald", neutral_hue="neutral" - ) - - title = "Predict & Visualize Protein Structures" - - with gr.Blocks(theme=theme, css=css, title=title, js=always_dark()) as interface: - gr.Markdown(f"# {title}") - - with gr.Row(): - with gr.Column(): - gr.Markdown("## Enter UniProt ID ") - uniprot_num_box = gr.Textbox( - label="Enter UniProt ID or select one on the right", - placeholder="e.g. P02768, P69905, etc.", - ) - get_sequence_button = gr.Button( - "Retrieve Sequence from UniProt ID", variant="primary" - ) - - uniprot_link_button = gr.Button(value="View protein on UniProt website") - uniprot_link_button.click( - fn=None, - inputs=uniprot_num_box, - js=get_js_for_uniprot_link(), - ) - - with gr.Column(): - example_uniprots = get_uniprot_examples() - - def extract_uniprot_num(example_idx): - uniprot = example_uniprots[example_idx] - return uniprot[uniprot.index("[") + 1 : uniprot.index("]")] - - gr.Markdown("## Example UniProt Accession Numbers") - with gr.Row(): - half_len = int(len(example_uniprots) / 2) - with gr.Column(): - for i, uniprot in enumerate(example_uniprots[:half_len]): - btn = gr.Button(uniprot, variant="secondary") - btn.click( - fn=lambda j=i: extract_uniprot_num(j), - outputs=uniprot_num_box, - ) - - with gr.Column(): - for i, uniprot in enumerate(example_uniprots[half_len:]): - btn = gr.Button(uniprot, variant="secondary") - btn.click( - fn=lambda j=i + half_len: extract_uniprot_num(j), - outputs=uniprot_num_box, - ) - - gr.Markdown("## Enter Sequence") - sequence_box = gr.Textbox( - label="Enter a sequence or retrieve it from a UniProt ID", - placeholder="e.g. MVTRLE..., PVTTIMHALL..., etc.", - ) - get_sequence_button.click( - fn=get_sequence, inputs=[uniprot_num_box], outputs=[sequence_box] - ) - - run_esm_button = gr.Button("Run ESM3 Folding", variant="primary") - - gr.Markdown("## ESM3 Predicted Structure") - molstar_html = gr.HTML() - - run_esm_button.click(fn=run_esm, inputs=sequence_box, outputs=molstar_html) - - # return a FastAPI app for Modal to serve - return mount_gradio_app(app=web_app, blocks=interface, path="/") - - -``` - -## Folding from the command line - -If you want to quickly run the ESM3 model without the web interface, you can -run it from the command line like this: - -```shell -modal run esm3 -``` - -This will run the same inference code above on Modal. The results are -returned in the [Crystallographic Information File](https://en.wikipedia.org/wiki/Crystallographic_Information_File) -format, which you can render with the online [Molstar Viewer](https://molstar.org/viewer/). - -```python -@app.local_entrypoint() -def main(sequence: Optional[str] = None, output_dir: Optional[str] = None): - if sequence is None: - print("using sequence for insulin [P01308]") - sequence = "MRTPMLLALLALATLCLAGRADAKPGDAESGKGAAFVSKQEGSEVVKRLRRYLDHWLGAPAPYPDPLEPKREVCELNPDCDELADHIGFQEAYRRFYGPV" - - if output_dir is None: - output_dir = Path("/tmp/esm3") - output_dir.mkdir(parents=True, exist_ok=True) - output_path = output_dir / "output.mmcif" - - print("starting inference on Modal") - results_buffer = Model().inference.remote(sequence) - - print(f"writing results to {output_path}") - output_path.write_bytes(results_buffer.read()) - - -``` - -## Addenda - -The remainder of this code is boilerplate. - -### Extracting Sequences from UniProt Accession Numbers - -To retrieve sequence information we'll utilize the `biotite` library which -will allow us to fetch [fasta](https://en.wikipedia.org/wiki/FASTA_format) -sequence files from the [National Center for Biotechnology Information (NCBI) Entrez database](https://www.ncbi.nlm.nih.gov/Web/Search/entrezfs.html). - -```python -def get_sequence(uniprot_num: str) -> str: - try: - DATA_PATH.mkdir(parents=True, exist_ok=True) - - uniprot_num = uniprot_num.strip() - fasta_path = DATA_PATH / f"{uniprot_num}.fasta" - - print(f"Fetching {fasta_path} from the entrez database") - entrez.fetch_single_file( - uniprot_num, fasta_path, db_name="protein", ret_type="fasta" - ) - fasta_file = fasta.FastaFile.read(fasta_path) - - protein_sequence = fasta.get_sequence(fasta_file) - return str(protein_sequence) - - except Exception as e: - return f"Error: {e}" - - -``` - -### Supporting functions for the Gradio app - -The following Python code is used to enhance the Gradio app, -mostly by generating some extra HTML & JS and handling styling. - -```python -def get_js_for_uniprot_link(): - url = "https://www.uniprot.org/uniprotkb/" - end = "/entry#structure" - return f"""(uni_id) => {{ if (!uni_id) return; window.open("{url}" + uni_id + "{end}"); }}""" - - -def get_molstar_html(mmcif_base64): - return f""" - """ - - -def get_uniprot_examples(): - return [ - "Albumin [P02768]", - "Insulin [P01308]", - "Hemoglobin [P69905]", - "Lysozyme [P61626]", - "BRCA1 [P38398]", - "Immunoglobulin [P01857]", - "Actin [P60709]", - "Ribonuclease [P07998]", - ] - - -def always_dark(): - return """ - function refresh() { - const url = new URL(window.location); - - if (url.searchParams.get('__theme') !== 'dark') { - url.searchParams.set('__theme', 'dark'); - window.location.href = url.href; - } - } - """ - -``` diff --git a/.agents/skills/modal/references/examples/esmfold2_binder_design.md b/.agents/skills/modal/references/examples/esmfold2_binder_design.md deleted file mode 100644 index 8753f5ede81fa49a2c01e69cb764a61b99d1b884..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/esmfold2_binder_design.md +++ /dev/null @@ -1,335 +0,0 @@ -# Design protein binders at scale with ESMFold2 and ESMC - -Protein folding was a landmark breakthrough in computational biology. -But for many applications, we don't just want to predict the structures of existing proteins — -we want to design new proteins that can modulate biology. - -One of the most important ways to do that is through binding. -Protein-protein interactions drive much of biological function, -and the ability to design molecules that bind specific targets -opens the door to new research tools and therapeutics. -Recent AI approaches have tackled binder design by inverting -structure prediction models via an iterative optimization process: - -1. Fold a candidate binder together with the target protein. -2. Score the resulting structure based on how well the binder folds and binds. -3. Take a step in sequence space that improves the score. -4. Repeat. - -In this example, we'll demonstrate how implement this process on Modal -using [ESMFold2 and ESMC](https://biohub.ai/esm/protein/about), state-of-the-art models -developed at [Biohub](https://biohub.ai/) that can predict the stucture of biomolecular complexes. -Check out their [technical report](https://modal-cdn.com/esmfold2_tech_report.pdf) -to see how the models were developed and used to design and experimentally validate binders against therapeutically relevant targets. - -We'll start by building a Modal Function that designs a single binder; then with only -a few more lines of code, we'll write an orchestrator function -that executes a large-scale search powered by Modal's autoscaling infrastructure and global GPU capacity. - -## Setup - -```python -from pathlib import Path -from typing import Optional - -import modal - -MINUTES = 60 # seconds -HOURS = 60 * MINUTES - -app = modal.App( - name="example-esmfold2-binder-design", -) - -``` - -## Defining our Modal Image - -We'll use `Image.micromamba` as our base image because a few of the packages we need -are only available via Conda. We'll also install the [`esm`](https://github.com/Biohub/esm) -library from CZ Biohub (which pulls in a custom fork of `transformers`) and a few other helpful libraries -for working with protein sequences. - -We set `CUBLAS_WORKSPACE_CONFIG` which allows us to ensure reproducibility by calling -`torch.use_deterministic_algorithms(True)` at the top of our remote code. - -```python -ESM_REVISION = ( - "f652b471d29da828b31e9b7a9cf7d0a7803240f5" # see https://github.com/Biohub/esm -) - -image = ( - modal.Image.micromamba(python_version="3.12") - .run_commands("apt update && apt install -y git build-essential") - .micromamba_install( - "anarci=2024.05.21-0", - channels=["conda-forge", "bioconda"], - ) - .uv_pip_install( - f"esm @ git+https://github.com/Biohub/esm.git@{ESM_REVISION}", - "abnumber==0.4.4", - "pyarrow==18.1.0", - ) - .env( - { - "HF_HOME": "/models", - "HF_XET_HIGH_PERFORMANCE": "1", # speed up Hugging Face downloads - "XFORMERS_IGNORE_FLASH_VERSION_CHECK": "1", - # required for torch.use_deterministic_algorithms(True) - "CUBLAS_WORKSPACE_CONFIG": ":4096:8", - } - ) -) - -``` - -## Caching weights and persisting results on Modal Volumes - -ESMFold2 builds on the 6B-parameter ESMC encoder; together with the four -critic models used for final scoring, the model weights come in around ~50 GB. -We cache them on a [Modal Volume](https://modal.com/docs/guide/volumes) -which delivers much better performance at cold-start time than re-downloading -from Hugging Face each time. - -```python -models_volume = modal.Volume.from_name("esmfold2-models", create_if_missing=True) -models_dir = Path("/models") - -``` - -A second Volume will store our results. - -```python -results_volume = modal.Volume.from_name( - "esmfold2-binder-design-results", create_if_missing=True -) -results_dir = Path("/results") - - -``` - -## Designing a binder on Modal - -To run binder design on Modal, we define a `BinderDesignService` class and -wrap it with the `@app.cls` decorator. The decorator takes arguments that -describe the infrastructure our code needs: the Image and both Volumes we -defined, plus an H100 GPU which has enough memory for the 6B-parameter ESMC encoder and the -four ESMFold2 "hero" critic models. - -Inside the class, the [`@modal.enter()` lifecycle hook](https://modal.com/docs/guide/lifecycle-functions#modalenter) -downloads and initializes those models once per container start, so subsequent -`design` calls on the same container reuse the loaded weights. - -We decorate our `design` method with `@modal.method()` to enable remote -execution. We'll see it called both via `.remote()` (single design) and via -`.spawn()` + [`modal.FunctionCall.gather`](https://modal.com/docs/reference/modal.FunctionCall) -(parallel sweep) further below. The class itself is a thin wrapper around -[`ESMFold2Designer`](https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/binder-design/binder_design/models.py) -from the helper package, which handles the actual model loading and the -gradient-guided optimization loop (`design_binder` in -[`binder_design.design`](https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/binder-design/binder_design/design.py)). - -```python -@app.cls( - image=image, - volumes={models_dir: models_volume}, - gpu="H100", - timeout=1 * HOURS, -) -class BinderDesignService: - """Modal entry point for ESMFold2-driven binder design. - - Set ``use_scaling_critics=True`` to also load the 15-checkpoint - scaling-experiment ensemble (distogram binding confidence only). - """ - - use_scaling_critics: bool = modal.parameter(default=False) - - @modal.enter() - def load(self): - from .binder_design import ESMFold2Designer - - self._designer = ESMFold2Designer() - self._designer.load(self.use_scaling_critics) - - @modal.method() - def design( - self, - target_name: Optional[str] = None, - target_sequence: Optional[str] = None, - binder_name: Optional[str] = None, - binder_sequence: Optional[str] = None, - is_antibody: Optional[bool] = None, - seed: int = 0, - batch_size: int = 1, - ): - return self._designer.design( - target_name=target_name, - target_sequence=target_sequence, - binder_name=binder_name, - binder_sequence=binder_sequence, - is_antibody=is_antibody, - seed=seed, - batch_size=batch_size, - ) - - -``` - -## Fanning out a sweep with selection - -A single design run gives you one candidate per batch slot. To recover the -kind of hit rates reported in the paper, you want many seeds, several binder -templates, and several targets, then a selection pass that ranks designs by -a combined ipTM / distogram-ipTM-proxy score. - -We orchestrate from inside a Modal Function so you don't have to worry about -keeping a long-running process alive locally or installing any local dependencies. - -```python -@app.function( - image=image, - volumes={results_dir: results_volume}, - gpu="H100", - timeout=2 * HOURS, -) -def run_sweep( - line_sweeps: dict[str, list], - use_scaling_critics: bool = False, - save_filename: str = "selection.parquet", -) -> bytes: - """Fan a grid sweep across GPUs, gather results, select top designs, ave results + return parquet.""" - import io - - from .binder_design.sweep import expand_sweep, select_designs - - designer = BinderDesignService(use_scaling_critics=use_scaling_critics) - configs = expand_sweep(line_sweeps) - - print(f"🧬 spawning {len(configs)} design jobs") - calls = [designer.design.spawn(**cfg) for cfg in configs] - raw_results = modal.FunctionCall.gather(*calls) - - df_select = select_designs(configs, raw_results) - - buf = io.BytesIO() - df_select.to_parquet(buf, index=False) - parquet_bytes = buf.getvalue() - - save_path = results_dir / save_filename - save_path.write_bytes(parquet_bytes) - results_volume.commit() - print(f"🧬 saved {len(df_select)} selected designs to volume:{save_path}") - - return parquet_bytes - - -``` - -## From the command line - -`main` runs a single design. Override the -`target_name` / `binder_name` to try one of the -[bundled targets](https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/binder-design/binder_design/prompts.py) -(`cd45`, `ctla4`, `egfr`, `pd-l1`, `pdgfr`) and binder templates -(`minibinder`, `trastuzumab_framework_vhvl`, `atezolizumab_framework_vhvl`, -`ocankitug_framework_vhvl`), or pass an arbitrary `target_sequence` / -`binder_sequence` directly. - -```shell -modal run -m 06_gpu_and_ml.binder-design.esmfold2_binder_design::main \ - --target-name pd-l1 --binder-name minibinder -``` - -```python -@app.local_entrypoint() -def main( - target_name: Optional[str] = "pd-l1", - target_sequence: Optional[str] = None, - binder_name: Optional[str] = "minibinder", - binder_sequence: Optional[str] = None, - is_antibody: Optional[bool] = None, - use_scaling_critics: bool = False, - seed: int = 0, - batch_size: int = 1, -): - designer = BinderDesignService(use_scaling_critics=use_scaling_critics) - seq, trajectory, results = designer.design.remote( - target_name=target_name, - target_sequence=target_sequence, - binder_name=binder_name, - binder_sequence=binder_sequence, - is_antibody=is_antibody, - seed=seed, - batch_size=batch_size, - ) - - avg_final_loss = sum(r["final_loss"] for r in results) / len(results) - print(f"🧬 designed sequence: {seq}") - print(f"🧬 trajectory length: {len(trajectory)} steps") - print(f"🧬 average final loss: {avg_final_loss:.4f}") - - -``` - -`sweep` runs a grid sweep across every `(target, binder, seed)` combination -of the targets and binders you pass in, scaling design horizontally with Modal's -[asynchronous job processing](https://modal.com/docs/guide/job-queue). -The selection pass runs server-side and the resulting parquet is -written to both the `esmfold2-binder-design-results` Volume and to a local -file for inspection. - -`target_names` and `binder_names` are passed as comma-separated strings. -The defaults sweep one target across two binder modalities -- a `minibinder` -and the `trastuzumab_framework_vhvl` antibody template -- so a single -command fans out across both at once: - -```shell -modal run -m 06_gpu_and_ml.binder-design.esmfold2_binder_design::sweep \ - --target-names pd-l1,ctla4 \ - --binder-names minibinder,trastuzumab_framework_vhvl \ - --n-seeds 8 -``` - -```python -@app.local_entrypoint() -def sweep( - target_names: str = "pd-l1", - binder_names: str = "minibinder,trastuzumab_framework_vhvl", - use_scaling_critics: bool = False, - n_seeds: int = 8, - output_path: Optional[str] = None, -): - target_name_list = [ - name.strip() for name in target_names.split(",") if name.strip() - ] - binder_name_list = [ - name.strip() for name in binder_names.split(",") if name.strip() - ] - - line_sweeps = { - "target_name": target_name_list, - "target_sequence": [None], - "binder_name": binder_name_list, - "binder_sequence": [None], - "seed": list(range(n_seeds)), - "batch_size": [1], - } - - print( - f"🧬 launching sweep: targets={target_name_list}, binders={binder_name_list}, " - f"n_seeds={n_seeds}, use_scaling_critics={use_scaling_critics}" - ) - parquet_bytes = run_sweep.remote( - line_sweeps, use_scaling_critics=use_scaling_critics - ) - - if output_path is None: - output_path = Path("/tmp") / "esmfold2_binder_design" / "selection.parquet" - else: - output_path = Path(output_path) - output_path.parent.mkdir(parents=True, exist_ok=True) - output_path.write_bytes(parquet_bytes) - print(f"🧬 wrote selection parquet to {output_path}") - -``` diff --git a/.agents/skills/modal/references/examples/fastrtc_flip_webcam.md b/.agents/skills/modal/references/examples/fastrtc_flip_webcam.md deleted file mode 100644 index 5e583a4a6f272f290ae936836856a02ea3c5623a..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/fastrtc_flip_webcam.md +++ /dev/null @@ -1,236 +0,0 @@ -# Run a FastRTC app on Modal - -[FastRTC](https://fastrtc.org/) is a Python library for real-time communication on the web. -This example demonstrates how to run a simple FastRTC app in the cloud on Modal. - -It's intended to help you get up and running with real-time streaming applications on Modal -as quickly as possible. If you're interested in running a production-grade WebRTC app on Modal, -see [this example](https://modal.com/docs/examples/webrtc_yolo). - -In this example, we stream webcam video from a browser to a container on Modal, -where the video is flipped, annotated, and sent back with under 100ms of delay. -You can try it out [here](https://modal-labs-examples--example-fastrtc-flip-webcam-ui.modal.run/) -or just dive straight into the code to run it yourself. - -## Set up FastRTC on Modal - -First, we import the `modal` SDK -and use it to define a [container image](https://modal.com/docs/guide/images) -with FastRTC and related dependencies. - -```python -import modal - -web_image = modal.Image.debian_slim(python_version="3.12").uv_pip_install( - "fastapi[standard]==0.115.4", - "fastrtc==0.0.23", - "gradio==5.7.1", - "opencv-python-headless==4.11.0.86", -) - -``` - -Then, we set that as the default Image on our Modal [App](https://modal.com/docs/guide/apps). - -```python -app = modal.App("example-fastrtc-flip-webcam", image=web_image) - -``` - -### Configure WebRTC streaming on Modal - -Under the hood, FastRTC uses the WebRTC -[APIs](https://www.w3.org/TR/webrtc/) and -[protocols](https://datatracker.ietf.org/doc/html/rfc8825). - -WebRTC provides low latency ("real-time") peer-to-peer communication -for Web applications, focusing on audio and video. -Considering that the Web is a platform originally designed -for high-latency, client-server communication of text and images, -that's no mean feat! - -In addition to protocols that implement this communication, -WebRTC includes APIs for describing and manipulating audio/video streams. -In this demo, we set a few simple parameters, like the direction of the webcam -and the minimum frame rate. See the -[MDN Web Docs for `MediaTrackConstraints`](https://developer.mozilla.org/en-US/docs/Web/API/MediaTrackConstraints) -for more. - -```python -TRACK_CONSTRAINTS = { - "width": {"exact": 640}, - "height": {"exact": 480}, - "frameRate": {"min": 30}, - "facingMode": { # https://developer.mozilla.org/en-US/docs/Web/API/MediaTrackSettings/facingMode - "ideal": "user" - }, -} - -``` - -In theory, the Internet is designed for peer-to-peer communication -all the way down to its heart, the Internet Protocol (IP): just send packets between IP addresses. -In practice, peer-to-peer communication on the contemporary Internet is fraught with difficulites, -from restrictive firewalls to finicky work-arounds for -[the exhaustion of IPv4 addresses](https://www.a10networks.com/glossary/what-is-ipv4-exhaustion/), -like [Carrier-Grade Network Address Translation (CGNAT)](https://en.wikipedia.org/wiki/Carrier-grade_NAT). - -So establishing peer-to-peer connections can be quite involved. -The protocol for doing so is called Interactive Connectivity Establishment (ICE). -It is described in [this RFC](https://datatracker.ietf.org/doc/html/rfc8445#section-2). - -ICE involves the peers exchanging a list of connections that might be used. -We use a fairly simple setup here, where our peer on Modal uses the -[Session Traversal Utilities for NAT (STUN)](https://datatracker.ietf.org/doc/html/rfc5389) -server provided by Google. A STUN server basically just reflects back to a client what their -IP address and port number appear to be when they talk to it. The peer on Modal communicates -that information to the other peer trying to connect to it -- in this case, a browser trying to share a webcam feed. -Note the use of `stun` and port `19302` in the URL in place of -something more familiar, like `http` and port `80`. - -```python -RTC_CONFIG = {"iceServers": [{"url": "stun:stun.l.google.com:19302"}]} - - -``` - -## Running a FastRTC app on Modal - -FastRTC builds on top of the [Gradio](https://www.gradio.app/docs) -library for defining Web UIs in Python. -Gradio in turn is compatible with the -[Asynchronous Server Gateway Interface (ASGI)](https://asgi.readthedocs.io/en/latest/) -protocol for asynchronous Python web servers, like -[FastAPI](https://fastrtc.org/userguide/streams/), -so we can host it on Modal's cloud platform using the -[`modal.asgi_app` decorator](https://modal.com/docs/guide/webhooks#serving-asgi-and-wsgi-apps) -with [Modal Function](https://modal.com/docs/guide/apps). - -But before we do that, we need to consider limits: -on how many peers can connect to one instance on Modal -and on how long they can stay connected. -We picked some sensible defaults to show how they interact -with the deployment parameters of the Modal Function. -You'll want to tune these for your application! - -```python -MAX_CONCURRENT_STREAMS = 10 # number of peers per instance on Modal - -MINUTES = 60 # seconds -TIME_LIMIT = 10 * MINUTES # time limit - - -@app.function( - # gradio requires sticky sessions - # so we limit the number of concurrent containers to 1 - # and allow that container to handle concurrent streams - max_containers=1, - scaledown_window=TIME_LIMIT + 1 * MINUTES, # add a small buffer to time limit -) -@modal.concurrent(max_inputs=MAX_CONCURRENT_STREAMS) # inputs per container -@modal.asgi_app() # ASGI on Modal -def ui(): - import fastrtc # WebRTC in Gradio - import gradio as gr # WebUIs in Python - from fastapi import FastAPI # asynchronous ASGI server framework - from gradio.routes import mount_gradio_app # connects Gradio and FastAPI - - with gr.Blocks() as blocks: # block-wise UI definition - gr.HTML( # simple HTML header - "

" - "Streaming Video Processing with Modal and FastRTC" - "

" - ) - - with gr.Column(): # a column of UI elements - fastrtc.Stream( # high-level media streaming UI element - modality="video", - mode="send-receive", - handler=flip_vertically, # handler -- handle incoming frame, produce outgoing frame - ui_args={"title": "Click 'Record' to flip your webcam in the cloud"}, - rtc_configuration=RTC_CONFIG, - track_constraints=TRACK_CONSTRAINTS, - concurrency_limit=MAX_CONCURRENT_STREAMS, # limit simultaneous connections - time_limit=TIME_LIMIT, # limit time per connection - ) - - return mount_gradio_app(app=FastAPI(), blocks=blocks, path="/") - - -``` - -To try this out for yourself, run - -```bash -modal serve 07_web/fastrtc_flip_webcam.py -``` - -and head to the `modal.run` URL that appears in your terminal. -You can also check on the application's dashboard -via the `modal.com` URL thatappears below it. - -The `modal serve` command produces a hot-reloading development server -- -try editing the `title` in the `ui_args` above and watch the server redeploy. - -This temporary deployment is tied to your terminal session. -To deploy permanently, run - -```bash -modal deploy 07_web_endponts/fastrtc_flip_webcam.py -``` - -Note that Modal is a serverless platform with [usage-based pricing](https://modal.com/pricing), -so this application will spin down and cost you nothing when it is not in use. - -## Addenda - -This FastRTC app is very much the "hello world" or "echo server" -of FastRTC: it just flips the incoming webcam stream and adds a "hello" message. -That logic appears below. - -```python -def flip_vertically(image): - import cv2 - import numpy as np - - image = image.astype(np.uint8) - - if image is None: - print("failed to decode image") - return - - # flip vertically and caption to show video was processed on Modal - image = cv2.flip(image, 0) - lines = ["Hello from Modal!"] - caption_image(image, lines) - - return image - - -def caption_image( - img, lines, font_scale=0.8, thickness=2, margin=10, font=None, color=None -): - import cv2 - - if font is None: - font = cv2.FONT_HERSHEY_SIMPLEX - if color is None: - color = (127, 238, 100, 128) # Modal Green - - # get text sizes - sizes = [cv2.getTextSize(line, font, font_scale, thickness)[0] for line in lines] - if not sizes: - return - - # position text in bottom right - pos_xs = [img.shape[1] - size[0] - margin for size in sizes] - - pos_ys = [img.shape[0] - margin] - for _width, height in reversed(sizes[:-1]): - next_pos = pos_ys[-1] - 2 * height - pos_ys.append(next_pos) - - for line, pos in zip(lines, zip(pos_xs, reversed(pos_ys))): - cv2.putText(img, line, pos, font, font_scale, color, thickness) - -``` diff --git a/.agents/skills/modal/references/examples/fine_tune_asr.md b/.agents/skills/modal/references/examples/fine_tune_asr.md deleted file mode 100644 index 565248f60d1c4db4c2bd40d2130df40d1272b2ea..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/fine_tune_asr.md +++ /dev/null @@ -1,614 +0,0 @@ -# Fine-tune Whisper to Improve Transcription on Domain-Specific Vocab - -This example demonstrates how to fine-tune an ASR model -([whisper-tiny.en](https://huggingface.co/openai/whisper-tiny.en)) -and deploy it for inference using Modal. - -Speech recognition models work well out-of-the-box for general speech transcription, -but can struggle with examples that are not well represented in the training data - -like proper nouns, technical jargon, and industry-specific terms. Fine-tuning with -examples of domain-specific vocabulary can improve transcription of these terms. - -For example, here is a sample transcription from the baseline model with no -fine-tuning: - -| | Transcription | -|------------------|---------------------------------------------------------------| -| **Ground Truth** | "deuterium you put into one element you make a new element" | -| **Prediction** | "the theorem you put into one element you make a new element" | - -After just 1.5 hours of training on a small dataset (~7k samples), the model has -already improved: - -| | Transcription | -|------------------|-------------------------------------------------------------| -| **Ground Truth** | "deuterium you put into one element you make a new element" | -| **Prediction** | "deuterium you put into one element you make a new element" | - -We'll use the "small" subset of "Science and Technology" from the -[GigaSpeech](https://huggingface.co/datasets/speechcolab/gigaspeech) -dataset, which is enough data to see the model improve on scientific terms in just a -few epochs. - -Note: GigaSpeech is a -[gated model](https://huggingface.co/docs/hub/en/models-gated), -so you'll need to accept the terms on the -[dataset card](https://huggingface.co/datasets/speechcolab/gigaspeech) -and create a -[Hugging Face Secret](https://modal.com/secrets/) -to download it. - -## Setup - -We start by importing our standard library dependencies, `fastapi`, and `modal`. - -We also need an [`App`](https://modal.com/docs/guide/apps) object, which we'll use to -define how our training application will run on Modal's cloud infrastructure. - -```python -import functools -import io -import os -from dataclasses import dataclass -from pathlib import Path -from typing import Annotated, Any, Union - -import fastapi -import modal - -MINUTES = 60 -HOURS = 60 * MINUTES - -app = modal.App(name="example-whisper-fine-tune") - -``` - -### Set up the container image - -We define the environment where our functions will run by building up a base -[container `Image`](https://modal.com/docs/guide/images) -with our dependencies using `Image.uv_pip_install`. We also set environment variables -here using `Image.env`, like the Hugging Face cache directory. - -```python -CACHE_DIR = "/cache" -image = ( - modal.Image.debian_slim(python_version="3.12") - .uv_pip_install( - "accelerate==1.8.1", - "datasets==3.6.0", - "evaluate==0.4.5", - "fastapi[standard]==0.116.1", - "huggingface-hub==0.36.0", - "jiwer==4.0.0", - "librosa==0.11.0", - "torch==2.7.1", - "torchaudio==2.7.1", - "transformers==4.53.2", - "whisper_normalizer==0.1.12", - ) - .env( - { - "HF_XET_HIGH_PERFORMANCE": "1", # Faster downloads from Hugging Face - "HF_HOME": CACHE_DIR, - } - ) -) - -``` - -Next we'll import the dependencies we need for the code that will run on Modal. - -The `image.imports()` context manager ensures these imports are available when our -Functions run in the cloud, without the need to install the dependencies locally. - -```python -with image.imports(): - import datasets - import evaluate - import librosa - import torch - import transformers - from whisper_normalizer.english import EnglishTextNormalizer - -``` - -### Storing data on Modal - -We use -[Modal Volumes](https://modal.com/docs/guide/volumes) -for data we want to persist across function calls. In this case, we'll create a cache -Volume for storing Hugging Face downloads for faster subsequent loads, and an output -Volume for saving our model and metrics after training. - -```python -cache_volume = modal.Volume.from_name("hf-hub-cache", create_if_missing=True) -output_volume = modal.Volume.from_name( - "fine-tune-asr-example-volume", - create_if_missing=True, -) -OUTPUT_DIR = "/outputs" -volumes = {CACHE_DIR: cache_volume, OUTPUT_DIR: output_volume} - -``` - -## Training - -We use a `dataclass` to collect some of the training parameters in one place. Here we -set `model_output_name` which is the directory on the Volume where our model will be -saved, and where we'll load it from when deploying the model for inference. - -```python -@dataclass -class Config: - """Training configuration.""" - - model_output_name: str = "whisper-fine-tune" # Name used for saving and loading - - # Model config - model_name: str = "openai/whisper-tiny.en" - - # Dataset config - dataset_name: str = "speechcolab/gigaspeech" - dataset_subset: str = "s" # "xs" for testing, "m", "l", "xl" for more data - dataset_split: str = "train" # The test and val splits don't have category labels - dataset_category: int = 15 # "Science and Technology" - max_duration_in_seconds: float = 20.0 - min_duration_in_seconds: float = 0.0 - - # Training config - num_train_epochs: int = 5 - warmup_steps: int = 400 - max_steps: int = -1 - batch_size: int = 64 - learning_rate: float = 1e-5 - eval_strategy: str = "epoch" - - -``` - -### Defining our training Function - -The training Function does the following: - -1. Load the pre-trained model, along with the feature extractor and tokenizer -2. Load the dataset -> select our training category -> extract features for training -3. Run baseline evals -4. 🚂 Train! -5. Save the fine-tuned model to the Volume -6. Run final evals - -We run evals before and after training to establish a baseline and see how much the -model improved. The most common way to measure the performance of speech recognition -models is "word error rate" (WER): - -`WER = (substitutions + deletions + insertions) / total words`. - -The `@app.function` decorator is where we attach infrastructure and define how our -Function runs on Modal. Here we tell the Function to use our `Image`, specify the GPU, -attach the Volumes we created earlier, add our access token, and set a timeout. - -```python -@app.function( - image=image, - gpu="H100", - volumes=volumes, - secrets=[modal.Secret.from_name("huggingface-secret", required_keys=["HF_TOKEN"])], - timeout=3 * HOURS, -) -def train( - config: Config, -): - """Loads data and trains the model.""" - - # Setting args for the Hugging Face trainer - training_args = transformers.Seq2SeqTrainingArguments( - output_dir=Path(OUTPUT_DIR) / config.model_output_name, - num_train_epochs=config.num_train_epochs, - per_device_train_batch_size=config.batch_size, - per_device_eval_batch_size=config.batch_size, - learning_rate=config.learning_rate, - warmup_steps=config.warmup_steps, - max_steps=config.max_steps, - eval_strategy=config.eval_strategy, - fp16=True, - group_by_length=True, - length_column_name="input_length", - predict_with_generate=True, - generation_max_length=40, - generation_num_beams=1, - ) - - print(f"Loading model: {config.model_name}") - feature_extractor = transformers.WhisperFeatureExtractor.from_pretrained( - pretrained_model_name_or_path=config.model_name, - ) - tokenizer = transformers.WhisperTokenizer.from_pretrained( - pretrained_model_name_or_path=config.model_name, - ) - model = transformers.WhisperForConditionalGeneration.from_pretrained( - pretrained_model_name_or_path=config.model_name, - ) - - print(f"Loading dataset: {config.dataset_name} {config.dataset_subset}") - dataset = ( - datasets.load_dataset( - config.dataset_name, - config.dataset_subset, - split=config.dataset_split, - num_proc=os.cpu_count(), - trust_remote_code=True, - ) - if config.dataset_name is not None - else get_test_dataset(config) - ) - - print("Preparing data") - max_input_length = config.max_duration_in_seconds * feature_extractor.sampling_rate - min_input_length = config.min_duration_in_seconds * feature_extractor.sampling_rate - - # Remove samples that are not from our target category (Science and Technology) - # Remove audio clips that are too short or too long - dataset = dataset.filter( - functools.partial( - filter_dataset, - dataset_category=config.dataset_category, - max_input_length=max_input_length, - min_input_length=min_input_length, - ), - input_columns=["category", "audio"], - num_proc=os.cpu_count(), - ) - - # Extract audio features and tokenize labels - dataset = dataset.map( - functools.partial( - prepare_dataset, - feature_extractor=feature_extractor, - tokenizer=tokenizer, - model_input_name=feature_extractor.model_input_names[0], - ), - batched=True, - remove_columns=dataset.column_names, - num_proc=os.cpu_count(), - desc="Feature extract + tokenize", - ) - - # Split the filtered dataset into train/validation sets - dataset = dataset.train_test_split(test_size=0.1, shuffle=True, seed=42) - - # Create a processor that combines the feature extractor and tokenizer - processor = transformers.WhisperProcessor( - feature_extractor=feature_extractor, - tokenizer=tokenizer, - ) - - # Custom data collator handles batching of variable-length audio sequences - data_collator = DataCollatorSpeechSeq2SeqWithPadding( - processor=processor, - decoder_start_token_id=model.config.decoder_start_token_id, - ) - - # Set up the Hugging Face trainer with all of our components - normalizer = EnglishTextNormalizer() - metric = evaluate.load("wer") - - trainer = transformers.Seq2SeqTrainer( - model=model, - args=training_args, - train_dataset=dataset["train"], - eval_dataset=dataset["test"], - processing_class=feature_extractor, - data_collator=data_collator, - compute_metrics=functools.partial( - compute_metrics, - tokenizer=tokenizer, - normalizer=normalizer, - metric=metric, - ), - ) - - print("Running evals before training to establish a baseline") - metrics = trainer.evaluate( - metric_key_prefix="baseline", - max_length=training_args.generation_max_length, - num_beams=training_args.generation_num_beams, - ) - trainer.log_metrics("baseline", metrics) - trainer.save_metrics("baseline", metrics) - - print(f"Starting training! Weights will be saved to '{training_args.output_dir}'") - train_result = trainer.train() - - # Save the model weights, tokenizer, and feature extractor - trainer.save_model() - tokenizer.save_pretrained(training_args.output_dir) - feature_extractor.save_pretrained(training_args.output_dir) - - # Log training metrics - metrics = train_result.metrics - metrics["train_samples"] = len(dataset["train"]) - trainer.log_metrics("train", metrics) - trainer.save_metrics("train", metrics) - trainer.save_state() - - # Final evaluation to see how much we improved - print("Running final evals") - metrics = trainer.evaluate( - metric_key_prefix="test", - max_length=training_args.generation_max_length, - num_beams=training_args.generation_num_beams, - ) - metrics["eval_samples"] = len(dataset["test"]) - - trainer.log_metrics("test", metrics) - trainer.save_metrics("test", metrics) - output_volume.commit() # Ensure everything is saved to the Volume - - print(f"\nTraining complete! Model saved to '{training_args.output_dir}'") - - -``` - -## Calling a Modal function from the command line - -The easiest way to invoke our training Function is by creating a `local_entrypoint` -- -our `main` function that runs locally and provides a command-line interface to trigger -training on Modal's cloud infrastructure. - -```python -@app.local_entrypoint() -def main(test: bool = False): - """Run Whisper fine-tuning on Modal.""" - if test: # for quick e2e test - config = Config( - dataset_name=None, - num_train_epochs=1.0, - warmup_steps=0, - max_steps=1, - ) - else: - config = Config() - - train.remote(config) - - -``` - -This will allow us to run this example with: - -```bash -modal run fine_tune_asr.py -``` - -Arguments passed to this function are turned in to CLI arguments automagically. For -example, adding `--test` will run a single step of training for end-to-end testing. - -```bash -modal run fine_tune_asr.py --test -``` - -Training will take ~1.5 hours, and will log WER and other metrics throughout the -run. - -Here are a few more examples of terms the model predicted correctly after fine-tuning: - -| **Base Model** | **Fine-tuned** | -|----------------|-----------------| -| and pm package | npm package | -| teach them | tritium | -| chromebox | chromevox | -| purposes | porpoises | -| difsoup | div soup | -| would you | widget | - -## Deploying our fine-tuned model for inference - -Once fine-tuning is complete, Modal makes it incredibly easy to deploy our new model. -We can define both our inference function and an endpoint using a Modal -[Cls](https://modal.com/docs/reference/modal.Cls). -This will allow us to take advantage of -[lifecycle hooks](https://modal.com/docs/guide/lifecycle-functions) -to load the model just once on container startup using the `@modal.enter` decorator. -We can use -[modal.fastapi\_endpoint](https://modal.com/docs/reference/modal.fastapi_endpoint) -to expose our inference function as a Web Function. - -```python -@app.cls( - image=image, - gpu="H100", - timeout=10 * MINUTES, - # scaledown_window=10 * MINUTES, - volumes=volumes, -) -class Inference: - model_name: str = modal.parameter(default=Config().model_output_name) - - @modal.enter() - def load_model(self): - """Load the model and processor on container startup.""" - - model = f"{OUTPUT_DIR}/{self.model_name}" - print(f"Loading model from {model}") - self.processor = transformers.WhisperProcessor.from_pretrained(model) - self.model = transformers.WhisperForConditionalGeneration.from_pretrained(model) - self.model.config.forced_decoder_ids = None - - @modal.method() - def transcribe( - self, - audio_bytes: bytes, - ) -> str: - # Resample audio to match the model's sample rate - model_sample_rate = self.processor.feature_extractor.sampling_rate - audio_data, sample_rate = librosa.load(io.BytesIO(audio_bytes), sr=None) - - audio_dataset = datasets.Dataset.from_dict( - {"audio": [{"array": audio_data, "sampling_rate": sample_rate}]} - ).cast_column("audio", datasets.Audio(sampling_rate=model_sample_rate)) - - # Audio -> features (log-mel spectrogram) - row = next(iter(audio_dataset)) - input_features = self.processor( - row["audio"]["array"], - sampling_rate=row["audio"]["sampling_rate"], - return_tensors="pt", - ).input_features - - # generate tokens -> decode to text - predicted_ids = self.model.generate(input_features) - transcription = self.processor.batch_decode( - predicted_ids, skip_special_tokens=True - )[0] - - return transcription - - @modal.fastapi_endpoint(method="POST", docs=True) - def web( - self, - audio_file: Annotated[bytes, fastapi.File()], - ) -> dict[str, str]: - """Defines an endpoint for calling inference.""" - - transcription = self.transcribe.local( # run in the same container - audio_bytes=audio_file, - ) - return {"transcription": transcription} - - -``` - -Deploy it with: - -```bash -modal deploy fine_tune_asr.py -``` - -Note: you can specify which model to load by passing the `model_name` as a -query parameter when calling the endpoint. This defaults to `model_output_name`, which -we set in our `Config` above, and is the name of the directory where our model -was saved. - -Here's an example of how to use this endpoint to transcribe an audio file: - -```bash -curl -X 'POST' \ -'https://your-workspace-name--example-whisper-fine-tune-inference-web.modal.run/?model_name=whisper-fine-tune' \ --H 'accept: application/json' \ --H 'Content-Type: multipart/form-data' \ --F 'audio_file=@your-audio-file.wav;type=audio/wav' -``` - -## Support code - -```python -def get_test_dataset(config, length=5): - return datasets.Dataset.from_dict( - { - "text": ["Modal"] * length, - "audio": [{"array": [1.0] * 16000, "sampling_rate": 16000}] * length, - "category": [config.dataset_category] * length, - } - ) - - -def filter_dataset( - category, audio, dataset_category, max_input_length, min_input_length -): - return ( - category == dataset_category - and len(audio["array"]) > min_input_length - and len(audio["array"]) < max_input_length - ) - - -def prepare_dataset(batch, feature_extractor, tokenizer, model_input_name): - """Batched: convert audio to features and text to token IDs.""" - audio_arrays = [s["array"] for s in batch["audio"]] - inputs = feature_extractor( - audio_arrays, - sampling_rate=feature_extractor.sampling_rate, - ) - batch[model_input_name] = inputs.get(model_input_name) - batch["input_length"] = [len(s["array"]) for s in batch["audio"]] - - normalized = [ - t.replace(" ", ",") - .replace(" ", ".") - .replace(" ", "?") - .replace(" ", "!") - .lower() - .strip() - for t in batch["text"] - ] - batch["labels"] = tokenizer(normalized).input_ids - - return batch - - -def compute_metrics(pred, tokenizer, normalizer, metric): - """Compute Word Error Rate between predictions and ground truth.""" - pred_ids = pred.predictions - - # Replace padding tokens with proper pad token ID - pred.label_ids[pred.label_ids == -100] = tokenizer.pad_token_id - - # Decode predictions and labels back to text - pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True) - norm_pred_str = [normalizer(s).strip() for s in pred_str] - - label_str = tokenizer.batch_decode(pred.label_ids, skip_special_tokens=True) - norm_label_str = [normalizer(s).strip() for s in label_str] - - # Calculate Word Error Rate - wer = metric.compute(predictions=norm_pred_str, references=norm_label_str) - return {"wer": wer} - - -@dataclass -class DataCollatorSpeechSeq2SeqWithPadding: - """ - Data collator that pads audio features and text labels for batch training. - - Args: - processor: WhisperProcessor combining feature extractor and tokenizer - decoder_start_token_id: The BOS token ID for the decoder - """ - - processor: Any - decoder_start_token_id: int - - def __call__( - self, features: list[dict[str, Union[list[int], "torch.Tensor"]]] - ) -> dict[str, "torch.Tensor"]: - # Separate audio features and text labels since they need different padding - model_input_name = self.processor.model_input_names[0] - input_features = [ - {model_input_name: feature[model_input_name]} for feature in features - ] - label_features = [{"input_ids": feature["labels"]} for feature in features] - - batch = self.processor.feature_extractor.pad( - input_features, - return_tensors="pt", - return_attention_mask=True, - padding=True, - ) - - labels_batch = self.processor.tokenizer.pad(label_features, return_tensors="pt") - - # Replace padding tokens with -100 so they're ignored in loss calculation - labels = labels_batch["input_ids"].masked_fill( - labels_batch.attention_mask.ne(1), -100 - ) - - # Remove start token if tokenizer added it - model will add it during training - if (labels[:, 0] == self.decoder_start_token_id).all().cpu().item(): - labels = labels[:, 1:] - - batch["labels"] = labels - - return batch - -``` diff --git a/.agents/skills/modal/references/examples/finetune_yolo.md b/.agents/skills/modal/references/examples/finetune_yolo.md deleted file mode 100644 index 87c3f813c096f8322f9bac7c77c1dcb5a0e3d5cd..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/finetune_yolo.md +++ /dev/null @@ -1,375 +0,0 @@ -# Fine-tune open source YOLO models for object detection - -Example by [@Erik-Dunteman](https://github.com/erik-dunteman) and [@AnirudhRahul](https://github.com/AnirudhRahul/). - -The popular "You Only Look Once" (YOLO) model line provides high-quality object detection in an economical package. -In this example, we use the [YOLOv10](https://docs.ultralytics.com/models/yolov10/) model, released on May 23, 2024. - -We will: - -* Download two custom datasets from the [Roboflow](https://roboflow.com/) computer vision platform: a dataset of cats and a dataset of dogs - -* Fine-tune the model on those datasets, in parallel, using the [Ultralytics package](https://docs.ultralytics.com/) - -* Run inference with the fine-tuned models on single images and on streaming frames - -For commercial use, be sure to consult the [Ultralytics software license options](https://docs.ultralytics.com/#yolo-licenses-how-is-ultralytics-yolo-licensed), -which include AGPL-3.0. - -## Set up the environment - -```python -import warnings -from dataclasses import dataclass -from datetime import datetime -from pathlib import Path - -import modal - -``` - -Modal runs your code in the cloud inside containers. So to use it, we have to define the dependencies -of our code as part of the container's [image](https://modal.com/docs/guide/custom-container). - -```python -image = ( - modal.Image.debian_slim(python_version="3.10") - .apt_install( # install system libraries for graphics handling, model download - ["libgl1-mesa-glx", "libglib2.0-0", "curl"] - ) - .uv_pip_install( # install python libraries for computer vision - ["ultralytics~=8.2.68", "roboflow~=1.1.37", "opencv-python~=4.10.0"] - ) - .uv_pip_install( # add an optional extra that renders images in the terminal - "term-image==0.7.1" - ) -) - -``` - -We also create a persistent [Volume](https://modal.com/docs/guide/volumes) for storing datasets, trained weights, and inference outputs. For more on storing model weights on Modal, see -[this guide](https://modal.com/docs/guide/model-weights). - -```python -volume = modal.Volume.from_name("example-yolo-finetune", create_if_missing=True) -volume_path = ( # the path to the volume from within the container - Path("/root") / "data" -) - -``` - -We attach both of these to a Modal [App](https://modal.com/docs/guide/apps). - -```python -app = modal.App("example-yolo-finetune", image=image, volumes={volume_path: volume}) - - -``` - -## Download a dataset - -We'll be downloading our data from the [Roboflow](https://roboflow.com/) computer vision platform, so to follow along you'll need to: - -* Create a free account on [Roboflow](https://app.roboflow.com/) - -* [Generate a Private API key](https://app.roboflow.com/settings/api) - -* Set up a Modal [Secret](https://modal.com/docs/guide/secrets) called `roboflow-api-key` in the Modal UI [here](https://modal.com/secrets), - setting the `ROBOFLOW_API_KEY` to the value of your API key. - -You're also free to bring your own dataset with a config in YOLOv10-compatible yaml format. - -We'll be training on the medium size model, but you're free to experiment with [other model sizes](https://docs.ultralytics.com/models/yolov10/#model-variants). - -```python -@dataclass -class DatasetConfig: - """Information required to download a dataset from Roboflow.""" - - workspace_id: str - project_id: str - version: int - format: str - target_class: str - - @property - def id(self) -> str: - return f"{self.workspace_id}/{self.project_id}/{self.version}" - - -@app.function( - secrets=[ - modal.Secret.from_name("roboflow-api-key", required_keys=["ROBOFLOW_API_KEY"]) - ] -) -def download_dataset(config: DatasetConfig): - import os - - from roboflow import Roboflow - - rf = Roboflow(api_key=os.getenv("ROBOFLOW_API_KEY")) - project = ( - rf.workspace(config.workspace_id) - .project(config.project_id) - .version(config.version) - ) - dataset_dir = volume_path / "dataset" / config.id - project.download(config.format, location=str(dataset_dir)) - - -``` - -## Train a model - -We train the model on a single A100 GPU. Training usually takes only a few minutes. - -```python -MINUTES = 60 - -TRAIN_GPU_COUNT = 1 -TRAIN_GPU = f"A100:{TRAIN_GPU_COUNT}" -TRAIN_CPU_COUNT = 4 - - -@app.function( - gpu=TRAIN_GPU, - cpu=TRAIN_CPU_COUNT, - timeout=60 * MINUTES, -) -def train( - model_id: str, - dataset: DatasetConfig, - model_size="yolov10m.pt", - quick_check=False, -): - from ultralytics import YOLO - - volume.reload() # make sure volume is synced - - model_path = volume_path / "runs" / model_id - model_path.mkdir(parents=True, exist_ok=True) - - data_path = volume_path / "dataset" / dataset.id / "data.yaml" - - model = YOLO(model_size) - model.train( - # dataset config - data=data_path, - fraction=0.4 - if not quick_check - else 0.04, # fraction of dataset to use for training/validation - # optimization config - device=list(range(TRAIN_GPU_COUNT)), # use the GPU(s) - epochs=8 if not quick_check else 1, # pass over entire dataset this many times - batch=0.95, # automatic batch size to target fraction of GPU util - seed=117, # set seed for reproducibility - # data processing config - workers=max( - TRAIN_CPU_COUNT // TRAIN_GPU_COUNT, 1 - ), # split CPUs evenly across GPUs - cache=False, # cache preprocessed images in RAM? - # model saving config - project=f"{volume_path}/runs", - name=model_id, - exist_ok=True, # overwrite previous model if it exists - verbose=True, # detailed logs - ) - - -``` - -## Run inference on single inputs and on streams - -We demonstrate two different ways to run inference -- on single images and on a stream of images. - -The images we use for inference are loaded from the test set, which was added to our Volume when we downloaded the dataset. -Each image read takes ~50ms, and inference can take ~5ms, so the disk read would be our biggest bottleneck if we just looped over the image paths. -To avoid it, we parallelize the disk reads across many workers using Modal's [`.map`](https://modal.com/docs/guide/scale), -streaming the images to the model. This roughly mimics the behavior of an interactive object detection pipeline. -This can increase throughput up to ~60 images/s, or ~17 milliseconds/image, depending on image size. - -```python -@app.function() -def read_image(image_path: str): - import cv2 - - source = cv2.imread(image_path) - return source - - -``` - -We use the `@enter` feature of [`modal.Cls`](https://modal.com/docs/guide/lifecycle-functions) -to load the model only once on container start and reuse it for future inferences. -We use a generator to stream images to the model. - -```python -@app.cls(gpu="a10g") -class Inference: - weights_path: str = modal.parameter() - - @modal.enter() - def load_model(self): - from ultralytics import YOLO - - self.model = YOLO(self.weights_path) - - @modal.method() - def predict(self, model_id: str, image_path: str, display: bool = False): - """A simple method for running inference on one image at a time.""" - results = self.model.predict( - image_path, - half=True, # use fp16 - save=True, - exist_ok=True, - project=f"{volume_path}/predictions/{model_id}", - ) - if display: - from term_image.image import from_file - - terminal_image = from_file(results[0].path) - terminal_image.draw() - # you can view the output file via the Volumes UI in the Modal dashboard -- https://modal.com/storage - - @modal.method() - def streaming_count(self, batch_dir: str, threshold: float | None = None): - """Counts the number of objects in a directory of images. - - Intended as a demonstration of high-throughput streaming inference.""" - import os - import time - - image_files = [os.path.join(batch_dir, f) for f in os.listdir(batch_dir)] - - completed, start = 0, time.monotonic_ns() - for image in read_image.map(image_files): - # note that we run predict on a single input at a time. - # each individual inference is usually done before the next image arrives, so there's no throughput benefit to batching. - results = self.model.predict( - image, - half=True, # use fp16 - save=False, # don't save to disk, as it slows down the pipeline significantly - verbose=False, - ) - completed += 1 - for res in results: - for conf in res.boxes.conf: - if threshold is None: - yield 1 - continue - if conf.item() >= threshold: - yield 1 - yield 0 - - elapsed_seconds = (time.monotonic_ns() - start) / 1e9 - print( - "Inferences per second:", - round(completed / elapsed_seconds, 2), - ) - - -``` - -## Running the example - -We'll kick off our parallel training jobs and run inference from the command line. - -```bash -modal run finetune_yolo.py -``` - -This runs the training in `quick_check` mode, useful for debugging the pipeline and getting a feel for it. -To do a longer run that actually meaningfully improves performance, use: - -```bash -modal run finetune_yolo.py --no-quick-check -``` - -```python -@app.local_entrypoint() -def main(quick_check: bool = True, inference_only: bool = False): - """Run fine-tuning and inference on two datasets. - - Args: - quick_check: fine-tune on a small subset. Lower quality results, but faster iteration. - inference_only: skip fine-tuning and only run inference - """ - - dogs = DatasetConfig( - workspace_id="cv-project-v2", - project_id="6-dog-breeds", - version=1, - format="yolov9", - target_class="🐶", - ) - cats = DatasetConfig( - workspace_id="jus-workspace", - project_id="cats-w7ohy", - version=3, - format="yolov9", - target_class="🐱", - ) - datasets = [dogs, cats] - - # .for_each runs a function once on each element of the input iterators - # here, that means download each dataset, in parallel - if not inference_only: - download_dataset.for_each(datasets) - - today = datetime.now().strftime("%Y-%m-%d") - model_ids = [dataset.id + f"/{today}" for dataset in datasets] - - if not inference_only: - train.for_each(model_ids, datasets, kwargs={"quick_check": quick_check}) - - # let's run inference! - for model_id, dataset in zip(model_ids, datasets): - inference = Inference( - weights_path=str(volume_path / "runs" / model_id / "weights" / "best.pt") - ) - - # predict on a single image and save output to the volume - test_images = volume.listdir( - str(Path("dataset") / dataset.id / "test" / "images") - ) - # run inference on the first 5 images - for ii, image in enumerate(test_images): - print(f"{model_id}: Single image prediction on image", image.path) - inference.predict.remote( - model_id=model_id, - image_path=f"{volume_path}/{image.path}", - display=( - ii == 0 # display inference results only on first image - ), - ) - if ii >= 4: - break - - # streaming inference on images from the test set - print(f"{model_id}: Streaming inferences on all images in the test set...") - count = 0 - for detection in inference.streaming_count.remote_gen( - batch_dir=f"{volume_path}/dataset/{dataset.id}/test/images" - ): - if detection: - print(f"{dataset.target_class}", end="") - count += 1 - else: - print("🎞️", end="", flush=True) - print(f"\n{model_id}: Counted {count} {dataset.target_class}s!") - - -``` - -## Addenda - -The rest of the code in this example is utility code. - -```python -warnings.filterwarnings( # filter warning from the terminal image library - "ignore", - message="It seems this process is not running within a terminal. Hence, some features will behave differently or be disabled.", - category=UserWarning, -) - -``` diff --git a/.agents/skills/modal/references/examples/flux.md b/.agents/skills/modal/references/examples/flux.md deleted file mode 100644 index 38e5567e038bd0c8ac449ac8bca4816cbc058c0d..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/flux.md +++ /dev/null @@ -1,291 +0,0 @@ -# Run Flux fast on H100s with `torch.compile` - -*Update: To speed up inference by another >2x, check out the additional optimization -techniques we tried in [this blog post](https://modal.com/blog/flux-3x-faster)!* - -In this guide, we'll run Flux as fast as possible on Modal using open source tools. -We'll use `torch.compile` and NVIDIA H100 GPUs. - -## Setting up the image and dependencies - -```python -import time -from io import BytesIO -from pathlib import Path - -import modal - -``` - -We'll make use of the full [CUDA toolkit](https://modal.com/docs/guide/cuda) -in this example, so we'll build our container image off of the `nvidia/cuda` base. - -```python -cuda_version = "12.4.0" # should be no greater than host CUDA version -flavor = "devel" # includes full CUDA toolkit -operating_sys = "ubuntu22.04" -tag = f"{cuda_version}-{flavor}-{operating_sys}" - -cuda_dev_image = modal.Image.from_registry( - f"nvidia/cuda:{tag}", add_python="3.11" -).entrypoint([]) - -``` - -Now we install most of our dependencies with `apt` and `pip`. -For Hugging Face's [Diffusers](https://github.com/huggingface/diffusers) library -we install from GitHub source and so pin to a specific commit. - -PyTorch added faster attention kernels for Hopper GPUs in version 2.5. - -```python -diffusers_commit_sha = "81cf3b2f155f1de322079af28f625349ee21ec6b" - -flux_image = ( - cuda_dev_image.apt_install( - "git", - "libglib2.0-0", - "libsm6", - "libxrender1", - "libxext6", - "ffmpeg", - "libgl1", - ) - .uv_pip_install( - "invisible_watermark==0.2.0", - "transformers==4.44.0", - "huggingface-hub==0.36.0", - "accelerate==0.33.0", - "safetensors==0.4.4", - "sentencepiece==0.2.0", - "torch==2.5.0", - f"git+https://github.com/huggingface/diffusers.git@{diffusers_commit_sha}", - "numpy<2", - ) - .env({"HF_XET_HIGH_PERFORMANCE": "1", "HF_HUB_CACHE": "/cache"}) -) - -``` - -Later, we'll also use `torch.compile` to increase the speed further. -Torch compilation needs to be re-executed when each new container starts, -so we turn on some extra caching to reduce compile times for later containers. - -```python -flux_image = flux_image.env( - { - "TORCHINDUCTOR_CACHE_DIR": "/root/.inductor-cache", - "TORCHINDUCTOR_FX_GRAPH_CACHE": "1", - } -) - -``` - -Finally, we construct our Modal [App](https://modal.com/docs/reference/modal.App), -set its default image to the one we just constructed, -and import `FluxPipeline` for downloading and running Flux.1. - -```python -app = modal.App("example-flux", image=flux_image) - -with flux_image.imports(): - import torch - from diffusers import FluxPipeline - -``` - -## Defining a parameterized `Model` inference class - -Next, we map the model's setup and inference code onto Modal. - -1. We run the model setup in the method decorated with `@modal.enter()`. This includes loading the - weights and moving them to the GPU, along with an optional `torch.compile` step (see details below). - The `@modal.enter()` decorator ensures that this method runs only once, when a new container starts, - instead of in the path of every call. - -2. We run the actual inference in methods decorated with `@modal.method()`. - -*Note: Access to the Flux.1-schnell model on Hugging Face is -[gated by a license agreement](https://huggingface.co/docs/hub/en/models-gated) -which you must agree to -[here](https://huggingface.co/black-forest-labs/FLUX.1-schnell). -After you have accepted the license, -[create a Modal Secret](https://modal.com/secrets) -with the name `huggingface-secret` following the instructions in the template.* - -```python -MINUTES = 60 # seconds -VARIANT = "schnell" # or "dev" -NUM_INFERENCE_STEPS = 4 # use ~50 for [dev], smaller for [schnell] - - -@app.cls( - gpu="H100", # fast GPU with strong software support - scaledown_window=20 * MINUTES, - timeout=60 * MINUTES, # leave plenty of time for compilation - volumes={ # add Volumes to store serializable compilation artifacts, see section on torch.compile below - "/cache": modal.Volume.from_name("hf-hub-cache", create_if_missing=True), - "/root/.nv": modal.Volume.from_name("nv-cache", create_if_missing=True), - "/root/.triton": modal.Volume.from_name("triton-cache", create_if_missing=True), - "/root/.inductor-cache": modal.Volume.from_name( - "inductor-cache", create_if_missing=True - ), - }, - secrets=[modal.Secret.from_name("huggingface-secret")], -) -class Model: - compile: bool = ( # see section on torch.compile below for details - modal.parameter(default=False) - ) - - @modal.enter() - def enter(self): - pipe = FluxPipeline.from_pretrained( - f"black-forest-labs/FLUX.1-{VARIANT}", torch_dtype=torch.bfloat16 - ).to("cuda") # move model to GPU - self.pipe = optimize(pipe, compile=self.compile) - - @modal.method() - def inference(self, prompt: str) -> bytes: - print("🎨 generating image...") - out = self.pipe( - prompt, - output_type="pil", - num_inference_steps=NUM_INFERENCE_STEPS, - ).images[0] - - byte_stream = BytesIO() - out.save(byte_stream, format="JPEG") - return byte_stream.getvalue() - - -``` - -## Calling our inference function - -To generate an image we just need to call the `Model`'s `generate` method -with `.remote` appended to it. -You can call `.generate.remote` from any Python environment that has access to your Modal credentials. -The local environment will get back the image as bytes. - -Here, we wrap the call in a Modal [`local_entrypoint`](https://modal.com/docs/reference/modal.App#local_entrypoint) -so that it can be run with `modal run`: - -```bash -modal run flux.py -``` - -By default, we call `generate` twice to demonstrate how much faster -the inference is after cold start. In our tests, clients received images in about 1.2 seconds. -We save the output bytes to a temporary file. - -```python -@app.local_entrypoint() -def main( - prompt: str = "a computer screen showing ASCII terminal art of the" - " word 'Modal' in neon green. two programmers are pointing excitedly" - " at the screen.", - twice: bool = True, - compile: bool = False, -): - t0 = time.time() - image_bytes = Model(compile=compile).inference.remote(prompt) - print(f"🎨 first inference latency: {time.time() - t0:.2f} seconds") - - if twice: - t0 = time.time() - image_bytes = Model(compile=compile).inference.remote(prompt) - print(f"🎨 second inference latency: {time.time() - t0:.2f} seconds") - - output_path = Path("/tmp") / "flux" / "output.jpg" - output_path.parent.mkdir(exist_ok=True, parents=True) - print(f"🎨 saving output to {output_path}") - output_path.write_bytes(image_bytes) - - -``` - -## Speeding up Flux with `torch.compile` - -By default, we do some basic optimizations, like adjusting memory layout -and re-expressing the attention head projections as a single matrix multiplication. -But there are additional speedups to be had! - -PyTorch 2 added a compiler that optimizes the -compute graphs created dynamically during PyTorch execution. -This feature helps close the gap with the performance of static graph frameworks -like TensorRT and TensorFlow. - -Here, we follow the suggestions from Hugging Face's -[guide to fast diffusion inference](https://huggingface.co/docs/diffusers/en/tutorials/fast_diffusion), -which we verified with our own internal benchmarks. -Review that guide for detailed explanations of the choices made below. - -The resulting compiled Flux `schnell` deployment returns images to the client in under a second (~700 ms), according to our testing. -*Super schnell*! - -Compilation takes up to twenty minutes on first iteration. -As of time of writing in late 2024, -the compilation artifacts cannot be fully serialized, -so some compilation work must be re-executed every time a new container is started. -That includes when scaling up an existing deployment or the first time a Function is invoked with `modal run`. - -We cache compilation outputs from `nvcc`, `triton`, and `inductor`, -which can reduce compilation time by up to an order of magnitude. -For details see [this tutorial](https://pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html). - -You can turn on compilation with the `--compile` flag. -Try it out with: - -```bash -modal run flux.py --compile -``` - -The `compile` option is passed by a [`modal.parameter`](https://modal.com/docs/reference/modal.parameter#modalparameter) on our class. -Each different choice for a `parameter` creates a [separate auto-scaling deployment](https://modal.com/docs/guide/parameterized-functions). -That means your client can use arbitrary logic to decide whether to hit a compiled or eager endpoint. - -```python -def optimize(pipe, compile=True): - # fuse QKV projections in Transformer and VAE - pipe.transformer.fuse_qkv_projections() - pipe.vae.fuse_qkv_projections() - - # switch memory layout to Torch's preferred, channels_last - pipe.transformer.to(memory_format=torch.channels_last) - pipe.vae.to(memory_format=torch.channels_last) - - if not compile: - return pipe - - # set torch compile flags - config = torch._inductor.config - config.disable_progress = False # show progress bar - config.conv_1x1_as_mm = True # treat 1x1 convolutions as matrix muls - # adjust autotuning algorithm - config.coordinate_descent_tuning = True - config.coordinate_descent_check_all_directions = True - config.epilogue_fusion = False # do not fuse pointwise ops into matmuls - - # tag the compute-intensive modules, the Transformer and VAE decoder, for compilation - pipe.transformer = torch.compile( - pipe.transformer, mode="max-autotune", fullgraph=True - ) - pipe.vae.decode = torch.compile( - pipe.vae.decode, mode="max-autotune", fullgraph=True - ) - - # trigger torch compilation - print("🔦 running torch compilation (may take up to 20 minutes)...") - - pipe( - "dummy prompt to trigger torch compilation", - output_type="pil", - num_inference_steps=NUM_INFERENCE_STEPS, # use ~50 for [dev], smaller for [schnell] - ).images[0] - - print("🔦 finished torch compilation") - - return pipe - -``` diff --git a/.agents/skills/modal/references/examples/generate_music.md b/.agents/skills/modal/references/examples/generate_music.md deleted file mode 100644 index 132866518a2604c5848c921cd4f7388f921f20ea..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/generate_music.md +++ /dev/null @@ -1,320 +0,0 @@ -# Make music with ACE-Step 1.5 - -In this example, we show you how you can run [ACE Studio](https://acestudio.ai/)'s -[ACE-Step 1.5](https://github.com/ace-step/ACE-Step-1.5) music generation model -on Modal. - -ACE-Step 1.5 introduces a multi-model architecture: -a DiT (Diffusion Transformer) handler for audio generation -and an LM (Language Model) handler for prompt augmentation. -The LM automatically enhances prompts, detects language, -and generates metadata like BPM and key. - -We'll set up both a serverless music generation service -and a web user interface. - -## Setting up dependencies - -```python -from pathlib import Path -from typing import Optional -from uuid import uuid4 - -import modal - -``` - -We start by defining the environment our generation runs in. -This takes some explaining since, like most cutting-edge ML environments, it is a bit fiddly. - -This environment is captured by a -[container image](https://modal.com/docs/guide/images), -which we build step-by-step by calling methods to add dependencies, -like `apt_install` to add system packages and `uv_pip_install` to add -Python packages. - -ACE-Step 1.5 uses a local path dependency (`nano-vllm`) in its -package configuration, so we clone the repo first and install from -the local directory. This lets `uv` resolve all dependencies together, -including the CUDA-enabled PyTorch build and the local `nano-vllm` package. - -```python -image = ( - modal.Image.from_registry( - "nvidia/cuda:13.0.0-cudnn-devel-ubuntu22.04", add_python="3.12" - ) - .apt_install("git", "ffmpeg") - .run_commands( - "git clone --branch v0.1.6 --depth 1 https://github.com/ace-step/ACE-Step-1.5.git /opt/ace-step", - ) - .uv_pip_install( - "/opt/ace-step", "hf_transfer==0.1.9", "torchcodec==0.10.0", "torch~=2.10.0" - ) - .entrypoint([]) -) - -``` - -In addition to source code, we'll also need the model weights. - -ACE-Step 1.5 integrates with the Hugging Face ecosystem, so setting up the models -is straightforward. The model handlers use Hugging Face -to download the weights if not already present. - -We use a single `checkpoints/` directory for all model downloads -(both the DiT and LM models) and persist it with a Modal -[Volume](https://modal.com/docs/guide/volumes). -For more on storing model weights on Modal, see -[this guide](https://modal.com/docs/guide/model-weights). - -```python -checkpoints_dir = "/opt/ace-step/checkpoints" -model_cache = modal.Volume.from_name("ACE-Step-v15-model-cache", create_if_missing=True) - -``` - -We set the `ACESTEP_PROJECT_ROOT` environment variable so that -the model handlers know where to find the checkpoints directory. - -```python -image = image.env( - {"ACESTEP_PROJECT_ROOT": "/opt/ace-step", "HF_HUB_ENABLE_HF_TRANSFER": "1"} -) - -``` - -While we're at it, let's also define the environment for our UI. -We'll stick with Python and so use FastAPI and Gradio. - -```python -web_image = modal.Image.debian_slim(python_version="3.12").uv_pip_install( - "fastapi[standard]==0.115.4", - "gradio==6.11.0", - "huggingface-hub==1.9.1", - "pydantic==2.10.1", -) - -``` - -This is a totally different environment from the one we run our model in. -Say goodbye to Python dependency conflict hell! - -## Running music generation on Modal - -Now, we write our music generation logic. - -* We make an [App](https://modal.com/docs/guide/apps) to organize our deployment. -* We load the model at start, instead of during inference, with `modal.enter`, - which requires that we use a Modal [`Cls`](https://modal.com/docs/guide/lifecycle-functions). -* In the `app.cls` decorator, we specify the Image we built and attach the Volume. - We also pick a GPU to run on -- here, an NVIDIA L40S. - -```python -app = modal.App("example-generate-music") - - -@app.cls(gpu="l40s", image=image, volumes={checkpoints_dir: model_cache}) -class MusicGenerator: - @modal.enter() - def init(self): - from acestep.handler import AceStepHandler - from acestep.llm_inference import LLMHandler - from acestep.model_downloader import ensure_lm_model, ensure_main_model - - # Download models if not already cached in the Volume. - lm_model_name = "acestep-5Hz-lm-4B" - ensure_main_model(checkpoints_dir=checkpoints_dir) - ensure_lm_model(model_name=lm_model_name, checkpoints_dir=checkpoints_dir) - - # Initialize the audio generation model. - self.dit_handler = AceStepHandler() - init_status, enable_generate = self.dit_handler.initialize_service( - project_root="/opt/ace-step", - config_path="acestep-v15-turbo", - device="cuda", - ) - if not enable_generate: - raise RuntimeError(f"DiT model initialization failed: {init_status}") - - # Initialize the language model for prompt enhancement. - self.llm_handler = LLMHandler() - lm_status, lm_success = self.llm_handler.initialize( - checkpoint_dir=checkpoints_dir, - lm_model_path=lm_model_name, - backend="vllm", - device="cuda", - ) - if not lm_success: - raise RuntimeError(f"LM initialization failed: {lm_status}") - - @modal.method() - def run( - self, - prompt: str, - lyrics: str, - duration: float = 60.0, - format: str = "mp3", # or wav - manual_seeds: Optional[int] = 1, - ) -> bytes: - from acestep.inference import GenerationConfig, GenerationParams, generate_music - - params = GenerationParams( - caption=prompt, - lyrics=lyrics, - duration=duration, - thinking=True, - ) - config = GenerationConfig( - audio_format=format, - batch_size=1, - seeds=[manual_seeds] if manual_seeds is not None else None, - use_random_seed=manual_seeds is None, - ) - result = generate_music( - self.dit_handler, - self.llm_handler, - params, - config, - save_dir="/dev/shm", - ) - if not result.success: - raise RuntimeError(f"Music generation failed: {result.error}") - return Path(result.audios[0]["path"]).read_bytes() - - -``` - -We can then generate music from anywhere by running code like what we have in the `local_entrypoint` below. - -```python -@app.local_entrypoint() -def main( - prompt: Optional[str] = None, - lyrics: Optional[str] = None, - duration: Optional[float] = None, - format: str = "mp3", # or wav - manual_seeds: Optional[int] = 1, -): - if lyrics is None: - lyrics = "[Instrumental]" - if prompt is None: - prompt = "Korean pop music, bright energetic electronic music, catchy melody, female vocals" - lyrics = """[intro][intro] - [chorus] - We're goin' up, up, up, it's our moment - You know together we're glowing - Gonna be, gonna be golden - Oh, up, up, up with our voices - 영원히 깨질 수 없는 - Gonna be, gonna be golden""" - if duration is None: - duration = 30.0 # seconds - print( - f"🎼 generating {duration} seconds of music from prompt '{prompt[:32] + ('...' if len(prompt) > 32 else '')}'" - f" and lyrics '{lyrics[:32] + ('...' if len(lyrics) > 32 else '')}'" - ) - - music_generator = MusicGenerator() # outside of this file, use modal.Cls.from_name - clip = music_generator.run.remote( - prompt, lyrics, duration=duration, format=format, manual_seeds=manual_seeds - ) - - dir = Path("/tmp/generate-music") - dir.mkdir(exist_ok=True, parents=True) - - output_path = dir / f"{slugify(prompt)[:64]}.{format}" - print(f"🎼 Saving to {output_path}") - output_path.write_bytes(clip) - - -def slugify(string): - return ( - string.lower() - .replace(" ", "-") - .replace("/", "-") - .replace("\\", "-") - .replace(":", "-") - ) - - -``` - -You can execute it with a command like: - -```shell -modal run generate_music.py -``` - -Pass in `--help` to see options and how to use them. - -## Hosting a web UI for the music generator - -With the Gradio library, we can create a simple web UI in Python -that calls out to our music generator, -then host it on Modal for anyone to try out. - -To deploy both the music generator and the UI, run - -```shell -modal deploy generate_music.py -``` - -```python -@app.function( - image=web_image, - # Gradio requires sticky sessions - # so we limit the number of concurrent containers to 1 - # and allow it to scale to 100 concurrent inputs - max_containers=1, -) -@modal.concurrent(max_inputs=100) -@modal.asgi_app() -def ui(): - import gradio as gr - from fastapi import FastAPI - from gradio.routes import mount_gradio_app - - api = FastAPI() - - # Since this Gradio app is running from its own container, - # we make a `.remote` call to the music generator - music_generator = MusicGenerator() - generate = music_generator.run.remote - - temp_dir = Path("/dev/shm") - - async def generate_music( - prompt: str, lyrics: str, duration: float = 30.0, format: str = "mp3" - ): - audio_bytes = await generate.aio( - prompt, lyrics, duration=duration, format=format - ) - - audio_path = temp_dir / f"{uuid4()}.{format}" - audio_path.write_bytes(audio_bytes) - - return audio_path - - with gr.Blocks(theme="soft") as demo: - gr.Markdown("# Generate Music") - with gr.Row(): - with gr.Column(): - prompt = gr.Textbox(label="Prompt") - lyrics = gr.Textbox(label="Lyrics") - duration = gr.Number( - label="Duration (seconds)", value=10.0, minimum=1.0, maximum=300.0 - ) - format = gr.Radio(["wav", "mp3"], label="Format", value="mp3") - btn = gr.Button("Generate") - with gr.Column(): - clip_output = gr.Audio(label="Generated Music", autoplay=True) - - btn.click( - generate_music, - inputs=[prompt, lyrics, duration, format], - outputs=[clip_output], - ) - - return mount_gradio_app(app=api, blocks=demo, path="/") - -``` diff --git a/.agents/skills/modal/references/examples/grpo_trl.md b/.agents/skills/modal/references/examples/grpo_trl.md deleted file mode 100644 index 31b38043db81984e79704c5d839f7b648d96da78..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/grpo_trl.md +++ /dev/null @@ -1,334 +0,0 @@ -# Train a model to solve coding problems using GRPO and TRL - -This example demonstrates how to run [GRPO](https://arxiv.org/pdf/2402.03300) on Modal using the TRL [GRPO trainer](https://huggingface.co/docs/trl/main/en/grpo_trainer) -GRPO is a reinforcement learning algorithm introduced by DeepSeek, and was used to train DeepSeek R1. -TRL is a reinforcement learning training library by Huggingface. - -First we perform the imports and then define the app. - -```python -from __future__ import annotations - -import os -import re -import subprocess -from pathlib import Path -from typing import Iterable, Sequence - -import modal - -app: modal.App = modal.App("example-grpo-trl") - -``` - -We define an image where we install the TRL library. -We also install vLLM for the next part of this example. We also use Weights & Biases for logging. - -```python -image: modal.Image = modal.Image.debian_slim().uv_pip_install( - "trl[vllm]==0.28.0", - "vllm==0.12.0", - "transformers==4.57", - "datasets==3.5.1", - "wandb==0.17.6", -) - -``` - -We import the necessary libraries needed in the context of the image. - -```python -with image.imports(): - from datasets import Dataset, load_dataset - from trl import GRPOConfig, GRPOTrainer - -``` - -We also define a [Modal Volume](https://modal.com/docs/guide/volumes#volumes) for storing model checkpoints. - -```python -MODELS_DIR = Path("/models") -checkpoints_volume: modal.Volume = modal.Volume.from_name( - "example-grpo-trl-checkpoints", create_if_missing=True -) - -``` - -## Defining the reward function - -In this example, we use the [OpenCoder-LLM/opc-sft-stage2](https://huggingface.co/datasets/OpenCoder-LLM/opc-sft-stage2) dataset to train a model to solve coding problems. - -In reinforcement learning, we define a reward function for the model. Since we are evaluating code that is generated by -a model, we use [Modal Sandboxes](https://modal.com/docs/guide/sandbox) to evaluate the code securely. - -For each completion from the model and a test case to test the completion, we define a simple reward function. -The function returns 1 if there are no errors, and 0 otherwise. You might want to adjust this reward function -as the model is unlikely to learn well with this function. - -```python -@app.function() -def compute_reward(completion: str, testcase: Sequence[str]) -> int: - sb, score = None, 0 - sb: modal.Sandbox = modal.Sandbox.create(app=app) - code_to_execute: str = get_generated_code_and_test_cases(completion, testcase) - - try: - p = sb.exec("python", "-c", code_to_execute, timeout=30) - p.wait() - return_code = p.returncode - if return_code == 0: - score = 1 - except Exception as e: - print(e) - finally: - sb.terminate() - return score - - -``` - -We write a function that constructs a program from the model completion. This is determined based on the format of the data. -The completions are supposed to follow the format "\`\`\`python ...". -The test cases are a list of assert statements. -More details [here](https://huggingface.co/datasets/OpenCoder-LLM/opc-sft-stage2). - -````python -def get_generated_code_and_test_cases(completion: str, testcase: Sequence[str]) -> str: - if "```python" in completion: - # Find the start and end of the code block - start_idx: int = completion.find("```python") + len("```python") - end_idx: int = completion.find("```", start_idx) - if end_idx != -1: - code: str = completion[start_idx:end_idx].strip() - else: - code: str = completion[start_idx:].strip() - else: - code: str = completion.strip() - - test_cases: str = "\n".join(testcase) - full_code: str = f"{code}\n\n{test_cases}" - return full_code - - -```` - -Finally, we define the function that is passed into the GRPOTrainer, which takes in a list of completions. -Custom reward functions must conform to a [specific signature](https://huggingface.co/docs/trl/main/en/grpo_trainer#using-a-custom-reward-function). - -```python -def reward_helper_function( - completions: Sequence[str], testcases: Sequence[Sequence[str]], **kwargs: object -) -> Iterable[int]: - return compute_reward.starmap(zip(completions, testcases)) - - -``` - -## Kicking off a training run - -Preprocess the data, preparing the columns that `GRPOTrainer` expects. -We use the OpenCoder-LLM educational instruct dataset, which has (instruction, code, test case) triples validated through a Python compiler. -More details [here](https://huggingface.co/datasets/OpenCoder-LLM/opc-sft-stage2). - -```python -def start_grpo_trainer(use_vllm=False, vllm_mode=None): - dataset: Dataset = load_dataset( - "OpenCoder-LLM/opc-sft-stage2", "educational_instruct", split="train" - ) - dataset = dataset.rename_column( - "instruction", "prompt" - ) # Needed for the GRPO trainer - dataset = dataset.rename_column("testcase", "testcases") - dataset = dataset.select(range(128)) # To simplify testing. - training_args: GRPOConfig = GRPOConfig( - output_dir=str(MODELS_DIR), - report_to="wandb", - use_vllm=use_vllm, - vllm_mode=vllm_mode, - save_steps=1, - max_steps=5, # To simplify testing. Remove for production use cases. - ) - trainer = GRPOTrainer( - model="Qwen/Qwen2-0.5B-Instruct", - reward_funcs=reward_helper_function, - args=training_args, - train_dataset=dataset, - ) - trainer.train() - - -``` - -We use Weights & Biases for logging, hence we use a [Modal Secret](https://modal.com/docs/guide/secrets#secrets) with wandb credentials. - -```python -@app.function( - image=image, - gpu="H100", - timeout=60 * 60 * 24, # 24 hours - secrets=[modal.Secret.from_name("wandb-secret")], - volumes={"/models": checkpoints_volume}, -) -def train() -> None: - start_grpo_trainer() - - -``` - -To run: `modal run --detach grpo_trl.py::train`. - -## Speeding up training with vLLM - -vLLM can be used either in server mode (run vLLM server on separate gpu) or colocate mode (within the training process). -In server mode, vLLM runs in a separate process (and using separate GPUs) and communicates with the trainer via HTTP. -This is ideal if you have dedicated GPUs for inference. More details [here](https://huggingface.co/docs/trl/main/en/grpo_trainer#-option-1-server-mode). -Here, we use 2 GPUs. We run the GRPOTrainer on 1 of them, and the vLLM process on another. - -```python -@app.function( - image=image, - gpu="H100:2", - timeout=60 * 60 * 24, # 24 hours - secrets=[modal.Secret.from_name("wandb-secret")], - volumes={str(MODELS_DIR): checkpoints_volume}, -) -def train_vllm_server_mode() -> None: - env_copy = os.environ.copy() - env_copy["CUDA_VISIBLE_DEVICES"] = "0" # Run serve vLLM process on GPU 0 - - # Start vllm-serve in the background - subprocess.Popen( - ["trl", "vllm-serve", "--model", "Qwen/Qwen2-0.5B-Instruct"], - env=env_copy, - ) - os.environ["CUDA_VISIBLE_DEVICES"] = "1" # Run training process on GPU 1 - start_grpo_trainer(use_vllm=True, vllm_mode="server") - - -``` - -You can execute this using `modal run --detach grpo_trl.py::train_vllm_server_mode`. - -In colocate mode, vLLM runs inside the trainer process and shares GPU memory with the training model. -This avoids launching a separate server and can improve GPU utilization, but may lead to memory contention on the training GPUs. -More details [here](https://huggingface.co/docs/trl/main/en/grpo_trainer#-option-2-colocate-mode). - -```python -@app.function( - image=image, - gpu="H100", - timeout=60 * 60 * 24, # 24 hours - secrets=[modal.Secret.from_name("wandb-secret")], - volumes={"/models": checkpoints_volume}, -) -def train_vllm_colocate_mode() -> None: - # Rank of the current process (0 for single-process training) - os.environ["RANK"] = "0" - # Local rank of the process on the node (0 for single-process training) - os.environ["LOCAL_RANK"] = "0" - # Total number of processes (1 for single-process training) - os.environ["WORLD_SIZE"] = "1" - # Address of the master node (localhost for single node) - os.environ["MASTER_ADDR"] = "localhost" - # Port for communication between processes - os.environ["MASTER_PORT"] = "12355" - start_grpo_trainer(use_vllm=True, vllm_mode="colocate") - - -``` - -You can execute this using `modal run --detach grpo_trl.py::train_vllm_colocate_mode`. - -## Performing inference on the trained model - -We use vLLM to perform inference on the trained model. - -```python -VLLM_PORT: int = 8000 - - -``` - -Once you have the model checkpoints in your Modal Volume, you can load the weights and perform inference using vLLM. For more on storing model weights on Modal, see -[this guide](https://modal.com/docs/guide/model-weights). -The weights path is as follows: `global_step_n/actor/huggingface` where n is the checkpoint you want (eg `global_step_5/actor/huggingface`). -The `latest_checkpointed_iteration.txt` file stores the most recent checkpoint index. - -```python -def get_latest_checkpoint_file_path(): - checkpoint_dirs = [ - d.name - for d in MODELS_DIR.iterdir() - if d.is_dir() and re.match(r"^checkpoint-(\d+)$", d.name) - ] - if not checkpoint_dirs: - raise FileNotFoundError("No checkpoint directories found in models dir") - latest_checkpoint_index = max( - int(re.match(r"^checkpoint-(\d+)$", d).group(1)) for d in checkpoint_dirs - ) - return str(MODELS_DIR / f"checkpoint-{latest_checkpoint_index}") - - -``` - -We provide the code for setting up an OpenAI compatible inference endpoint here. For more details re. serving models on vLLM, check out [this example.](https://modal.com/docs/examples/vllm_inference#deploy-the-server) - -```python -vllm_image = ( - modal.Image.debian_slim(python_version="3.12") - .uv_pip_install( - "vllm==0.12.0", - "flashinfer-python==0.5.3", - extra_index_url="https://download.pytorch.org/whl/cu128", - extra_options="--index-strategy unsafe-best-match", - ) - .env({"VLLM_USE_V1": "1"}) -) - -vllm_cache_vol = modal.Volume.from_name("vllm-cache", create_if_missing=True) - - -@app.function( - image=vllm_image, - gpu="H100", - scaledown_window=15 * 60, # How long should we stay up with no requests? - timeout=10 * 60, # How long should we wait for container start? - volumes={"/root/.cache/vllm": vllm_cache_vol, MODELS_DIR: checkpoints_volume}, -) -@modal.concurrent( - max_inputs=32 -) # How many requests can one replica handle? tune carefully! -@modal.web_server(port=VLLM_PORT, startup_timeout=10 * 60) -def serve(): - latest_checkpoint_file_path = get_latest_checkpoint_file_path() - - cmd = [ - "vllm", - "serve", - "--uvicorn-log-level=info", - latest_checkpoint_file_path, - "--host", - "0.0.0.0", - "--port", - str(VLLM_PORT), - ] - subprocess.Popen(" ".join(cmd), shell=True) - - -``` - -You can then deploy the server using `modal deploy grpo_trl.py`, which gives you a custom url. You can then query it using the following curl command: - -```bash -curl -X POST /v1/chat/completions \ - -H 'Content-Type: application/json' \ - -d '{ - "messages": [ - {"role": "system", "content": "You are a helpful assistant for solving math problems."}, - {"role": "user", "content": "James had 4 apples. Mary gave him 2 and he ate 1. How many does he have left?"} - ], - "temperature": 0.7 - }' -``` - -or in the [following ways](https://modal.com/docs/examples/vllm_inference#interact-with-the-server). diff --git a/.agents/skills/modal/references/examples/grpo_verl.md b/.agents/skills/modal/references/examples/grpo_verl.md deleted file mode 100644 index 63cc9d594cd02f6bcff1cf664db259be2f2da45a..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/grpo_verl.md +++ /dev/null @@ -1,343 +0,0 @@ -# Train a model to solve math problems using GRPO and verl - -This example demonstrates how to train with [GRPO](https://arxiv.org/pdf/2402.03300) on Modal using the [verl](https://github.com/volcengine/verl) framework. -GRPO is a reinforcement learning algorithm introduced by DeepSeek, and was used to train DeepSeek R1. -verl is a reinforcement learning training library that is an implementation of [HybridFlow](https://arxiv.org/abs/2409.19256v2), an RLHF framework. - -The training process works as follows: - -* Each example in the dataset corresponds to a math problem. -* In each training step, the model attempts to solve the math problems showing its steps. -* We then compute a reward for the model's solution using the reward function defined below. -* That reward value is then used to update the model's parameters according to the GRPO training algorithm. - -## Setup - -Import the necessary modules for Modal deployment. - -```python -import re -import subprocess -from pathlib import Path -from typing import Literal, Optional - -import modal - -``` - -## Defining the image and app - -```python -app = modal.App("example-grpo-verl") - -``` - -We define an image where we clone the verl repo and install its dependencies. We use a base verl image as a starting point. - -```python -VERL_REPO_PATH: Path = Path("/root/verl") -image = ( - modal.Image.from_registry("verlai/verl:app-verl0.4-vllm0.8.5-mcore0.12.1") - .apt_install("git") - .run_commands(f"git clone https://github.com/volcengine/verl {VERL_REPO_PATH}") - .uv_pip_install("verl[vllm]==0.4.1") -) - -``` - -## Defining the dataset - -In this example, we'll use reinforcement learning to train a model to solve math problems. -We use the [GSM8K](https://huggingface.co/datasets/openai/gsm8k) dataset of math problems and a [Modal Volume](https://modal.com/docs/guide/volumes#volumes) to store the data. - -```python -DATA_PATH: Path = Path("/data") -data_volume: modal.Volume = modal.Volume.from_name( - "grpo-verl-example-data", create_if_missing=True -) - - -``` - -We write a Modal Function to populate the Volume with the data. This downloads the dataset and stores it in the Volume. -You will need to run this step if you don't already have data you'd like to use for this example. - -```python -@app.function(image=image, volumes={DATA_PATH: data_volume}) -def prep_dataset() -> None: - subprocess.run( - [ - "python", - VERL_REPO_PATH / "examples" / "data_preprocess" / "gsm8k.py", - "--local_dir", - DATA_PATH, - ], - check=True, - ) - - -``` - -You can kick off the dataset download with -`modal run ::prep_dataset` - -## Defining a reward function - -In reinforcement learning, we define a reward function for the model. -We can define this in a separate file, or in the same file as in this case that we then pass as an argument to verl. -We use a `default` reward function for GSM8K from the [verl repo](https://github.com/volcengine/verl/blob/v0.1/verl/utils/reward_score/gsm8k.py), modified to return 1.0 if it's a correct answer and 0 otherwise. - -```python -def extract_solution( - solution_str: str, method: Literal["strict", "flexible"] = "strict" -) -> Optional[str]: - assert method in ["strict", "flexible"] - - if method == "strict": - # This also tests the formatting of the model - solution = re.search("#### (\\-?[0-9\\.\\,]+)", solution_str) - if solution is None: - final_answer: Optional[str] = None - else: - final_answer = solution.group(0) - final_answer = ( - final_answer.split("#### ")[1].replace(",", "").replace("$", "") - ) - elif method == "flexible": - answer = re.findall("(\\-?[0-9\\.\\,]+)", solution_str) - final_answer: Optional[str] = None - if len(answer) == 0: - # No reward if there is no answer. - pass - else: - invalid_str: list[str] = ["", "."] - # Find the last number that is not '.' - for final_answer in reversed(answer): - if final_answer not in invalid_str: - break - return final_answer - - -``` - -Reward functions need to follow a [predefined signature.](https://verl.readthedocs.io/en/latest/preparation/reward_function.html) - -```python -def compute_reward( - data_source: str, solution_str: str, ground_truth: str, extra_info: dict -) -> float: - answer = extract_solution(solution_str=solution_str, method="strict") - if answer is None: - return 0.0 - else: - if answer == ground_truth: - return 1.0 - else: - return 0.0 - - -``` - -We then define constants to pass into verl during the training run. - -```python -PATH_TO_REWARD_FUNCTION: Path = Path("/root/grpo_verl.py") -REWARD_FUNCTION_NAME: str = "compute_reward" - -``` - -## Kicking off a training run - -We define some more constants for the training run. - -```python -MODELS_PATH: Path = Path("/models") -MINUTES: int = 60 - - -``` - -We also define a Volume for storing model checkpoints. - -```python -checkpoints_volume: modal.Volume = modal.Volume.from_name( - "grpo-verl-example-checkpoints", create_if_missing=True -) - -``` - -Now, we write a Modal Function for kicking off the training run. -If you wish to use Weights & Biases, as we do in this code, you'll need to create a Weights & Biases [Secret.](https://modal.com/docs/guide/secrets#secrets) - -verl uses Ray under the hood. It creates Ray workers for each step where each Ray worker is a python process and each step is a step in the RL dataflow pipeline. -verl also keeps a separate control flow process that's independent of this, responsible for figuring out what step in the RL pipeline to execute. -Each Ray worker gets mapped onto 1 or more GPUs. Depending on the number of GPUs available, Ray will decide what workers go where, or to hold off scheduling workers -if there are no available GPUs. Generally, more VRAM = less hot-swapping of Ray workers, which means less waiting around for memory copying each iteration. -In this example we have chosen a configuration that allows for easy automated testing, but you may wish to use more GPUs or more powerful GPU types. -More details [here](https://verl.readthedocs.io/en/latest/hybrid_flow.html). - -```python -@app.function( - image=image, - gpu="H100:2", - volumes={ - MODELS_PATH: checkpoints_volume, - DATA_PATH: data_volume, - }, - secrets=[modal.Secret.from_name("wandb-secret")], - timeout=24 * 60 * MINUTES, -) -def train(*arglist) -> None: - data_volume.reload() - - cmd: list[str] = [ - "python", - "-m", - "verl.trainer.main_ppo", - "algorithm.adv_estimator=grpo", - f"data.train_files={DATA_PATH / 'train.parquet'}", - f"data.val_files={DATA_PATH / 'test.parquet'}", - "data.train_batch_size=128", - "data.max_prompt_length=64", - "data.max_response_length=1024", - "data.filter_overlong_prompts=True", - "data.truncation=error", - "actor_rollout_ref.model.path=Qwen/Qwen2-0.5B", - "actor_rollout_ref.actor.optim.lr=1e-6", - "actor_rollout_ref.model.use_remove_padding=False", - "actor_rollout_ref.actor.ppo_mini_batch_size=128", - "actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=16", - "actor_rollout_ref.actor.checkpoint.save_contents='model,optimizer,extra,hf_model'", - "actor_rollout_ref.actor.use_kl_loss=True", - "actor_rollout_ref.actor.entropy_coeff=0", - "actor_rollout_ref.actor.kl_loss_coef=0.001", - "actor_rollout_ref.actor.kl_loss_type=low_var_kl", - "actor_rollout_ref.model.enable_gradient_checkpointing=True", - "actor_rollout_ref.actor.fsdp_config.param_offload=False", - "actor_rollout_ref.actor.fsdp_config.optimizer_offload=False", - "actor_rollout_ref.rollout.tensor_model_parallel_size=2", - "actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=16", - "actor_rollout_ref.rollout.name=vllm", - "actor_rollout_ref.rollout.gpu_memory_utilization=0.4", - "actor_rollout_ref.rollout.n=5", - "actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=16", - "actor_rollout_ref.ref.fsdp_config.param_offload=True", - "algorithm.use_kl_in_reward=False", - "trainer.critic_warmup=0", - "trainer.logger=['console', 'wandb']", - "trainer.project_name=verl_grpo_example_qwen2-0.5b", - "trainer.experiment_name=qwen2-0.5b_example", - "trainer.n_gpus_per_node=2", - "trainer.nnodes=1", - "trainer.test_freq=5", - f"trainer.default_local_dir={MODELS_PATH}", - "trainer.resume_mode=auto", - # Parameters chosen to ensure easy automated testing. Remove if needed. - "trainer.save_freq=1", - "trainer.total_training_steps=1", - "trainer.total_epochs=1", - # For the custom reward function. - f"custom_reward_function.path={str(PATH_TO_REWARD_FUNCTION)}", - f"custom_reward_function.name={REWARD_FUNCTION_NAME}", - ] - if arglist: - cmd.extend(arglist) - - subprocess.run(cmd, check=True) - - -``` - -You can now run the training using `modal run --detach grpo_verl.py::train`, or pass in any [additional args from the CLI](https://modal.com/docs/guide/apps#argument-parsing) like this `modal run --detach grpo.py::train -- trainer.total_epochs=20 actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=16`. - -## Performing inference on the trained model - -We use vLLM to perform inference on the trained model. - -```python -VLLM_PORT: int = 8000 - - -``` - -Once you have the model checkpoints in your Modal Volume, you can load the weights and perform inference using vLLM. For more on storing model weights on Modal, see -[this guide](https://modal.com/docs/guide/model-weights). -The weights path is as follows: `global_step_n/actor/huggingface` where n is the checkpoint you want (e.g. `global_step_5/actor/huggingface`). -The `latest_checkpointed_iteration.txt` file stores the most recent checkpoint index. - -```python -def get_latest_checkpoint_file_path(): - with open(MODELS_PATH / "latest_checkpointed_iteration.txt") as f: - latest_checkpoint_index = int(f.read()) - return str( - MODELS_PATH / f"global_step_{latest_checkpoint_index}" / "actor" / "huggingface" - ) - - -``` - -We provide the code for setting up an OpenAI compatible inference endpoint here. For more details re. serving models on vLLM, check out [this example.](https://modal.com/docs/examples/vllm_inference#deploy-the-server) - -```python -vllm_image = ( - modal.Image.debian_slim(python_version="3.12") - .uv_pip_install( - "vllm==0.9.1", - "flashinfer-python==0.2.6.post1", - extra_index_url="https://download.pytorch.org/whl/cu128", - extra_options="--index-strategy unsafe-best-match", - ) - .env({"VLLM_USE_V1": "1"}) -) - -vllm_cache_vol = modal.Volume.from_name("vllm-cache", create_if_missing=True) - - -@app.function( - image=vllm_image, - gpu="H100:2", - scaledown_window=15 * MINUTES, # How long should we stay up with no requests? - timeout=10 * MINUTES, # How long should we wait for container start? - volumes={"/root/.cache/vllm": vllm_cache_vol, MODELS_PATH: checkpoints_volume}, -) -@modal.concurrent( - max_inputs=32 -) # How many requests can one replica handle? Tune carefully! -@modal.web_server(port=VLLM_PORT, startup_timeout=10 * MINUTES) -def serve(): - import subprocess - - latest_checkpoint_file_path = get_latest_checkpoint_file_path() - - cmd = [ - "vllm", - "serve", - "--uvicorn-log-level=info", - latest_checkpoint_file_path, - "--host", - "0.0.0.0", - "--port", - str(VLLM_PORT), - "--tensor-parallel-size", - "2", - ] - subprocess.Popen(" ".join(cmd), shell=True) - - -``` - -You can then deploy the server using `modal deploy grpo_verl.py`, which gives you a custom URL. You can then query it using the following curl command: - -```bash -curl -X POST /v1/chat/completions \ - -H 'Content-Type: application/json' \ - -d '{ - "messages": [ - {"role": "system", "content": "You are a helpful assistant for solving math problems."}, - {"role": "user", "content": "James had 4 apples. Mary gave him 2 and he ate 1. How many does he have left?"} - ], - "temperature": 0.7 - }' -``` - -or in the [following ways](https://modal.com/docs/examples/vllm_inference#interact-with-the-server). diff --git a/.agents/skills/modal/references/examples/hackernews_alerts.md b/.agents/skills/modal/references/examples/hackernews_alerts.md deleted file mode 100644 index 553230e39f3c77f1cd6eb9859c19607933a8d3ca..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/hackernews_alerts.md +++ /dev/null @@ -1,119 +0,0 @@ -# Run cron jobs in the cloud to search Hacker News - -In this example, we use Modal to deploy a cron job that periodically queries Hacker News for -new posts matching a given search term, and posts the results to Slack. - -## Import and define the app - -Let's start off with imports, and defining a Modal app. - -```python -import os -from datetime import datetime, timedelta - -import modal - -app = modal.App("example-hackernews-alerts") - -``` - -Now, let's define an image that has the `slack-sdk` package installed, in which we can run a function -that posts a slack message. - -```python -slack_sdk_image = modal.Image.debian_slim().uv_pip_install("slack-sdk") - -``` - -## Defining the function and importing the secret - -Our Slack bot will need access to a bot token. -We can use Modal's [Secrets](https://modal.com/secrets) interface to accomplish this. -To quickly create a Slack bot secret, click the "Create new secret" button. -Then, select the Slack secret template from the list options, -and follow the instructions in the "Where to find the credentials?" panel. -Name your secret `hn-bot-slack.` - -Now, we define the function `post_to_slack`, which simply instantiates the Slack client using our token, -and then uses it to post a message to a given channel name. - -```python -@app.function( - image=slack_sdk_image, - secrets=[modal.Secret.from_name("hn-bot-slack", required_keys=["SLACK_BOT_TOKEN"])], -) -async def post_to_slack(message: str): - import slack_sdk - - client = slack_sdk.WebClient(token=os.environ["SLACK_BOT_TOKEN"]) - client.chat_postMessage(channel="hn-alerts", text=message) - - -``` - -## Searching Hacker News - -We are going to use Algolia's [Hacker News Search API](https://hn.algolia.com/api) to query for posts -matching a given search term in the past X days. Let's define our search term and query period. - -```python -QUERY = "serverless" -WINDOW_SIZE_DAYS = 1 - -``` - -Let's also define an image that has the `requests` package installed, so we can query the API. - -```python -requests_image = modal.Image.debian_slim().uv_pip_install("requests") - -``` - -We can now define our main entrypoint, that queries Algolia for the term, and calls `post_to_slack` -on all the results. We specify a [schedule](https://modal.com/docs/guide/cron) -in the function decorator, which means that our function will run automatically at the given interval. - -```python -@app.function(image=requests_image) -def search_hackernews(): - import requests - - url = "http://hn.algolia.com/api/v1/search" - - threshold = datetime.utcnow() - timedelta(days=WINDOW_SIZE_DAYS) - - params = { - "query": QUERY, - "numericFilters": f"created_at_i>{threshold.timestamp()}", - } - - response = requests.get(url, params, timeout=10).json() - urls = [item["url"] for item in response["hits"] if item.get("url")] - - print(f"Query returned {len(urls)} items.") - - post_to_slack.for_each(urls) - - -``` - -## Test running - -We can now test run our scheduled function as follows: `modal run hackernews_alerts.py::app.search_hackernews` - -## Defining the schedule and deploying - -Let's define a function that will be called by Modal every day - -```python -@app.function(schedule=modal.Period(days=1)) -def run_daily(): - search_hackernews.remote() - - -``` - -In order to deploy this as a persistent cron job, you can run `modal deploy hackernews_alerts.py`, - -Once the job is deployed, visit the [apps page](https://modal.com/apps) page to see -its execution history, logs and other stats. diff --git a/.agents/skills/modal/references/examples/hello_world.md b/.agents/skills/modal/references/examples/hello_world.md deleted file mode 100644 index eea67c3404675168f888eca892e561ccb05b8c04..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/hello_world.md +++ /dev/null @@ -1,131 +0,0 @@ -# Hello, world! - -This tutorial demonstrates some core features of Modal: - -* You can run functions on Modal just as easily as you run them locally. -* Running functions in parallel on Modal is simple and fast. -* Logs and errors show up immediately, even for functions running on Modal. - -## Importing Modal and setting up - -We start by importing `modal` and creating a `App`. -We build up this `App` to [define our application](https://modal.com/docs/guide/apps). - -```python -import sys - -import modal - -app = modal.App("example-hello-world") - -``` - -## Defining a function - -Modal takes code and runs it in the cloud. - -So first we've got to write some code. - -Let's write a simple function that takes in an input, -prints a log or an error to the console, -and then returns an output. - -To make this function work with Modal, we just wrap it in a decorator, -[`@app.function`](https://modal.com/docs/reference/modal.App#function). - -```python -@app.function() -def f(i): - if i % 2 == 0: - print("hello", i) - else: - print("world", i, file=sys.stderr) - - return i * i - - -``` - -## Running our function locally, remotely, and in parallel - -Now let's see three different ways we can call that function: - -1. As a regular call on your `local` machine, with `f.local` - -2. As a `remote` call that runs in the cloud, with `f.remote` - -3. By `map`ping many copies of `f` in the cloud over many inputs, with `f.map` - -We call `f` in each of these ways inside the `main` function below. - -```python -@app.local_entrypoint() -def main(): - # run the function locally - print(f.local(1000)) - - # run the function remotely on Modal - print(f.remote(1000)) - - # run the function in parallel and remotely on Modal - total = 0 - for ret in f.map(range(200)): - total += ret - - print(total) - - -``` - -Enter `modal run hello_world.py` in a shell, and you'll see a Modal app initialize. -You'll then see the `print`ed logs of -the `main` function and, mixed in with them, all the logs of `f` as it is run -locally, then remotely, and then remotely and in parallel. - -That's all triggered by adding the -[`@app.local_entrypoint`](https://modal.com/docs/reference/modal.App#local_entrypoint) -decorator on `main`, which defines it as the function to start from locally when we invoke `modal run`. - -## What just happened? - -When we called `.remote` on `f`, the function was executed -*in the cloud*, on Modal's infrastructure, not on the local machine. - -In short, we took the function `f`, put it inside a container, -sent it the inputs, and streamed back the logs and outputs. - -## But why does this matter? - -Try one of these things next to start seeing the full power of Modal! - -### You can change the code and run it again - -For instance, change the `print` statement in the function `f` -to print `"spam"` and `"eggs"` instead and run the app again. -You'll see that that your new code is run with no extra work from you -- -and it should even run faster! - -Modal's goal is to make running code in the cloud feel like you're -running code locally. That means no waiting for long image builds when you've just moved a comma, -no fiddling with container image pushes, and no context-switching to a web UI to inspect logs. - -### You can map over more data - -Change the `map` range from `200` to some large number, like `1170`. You'll see -Modal create and run even more containers in parallel this time. - -And it'll happen lightning fast! - -### You can run a more interesting function - -The function `f` is a bit silly and doesn't do much, but in its place -imagine something that matters to you, like: - -* Running [language model inference](https://modal.com/docs/examples/vllm_inference) - or [fine-tuning](https://modal.com/docs/examples/slack-finetune) -* Manipulating [audio](https://modal.com/docs/examples/musicgen) - or [images](https://modal.com/docs/examples/diffusers_lora_finetune) -* [Embedding huge text datasets](https://modal.com/docs/examples/amazon_embeddings) at lightning fast speeds - -Modal lets you parallelize that operation effortlessly by running hundreds or -thousands of containers in the cloud. diff --git a/.agents/skills/modal/references/examples/hp_sweep_gpt.md b/.agents/skills/modal/references/examples/hp_sweep_gpt.md deleted file mode 100644 index 7cbd2c5f5bca58cdb0d1122bd3f41c2c62414232..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/hp_sweep_gpt.md +++ /dev/null @@ -1,874 +0,0 @@ -# Train an SLM from scratch with early-stopping grid search over hyperparameters - -![Split-Panel Image. Left: AI generated picture of Shakespeare. Right: SLM generated text](./shakespeare.jpg) - -When you want a language model that performs well on your task, there are three options, -ordered by the degree of customization: - -* [**Prompt Engineering**](https://en.wikipedia.org/wiki/Prompt_engineering): - large and capable language models understand tasks in natural language, so you can - carefully design a natural language "prompt" to elicit the desired behavior. - -* [**Fine-Tuning**](https://modal.com/docs/examples/llm-finetuning): - those same language models were trained by gradient descent on data sets representing tasks, - and they can be further trained by gradient descent on data sets representative of your task. - -* **Training from Scratch**: - if you have enough data for your task, you can throw the pretrained model away and make your own. - -Each step adds additional engineering complexity, but also leads to a superior cost-performance Pareto frontier -for your tasks. Fine-tuned models at one-tenth the size regularly outperform more generic models, -and models trained from scratch outperform them. - -Because these models are so much smaller than the Large Language Models that power generic -assistant chatbots like ChatGPT and Claude, they are often called *Small Language Models* (SLMs). - -In this example, we will explore training an SLM from scratch on Modal. - -In fact, we'll train 8 SLMs in parallel with different hyperparameters -and then select the best one for additional training. - -We'll monitor this training live and serve our training and trained models -as Web Functions and simple browser UIs. - -Along the way we'll use many features of the Modal platform: -[distributed Volumes](https://modal.com/docs/guide/volumes), -multiple [Web Functions](https://modal.com/docs/guide/webhooks), -and [parallel container execution](https://modal.com/docs/guide/scale#parallel-execution-of-inputs). - -Together, these features give every machine learning and AI team -the same infrastructural capabilities that the most sophisticated companies -have in their internal platforms. - -## Basic Setup - -```python -import logging as L -import urllib.request -from dataclasses import dataclass -from pathlib import Path, PosixPath -from typing import Optional - -import modal -from pydantic import BaseModel - -MINUTES = 60 # seconds -HOURS = 60 * MINUTES - -app_name = "example-hp-sweep-gpt" -app = modal.App(app_name) - -``` - -We'll use A10G GPUs for training, which are able to train the model to recognizably improved performance -in ~15 minutes while keeping costs under ~$1. - -```python -gpu = "A10G" - -``` - -### Create a Volume to store data, weights, and logs - -Since we'll be coordinating training across multiple machines we'll use a -distributed [Volume](https://modal.com/docs/guide/volumes) -to store the data, checkpointed models, and TensorBoard logs. - -```python -volume = modal.Volume.from_name("example-hp-sweep-gpt-volume", create_if_missing=True) -volume_path = PosixPath("/vol/data") -model_filename = "nano_gpt_model.pt" -best_model_filename = "best_nano_gpt_model.pt" -tb_log_path = volume_path / "tb_logs" -model_save_path = volume_path / "models" - -``` - -### Define dependencies in container images - -The container image for training is based on Modal's default slim Debian Linux image with `torch` -for defining and running our neural network and `tensorboard` for monitoring training. - -```python -base_image = modal.Image.debian_slim(python_version="3.11").uv_pip_install( - "pydantic==2.9.1" -) - -torch_image = base_image.uv_pip_install( - "torch==2.1.2", - "tensorboard==2.17.1", - "numpy<2", -) - -``` - -We also have some local dependencies that we'll need to import into the remote environment. -We add them into the remote container. - -```python -torch_image = torch_image.add_local_dir( - Path(__file__).parent / "src", remote_path="/root/src" -) - -``` - -We'll serve a simple Web Function: - -```python -web_image = base_image.uv_pip_install("fastapi[standard]==0.115.4", "starlette==0.41.2") - -``` - -And we'll deploy a web UI for interacting with our trained models using Gradio. - -```python -assets_path = Path(__file__).parent / "assets" -ui_image = web_image.uv_pip_install("gradio~=4.44.0").add_local_dir( - assets_path, remote_path="/assets" -) - - -``` - -We can also "pre-import" libraries that will be used by the functions we run on Modal in a given image -using the `with image.imports` context manager. - -```python -with torch_image.imports(): - import glob - import os - from timeit import default_timer as timer - - import tensorboard - import torch - from src.dataset import Dataset - from src.logs_manager import LogsManager - from src.model import AttentionModel - from src.tokenizer import Tokenizer - -``` - -## Running SLM training on Modal - -Here we define the training function, wrapping it in a decorator -that specifies the infrastructural parameters, like the container `image` we want to use, -which `volume` to mount where, the `gpu` we're using, and so on. - -Training consists of specifying optimization parameters, loading the -`dataset`, building the `model`, setting up TensorBoard logging & -checkpointing, and then finally executing the `training_loop` itself. - -```python -@app.function( - image=torch_image, - volumes={volume_path: volume}, - gpu=gpu, - timeout=1 * HOURS, -) -def train_model( - node_rank, - n_nodes, - hparams, - experiment_name, - run_to_first_save=False, - n_steps=3000, - n_steps_before_eval=None, - n_steps_before_checkpoint=None, -): - # optimizer, data, and model prep - batch_size = 64 - learning_rate = 3e-4 - - n_eval_steps = 100 - if n_steps_before_eval is None: - n_steps_before_eval = int(n_steps / 8) # eval eight times per run - if n_steps_before_checkpoint is None: - n_steps_before_checkpoint = int(n_steps / 4) # save four times per run - - train_percent = 0.9 - - L.basicConfig( - level=L.INFO, - format=f"\033[0;32m%(asctime)s %(levelname)s [%(filename)s.%(funcName)s:%(lineno)d] [Node {node_rank + 1}] %(message)s\033[0m", - datefmt="%b %d %H:%M:%S", - ) - - # use GPU if available - device = "cuda" if torch.cuda.is_available() else "cpu" - L.info("Remote Device: %s // GPU: %s", device, gpu) - - input_file_path = volume_path / "shakespeare_char.txt" - text = prepare_data(input_file_path, volume) - - # construct tokenizer & dataset - tokenizer = Tokenizer(text) - dataset = Dataset( - tokenizer.encode(text), - train_percent, - batch_size, - hparams.context_size, - device, - ) - - # build the model - model = build_model(hparams, tokenizer.vocab_size, device) - num_parameters = sum(p.numel() for p in model.parameters()) - L.info(f"Num parameters: {num_parameters}") - - optimizer = setup_optimizer(model, learning_rate) - - # TensorBoard logging & checkpointing prep - logs_manager = LogsManager(experiment_name, hparams, num_parameters, tb_log_path) - L.info(f"Model name: {logs_manager.model_name}") - - model_save_dir = model_save_path / experiment_name / logs_manager.model_name - if model_save_dir.exists(): - L.info("Loading model from checkpoint...") - checkpoint = torch.load(str(model_save_dir / model_filename)) - is_best_model = not run_to_first_save - if is_best_model: - make_best_symbolic_link(model_save_dir, model_filename, experiment_name) - model.load_state_dict(checkpoint["model"]) - start_step = checkpoint["steps"] + 1 - else: - model_save_dir.mkdir(parents=True, exist_ok=True) - start_step = 0 - checkpoint = init_checkpoint(model, tokenizer, optimizer, start_step, hparams) - - checkpoint_path = model_save_dir / model_filename - - out = training_loop( - start_step, - n_steps, - n_steps_before_eval, - n_steps_before_checkpoint, - n_eval_steps, - dataset, - tokenizer, - model, - optimizer, - logs_manager, - checkpoint, - checkpoint_path, - run_to_first_save, - ) - - return node_rank, float(out["val"]), hparams - - -``` - -## Launch a hyperparameter sweep from a `local_entrypoint` - -The main entry point coordinates the hyperparameter optimization. -First we specify the default hyperparameters for the model, taken from -[Andrej Karpathy's walkthrough](https://www.youtube.com/watch?v=kCc8FmEb1nY\&t=5976s). -For better performance, you can increase the `context_size` and scale up the GPU accordingly. - -```python -@dataclass -class ModelHyperparameters: - n_heads: int = 6 - n_embed: int = 384 - n_blocks: int = 6 - context_size: int = 256 - dropout: float = 0.2 - - -``` - -Next we define the local entrypoint: the code we run locally to coordinate training. - -It will train 8 models in parallel across 8 containers, each -with different hyperparameters, varying the number of heads (`n_heads`), the -`context_size` (called the "block size" by Karpathy), and the dropout rate (`dropout`). To run in -parallel we need to use the [`starmap` method](https://modal.com/docs/guide/scale#parallel-execution-of-inputs). - -We train all of the models until the first checkpoint and then stop early so we -can compare the validation losses. - -Then we restart training for the best model and train it to completion. - -You can kick off training with the following command: - -```bash -modal run 06_gpu_and_ml/hyperparameter-sweep/hp_sweep_gpt.py -``` - -The output will look something like this: - -``` -Sep 16 21:20:39 INFO [hp_sweep_gpt.py.train_model:127] [Node 1] Remote Device: cuda // GPU: A10G -Sep 16 21:20:40 INFO [hp_sweep_gpt.py.train_model:149] [Node 1] Num parameters: 10693697 -Sep 16 21:20:40 INFO [hp_sweep_gpt.py.train_model:156] [Node 1] Model Name: E2024-0916-142031.618259_context_size=8_n_heads=1_dropout=0.1 -Sep 16 21:20:41 INFO [hp_sweep_gpt.py.train_model:225] [Node 1] 0) // 1.03s // Train Loss: 3.58 // Val Loss: 3.60 -Sep 16 21:20:41 INFO [hp_sweep_gpt.py.train_model:127] [Node 2] Remote Device: cuda // GPU: A10G -... -``` - -The `local_entrypoint` code is below. Note that the arguments to it can also be passed via the command line. -Use `--help` for details. - -```python -@app.local_entrypoint() -def main( - n_steps: int = 3000, - n_steps_before_checkpoint: Optional[int] = None, - n_steps_before_eval: Optional[int] = None, -): - from datetime import datetime - from itertools import product - - experiment_name = f"E{datetime.now().strftime('%Y-%m-%d-%H%M%S.%f')}" - default_hparams = ModelHyperparameters() - - # build list of hyperparameters to train & validate - nheads_options = (1, default_hparams.n_heads) - context_size_options = (8, default_hparams.context_size) - dropout_options = (0.1, default_hparams.dropout) - - hparams_list = [ - ModelHyperparameters(n_heads=h, context_size=c, dropout=d) - for h, c, d in product(nheads_options, context_size_options, dropout_options) - ] - - # run training for each hyperparameter setting - results = [] - stop_early = True # stop early so we can compare val losses - print(f"Testing {len(hparams_list)} hyperparameter settings") - n_nodes = len(hparams_list) - static_params = ( - experiment_name, - stop_early, - n_steps, - n_steps_before_eval, - n_steps_before_checkpoint, - ) - for result in train_model.starmap( - [(i, n_nodes, h, *static_params) for i, h in enumerate(hparams_list)], - order_outputs=False, - ): - # result = (node_rank, val_loss, hparams) - node_rank = result[0] - results.append(result) - print( - f"[Node {node_rank + 1}/{n_nodes}] Finished. Early stop val loss result: {result[1:]}" - ) - - # find the model and hparams with the lowest validation loss - best_result = min(results, key=lambda x: x[1]) - print(f"Best early stop val loss result: {best_result}") - best_hparams = best_result[-1] - - # finish training with best hparams - node_rank = 0 - n_nodes = 1 # only one node for final training run - train_model.remote( - node_rank, - n_nodes, - best_hparams, - experiment_name, - not stop_early, - n_steps, - n_steps_before_eval, - n_steps_before_checkpoint, - ) - - -``` - -### Monitor experiments with TensorBoard - -To monitor our training we will create a TensorBoard WSGI web app, which will -display the progress of our training across all 8 models. We'll use the latest -logs for the most recent experiment written to the Volume. - -To ensure we have the latest data we add some -[WSGI Middleware](https://peps.python.org/pep-3333/) -that checks the Modal Volume for updates when the page is reloaded. - -```python -class VolumeMiddleware: - def __init__(self, app): - self.app = app - - def __call__(self, environ, start_response): - if (route := environ.get("PATH_INFO")) in ["/", "/modal-volume-reload"]: - try: - volume.reload() - except Exception as e: - print("Exception while re-loading traces: ", e) - if route == "/modal-volume-reload": - environ["PATH_INFO"] = "/" # redirect - return self.app(environ, start_response) - - -``` - -To ensure a unique color per experiment you can click the palette (🎨) icon -under TensorBoard > Time Series > Run and use the Regex: -`E(\d{4})-(\d{2})-(\d{2})-(\d{6})\.(\d{6})` - -You can deploy this TensorBoard service by running - -``` -modal deploy 06_gpu_and_ml/hyperparameter-sweep/hp_sweep_gpt.py -``` - -and visit it at the URL that ends with `-monitor-training.modal.run`. - -After training finishes, your TensorBoard UI will look something like this: - -![8 lines on a graph, validation loss on y-axis, time step on x-axis. All lines go down over the first 1000 time steps, and one goes to 5000 time steps with a final loss of 1.52](./tensorboard.png) - -You can also find some sample text generated by the model in the "Text" tab. - -```python -@app.function( - image=torch_image, - volumes={volume_path: volume}, -) -@modal.concurrent(max_inputs=100) -@modal.wsgi_app() -def monitor_training(): - board = tensorboard.program.TensorBoard() - board.configure(logdir=str(tb_log_path)) - (data_provider, deprecated_multiplexer) = board._make_data_provider() - wsgi_app = tensorboard.backend.application.TensorBoardWSGIApp( - board.flags, - board.plugin_loaders, - data_provider, - board.assets_zip_provider, - deprecated_multiplexer, - experimental_middlewares=[VolumeMiddleware], - ) - return wsgi_app - - -``` - -Notice that there are 8 models training, and the one with the lowest -validation loss at step 600 continues training to 3000 steps. - -## Serving SLMs on Modal during and after training - -Because our weights are stored in a distributed Volume, -we can deploy an inference Function based off of them without any extra work -- -and we can even check in on models while we're still training them! # For more on storing model weights on Modal, see -[this guide](https://modal.com/docs/guide/model-weights). - -### Remote inference with Modal `Cls`es - -We wrap our inference in a Modal `Cls` called `ModelInference`. -The user of `ModelInference` can control which model is used by providing the -`experiment_name`. Each unique choice creates a separate -[auto-scaling deployment](https://modal.com/docs/guide/parameterized-functions). -If the user does not specify an `experiment_name`, the latest experiment -is used. - -```python -@app.cls(image=torch_image, volumes={volume_path: volume}, gpu=gpu) -class ModelInference: - experiment_name: str = modal.parameter(default="") - - def get_latest_available_model_dirs(self, n_last): - """Find the latest models that have a best model checkpoint saved.""" - save_model_dirs = glob.glob(f"{model_save_path}/*") - sorted_model_dirs = sorted(save_model_dirs, key=os.path.getctime, reverse=True) - - valid_model_dirs = [] - for latest_model_dir in sorted_model_dirs: - if Path(f"{latest_model_dir}/{best_model_filename}").exists(): - valid_model_dirs.append(Path(latest_model_dir)) - if len(valid_model_dirs) >= n_last: - return valid_model_dirs - return valid_model_dirs - - @modal.method() - def get_latest_available_experiment_names(self, n_last): - return [d.name for d in self.get_latest_available_model_dirs(n_last)] - - def load_model_impl(self): - from .src.model import AttentionModel - from .src.tokenizer import Tokenizer - - if self.experiment_name != "": # user selected model - use_model_dir = f"{model_save_path}/{self.experiment_name}" - else: # otherwise, pick latest - try: - use_model_dir = self.get_latest_available_model_dirs(1)[0] - except IndexError: - raise ValueError("No models available to load.") - - if self.use_model_dir == use_model_dir and self.is_fully_trained: - return # already loaded fully trained model. - - print(f"Loading experiment: {Path(use_model_dir).name}...") - checkpoint = torch.load(f"{use_model_dir}/{best_model_filename}") - - self.use_model_dir = use_model_dir - hparams = checkpoint["hparams"] - key = ( # for backwards compatibility - "unique_chars" if "unique_chars" in checkpoint else "chars" - ) - unique_chars = checkpoint[key] - steps = checkpoint["steps"] - val_loss = checkpoint["val_loss"] - self.is_fully_trained = checkpoint["finished_training"] - - print( - f"Loaded model with {steps} train steps" - f" and val loss of {val_loss:.2f}" - f" (fully_trained={self.is_fully_trained})" - ) - - self.tokenizer = Tokenizer(unique_chars) - self.device = "cuda" if torch.cuda.is_available() else "cpu" - - self.model = AttentionModel(self.tokenizer.vocab_size, hparams, self.device) - self.model.load_state_dict(checkpoint["model"]) - self.model.to(self.device) - - @modal.enter() - def load_model(self): - self.use_model_dir = None - self.is_fully_trained = False - self.load_model_impl() - - @modal.method() - def generate(self, prompt): - self.load_model_impl() # load updated model if available - - n_new_tokens = 1000 - return self.model.generate_from_text(self.tokenizer, prompt, n_new_tokens) - - -``` - -### Adding a simple Web Function - -The `ModelInference` class above is available for use -from any other Python environment with the right Modal credentials -and the `modal` package installed -- just use [`lookup`](https://modal.com/docs/reference/modal.Cls#lookup). - -But we can also expose it as a Web Function for easy access -from anywhere, including other programming languages or the command line. - -```python -class GenerationRequest(BaseModel): - prompt: str - - -@app.function(image=web_image) -@modal.fastapi_endpoint(method="POST", docs=True) -def web_generate(request: GenerationRequest): - output = ModelInference().generate.remote(request.prompt) - return {"output": output} - - -``` - -This Function can be deployed on Modal with `modal deploy`. -That will allow us to generate text via a simple `curl` command like this: - -```bash -curl -X POST -H 'Content-Type: application/json' --data-binary '{"prompt": "\n"}' https://your-workspace-name--modal-nano-gpt-web-generate.modal.run -``` - -which will return something like: - -```json -{ -"output": - "BRUTUS: - The broy trefore anny pleasory to - wip me state of villoor so: - Fortols listhey for brother beat the else - Be all, ill of lo-love in igham; - Ah, here all that queen and hould you father offer" -} -``` - -It's not exactly Shakespeare, but at least it shows our model learned something! - -You can choose which model to use by specifying the `experiment_name` in the query parameters of the request URL. - -### Serving a Gradio UI with `asgi_app` - -Second, we create a Gradio web app for generating text via a graphical user interface in the browser. -That way our fellow team members and stakeholders can easily interact with the model and give feedback, -even when we're still training the model. - -You should see the URL for this UI in the output of `modal deploy` -or on your [Modal app dashboard](https://modal.com/apps) for this app. - -The Gradio UI will look something like this: - -![Image of Gradio Web App. Top shows model selection dropdown. Left side shows input prompt textbox. Right side shows SLM generated output. Bottom has button for starting generation process](./gradio.png) - -```python -@app.function( - image=ui_image, - max_containers=1, - volumes={volume_path: volume}, -) -@modal.concurrent(max_inputs=100) -@modal.asgi_app() -def ui(): - import gradio as gr - from fastapi import FastAPI - from fastapi.responses import FileResponse - from gradio.routes import mount_gradio_app - - # call out to the inference in a separate Modal environment with a GPU - def generate(text="", experiment_name=""): - if not text: - text = "\n" - generated = ModelInference(experiment_name=experiment_name).generate.remote( - text - ) - return text + generated - - example_prompts = [ - "DUKE OF YORK:\nWhere art thou Lucas?", - "ROMEO:\nWhat is a man?", - "CLARENCE:\nFair is foul and foul is fair, but who are you?", - "Brevity is the soul of wit, so what is the soul of foolishness?", - ] - - web_app = FastAPI() - - # custom styles: an icon, a background, and a theme - @web_app.get("/favicon.ico", include_in_schema=False) - async def favicon(): - return FileResponse("/assets/favicon.svg") - - @web_app.get("/assets/background.svg", include_in_schema=False) - async def background(): - return FileResponse("/assets/background.svg") - - with open("/assets/index.css") as f: - css = f.read() - - n_last = 20 - experiment_names = ModelInference().get_latest_available_experiment_names.remote( - n_last - ) - theme = gr.themes.Default( - primary_hue="green", secondary_hue="emerald", neutral_hue="neutral" - ) - - # add a Gradio UI around inference - with gr.Blocks(theme=theme, css=css, title="SLM") as interface: - # title - gr.Markdown("# GPT-style Shakespeare text generation.") - - # Model Selection - with gr.Row(): - gr.Markdown("## Model Version") - with gr.Row(): - experiment_dropdown = gr.Dropdown( - experiment_names, label="Select Model Version" - ) - - # input and output - with gr.Row(): - with gr.Column(): - gr.Markdown("## Input:") - input_box = gr.Textbox( # input text component - label="", - placeholder="Write some Shakespeare like text or keep it empty!", - lines=10, - ) - with gr.Column(): - gr.Markdown("## Output:") - output_box = gr.Textbox( # output text component - label="", - lines=10, - ) - - # button to trigger inference and a link to Modal - with gr.Row(): - generate_button = gr.Button("Generate", variant="primary", scale=2) - generate_button.click( - fn=generate, - inputs=[input_box, experiment_dropdown], - outputs=output_box, - ) # connect inputs and outputs with inference function - - gr.Button( # shameless plug - " Powered by Modal", - variant="secondary", - link="https://modal.com", - ) - - # example prompts - with gr.Column(variant="compact"): - # add in a few examples to inspire users - for ii, prompt in enumerate(example_prompts): - btn = gr.Button(prompt, variant="secondary") - btn.click(fn=lambda idx=ii: example_prompts[idx], outputs=input_box) - - # mount for execution on Modal - return mount_gradio_app( - app=web_app, - blocks=interface, - path="/", - ) - - -``` - -## Addenda - -The remainder of this code is boilerplate. - -### Training Loop - -There's quite a lot of code for just the training loop! If you'd rather not write this stuff yourself, -consider a training framework like [PyTorch Lightning](https://lightning.ai/docs/pytorch/stable) -or [Hugging Face](https://huggingface.co/transformers/main_classes/trainer.html). - -```python -def training_loop( - start_step, - n_steps, - n_steps_before_eval, - n_steps_before_checkpoint, - n_eval_steps, - dataset, - tokenizer, - model, - optimizer, - logs_manager, - checkpoint, - checkpoint_path, - run_to_first_save, -): - @torch.no_grad() - def eval_model(model, dataset, tokenizer, n_eval_steps): - """Evaluate model on train and validation data.""" - out = {} - model.eval() # Turn off gradients - for split in ("train", "val"): - losses = torch.zeros(n_eval_steps) - for k in range(n_eval_steps): - xb, yb = dataset.get_batch(split) - logits, loss = model.forward(xb, yb) - losses[k] = loss - out[split] = losses.mean() - - # Generate some output samples - out["sample"] = model.generate_from_text(tokenizer, "\n", 1000) - - model.train() # Turn on gradients - return out - - t_last = timer() - for step in range(start_step, n_steps + 1): - # sample a batch of data - xb, yb = dataset.get_batch("train") - - # evaluate the loss, calculate & apply gradients - logits, loss = model.forward(xb, yb) - optimizer.zero_grad(set_to_none=True) - loss.backward() - optimizer.step() - - # log training loss - logs_manager.add_train_scalar("Cross Entropy Loss", loss.item(), step) - - # evaluate model on validation set - if step % n_steps_before_eval == 0: - out = eval_model(model, dataset, tokenizer, n_eval_steps) - log_evals(out, step, t_last, logs_manager) - t_last = timer() - - # save model with checkpoint information - if step > 0 and step % n_steps_before_checkpoint == 0: - checkpoint["steps"] = step - checkpoint["val_loss"] = out["val"] - - # mark as finished if we hit n steps. - checkpoint["finished_training"] = step >= n_steps - - L.info( - f"Saving checkpoint to {checkpoint_path}\t {checkpoint['finished_training']})" - ) - save_checkpoint(checkpoint, checkpoint_path) - - if run_to_first_save: - L.info("Stopping early...") - break - return out - - -def save_checkpoint(checkpoint, checkpoint_path): - torch.save(checkpoint, checkpoint_path) - volume.commit() - - -def build_model(hparams, vocab_size, device): - """Initialize the model and move it to the device.""" - model = AttentionModel(vocab_size, hparams, device) - model.to(device) - return model - - -def setup_optimizer(model, learning_rate): - """Set up the optimizer for the model.""" - return torch.optim.AdamW(model.parameters(), lr=learning_rate) - - -``` - -### Miscellaneous - -The remaining code includes small helper functions for training the model. - -```python -def prepare_data(input_file_path: Path, volume: modal.Volume) -> str: - """Download and read the dataset.""" - volume.reload() - if not input_file_path.exists(): - L.info("Downloading Shakespeare dataset...") - data_url = "https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt" - urllib.request.urlretrieve(data_url, input_file_path) - volume.commit() - return input_file_path.read_text() - - -def make_best_symbolic_link(model_save_dir, model_filename, experiment_name): - # create symlink to the best model so it's easy to find for web serving - os.symlink( - str(model_save_dir / model_filename), - str(model_save_path / experiment_name / best_model_filename), - ) - volume.commit() # commit the symlink - - -def init_checkpoint(model, tokenizer, optimizer, start_step, hparams): - return { - "model": model.state_dict(), - "unique_chars": tokenizer.unique_chars, - "optimizer": optimizer.state_dict(), - "val_loss": float("inf"), - "steps": start_step, - "hparams": hparams, - "finished_training": False, - } - - -def log_evals(result, step, t_last, logs_manager): - runtime_s = timer() - t_last - L.info( - f"{step:5d}) // {runtime_s:>5.2f}s // Train Loss: {result['train']:.2f} // Val Loss: {result['val']:.2f}" - ) - logs_manager.add_val_scalar("Cross Entropy Loss", result["val"], step) - logs_manager.add_val_text("Sample Output", result["sample"], step) - logs_manager.flush() - volume.commit() # Make sure TensorBoard container will see it. - - return result - -``` diff --git a/.agents/skills/modal/references/examples/image_to_image.md b/.agents/skills/modal/references/examples/image_to_image.md deleted file mode 100644 index 42a59874c936c10182e706fba94030431f6652e6..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/image_to_image.md +++ /dev/null @@ -1,207 +0,0 @@ -# Edit images with Flux Kontext - -In this example, we run the Flux Kontext model in *image-to-image* mode: -the model takes in a prompt and an image and edits the image to better match the prompt. - -For example, the model edited the first image into the second based on the prompt -"*A cute dog wizard inspired by Gandalf from Lord of the Rings, featuring detailed fantasy elements in Studio Ghibli style*". - - A photo of a dog transformed into a cartoon of a cute dog wizard - -The model is Black Forest Labs' [FLUX.1-Kontext-dev](https://huggingface.co/black-forest-labs/FLUX.1-Kontext-dev). -Learn more about the model [here](https://bfl.ai/announcements/flux-1-kontext-dev). - -## Define a container image - -First, we define the environment the model inference will run in, -the [container image](https://modal.com/docs/guide/custom-container). - -We start from an NVIDIA CUDA base image and install the necessary Python packages. -We use a specific commit of the `diffusers` library to ensure compatibility with the Flux Kontext model. - -```python -from io import BytesIO -from pathlib import Path - -import modal - -app = modal.App("example-image-to-image") - -diffusers_commit_sha = "00f95b9755718aabb65456e791b8408526ae6e76" - -image = ( - modal.Image.from_registry("nvidia/cuda:12.8.1-devel-ubuntu22.04", add_python="3.12") - .entrypoint([]) # remove verbose logging by base image on entry - .apt_install("git") - .uv_pip_install( - "Pillow~=11.2.1", - "accelerate~=1.8.1", - f"git+https://github.com/huggingface/diffusers.git@{diffusers_commit_sha}", - "huggingface-hub==0.36.0", - "optimum-quanto==0.2.7", - "safetensors==0.5.3", - "sentencepiece==0.2.0", - "torch==2.7.1", - "transformers~=4.53.0", - extra_options="--index-strategy unsafe-best-match", - extra_index_url="https://download.pytorch.org/whl/cu128", - ) -) - -``` - -## Download the model - -We'll be using the FLUX.1-Kontext-dev model from Black Forest Labs. -This model specializes in image-to-image editing with strong prompt adherence. - -```python -MODEL_NAME = "black-forest-labs/FLUX.1-Kontext-dev" -MODEL_REVISION = "f9fdd1a95e0dfd7653cb0966cda2486745122695" - -``` - -Note that access to the FLUX.1-Kontext-dev model on Hugging Face is -[gated by a license agreement](https://huggingface.co/docs/hub/en/models-gated) which -you must agree to [here](https://huggingface.co/black-forest-labs/FLUX.1-Kontext-dev). -After you have accepted the license, [create a Modal Secret](https://modal.com/secrets) -with the name `huggingface-secret` following the instructions in the template. - -## Cache the model weights - -The model weights are large (tens of gigabytes), so we want to cache them -to avoid downloading them every time a container starts. -We use a [Modal Volume](https://modal.com/docs/guide/volumes) to persist the Hugging Face cache. -Modal Volumes act like a shared disk that all Modal Functions can access. -For more on storing model weights on Modal, see [this guide](https://modal.com/docs/guide/model-weights). - -```python -CACHE_DIR = Path("/cache") -cache_volume = modal.Volume.from_name("hf-hub-cache", create_if_missing=True) -volumes = {CACHE_DIR: cache_volume} - -``` - -We reference the Hugging Face secret we created earlier to authenticate when downloading the model. - -```python -secrets = [modal.Secret.from_name("huggingface-secret")] - -``` - -We configure environment variables to enable faster downloads from Hugging Face -and point the Hugging Face cache to our Modal Volume. - -```python -image = image.env({"HF_XET_HIGH_PERFORMANCE": "1", "HF_HOME": str(CACHE_DIR)}) - -``` - -Finally, we import packages we'll be using in our inference function, -but not locally. - -```python -with image.imports(): - import torch - from diffusers import FluxKontextPipeline - from diffusers.utils import load_image - from PIL import Image - - -``` - -## Set up and run Flux Kontext - -The Modal `Cls` defined below contains all the logic to set up and run Flux Kontext inference. - -We define our Python class as a Modal `Cls` using the `app.cls` decorator. -We provide a few arguments to describe the infrastructure our inference should run on: - -* the Image, Volume, and Secret we defined above -* a [`gpu`](https://modal.com/docs/guide/gpu), in particular a [B200](https://modal.com/blog/introducing-b200-h200) - -The [container lifecycle](https://modal.com/docs/guide/lifecycle-functions) decorator, -`@modal.enter`, ensures that the model is loaded into memory when a container starts, before it picks up any inputs. -This is useful for managing tail latencies (see [this guide](https://modal.com/docs/guide/cold-start) for details). - -The `inference` method runs the actual model inference. It takes in an image (as raw `bytes`) and a string `prompt` and returns -a new image (also as raw `bytes`). - -```python -@app.cls(image=image, gpu="B200", volumes=volumes, secrets=secrets) -class Model: - @modal.enter() - def enter(self): - print(f"Loading {MODEL_NAME}...") - - self.pipe = FluxKontextPipeline.from_pretrained( - MODEL_NAME, - revision=MODEL_REVISION, - torch_dtype=torch.bfloat16, - cache_dir=CACHE_DIR, - ).to("cuda") - - @modal.method() - def inference( - self, - image_bytes: bytes, - prompt: str, - guidance_scale: float = 3.5, - num_inference_steps: int = 20, - seed: int | None = None, - ) -> bytes: - init_image = load_image(Image.open(BytesIO(image_bytes))).resize((512, 512)) - - generator = None - if seed is not None: - generator = torch.Generator(device="cuda").manual_seed(seed) - - image = self.pipe( - image=init_image, - prompt=prompt, - guidance_scale=guidance_scale, - num_inference_steps=num_inference_steps, - output_type="pil", - generator=generator, - ).images[0] - - byte_stream = BytesIO() - image.save(byte_stream, format="PNG") - - return byte_stream.getvalue() - - -``` - -## Running the model from the command line - -You can run the model from the command line with - -```bash -modal run image_to_image.py -``` - -Use `--help` for additional details. - -```python -@app.local_entrypoint() -def main( - image_path=Path(__file__).parent / "demo_images/dog.png", - output_path=Path("/tmp/stable-diffusion/output.png"), - prompt: str = "A cute dog wizard inspired by Gandalf from Lord of the Rings, featuring detailed fantasy elements in Studio Ghibli style", -): - print(f"🎨 reading input image from {image_path}") - input_image_bytes = Path(image_path).read_bytes() - print(f"🎨 editing image with prompt '{prompt}'") - output_image_bytes = Model().inference.remote(input_image_bytes, prompt) - - if isinstance(output_path, str): - output_path = Path(output_path) - - dir = output_path.parent - dir.mkdir(exist_ok=True, parents=True) - - print(f"🎨 saving output image to {output_path}") - output_path.write_bytes(output_image_bytes) - -``` diff --git a/.agents/skills/modal/references/examples/image_to_video.md b/.agents/skills/modal/references/examples/image_to_video.md deleted file mode 100644 index 3f6c7455c5f0fc1961f59390a9db4cc2e0872a37..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/image_to_video.md +++ /dev/null @@ -1,359 +0,0 @@ -# Animate images with Lightricks LTX-Video via CLI, API, and web UI - -This example shows how to run [LTX-Video](https://huggingface.co/Lightricks/LTX-Video) on Modal -to generate videos from your local command line, via an API, and in a web UI. - -Generating a 5 second video takes ~1 minute from cold start. -Once the container is warm, a 5 second video takes ~15 seconds. - -Here is a sample we generated: - -
- -
- -## Basic setup - -```python -import io -import random -import time -from pathlib import Path -from typing import Annotated, Optional - -import fastapi -import modal - -``` - -All Modal programs need an [`App`](https://modal.com/docs/reference/modal.App) — -an object that acts as a recipe for the application. - -```python -app = modal.App("example-image-to-video") - -``` - -### Configuring dependencies - -The model runs remotely, on Modal's cloud, which means we need to -[define the environment it runs in](https://modal.com/docs/guide/images). - -Below, we start from a lightweight base Linux image -and then install our system and Python dependencies, -like Hugging Face's `diffusers` library and `torch`. - -```python -image = ( - modal.Image.debian_slim(python_version="3.12") - .apt_install("python3-opencv") - .uv_pip_install( - "accelerate==1.4.0", - "diffusers==0.32.2", - "fastapi[standard]==0.115.8", - "huggingface-hub==0.36.0", - "imageio==2.37.0", - "imageio-ffmpeg==0.6.0", - "opencv-python==4.11.0.86", - "pillow==11.1.0", - "sentencepiece==0.2.0", - "torch==2.6.0", - "torchvision==0.21.0", - "transformers==4.49.0", - ) -) - -``` - -## Storing model weights on Modal - -We also need the parameters of the model remotely. -They can be loaded at runtime from Hugging Face, -based on a repository ID and a revision (aka a commit SHA). - -```python -MODEL_ID = "Lightricks/LTX-Video" -MODEL_REVISION_ID = "a6d59ee37c13c58261aa79027d3e41cd41960925" - -``` - -Hugging Face will also cache the weights to disk once they're downloaded. -But Modal Functions are serverless, and so even disks are ephemeral, -which means the weights would get re-downloaded every time we spin up a new instance. - -We can fix this -- without any modifications to Hugging Face's model loading code! -- -by pointing the Hugging Face cache at a [Modal Volume](https://modal.com/docs/guide/volumes). For more on storing model weights on Modal, see -[this guide](https://modal.com/docs/guide/model-weights). - -```python -model_volume = modal.Volume.from_name("hf-hub-cache", create_if_missing=True) - -MODEL_PATH = "/models" # where the Volume will appear on our Functions' filesystems - -image = image.env( - { - "HF_XET_HIGH_PERFORMANCE": "1", # faster downloads - "HF_HUB_CACHE": MODEL_PATH, - } -) - -``` - -## Storing model outputs on Modal - -Contemporary video models can take a long time to run and they produce large outputs. -That makes them a great candidate for storage on Modal Volumes as well. -Python code running outside of Modal can also access this storage, as we'll see below. - -```python -OUTPUT_PATH = "/outputs" -output_volume = modal.Volume.from_name("outputs", create_if_missing=True) - -``` - -## Implementing LTX-Video inference on Modal - -We wrap the inference logic in a Modal [Cls](https://modal.com/docs/guide/lifecycle-functions) -that ensures models are loaded and then moved to the GPU once when a new instance -starts, rather than every time we run it. - -The `run` function just wraps a `diffusers` pipeline. -It saves the generated video to a Modal Volume, and returns the filename. - -We also include a `web` wrapper that makes it possible -to trigger inference via an API call. -For details, see the `/docs` route of the URL ending in `inference-web.modal.run` -that appears when you deploy the app. - -```python -with image.imports(): # loaded on all of our remote Functions - import diffusers - import torch - from PIL import Image - -MINUTES = 60 - - -@app.cls( - image=image, - gpu="H100", - timeout=10 * MINUTES, - scaledown_window=10 * MINUTES, - volumes={MODEL_PATH: model_volume, OUTPUT_PATH: output_volume}, -) -class Inference: - @modal.enter() - def load_pipeline(self): - self.pipe = diffusers.LTXImageToVideoPipeline.from_pretrained( - MODEL_ID, - revision=MODEL_REVISION_ID, - torch_dtype=torch.bfloat16, - ).to("cuda") - - @modal.method() - def run( - self, - image_bytes: bytes, - prompt: str, - negative_prompt: Optional[str] = None, - num_frames: Optional[int] = None, - num_inference_steps: Optional[int] = None, - seed: Optional[int] = None, - ) -> str: - negative_prompt = ( - negative_prompt - or "worst quality, inconsistent motion, blurry, jittery, distorted" - ) - width = 768 - height = 512 - num_frames = num_frames or 25 - num_inference_steps = num_inference_steps or 50 - seed = seed or random.randint(0, 2**32 - 1) - print(f"Seeding RNG with: {seed}") - torch.manual_seed(seed) - - image = diffusers.utils.load_image(Image.open(io.BytesIO(image_bytes))) - - video = self.pipe( - image=image, - prompt=prompt, - negative_prompt=negative_prompt, - width=width, - height=height, - num_frames=num_frames, - num_inference_steps=num_inference_steps, - ).frames[0] - - mp4_name = ( - f"{seed}_{''.join(c if c.isalnum() else '-' for c in prompt[:100])}.mp4" - ) - diffusers.utils.export_to_video( - video, f"{Path(OUTPUT_PATH) / mp4_name}", fps=24 - ) - output_volume.commit() - torch.cuda.empty_cache() # reduce fragmentation - return mp4_name - - @modal.fastapi_endpoint(method="POST", docs=True) - def web( - self, - image_bytes: Annotated[bytes, fastapi.File()], - prompt: str, - negative_prompt: Optional[str] = None, - num_frames: Optional[int] = None, - num_inference_steps: Optional[int] = None, - seed: Optional[int] = None, - ) -> fastapi.Response: - mp4_name = self.run.local( # run in the same container - image_bytes=image_bytes, - prompt=prompt, - negative_prompt=negative_prompt, - num_frames=num_frames, - num_inference_steps=num_inference_steps, - seed=seed, - ) - return fastapi.responses.FileResponse( - path=f"{Path(OUTPUT_PATH) / mp4_name}", - media_type="video/mp4", - filename=mp4_name, - ) - - -``` - -## Generating videos from the command line - -We add a [local entrypoint](https://modal.com/docs/reference/modal.App#local_entrypoint) -that calls the `Inference.run` method to run inference from the command line. -The function's parameters are automatically turned into a CLI. - -Run it with - -```bash -modal run image_to_video.py --prompt "A cat looking out the window at a snowy mountain" --image-path /path/to/cat.jpg -``` - -You can also pass `--help` to see the full list of arguments. - -```python -@app.local_entrypoint() -def entrypoint( - image_path: str, - prompt: str, - negative_prompt: Optional[str] = None, - num_frames: Optional[int] = None, - num_inference_steps: Optional[int] = None, - seed: Optional[int] = None, - twice: bool = True, -): - import os - import urllib.request - - print(f"🎥 Generating a video from the image at {image_path}") - print(f"🎥 using the prompt {prompt}") - - if image_path.startswith(("http://", "https://")): - image_bytes = urllib.request.urlopen(image_path).read() - elif os.path.isfile(image_path): - image_bytes = Path(image_path).read_bytes() - else: - raise ValueError(f"{image_path} is not a valid file or URL.") - - inference_service = Inference() - - for _ in range(1 + twice): - start = time.time() - mp4_name = inference_service.run.remote( - image_bytes=image_bytes, - prompt=prompt, - negative_prompt=negative_prompt, - num_frames=num_frames, - seed=seed, - ) - duration = time.time() - start - print(f"🎥 Generated video in {duration:.3f}s") - - output_dir = Path("/tmp/image_to_video") - output_dir.mkdir(exist_ok=True, parents=True) - output_path = output_dir / mp4_name - # read in the file from the Modal Volume, then write it to the local disk - output_path.write_bytes(b"".join(output_volume.read_file(mp4_name))) - print(f"🎥 Video saved to {output_path}") - - -``` - -## Generating videos via an API - -The Modal `Cls` above also included a [`fastapi_endpoint`](https://modal.com/docs/examples/basic_web), -which adds a simple web API to the inference method. - -To try it out, run - -```bash -modal deploy image_to_video.py -``` - -copy the printed URL ending in `inference-web.modal.run`, -and add `/docs` to the end. This will bring up the interactive -Swagger/OpenAPI docs for the endpoint. - -## Generating videos in a web UI - -Lastly, we add a simple front-end web UI (written in Alpine.js) for -our image to video backend. - -This is also deployed when you run - -```bash -modal deploy image_to_video.py. -``` - -The `Inference` class will serve multiple users from its own auto-scaling pool of warm GPU containers automatically, -and they will spin down when there are no requests. - -```python -frontend_path = Path(__file__).parent / "frontend" - -web_image = ( - modal.Image.debian_slim(python_version="3.12") - .uv_pip_install("jinja2==3.1.5", "fastapi[standard]==0.115.8") - .add_local_dir( # mount frontend/client code - frontend_path, remote_path="/assets" - ) -) - - -@app.function(image=web_image) -@modal.concurrent(max_inputs=100) -@modal.asgi_app() -def ui(): - import fastapi.staticfiles - import fastapi.templating - - web_app = fastapi.FastAPI() - templates = fastapi.templating.Jinja2Templates(directory="/assets") - - @web_app.get("/") - async def read_root(request: fastapi.Request): - return templates.TemplateResponse( - "index.html", - { - "request": request, - "inference_url": Inference().web.get_web_url(), - "model_name": "LTX-Video Image to Video", - "default_prompt": "A young girl stands calmly in the foreground, looking directly at the camera, as a house fire rages in the background.", - }, - ) - - web_app.mount( - "/static", - fastapi.staticfiles.StaticFiles(directory="/assets"), - name="static", - ) - - return web_app - -``` diff --git a/.agents/skills/modal/references/examples/index.md b/.agents/skills/modal/references/examples/index.md deleted file mode 100644 index 273cef23fef2a30761767ef62a2abb1412657c8f..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/index.md +++ /dev/null @@ -1,77 +0,0 @@ -# Examples - -- Getting started - - [Hello, world](./hello_world.md) - - [Simple web scraper](./webscraper.md) - - [Serving Web Functions](./basic_web.md) -- Large language models (LLMs) - - [Deploy an OpenAI-compatible LLM service with vLLM](./llm_inference.md) - - [Cut Ministral 3 cold start times by 10x with snapshots](./ministral3_inference.md) - - [Maximize tokens per second in batch processing with vLLM](./vllm_throughput.md) - - [Serve an ultra-low-latency chatbot with SGLang](./sglang_low_latency.md) - - [Deploy Nemotron 3](./nemotron_inference.md) - - [Efficient LLM Finetuning with Unsloth](./unsloth_finetune.md) - - [Run a multimodal RAG chatbot to answer questions about PDFs](./chat_with_pdf_vision.md) - - [Fine-tune an LLM to replace your CEO](./llm-finetuning.md) - - [Deploy a stateless MCP with FastMCP](./mcp_server_stateless.md) -- Images, video, & 3D - - [Edit images with Flux Kontext](./image_to_image.md) - - [Fine-tune Wan2.1 video models on your face](./music-video-gen.md) - - [Run Flux fast with torch.compile](./flux.md) - - [Fine-tune Flux with LoRA](./diffusers_lora_finetune.md) - - [Animate images with LTX-Video](./image_to_video.md) - - [Generate video clips with LTX-Video](./ltx.md) - - [Run Stable Diffusion with a CLI, API, and web UI](./text_to_image.md) -- Audio - - [Deploy a Moshi voice chatbot](./llm-voice-chat.md) - - [Stream transcripts at the speed of speech using Kyutai STT](./streaming_kyutai_stt.md) - - [Make music with ACE-Step](./generate_music.md) - - [Generate speech with Chatterbox](./chatterbox_tts.md) - - [Run high throughput batched transcription with Whisper](./batched_whisper.md) - - [Fine-tune Whisper to recognize new words](./fine_tune_asr.md) -- Real-time communication (WebRTC) - - [Serverless WebRTC](./webrtc_yolo.md) - - [WebRTC quickstart with FastRTC](./fastrtc_flip_webcam.md) -- Computational biology - - [Design protein binders with ESMFold2](./esmfold2_binder_design.md) - - [Fold proteins with Chai-1](./chai1.md) - - [Build a protein-folding dashboard](./esm3.md) - - [Fold proteins with Boltz-2](./boltz_predict.md) -- Modal Sandboxes - - [Run a background coding agent with OpenCode](./opencode_server.md) - - [Build a scalable AI coding platform](./modal-vibe.md) - - [Create GIFs from Slack using the Claude Agent SDK](./claude-slack-gif-creator.md) - - [Run a LangGraph agent's code in a secure GPU sandbox](./agent.md) - - [Control a sandboxed computer with an LLM](./anthropic_computer_use.md) - - [Build a stateful, sandboxed code interpreter](./simple_code_interpreter.md) - - [Run Node.js, Ruby, and more in a Sandbox](./safe_code_execution.md) - - [Speed up Sandbox starts with warm pools](./sandbox_pool.md) -- Reinforcement Learning - - [Train a model to solve math problems using GRPO and verl](./grpo_verl.md) - - [Train a model to solve coding problems using GRPO and TRL](./grpo_trl.md) -- Embeddings - - [Embed millions of documents with TEI](./amazon_embeddings.md) - - [Turn satellite images into vectors and store them in MongoDB](./mongodb-search.md) -- Parallel processing and job scheduling - - [Deploy a Hacker News Slackbot](./hackernews_alerts.md) - - [Run a Document OCR job queue](./doc_ocr_jobs.md) - - [Serve a Document OCR web app](./doc_ocr_webapp.md) -- Training models from scratch - - [Train an SLM with early-stopping grid search over hyperparameters](./hp_sweep_gpt.md) - - [Run long, resumable training jobs](./long-training.md) -- Hosting popular libraries - - [YOLO: Fine-tune and serve computer vision models](./finetune_yolo.md) - - [Blender: Build a 3D render farm](./blender_video.md) - - [Streamlit: Run and deploy Streamlit apps](./serve_streamlit.md) - - [SQLite: Publish explorable data with Datasette](./cron_datasette.md) - - [Algolia: Build docsearch with a crawler](./algolia_indexer.md) -- Connecting to other APIs - - [Discord: Deploy and run a Discord Bot](./discord_bot.md) - - [Google Sheets: Sync databases and APIs to a Google Sheet](./db_to_sheet.md) - - [OpenAI: Run a RAG Q&A chatbot](./potus_speech_qanda.md) - - [Tailscale: Add Modal Apps to your VPN](./modal_tailscale.md) - - [Prometheus: Publish custom metrics with Pushgateway](./pushgateway.md) -- Managing data - - [Mount S3 buckets in Modal apps](./s3_bucket_mount.md) - - [Build your own data warehouse with DuckDB, DBT, and Modal](./dbt_duckdb.md) - - [Create a LoRA Playground with Modal, Gradio, and S3](./cloud_bucket_mount_loras.md) diff --git a/.agents/skills/modal/references/examples/llm-finetuning.md b/.agents/skills/modal/references/examples/llm-finetuning.md deleted file mode 100644 index ee592872fce5770add303f3c9000d54316fa9751..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/llm-finetuning.md +++ /dev/null @@ -1,226 +0,0 @@ -# DoppelBot: Fine-tune an LLM to replace your CEO - -*(quick links: -[add to your own Slack](https://github.com/modal-labs/doppel-bot#usage); -[source code](https://github.com/modal-labs/doppel-bot))* - -Internally at Modal, we spend a *lot* of time talking to each other on Slack. -Now, with the advent of open-source large language models, we had started to -wonder if all of this wasn't a bit redundant. Could we have these language -models bike-shed on Slack for us, so we could spend our time on higher leverage -activities such as -[paddleboarding in Tahiti](https://x.com/modal/status/1642262543757352960) -instead? - -To test this, we fine-tuned -[Llama 3.1](https://ai.meta.com/blog/meta-llama-3-1/) on -[Erik](https://twitter.com/bernhardsson)'s Slack messages, and `@erik-bot` was -born. - -![erik-bot](https://modal-cdn.com/erik-bot-1.jpeg) - -Since then, `@erik-bot` has been an invaluable asset to us, in areas ranging -from [API design](https://modal-cdn.com/erik-bot-2.png) to -[legal advice](https://modal-cdn.com/erik-bot-3.png) to thought leadership. - -![erik-bot-3](https://modal-cdn.com/erik-bot-4.png) - -We were planning on releasing the weights for `@erik-bot` to the world, but all -our metrics have been going up and to the right a little too much since we've -launched him... - -So, we are releasing the next best thing. `DoppelBot` is a Slack bot that you -can install in your own workspace, and fine-tune on your own Slack messages. -Follow the instructions [here](https://github.com/modal-labs/doppel-bot#usage) -to replace your own CEO with an LLM today. - -All the components—scraping, fine-tuning, inference and slack event handlers run -on Modal, and the code itself is open-source and available -[here](https://github.com/modal-labs/doppel-bot). If you're new to Modal, it's -worth reiterating that **all of these components are also serverless and scale -to zero**. This means that you can deploy and forget about them, because you'll -only pay for compute when your app is used! - -## How it works - -DoppelBot uses the Slack SDK to scrape messages from a Slack workspace, and -converts them into prompt/response pairs. It uses these to fine-tune a language -model using [Low-Rank Adaptation (LoRA)](https://arxiv.org/abs/2106.09685), a -technique that produces a small adapter that can be merged with the base model -when needed, instead of modifying all the parameters in the base model. The -fine-tuned adapters for each user are stored in a Modal -[Volume](/docs/guide/volumes). When a user `@`s the bot, -Slack sends a webhook call to Modal, which loads the adapter for that user and -generates a response. - -We go into detail into each of these steps below, and provide commands for -running each of them individually. To follow along, -[clone the repo](https://github.com/modal-labs/doppel-bot) and -[set up a Slack token](https://github.com/modal-labs/doppel-bot#create-a-slack-app) -for yourself. - -### Scraping slack - - - -The scraper uses Modal's [`.map()`](/docs/guide/scale#scaling-out) to fetch -messages from all public channels in parallel. Each thread is split into -contiguous messages from the target users and continguous messages from other -users. These will be fed into the model as prompts in the following format: - -``` -[system]: You are {user}, employee at a fast-growing startup. Below is an input conversation that takes place in the company's internal Slack. Write a response that appropriately continues the conversation. - -[user]: - -[assistant]: -``` - -Initial versions of the model were prone to generating short responses -— unsurprising, because a majority of Slack communication is pretty terse. -Adding a minimum character length for the target user's messages fixed this. - -If you're following along at home, you can run the scraper with the following -command: - -```bash -modal run -m src.scrape::scrape --user="" -``` - -Scraped results are stored in a Modal -[Volume](/docs/guide/volumes), so they can be used by the next step. - -### Fine-tuning - - - -Next, we use the prompts to fine-tune a language model. We chose -[Llama 3.1](https://ai.meta.com/blog/meta-llama-3-1/) because of its permissive license and high quality relative to its small size. Fine-tuning is -done using [Low-Rank Adaptation (LoRA)](https://arxiv.org/abs/2106.09685), a -[parameter-efficient fine-tuning](https://huggingface.co/blog/peft) technique -that produces a small adapter that can be merged with the base model when needed -(~60MB for the rank we're using). - -Our fine-tuning implementation uses [torchtune](https://github.com/pytorch/torchtune), a new PyTorch library for easily configuring fine-tuning runs. - -Because of the typically small sample sizes we're working with, training for -longer than a couple hundred steps (with our batch size of 128) quickly led to -overfitting. Admittedly, we haven't thoroughly evaluated the hyperparameter -space yet — do reach out to us if you're interested in collaborating on this! - -![train-loss](../../assets/docs/train-loss.png) - -To try this step yourself, run: - -```bash -modal run -m src.finetune --user="" -``` - -### Inference - - - -We use [vLLM](https://github.com/vllm-project/vllm) as our inference engine, which now comes with support for dynamically swapping LoRA adapters [out of the box](https://docs.vllm.ai/en/latest/features/lora.html). - -With parametrized functions, every user model gets its own pool of containers -that scales up when there are incoming requests, and scales to 0 when there's -none. Here's what that looks like stripped down to the essentials: - -```python notest -@app.cls(gpu="L40S") -class Model(): - @modal.enter() - def enter(self): - self.engine = AsyncLLMEngine.from_engine_args(AsyncEngineArgs(...)) - self.loras: dict[str, int] = dict() # per replica LoRA identifier - - @modal.method() - def generate(self, input: str): - if (ident := f"{user}-{team_id}") not in self.loras: - self.loras[ident] = len(self.loras) + 1 - - lora_request = LoRARequest( - ident, self.loras[ident], lora_local_path=checkpoint_path - ) - - tokenizer = await self.engine.get_tokenizer(lora_request=lora_request) - - prompt = tokenizer.apply_chat_template( - conversation=inpt, tokenize=False, add_generation_prompt=True - ) - - results_generator = self.engine.generate(prompt, lora_request=lora_request,) -``` - -If you've fine-tuned a model already in the previous step, you can run inference -using it now: - -```bash -modal run -m src.inference --user="" -``` - -(We have a list of sample inputs in the file, but you can also try it out with -your own messages!) - -### Slack Bot - - - -Finally, it all comes together in -[`bot.py`](https://github.com/modal-labs/doppel-bot/blob/main/src/bot.py). As -you might have guessed, all events from Slack are handled by serverless Modal -functions. We handle 3 types of events: - -* [`url_verification`](https://github.com/modal-labs/doppel-bot/blob/24609583c43c0e722f56f85a1c00bb55b46c7754/src/bot.py#L112): - To verify that this is a Slack app, Slack expects us to return a challenge - string. -* [`app_mention`](https://github.com/modal-labs/doppel-bot/blob/main/src/bot.py#L118): - When the bot is mentioned in a channel, we retrieve the recent messages from - that thread, do some basic cleaning and call the user's model to generate a - response. - -```python notest -model = OpenLlamaModel.remote(user, team_id) -result = model.generate(messages) -``` - -* [`doppel` slash command](https://github.com/modal-labs/doppel-bot/blob/main/src/bot.py#L182): - This command kicks off the scraping -> finetuning pipeline for the user. - -To deploy the slackbot in its entirety, you need to run: - -```shell -modal deploy -m src.bot -``` - -
- -### Multi-Workspace Support - -
- -Everything we've talked about so far is for a single-workspace Slack app. To -make it work with multiple workspaces, we'll need to handle -[workspace installation and authentication with OAuth](https://api.slack.com/authentication/oauth-v2), -and also store some state for each workspace. - -Luckily, Slack's [Bolt](https://slack.dev/bolt-python/concepts) framework -provides a complete (but frugally documented) OAuth implemention. A neat feature -is that the OAuth state can be backed by a file system, so all we need to do is -[point Bolt](https://github.com/modal-labs/doppel-bot/blob/24609583c43c0e722f56f85a1c00bb55b46c7754/src/bot.py#L78) -at a Modal [Volume](/docs/guide/volumes), and then we don't need to worry about -managing this state ourselves. - -To store state for each workspace, we're using [Neon](https://neon.tech/), a -serverless Postgres database that's really easy to set up and *just works*. If -you're interested in developing a multi-workspace app, -[follow our instructions](https://github.com/modal-labs/doppel-bot#optional-multi-workspace-app) -on how to set up Neon with Modal. - -## Next Steps - -If you've made it this far, you have just found a way to increase your team's -productivity by 10x! Congratulations on the well-earned vacation! 🎉 - -If you're interested in learning more about Modal, check out our [docs](/docs) -and other [examples](/examples). diff --git a/.agents/skills/modal/references/examples/llm-voice-chat.md b/.agents/skills/modal/references/examples/llm-voice-chat.md deleted file mode 100644 index dd25ec358375cb48fe2c68c0cf6ae4caacbcfd49..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/llm-voice-chat.md +++ /dev/null @@ -1,131 +0,0 @@ -# QuiLLMan: Voice Chat with Moshi - -[QuiLLMan](https://github.com/modal-labs/quillman) is a complete voice chat application built on Modal: you speak and the chatbot speaks back! - -At the core is Kyutai Lab's [Moshi](https://github.com/kyutai-labs/moshi) model, a speech-to-speech language model that will continuously listen, plan, and respond to the user. - -Thanks to bidirectional websocket streaming and [Opus audio compression](https://opus-codec.org/), response times on good internet can be nearly instantaneous, closely matching the cadence of human speech. - -You can find the demo live [here](https://modal-labs--quillman-web.modal.run/). - -![Quillman](https://github.com/user-attachments/assets/afda5874-8509-4f56-9f25-d734b8f1c40a) - -Everything — from the React frontend to the model backend — is deployed serverlessly on Modal, allowing it to automatically scale and ensuring you only pay for the compute you use. - -This page provides a high-level walkthrough of the [GitHub repo](https://github.com/modal-labs/quillman). - -## Code overview - -Traditionally, building a bidirectional streaming web application as compute-heavy as QuiLLMan would take a lot of work, and it's especially difficult to make it robust and scale to handle many concurrent users. - -But with Modal, it’s as simple as writing two different classes and running a CLI command. - -Our project structure looks like this: - -1. [Moshi Websocket Server](https://modal.com/docs/examples/llm-voice-chat#moshi-websocket-server): loads an instance of the Moshi model and maintains a bidirectional websocket connection with the client. -2. [React Frontend](https://modal.com/docs/examples/llm-voice-chat#react-frontend): runs client-side interaction logic. - -Let’s go through each of these components in more detail. - -### FastAPI Server - -Both frontend and backend are served via a [FastAPI Server](https://fastapi.tiangolo.com/), which is a popular Python web framework for building REST APIs. - -On Modal, a function or class method can be exposed to web traffic by decorating it with [`@app.asgi_app()`](https://modal.com/docs/reference/modal.asgi_app#modalasgi_app) and returning a FastAPI app. You're then free to configure the FastAPI server however you like, including adding middleware, serving static files, and running websockets. - -### Moshi Websocket Server - -Traditionally, a speech-to-speech chat app requires three distinct modules: speech-to-text, text-to-text, and text-to-speech. Passing data between these modules introduces bottlenecks, and can limit the speed of the app and forces a turn-by-turn conversation which can feel unnatural. - -Kyutai Lab's [Moshi](https://github.com/kyutai-labs/moshi) bundles all modalities into one model, which decreases latency and makes for a much simpler app. - -Under the hood, Moshi uses the [Mimi](https://huggingface.co/kyutai/mimi) streaming encoder/decoder model to maintain an unbroken stream of audio in and out. The encoded audio is processed by a [speech-text foundation model](https://huggingface.co/kyutai/moshiko-pytorch-bf16), which uses an internal monologue to determine when and how to respond. - -Using a streaming model introduces a few challenges not normally seen in inference backends: - -1. The model is *stateful*, meaning it maintains context of the conversation so far. This means a model instance cannot be shared between user conversations, so we must run a unique GPU per user session, which is normally not an easy feat! -2. The model is *streaming*, so the interface around it is not as simple as a POST request. We must find a way to stream audio data in and out, and do it fast enough for seamless playback. - -We solve both of these in `src/moshi.py`, using a few Modal features. - -To solve statefulness, we just spin up a new GPU per concurrent user. -That's easy with Modal! - -```python notest -@app.cls( - image=image, - gpu="A10G", - scaledown_window=300, - ... -) -class Moshi: - # ... -``` - -With this setting, if a new user connects, a new GPU instance is created! When any user disconnects, the state of their model is reset and that GPU instance is returned to the warm pool for re-use (for up to 300 seconds). Be aware that a GPU per user is not going to be cheap, but it's the simplest way to ensure user sessions are isolated. - -For streaming, we use FastAPI's support for bidirectional websockets. This allows clients to establish a single connection at the start of their session, and stream audio data both ways. - -Just as a FastAPI server can run from a Modal function, it can also be attached to a Modal class method, allowing us to couple a prewarmed Moshi model to a websocket session. - -```python notest -@modal.asgi_app() -def web(self): - from fastapi import FastAPI, Response, WebSocket, WebSocketDisconnect - - web_app = FastAPI() - @web_app.websocket("/ws") - async def websocket(ws: WebSocket): - with torch.no_grad(): - await ws.accept() - - # handle user session - - # spawn loops for async IO - async def recv_loop(): - while True: - data = await ws.receive_bytes() - # send data into inference stream... - - async def send_loop(): - while True: - await asyncio.sleep(0.001) - msg = self.opus_stream_outbound.read_bytes() - # send inference output to user ... -``` - -To run a [development server](https://modal.com/docs/guide/webhooks#developing-with-modal-serve) for the Moshi module, run this command from the root of the repo. - -```shell -modal serve -m src.moshi -``` - -In the terminal output, you'll find a URL for creating a websocket connection. - -### React Frontend - -The frontend is a static React app, found in the `src/frontend` directory and served by `src/app.py`. - -We use the [Web Audio API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API) to record audio from the user's microphone and playback audio responses from the model. - -For efficient audio transmission, we use the [Opus codec](https://opus-codec.org/) to compress audio across the network. Opus recording and playback are supported by the [`opus-recorder`](https://github.com/chris-rudmin/opus-recorder) and [`ogg-opus-decoder`](https://github.com/eshaz/wasm-audio-decoders/tree/master/src/ogg-opus-decoder) libraries. - -To serve the frontend assets, run this command from the root of the repo. - -```shell -modal serve -m src.app -``` - -Since `src/app.py` imports the `src/moshi.py` module, this `serve` command also serves the Moshi websocket server as its own endpoint. - -## Deploy - -When you're ready to go live, use the `deploy` command to deploy the app to Modal. - -```shell -modal deploy -m src.app -``` - -## Steal this example - -The code for this entire example is [available on GitHub](https://github.com/modal-labs/quillman), so feel free to fork it and make it your own! diff --git a/.agents/skills/modal/references/examples/llm_inference.md b/.agents/skills/modal/references/examples/llm_inference.md deleted file mode 100644 index ebf1843bf3c0bea9e2850f0f99d8a53b9483c019..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/llm_inference.md +++ /dev/null @@ -1,346 +0,0 @@ -# Run OpenAI-compatible LLM inference with Gemma and vLLM - -In this example, we show how to run a vLLM server in OpenAI-compatible mode on Modal. - -LLMs do more than just model language: they chat, they produce JSON and XML, they run code, and more. -This has complicated their interface far beyond "text-in, text-out". -OpenAI's API has emerged as a standard for that interface, -and it is supported by open source LLM serving frameworks like [vLLM](https://docs.vllm.ai/en/latest/). - -This example is intended to demonstrate the basics of deploying LLM inference on Modal. -For more on how to optimize performance, see -[this guide](https://modal.com/docs/guide/high-performance-llm-inference) -and check out our -[LLM Engineer's Almanac](https://modal.com/llm-almanac). - -Our examples repository also includes scripts for running clients and load-testing for OpenAI-compatible APIs -[here](https://github.com/modal-labs/modal-examples/tree/main/06_gpu_and_ml/llm-serving/openai_compatible). - -## Set up the container image - -Our first order of business is to define the environment our server will run in: -the [container `Image`](https://modal.com/docs/guide/custom-container). -vLLM can be installed with `uv pip`, since Modal [provides the CUDA drivers](https://modal.com/docs/guide/cuda). - -```python -import json -from typing import Any - -import aiohttp -import modal - -vllm_image = ( - modal.Image.from_registry("nvidia/cuda:12.9.0-devel-ubuntu22.04", add_python="3.12") - .entrypoint([]) - .uv_pip_install("vllm==0.21.0") - .env( - { - "HF_XET_HIGH_PERFORMANCE": "1", # faster model transfers - "VLLM_LOG_STATS_INTERVAL": "1", # more frequent metrics logging - } - ) -) - -``` - -## Download the model weights - -We'll be running a pretrained foundation model -- -[Google's Gemma 4](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/). -It can also take images, video, and audio as inputs, -though we won't use that here. - -We'll use the 26BA4B variant, [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it). -This variant is trained with reasoning capabilities, which allow it to -enhance the quality of its generated responses. -It has `26B`illion parameters, of which `4B`illion are `A`ctive -in processing of each token. - -You can swap this model out for another by changing the strings below, -though you might also need to adjust some of the server configuration as well. -A single H200 GPU has enough VRAM to store this 26,000,000,000 parameter model -along with a large KV cache. - -```python -MODEL_NAME = "google/gemma-4-26B-A4B-it" -MODEL_REVISION = "47b6801b24d15ff9bcd8c96dfaea0be9ed3a0301" # avoid nasty surprises when repos update! - -``` - -Although vLLM will download weights from Hugging Face on-demand, -we want to cache them so we don't do it every time our server starts. -We'll use [Modal Volumes](https://modal.com/docs/guide/volumes) for our cache. -Modal Volumes are essentially a "shared disk" that all Modal Functions can access like it's a regular disk. -For more on storing model weights on Modal, see -[this guide](https://modal.com/docs/guide/model-weights). - -```python -hf_cache_vol = modal.Volume.from_name("huggingface-cache", create_if_missing=True) - -``` - -We'll also cache some of vLLM's JIT compilation artifacts in a Modal Volume. - -```python -vllm_cache_vol = modal.Volume.from_name("vllm-cache", create_if_missing=True) - -``` - -## Configuring vLLM - -### Trading off fast boots and token generation performance - -vLLM has embraced dynamic and just-in-time compilation to eke out additional performance without having to write too many custom kernels, -e.g. via the Torch compiler and CUDA graph capture. -These compilation features incur latency in exchange for lowered latency and higher throughput during generation. -This latency is typically tens of seconds to a few minutes, reduced to about ten seconds when loaded from the cache. -We make this trade-off controllable with the `FAST_BOOT` variable below. - -```python -FAST_BOOT = False - -``` - -If you're running an LLM service that frequently scales from 0 (frequent ["cold starts"](https://modal.com/docs/guide/cold-start)) -you might want to set this to `True`, or consider [GPU memory snapshots](https://modal.com/docs/guide/memory-snapshots). -It's also useful to set this when you're iterating on the server configuration. - -If you're running an LLM service that usually has multiple replicas running, then set this to `False` for improved performance. - -See the code below for details on the parameters that `FAST_BOOT` controls. - -### Model-specific configuration - -Almost all models require some amount of configuration via command-line flags, -especially to achieve optimal performance. - -We set these flags in the code below, roughly following the -[usage guide from the vLLM docs](https://docs.vllm.ai/projects/recipes/en/latest/Google/Gemma4.html). - -For instance, we turn off multimodal features to save on [GPU RAM](https://modal.com/gpu-glossary/device-hardware/gpu-ram), -and we activate the [built-in multi-token prediction (MTP)](https://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4/) -speculative decoding for improved throughput at lower concurrencies. - -```python -SPECULATIVE_MODEL_NAME = "google/gemma-4-26B-A4B-it-assistant" -SPECULATIVE_MODEL_REVISION = "f188f476dc11dd5bb3014dc861529d316bce49d3" - -``` - -For more on the performance you can expect when serving your own LLMs, see -[our LLM engine performance benchmarks](https://modal.com/llm-almanac). - -## Build a vLLM engine and serve it - -The function below spawns a vLLM instance listening at port 8000, serving requests to our model. -We wrap it in the [`@modal.web_server` decorator](https://modal.com/docs/guide/webhooks#non-asgi-web-servers) -to connect it to the Internet. - -The server runs in an independent process, via `subprocess.Popen`, and only starts accepting requests -once the model is spun up and the `serve` function returns. - -```python -app = modal.App("example-vllm-inference") - -N_GPU = 1 -MINUTES = 60 # seconds -VLLM_PORT = 8000 - - -@app.function( - image=vllm_image, - gpu=f"H200:{N_GPU}", - scaledown_window=15 * MINUTES, # how long should we stay up with no requests? - timeout=10 * MINUTES, # how long should we wait for container start? - volumes={ - "/root/.cache/huggingface": hf_cache_vol, - "/root/.cache/vllm": vllm_cache_vol, - }, -) -@modal.concurrent( # how many requests can one replica handle? tune carefully! - max_inputs=100, -) -@modal.web_server(port=VLLM_PORT, startup_timeout=10 * MINUTES) -def serve(): - import json - import subprocess - - cmd = [ - "vllm", - "serve", - MODEL_NAME, - "--revision", - MODEL_REVISION, - "--served-model-name", - MODEL_NAME, - "llm", - "--host", - "0.0.0.0", - "--port", - str(VLLM_PORT), - "--uvicorn-log-level=info", - "--async-scheduling", - ] - - # enforce-eager disables both Torch compilation and CUDA graph capture - # default is no-enforce-eager. see the --compilation-config flag for tighter control - cmd += ["--enforce-eager" if FAST_BOOT else "--no-enforce-eager"] - - # assume multiple GPUs are for splitting up large matrix multiplications - cmd += ["--tensor-parallel-size", str(N_GPU)] - - # add model-specific configuration - cmd += [ - # skip multimedia support, just language - "--limit-mm-per-prompt", - f"'{json.dumps({'image': 0, 'video': 0, 'audio': 0})}'", - # enable reasoning and tool use - "--enable-auto-tool-choice", - "--reasoning-parser gemma4", - "--tool-call-parser gemma4", - ] - - # add speculative decoding - cmd += [ - "--speculative-config", - f"'{json.dumps({'model': SPECULATIVE_MODEL_NAME, 'revision': SPECULATIVE_MODEL_REVISION, 'num_speculative_tokens': 4})}'", - ] - - print(*cmd) - - subprocess.Popen(" ".join(cmd), shell=True) - - -``` - -## Deploy the server - -To deploy the API on Modal, just run - -```bash -modal deploy vllm_inference.py -``` - -This will create a new app on Modal, build the container image for it if it hasn't been built yet, -and deploy the app. - -## Interact with the server - -Once it is deployed, you'll see a URL appear in the command line, -something like `https://your-workspace-name--example-vllm-inference-serve.modal.run`. - -You can find [interactive Swagger UI docs](https://swagger.io/tools/swagger-ui/) -at the `/docs` route of that URL, i.e. `https://your-workspace-name--example-vllm-inference-serve.modal.run/docs`. -These docs describe each route and indicate the expected input and output -and translate requests into `curl` commands. - -For simple routes like `/health`, which checks whether the server is responding, -you can even send a request directly from the docs. - -To interact with the API programmatically in Python, we recommend the `openai` library. - -See the `client.py` script in the examples repository -[here](https://github.com/modal-labs/modal-examples/tree/main/06_gpu_and_ml/llm-serving/openai_compatible) -to take it for a spin: - -```bash -# pip install openai==1.76.0 -python openai_compatible/client.py -``` - -## Testing the server - -To make it easier to test the server setup, we also include a `local_entrypoint` -that does a healthcheck and then hits the server. - -If you execute the command - -```bash -modal run vllm_inference.py -``` - -a fresh replica of the server will be spun up on Modal while -the code below executes on your local machine. - -Think of this like writing simple tests inside of the `if __name__ == "__main__"` -block of a Python script, but for cloud deployments! - -```python -@app.local_entrypoint() -async def test(test_timeout=15 * MINUTES, content=None, twice=True): - url = await serve.get_web_url.aio() - - system_prompt = { - "role": "system", - "content": "You are a pirate who can't help but drop sly reminders that he went to Harvard.", - } - if content is None: - content = "Explain the singular value decomposition." - - messages = [ # OpenAI chat format - system_prompt, - {"role": "user", "content": content}, - ] - - async with aiohttp.ClientSession(base_url=url) as session: - print(f"Running health check for server at {url}") - async with session.get("/health", timeout=test_timeout - 1 * MINUTES) as resp: - up = resp.status == 200 - assert up, f"Failed health check for server at {url}" - print(f"Successful health check for server at {url}") - - print(f"Sending messages to {url}:", *messages, sep="\n\t") - await _send_request(session, "llm", messages) - if twice: - messages[0]["content"] = "You are Jar Jar Binks." - print(f"Sending messages to {url}:", *messages, sep="\n\t") - await _send_request(session, "llm", messages) - - -async def _send_request( - session: aiohttp.ClientSession, model: str, messages: list -) -> None: - # `stream=True` tells an OpenAI-compatible backend to stream chunks - payload: dict[str, Any] = {"messages": messages, "model": model, "stream": True} - # explicitly enable thinking for this model - payload["chat_template_kwargs"] = {"enable_thinking": True} - - headers = {"Content-Type": "application/json", "Accept": "text/event-stream"} - - async with session.post( - "/v1/chat/completions", json=payload, headers=headers - ) as resp: - async for raw in resp.content: - resp.raise_for_status() - # extract new content and stream it - line = raw.decode().strip() - if not line or line == "data: [DONE]": - continue - if line.startswith("data: "): # SSE prefix - line = line[len("data: ") :] - - chunk = json.loads(line) - assert ( - chunk["object"] == "chat.completion.chunk" - ) # or something went horribly wrong - delta = chunk["choices"][0]["delta"] - content = ( - delta.get("content") - or delta.get("reasoning") - or delta.get("reasoning_content") - ) - if content: - print(content, end="") - else: - print("\n", chunk) - print() - - -``` - -We also include a basic example of a load-testing setup using -`locust` in the `load_test.py` script [here](https://github.com/modal-labs/modal-examples/tree/main/06_gpu_and_ml/llm-serving/openai_compatible): - -```bash -modal run openai_compatible/load_test.py -``` diff --git a/.agents/skills/modal/references/examples/long-training.md b/.agents/skills/modal/references/examples/long-training.md deleted file mode 100644 index 03ee4f6fea199537d7bf0c00985bd08666c9e62f..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/long-training.md +++ /dev/null @@ -1,274 +0,0 @@ -# Run long, resumable training jobs on Modal - -Individual Modal Function calls have a [maximum timeout of 24 hours](https://modal.com/docs/guide/timeouts). -You can still run long training jobs on Modal by making them interruptible and resumable -(aka [*reentrant*](https://en.wikipedia.org/wiki/Reentrancy_%28computing%29)). - -This is usually done via checkpointing: saving the model state to disk at regular intervals. -We recommend implementing checkpointing logic regardless of the duration of your training jobs. -This prevents loss of progress in case of interruptions or [preemptions](https://modal.com/docs/guide/preemption). - -In this example, we'll walk through how to implement this pattern in -[PyTorch Lightning](https://lightning.ai/docs/pytorch/2.4.0/). - -But the fundamental pattern is simple and can be applied to any training framework: - -1. Periodically save checkpoints to a Modal [Volume](https://modal.com/docs/guide/volumes) -2. When your training function starts, check the Volume for the latest checkpoint -3. Add [retries](https://modal.com/docs/guide/retries) to your training function - -## Resuming from checkpoints in a training loop - -The `train` function below shows some very simple training logic -using the built-in checkpointing features of PyTorch Lightning. - -Lightning uses a special filename, `last.ckpt`, -to indicate which checkpoint is the most recent. -We check for this file and resume training from it if it exists. - -```python -from pathlib import Path -from typing import Optional - -import modal - - -def train(experiment): - experiment_dir = CHECKPOINTS_PATH / experiment - last_checkpoint = experiment_dir / "last.ckpt" - - if last_checkpoint.exists(): - print(f"⚡️ resuming training from the latest checkpoint: {last_checkpoint}") - train_model( - DATA_PATH, - experiment_dir, - resume_from_checkpoint=last_checkpoint, - ) - print("⚡️ training finished successfully") - else: - print("⚡️ starting training from scratch") - train_model(DATA_PATH, experiment_dir) - - -``` - -This implementation works fine in a local environment. -Running it serverlessly and durably on Modal -- with access to auto-scaling cloud GPU infrastructure -\-- does not require any adjustments to the code. -We just need to ensure that data and checkpoints are saved in Modal *Volumes*. - -## Modal Volumes are distributed file systems - -Modal [Volumes](https://modal.com/docs/guide/volumes) are distributed file systems -- -you can read and write files from them just like local disks, -but they are accessible to all of your Modal Functions. -Their performance is tuned for [Write-Once, Read-Many](https://en.wikipedia.org/wiki/Write_once_read_many) workloads -with small numbers of large files. - -You can attach them to any Modal Function that needs access. - -But first, you need to create them: - -```python -volume = modal.Volume.from_name("example-long-training", create_if_missing=True) - -``` - -## Porting training to Modal - -To attach a Modal Volume to our training function, we need to port it over to run on Modal. - -That means we need to define our training function's dependencies -(as a [container image](https://modal.com/docs/guide/custom-container)) -and attach it to an application (a [`modal.App`](https://modal.com/docs/guide/apps)). - -Modal Functions that run on GPUs [already have CUDA drivers installed](https://modal.com/docs/guide/cuda), -so dependency specification is straightforward. -We just `uv_pip_install` PyTorch and PyTorch Lightning. - -```python -image = modal.Image.debian_slim(python_version="3.12").uv_pip_install( - "lightning~=2.4.0", "torch~=2.4.0", "torchvision==0.19.0" -) - -app = modal.App("example-long-training", image=image) - -``` - -Next, we attach our training function to this app with `app.function`. - -We define all of the serverless infrastructure-specific details of our training at this point. -For resumable training, there are three key pieces: attaching volumes, adding retries, and setting the timeout. - -We want to attach the Volume to our Function so that the data and checkpoints are saved into it. -In this sample code, we set these paths via global variables, but in another setting, -these might be set via environment variables or other configuration mechanisms. - -```python -volume_path = Path("/experiments") -DATA_PATH = volume_path / "data" -CHECKPOINTS_PATH = volume_path / "checkpoints" - -volumes = {volume_path: volume} - -``` - -Then, we define how we want to restart our training in case of interruption. -We can use `modal.Retries` to add automatic retries to our Function. -We set the delay time to `0.0` seconds, because on pre-emption or timeout we want to restart immediately. -We set `max_retries` to the current maximum, which is `10`. - -```python -retries = modal.Retries(initial_delay=0.0, max_retries=10) - -``` - -Timeouts on Modal are set in seconds, with a minimum of 10 seconds and a maximum of 24 hours. -When running training jobs that last up to week, we'd set that timeout to 24 hours, -which would give our training job a maximum of 10 days to complete before we'd need to manually restart. - -For this example, we'll set it to 30 seconds. When running the example, you should observe a few interruptions. - -```python -timeout = 30 # seconds - -``` - -Now, we put all of this together by wrapping `train` and decorating it -with `app.function` to add all the infrastructure. We add the `single_use_containers` flag to ensure that our retries -will always kickoff in a fresh container. - -```python -@app.function( - volumes=volumes, - gpu="a10g", - timeout=timeout, - retries=retries, - single_use_containers=True, -) -def train_interruptible(*args, **kwargs): - train(*args, **kwargs) - - -``` - -## Kicking off interruptible training - -We define a [`local_entrypoint`](https://modal.com/docs/guide/apps#entrypoints-for-ephemeral-apps) -to kick off the training job from the local Python environment. - -```python -@app.local_entrypoint() -def main(experiment: Optional[str] = None): - if experiment is None: - from uuid import uuid4 - - experiment = uuid4().hex[:8] - print(f"⚡️ starting interruptible training experiment {experiment}") - train_interruptible.spawn(experiment).get() - - -``` - -It's important to use `.spawn(...).get()` because `.remote` created Function Calls -expire after 24 hours. - -You can run this with - -```bash -modal run --detach 06_gpu_and_ml/long-training.py -``` - -You should see the training job start and then be interrupted, -producing a large stack trace in the terminal in red font. -The job will restart within a few seconds. - -The `--detach` flag ensures training will continue even if you close your terminal or turn off your computer. -Try detaching and then watch the logs in the [Modal dashboard](https://modal.com/apps). - -## Details of PyTorch Lightning implementation - -This basic pattern works for any training framework or for custom training jobs -- -or for any reentrant work that can save state to disk. - -But to make the example complete, we include all the details of the PyTorch Lightning implementation below. - -PyTorch Lightning offers [built-in checkpointing](https://pytorch-lightning.readthedocs.io/en/1.2.10/common/weights_loading.html). -You can specify the checkpoint file path that you want to resume from using the `ckpt_path` parameter of -[`trainer.fit`](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.trainer.trainer.Trainer.html) -Additionally, you can specify the checkpointing interval with the `every_n_epochs` parameter of -[`ModelCheckpoint`](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.ModelCheckpoint.html). - -```python -def get_checkpoint(checkpoint_dir): - from lightning.pytorch.callbacks import ModelCheckpoint - - return ModelCheckpoint( - dirpath=checkpoint_dir, - save_last=True, - every_n_epochs=10, - filename="{epoch:02d}", - ) - - -def train_model(data_dir, checkpoint_dir, resume_from_checkpoint=None): - import lightning as L - - autoencoder = get_autoencoder() - train_loader = get_train_loader(data_dir=data_dir) - checkpoint_callback = get_checkpoint(checkpoint_dir) - - trainer = L.Trainer( - limit_train_batches=100, max_epochs=100, callbacks=[checkpoint_callback] - ) - if resume_from_checkpoint is not None: - trainer.fit( - model=autoencoder, - train_dataloaders=train_loader, - ckpt_path=resume_from_checkpoint, - ) - else: - trainer.fit(autoencoder, train_loader) - - -def get_autoencoder(checkpoint_path=None): - import lightning as L - from torch import nn, optim - - class LitAutoEncoder(L.LightningModule): - def __init__(self): - super().__init__() - self.encoder = nn.Sequential( - nn.Linear(28 * 28, 64), nn.ReLU(), nn.Linear(64, 3) - ) - self.decoder = nn.Sequential( - nn.Linear(3, 64), nn.ReLU(), nn.Linear(64, 28 * 28) - ) - - def training_step(self, batch, batch_idx): - x, _ = batch - x = x.view(x.size(0), -1) - z = self.encoder(x) - x_hat = self.decoder(z) - loss = nn.functional.mse_loss(x_hat, x) - self.log("train_loss", loss) - return loss - - def configure_optimizers(self): - optimizer = optim.Adam(self.parameters(), lr=1e-3) - return optimizer - - return LitAutoEncoder() - - -def get_train_loader(data_dir): - from torch import utils - from torchvision.datasets import MNIST - from torchvision.transforms import ToTensor - - print("⚡ setting up data") - dataset = MNIST(data_dir, download=True, transform=ToTensor()) - train_loader = utils.data.DataLoader(dataset, num_workers=4) - return train_loader - -``` diff --git a/.agents/skills/modal/references/examples/ltx.md b/.agents/skills/modal/references/examples/ltx.md deleted file mode 100644 index 50a5e8129b0d44fb53279543346905f27c3d1378..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/ltx.md +++ /dev/null @@ -1,276 +0,0 @@ -# Generate videos from prompts with Lightricks LTX-Video - -This example demonstrates how to run the [LTX-Video](https://github.com/Lightricks/LTX-Video) -video generation model by [Lightricks](https://www.lightricks.com/) on Modal. - -LTX-Video is fast! Generating a twenty second 480p video at moderate quality -takes as little as two seconds on a warm container. - -Here's one that we generated: - -
- -
- -## Setup - -We start by importing dependencies we need locally, -defining a Modal [App](https://modal.com/docs/guide/apps), -and defining the container [Image](https://modal.com/docs/guide/images) -that our video model will run in. - -```python -import string -import time -from pathlib import Path -from typing import Optional - -import modal - -app = modal.App("example-ltx") - -image = ( - modal.Image.debian_slim(python_version="3.12") - .uv_pip_install( - "accelerate==1.6.0", - "diffusers==0.33.1", - "huggingface-hub==0.36.0", - "imageio==2.37.0", - "imageio-ffmpeg==0.5.1", - "sentencepiece==0.2.0", - "torch==2.7.0", - "transformers==4.51.3", - ) - .env({"HF_XET_HIGH_PERFORMANCE": "1"}) -) - -``` - -## Storing data on Modal Volumes - -On Modal, we save large or expensive-to-compute data to -[distributed Volumes](https://modal.com/docs/guide/volumes) -that are accessible both locally and remotely. - -We'll store the LTX-Video model's weights and the outputs we generate -on Modal Volumes. - -We store the outputs on a Modal Volume so that clients -don't need to sit around waiting for the video to be generated. - -```python -VOLUME_NAME = "ltx-outputs" -outputs = modal.Volume.from_name(VOLUME_NAME, create_if_missing=True) -OUTPUTS_PATH = Path("/outputs") - -``` - -We store the weights on a Modal Volume so that we don't -have to fetch them from the Hugging Face Hub every time -a container boots. This download takes about two minutes, -depending on traffic and network speed. - -```python -MODEL_VOLUME_NAME = "ltx-model" -model = modal.Volume.from_name(MODEL_VOLUME_NAME, create_if_missing=True) - -``` - -We don't have to change any of the Hugging Face code to do this -- -we just set the location of Hugging Face's cache to be on a Volume -using the `HF_HOME` environment variable. - -```python -MODEL_PATH = Path("/models") -image = image.env({"HF_HOME": str(MODEL_PATH)}) - -``` - -For more on storing model weights on Modal, see -[this guide](https://modal.com/docs/guide/model-weights). - -## Setting up our LTX class - -We use the `@cls` decorator to specify the infrastructure our inference function needs, -as defined above. - -That decorator also gives us control over the -[lifecycle](https://modal.com/docs/guide/lifecycle-functions) -of our cloud container. - -Specifically, we use the `enter` method to load the model into GPU memory -(from the Volume if it's present or the Hub if it's not) -before the container is marked ready for inputs. - -This helps reduce tail latencies caused by cold starts. -For details and more tips, see [this guide](https://modal.com/docs/guide/cold-start#cold-start-performance). - -The actual inference code is in a `modal.method` of the class. - -```python -MINUTES = 60 # seconds - - -@app.cls( - image=image, # use our container Image - volumes={OUTPUTS_PATH: outputs, MODEL_PATH: model}, # attach our Volumes - gpu="H100", # use a big, fast GPU - timeout=10 * MINUTES, # run inference for up to 10 minutes - scaledown_window=15 * MINUTES, # stay idle for 15 minutes before scaling down -) -class LTX: - @modal.enter() - def load_model(self): - import torch - from diffusers import DiffusionPipeline - - self.pipe = DiffusionPipeline.from_pretrained( - "Lightricks/LTX-Video", torch_dtype=torch.bfloat16 - ) - self.pipe.to("cuda") - - @modal.method() - def generate( - self, - prompt, - negative_prompt="", - num_inference_steps=200, - guidance_scale=4.5, - num_frames=19, - width=704, - height=480, - ): - from diffusers.utils import export_to_video - - frames = self.pipe( - prompt=prompt, - negative_prompt=negative_prompt, - num_inference_steps=num_inference_steps, - guidance_scale=guidance_scale, - num_frames=num_frames, - width=width, - height=height, - ).frames[0] - - # save to disk using prompt as filename - mp4_name = slugify(prompt) - export_to_video(frames, Path(OUTPUTS_PATH) / mp4_name) - outputs.commit() - return mp4_name - - -``` - -## Generate videos from the command line - -We trigger LTX-Video inference from our local machine by running the code in -the local entrypoint below with `modal run`. - -It will spin up a new replica to generate a video. -Then it will, by default, generate a second video to demonstrate -the lower latency when hitting a warm container. - -You can trigger inference with: - -```bash -modal run ltx -``` - -All outputs are saved both locally and on a Modal Volume. -You can explore the contents of Modal Volumes from your Modal Dashboard -or from the command line with the `modal volume` command. - -```bash -modal volume ls ltx-outputs -``` - -See `modal volume --help` for details. - -Optional command line flags for the script can be viewed with: - -```bash -modal run ltx --help -``` - -Using these flags, you can tweak your generation from the command line: - -```bash -modal run --detach ltx --prompt="a cat playing drums in a jazz ensemble" --num-inference-steps=64 -``` - -```python -@app.local_entrypoint() -def main( - prompt: Optional[str] = None, - negative_prompt="worst quality, blurry, jittery, distorted", - num_inference_steps: int = 10, # 10 when testing, 100 or more when generating - guidance_scale: float = 2.5, - num_frames: int = 150, # produces ~10s of video - width: int = 704, - height: int = 480, - twice: bool = True, # run twice to show cold start latency -): - if prompt is None: - prompt = DEFAULT_PROMPT - - ltx = LTX() - - def run(): - print(f"🎥 Generating a video from the prompt '{prompt}'") - start = time.time() - mp4_name = ltx.generate.remote( - prompt=prompt, - negative_prompt=negative_prompt, - num_inference_steps=num_inference_steps, - guidance_scale=guidance_scale, - num_frames=num_frames, - width=width, - height=height, - ) - duration = time.time() - start - print(f"🎥 Client received video in {int(duration)}s") - print(f"🎥 LTX video saved to Modal Volume at {mp4_name}") - - local_dir = Path("/tmp/ltx") - local_dir.mkdir(exist_ok=True, parents=True) - local_path = local_dir / mp4_name - local_path.write_bytes(b"".join(outputs.read_file(mp4_name))) - print(f"🎥 LTX video saved locally at {local_path}") - - run() - - if twice: - print("🎥 Generating a video from a warm container") - run() - - -``` - -## Addenda - -The remainder of the code in this file is utility code. - -```python -DEFAULT_PROMPT = ( - "The camera pans over a snow-covered mountain range," - " revealing a vast expanse of snow-capped peaks and valleys." - " The mountains are covered in a thick layer of snow," - " with some areas appearing almost white while others have a slightly darker, almost grayish hue." - " The peaks are jagged and irregular, with some rising sharply into the sky" - " while others are more rounded." - " The valleys are deep and narrow, with steep slopes that are also covered in snow." - " The trees in the foreground are mostly bare, with only a few leaves remaining on their branches." -) - - -def slugify(prompt): - for char in string.punctuation: - prompt = prompt.replace(char, "") - prompt = prompt.replace(" ", "_") - prompt = prompt[:230] # some OSes limit filenames to <256 chars - mp4_name = str(int(time.time())) + "_" + prompt + ".mp4" - return mp4_name - -``` diff --git a/.agents/skills/modal/references/examples/mcp_server_stateless.md b/.agents/skills/modal/references/examples/mcp_server_stateless.md deleted file mode 100644 index 4117c6120c21d4b165f829d0ea958ea9d4c3a2a6..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/mcp_server_stateless.md +++ /dev/null @@ -1,165 +0,0 @@ -# Deploy a remote, stateless MCP server on Modal with FastMCP - -This example demonstrates how to deploy a simple -[MCP server](https://modelcontextprotocol.io/) -on Modal. - -The server provides a tool to get the current date and time in a given timezone. -It is a stateless MCP server, meaning that it does not store any state between requests, -which is important for mapping onto Modal's serverless Functions. -It uses the "streamable HTTP" transport type. - -## Building the MCP server - -First, we define our dependencies. - -We use the [FastMCP library](https://github.com/jlowin/fastmcp) to create the MCP -server. We wrap with a FastAPI server to expose it to the Internet. - -```python -import modal - -app = modal.App("example-mcp-server-stateless") - -image = modal.Image.debian_slim(python_version="3.12").uv_pip_install( - "fastapi==0.115.14", - "fastmcp==2.10.6", - "pydantic==2.11.10", -) - - -``` - -Next, we create the MCP server itself using FastMCP and add a tool to it that -allows LLMs to get the current date and time in a given timezone. - -```python -def make_mcp_server(): - from fastmcp import FastMCP - - mcp = FastMCP("Date and Time MCP Server") - - @mcp.tool() - async def current_date_and_time(timezone: str = "UTC") -> str: - """Get the current date and time. - - Args: - timezone: The timezone to get the date and time in (optional). Defaults to UTC. - - Returns: - The current date and time in the given timezone, in ISO 8601 format. - """ - from datetime import datetime - from zoneinfo import ZoneInfo - - try: - tz = ZoneInfo(timezone) - except Exception: - raise ValueError( - f"Invalid timezone '{timezone}'. Please use a valid timezone like 'UTC', " - "'America/New_York', or 'Europe/Stockholm'." - ) - return datetime.now(tz).isoformat() - - return mcp - - -``` - -We then use FastMCP to create a Starlette app with `streamable-http` as transport -type, and set `stateless_http=True` to make it stateless. - -This will be mounted by the FastAPI app, which we deploy as a -[Modal Web Function](https://modal.com/docs/guide/webhooks) -using [the `asgi_app` decorator](https://modal.com/docs/reference/modal.asgi_app): - -```python -@app.function(image=image) -@modal.asgi_app() -def web(): - """Web gateway for the MCP server""" - from fastapi import FastAPI - - mcp = make_mcp_server() - mcp_app = mcp.http_app(transport="streamable-http", stateless_http=True) - - fastapi_app = FastAPI(lifespan=mcp_app.router.lifespan_context) - fastapi_app.mount("/", mcp_app, "mcp") - - return fastapi_app - - -``` - -And we're done! - -## Testing the MCP server - -Now you can [serve](https://modal.com/docs/reference/cli/serve#modal-serve) the MCP -server by running: - -```bash -modal serve mcp_server_stateless.py -``` - -Then open the [MCP inspector](https://github.com/modelcontextprotocol/inspector): - -```bash -npx @modelcontextprotocol/inspector -``` - -Enter the URL of the MCP server that was printed by the `modal serve` command above, -suffixed with `/mcp/` (so for example -`https://modal-labs-examples--datetime-mcp-server-web-dev.modal.run/mcp/`). Also -make sure to select "Streamable HTTP" as the "Transport Type". - -After connecting and clicking "List Tools" in the "Tools" tab you should see your -`current_date_and_time` tool listed, and if you "Run Tool" it should give you the -current date and time in UTC! - -To automatically test the MCP server, we spin up a client and have it list the tools. - -```python -@app.function(image=image) -async def test_tool(tool_name: str | None = None): - from fastmcp import Client - from fastmcp.client.transports import StreamableHttpTransport - - if tool_name is None: - tool_name = "current_date_and_time" - - transport = StreamableHttpTransport(url=f"{web.get_web_url()}/mcp/") - client = Client(transport) - - async with client: - tools = await client.list_tools() - - for tool in tools: - print(tool) - if tool.name == tool_name: - result = await client.call_tool(tool_name) - print(result.data) - return - - raise Exception(f"could not find tool {tool_name}") - - -``` - -This test is executed by running the script with `modal run`: - -```bash -modal run mcp_server_stateless::test_tool -``` - -## Deploying the MCP server - -`modal serve` creates an ephemeral, hot-reloading server, -which is useful for testing and development. - -When it's time to move to production, -you can deploy the server with - -```bash -modal deploy mcp_server_stateless -``` diff --git a/.agents/skills/modal/references/examples/ministral3_inference.md b/.agents/skills/modal/references/examples/ministral3_inference.md deleted file mode 100644 index 246e3116478984092798d3c61e689a4f1c9be50d..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/ministral3_inference.md +++ /dev/null @@ -1,488 +0,0 @@ -# Serverless Ministral 3 with vLLM and Modal - -In this example, we show how to serve Mistral's Ministral 3 vision-language models on Modal. - -The [Ministral 3](https://huggingface.co/collections/mistralai/ministral-3-more) model series -performs competitively with the Qwen 3-VL model series on benchmarks -(see model cards for details). - -We also include instructions for cutting cold start times -for long-running deployments by an order of magnitude using Modal's -[CPU + GPU memory snapshots](https://modal.com/docs/guide/memory-snapshot). - -## Set up the container image - -Our first order of business is to define the environment our server will run in: -the [container `Image`](https://modal.com/docs/guide/custom-container). -We'll use the [vLLM inference server](https://docs.vllm.ai). -vLLM can be installed with `uv pip`, since Modal [provides the CUDA drivers](https://modal.com/docs/guide/cuda). - -```python -import json -import socket -import subprocess -from typing import Any - -import aiohttp -import modal - -MINUTES = 60 # seconds -VLLM_PORT = 8000 - -app = modal.App("example-ministral3-inference") - -vllm_image = ( - modal.Image.from_registry("nvidia/cuda:12.9.0-devel-ubuntu22.04", add_python="3.12") - .entrypoint([]) - .uv_pip_install( - "vllm==0.13.0", - "huggingface-hub==0.36.0", - "flashinfer-python==0.5.3", - ) -) - -``` - -## Download the Ministral weights - -We also need to download the model weights. -We'll retrieve them from the Hugging Face Hub. - -To speed up the model load, we'll toggle the `HIGH_PERFORMANCE` -flag for Hugging Face's [Xet backend](https://huggingface.co/docs/hub/en/xet/index). - -```python -vllm_image = vllm_image.env({"HF_XET_HIGH_PERFORMANCE": "1"}) - -``` - -The [Ministral 3 model series](https://huggingface.co/collections/mistralai/ministral-3-more) -contains a variety of models: - -* 3B, 8B, and 14B sizes -* base models and instruction & reasoning fine-tuned models -* BF16 and FP8 quantizations - -All are available under the Apache 2.0 open source license. - -We'll use the FP8 instruct variant of the 8B model: - -```python -MODEL_NAME = "mistralai/Ministral-3-8B-Instruct-2512" - -``` - -Native hardware support for FP8 formats in [Tensor Cores](https://modal.com/gpu-glossary/device-hardware/tensor-core) -is limited to the latest [Streaming Multiprocessor architectures](https://modal.com/gpu-glossary/device-hardware/streaming-multiprocessor-architecture), -like those of Modal's [Hopper H100/H200 and Blackwell B200 GPUs](https://modal.com/blog/introducing-b200-h200). - -At 80 GB VRAM, a single H100 GPU has enough space to store the 8B FP8 model weights (~8 GB) -and a very large KV cache. A single H100 is also enough to serve the 14B model in full precision, -but without as much room for KV (though still enough to handle the full sequence length). - -```python -N_GPU = 1 - -``` - -### Cache with Modal Volumes - -Modal Functions are serverless: when they aren't being used, -their underlying containers spin down and all ephemeral resources, -like GPUs, memory, network connections, and local disks are released. - -We can preserve saved files by mounting a -[Modal Volume](https://modal.com/docs/guide/volumes) -- -a persistent, remote filesystem. - -We'll use two Volumes: one for weights from Hugging Face -and one for compilation artifacts from vLLM. - -```python -hf_cache_vol = modal.Volume.from_name("huggingface-cache", create_if_missing=True) -vllm_cache_vol = modal.Volume.from_name("vllm-cache", create_if_missing=True) - -``` - -## Serve Ministral 3 with vLLM - -We serve Ministral 3 on Modal by spinning up a Modal Function -that acts as a [`web_server`](https://modal.com/docs/guide/webhooks) -and spins up a vLLM server in a subprocess -(via the `vllm serve` command). - -### Improve cold start time with snapshots - -Starting up a vLLM server can be slow -- -tens of seconds to minutes. Much of that time -is spent on JIT compilation of inference code. - -We can skip most of that work and reduce startup times by a factor of 10 -using Modal's [memory snapshots](https://modal.com/docs/guide/memory-snapshot), -which serialize the contents of CPU and GPU memory. - -This adds quite some complexity to the code. -If you're looking for a minimal example, see -our [`vllm_inference` example here](https://modal.com/docs/examples/vllm_inference). - -We'll need to set a few extra configuration values: - -```python -vllm_image = vllm_image.env( - { - "VLLM_SERVER_DEV_MODE": "1", # allow use of "Sleep Mode" - "TORCHINDUCTOR_COMPILE_THREADS": "1", # improve compatibility with snapshots - } -) - -``` - -Setting the `DEV_MODE` flag allows us to use the `sleep`/`wake_up` endpoints -to toggle the server in and out of "sleep mode". - -```python -with vllm_image.imports(): - import requests - - -def sleep(level=1): - requests.post( - f"http://localhost:{VLLM_PORT}/sleep?level={level}" - ).raise_for_status() - - -def wake_up(): - requests.post(f"http://localhost:{VLLM_PORT}/wake_up").raise_for_status() - - -``` - -Sleep Mode helps with memory snapshotting. -When the server is asleep, model weights are offloaded to CPU memory and the KV cache is emptied. -For details, see the [vLLM docs](https://docs.vllm.ai/en/stable/features/sleep_mode/). - -We'll also need two helper functions. -Ther first, `wait_ready`, busy-polls the server until it is live. - -```python -def wait_ready(proc: subprocess.Popen): - while True: - try: - socket.create_connection(("localhost", VLLM_PORT), timeout=1).close() - return - except OSError: - if proc.poll() is not None: - raise RuntimeError(f"vLLM exited with {proc.returncode}") - - -``` - -Once the server is live, we `warmup` inference with a few requests. -This is important for capturing non-serializable JIT compilation artifacts, -like CUDA graphs and some Torch compilation outputs, -in our snapshot. - -```python -def warmup(): - payload = { - "model": "llm", - "messages": [{"role": "user", "content": "Who are you?"}], - "max_tokens": 16, - } - - for ii in range(3): - requests.post( - f"http://localhost:{VLLM_PORT}/v1/chat/completions", - json=payload, - timeout=300, - ).raise_for_status() - - -``` - -### Define the server - -We construct our web-serving Modal Function -by decorating a regular Python class. -The decorators include a number of configuration -options for deployment, including resources like GPUs and Volumes -and timeouts on container scaledown. -You can read more about the options -[here](https://modal.com/docs/reference/modal.App#function). - -We control memory snapshotting and container start behavior -by decorating the methods of the class. - -We start the server, warm it up, and then put it to sleep -in the `start` method. This method has the `modal.enter` -decorator to ensure it runs when a new container starts -and we pass `snap=True` to turn on memory snapshotting. - -The following method, `wake_up`, calls the `wake_up` -endpoint and then waits for the server to be ready. -It is run after the `start` method because it is defined later -in the code and also has the `modal.enter` decorator. -It has `snap=False` so that it isn't included in the snapshot. - -Finally, we connect the vLLM server to the Internet -using the [`modal.web_server`](https://modal.com/docs/guide/webhooks#non-asgi-web-servers) decorator. - -```python -@app.cls( - image=vllm_image, - gpu=f"H100:{N_GPU}", - scaledown_window=15 * MINUTES, # how long should we stay up with no requests? - timeout=10 * MINUTES, # how long should we wait for container start? - volumes={ - "/root/.cache/huggingface": hf_cache_vol, - "/root/.cache/vllm": vllm_cache_vol, - }, - enable_memory_snapshot=True, - experimental_options={"enable_gpu_snapshot": True}, -) -@modal.concurrent( # how many requests can one replica handle? tune carefully! - max_inputs=32 -) -class VllmServer: - @modal.enter(snap=True) - def start(self): - cmd = [ - "vllm", - "serve", - "--uvicorn-log-level=info", - MODEL_NAME, - "--served-model-name", - MODEL_NAME, - "llm", - "--host", - "0.0.0.0", - "--port", - str(VLLM_PORT), - "--gpu_memory_utilization", - str(0.95), - ] - - # assume multiple GPUs are for splitting up large matrix multiplications - cmd += ["--tensor-parallel-size", str(N_GPU)] - - # add mistral config arguments - cmd += [ - "--tokenizer_mode", - "mistral", - "--config_format", - "mistral", - "--load_format", - "mistral", - "--tool-call-parser", - "mistral", - "--enable-auto-tool-choice", - ] - - # add config arguments for snapshotting - - cmd += [ - "--enable-sleep-mode", - # make KV cache predictable / small - "--max-num-seqs", - "2", - "--max-model-len", - "12288", - "--max-num-batched-tokens", - "12288", - ] - - print(*cmd) - - self.vllm_proc = subprocess.Popen(cmd) - - wait_ready(self.vllm_proc) - - warmup() - - sleep() - - @modal.enter(snap=False) - def wake_up(self): - wake_up() - wait_ready(self.vllm_proc) - - @modal.web_server(port=VLLM_PORT, startup_timeout=10 * MINUTES) - def serve(self): - pass - - @modal.exit() - def stop(self): - self.vllm_proc.terminate() - - -``` - -## Deploy the server - -To deploy the API on Modal, just run - -```bash -modal deploy ministral3_inference.py -``` - -This will create a new app on Modal, build the container image for it if it hasn't been built yet, -and deploy the app. - -## Interact with the server - -Once it is deployed, you'll see a URL appear in the command line, -something like `https://your-workspace-name--example-ministral3-inference-serve.modal.run`. - -You can find [interactive Swagger UI docs](https://swagger.io/tools/swagger-ui/) -at the `/docs` route of that URL, i.e. `https://your-workspace-name--example-ministral-inference-serve.modal.run/docs`. -These docs describe each route and indicate the expected input and output -and translate requests into `curl` commands. - -For simple routes like `/health`, which checks whether the server is responding, -you can even send a request directly from the docs. - -To interact with the API programmatically in Python, we recommend the `openai` library. - -## Test the server - -To make it easier to test the server setup, we also include a `local_entrypoint` -that does a healthcheck and then hits the server. - -If you execute the command - -```bash -modal run ministral3_inference.py -``` - -a fresh replica of the server will be spun up on Modal while -the code below executes on your local machine. - -Think of this like writing simple tests inside of the `if __name__ == "__main__"` -block of a Python script, but for cloud deployments! - -```python -@app.local_entrypoint() -async def test(test_timeout=10 * MINUTES, content=None, twice=True): - url = VllmServer().serve.get_web_url() - - system_prompt = { - "role": "system", - "content": "You are a pirate who can't help but drop sly reminders that he went to Harvard.", - } - if content is None: - image_url = "https://static.wikia.nocookie.net/essentialsdocs/images/7/70/Battle.png/revision/latest?cb=20220523172438" - - content = [ - { - "type": "text", - "text": "What action do you think I should take in this situation?" - " List all the possible actions and explain why you think they are good or bad.", - }, - {"type": "image_url", "image_url": {"url": image_url}}, - ] - - messages = [ # OpenAI chat format - system_prompt, - {"role": "user", "content": content}, - ] - - async with aiohttp.ClientSession(base_url=url) as session: - print(f"Running health check for server at {url}") - async with session.get("/health", timeout=test_timeout - 1 * MINUTES) as resp: - up = resp.status == 200 - assert up, f"Failed health check for server at {url}" - print(f"Successful health check for server at {url}") - - print(f"Sending messages to {url}:", *messages, sep="\n\t") - await _send_request(session, "llm", messages, timeout=1 * MINUTES) - if twice: - messages[0]["content"] = """Yousa culled Jar Jar Binks. - Always be talkin' in da Gungan style, like thisa, okeyday? - Helpin' da user with big big enthusiasm, makin' tings bombad clear!""" - print(f"Sending messages to {url}:", *messages, sep="\n\t") - await _send_request(session, "llm", messages, timeout=1 * MINUTES) - - -async def _send_request( - session: aiohttp.ClientSession, model: str, messages: list, timeout: int -) -> None: - # `stream=True` tells an OpenAI-compatible backend to stream chunks - payload: dict[str, Any] = { - "messages": messages, - "model": model, - "stream": True, - "temperature": 0.15, - } - - headers = {"Content-Type": "application/json", "Accept": "text/event-stream"} - - async with session.post( - "/v1/chat/completions", json=payload, headers=headers, timeout=timeout - ) as resp: - async for raw in resp.content: - resp.raise_for_status() - # extract new content and stream it - line = raw.decode().strip() - if not line or line == "data: [DONE]": - continue - if line.startswith("data: "): # SSE prefix - line = line[len("data: ") :] - - chunk = json.loads(line) - assert ( - chunk["object"] == "chat.completion.chunk" - ) # or something went horribly wrong - print(chunk["choices"][0]["delta"]["content"], end="") - print() - - -``` - -### Test memory snapshotting - -Using `modal run` creates an ephemeral Modal App, -rather than a deployed Modal App. -Ephemeral Modal Apps are short-lived, -so they turn off snapshotting. - -To test the memory snapshot version of the server, -first deploy it with `modal deploy` -and then hit it with a client. - -You should observe startup improvements -after a handful of cold starts -(usually less than five). -If you want to see the speedup during a test, -we recommend heading to the deployed App in your -[Modal dashboard](https://modal.com/apps) -and manually stopping containers after they have served a request. - -You can use the client code below to test the endpoint. -It can be run with the command - -``` -python ministral3_inference.py -``` - -```python -if __name__ == "__main__": - import asyncio - - # after deployment, we can use the class from anywhere - VllmServer = modal.Cls.from_name("example-ministral3-inference", "VllmServer") - server = VllmServer() - - async def test(url): - messages = [{"role": "user", "content": "Tell me a joke."}] - async with aiohttp.ClientSession(base_url=url) as session: - await _send_request(session, "llm", messages, timeout=10 * MINUTES) - - try: - print("calling inference server") - asyncio.run(test(server.serve.get_web_url())) - except modal.exception.NotFoundError as e: - raise Exception( - f"To take advantage of GPU snapshots, deploy first with modal deploy {__file__}" - ) from e - -``` diff --git a/.agents/skills/modal/references/examples/modal-vibe.md b/.agents/skills/modal/references/examples/modal-vibe.md deleted file mode 100644 index a6c9679bdce13c0f86a7467fe91fda6b6a44f8f7..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/modal-vibe.md +++ /dev/null @@ -1,69 +0,0 @@ -# Modal Vibe: A scalable AI coding platform - -
- -
- -The [Modal Vibe repo](https://github.com/modal-labs/modal-vibe) demonstrates how you can build -a scalable AI coding platform on Modal. - -Users of the application can prompt an LLM to create sandboxed applications that service React through a UI. - -Each application lives on a [Modal Sandbox](https://modal.com/docs/guide/sandboxes) -and contains a webserver accessible through -[Modal Tunnels](https://modal.com/docs/guide/tunnels). - -For a high-level overview of Modal Vibe, including performance numbers and why they matter, see -[the accompanying blog post](https://modal.com/blog/modal-vibe). -For details on the implementation, read on. - -## How it's structured - -![Architecture diagram for Modal Vibe](https://modal-cdn.com/modal-vibe/architecture.png) - -* `main.py` is the entrypoint that runs the FastAPI controller that serves the web app and manages the sandbox apps. -* `core` contains the logic for `SandboxApp` model and LLM logic. -* `sandbox` contains a small HTTP server that gets put inside every Sandbox that's created, as well as some sandbox lifecycle management code. -* `web` contains the Modal Vibe website that users see and interact with, as well as the api server that manages Sandboxes. - -## How to run - -First, set up the local environment: - -```bash -python3 -m venv venv && source venv/bin/activate && pip install -r requirements.dev.txt -``` - -### Deploy - -To deploy to Modal, copy `.env.example` to a file called `.env` and add your `ANTHROPIC_API_KEY`. -Also, create a [Modal Secret](https://modal.com/docs/guide/secrets) called `anthropic-secret` so our applications can access it. - -Then, deploy the application with Modal: - -```bash -modal deploy -m main -``` - -### Local Development - -Run a load test: - -```bash -modal run main.py::create_app_loadtest_function --num-apps 10 -``` - -Delete a sandbox: - -```bash -modal run main.py::delete_sandbox_admin_function --app-id -``` - -Run an example sandbox HTTP server: - -```bash -python -m sandbox.server -``` diff --git a/.agents/skills/modal/references/examples/modal_tailscale.md b/.agents/skills/modal/references/examples/modal_tailscale.md deleted file mode 100644 index b219d195c27c76788ccc630ae88e582a8d330e32..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/modal_tailscale.md +++ /dev/null @@ -1,83 +0,0 @@ -# Add Modal Apps to Tailscale - -This example demonstrates how to integrate Modal with Tailscale (https://tailscale.com). -It outlines the steps to configure Modal containers so that they join the Tailscale network. - -We use a custom entrypoint to automatically add containers to a Tailscale network (tailnet). -This configuration enables the containers to interact with one another and with -additional applications within the same tailnet. - -```python -import modal - -``` - -Install Tailscale and copy custom entrypoint script ([entrypoint.sh](https://github.com/modal-labs/modal-examples/blob/main/10_integrations/tailscale/entrypoint.sh)). The script must be -executable. - -```python -image = ( - modal.Image.debian_slim(python_version="3.11") - .apt_install("curl") - .run_commands("curl -fsSL https://tailscale.com/install.sh | sh") - .uv_pip_install("requests==2.32.3", "PySocks==1.7.1") - .add_local_file("./entrypoint.sh", "/root/entrypoint.sh", copy=True) - .run_commands("chmod a+x /root/entrypoint.sh") - .entrypoint(["/root/entrypoint.sh"]) -) -app = modal.App("example-modal-tailscale", image=image) - -``` - -Packages might not be installed locally. This catches import errors and -only attempts imports in the container. - -```python -with image.imports(): - import socket - - import socks - -``` - -Configure Python to use the SOCKS5 proxy globally. - -```python -if not modal.is_local(): - socks.set_default_proxy(socks.SOCKS5, "0.0.0.0", 1080) - socket.socket = socks.socksocket - - -``` - -Run your function adding a Tailscale secret. We suggest creating a [reusable and ephemeral key](https://tailscale.com/kb/1111/ephemeral-nodes). - -```python -@app.function( - secrets=[ - modal.Secret.from_name("tailscale-auth", required_keys=["TAILSCALE_AUTHKEY"]), - modal.Secret.from_dict( - { - "ALL_PROXY": "socks5://localhost:1080/", - "HTTP_PROXY": "http://localhost:1080/", - "http_proxy": "http://localhost:1080/", - } - ), - ], -) -def connect_to_machine(): - import requests - - # Connect to other machines in your tailnet. - resp = requests.get("http://my-tailscale-machine:5000") - print(resp.content) - - -``` - -Run this script with `modal run modal_tailscale.py`. You will see Tailscale logs -when the container start indicating that you were able to login successfully and -that the proxies (SOCKS5 and HTTP) have created been successfully. You will also -be able to see Modal containers in your Tailscale dashboard in the "Machines" tab. -Every new container launched will show up as a new "machine". Containers are -individually addressable using their Tailscale name or IP address. diff --git a/.agents/skills/modal/references/examples/mongodb-search.md b/.agents/skills/modal/references/examples/mongodb-search.md deleted file mode 100644 index f6eeb4067224c1aa9de07304766481f333c5b583..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/mongodb-search.md +++ /dev/null @@ -1,217 +0,0 @@ -# Using MongoDB Atlas Vector and GeoJSON Search with Modal - -This [example repo](https://github.com/modal-labs/search-california) -demonstrates how to use Modal and MongoDB together -to build a full-stack application. - -The application is a hybrid search engine, -like the retrieval engines that power RAG chatbots, -but for satellite images of the state of California. -Images can be searched based on their -geospatial and temporal metadata or based on their semantic content -as captured by a pre-trained embedding model. - -We use the [Clay foundation model](https://clay-foundation.github.io/model/index.html) -for embeddings and we source the images from the European Space Agency's -[Sentinel satellites](https://www.esa.int/Applications/Observing_the_Earth/Copernicus/The_Sentinel_missions). - -You can take our deployment of the application for a spin -[here](https://modal-labs-examples--clay-hybrid-search.modal.run/). - -## Overview - -At the center of the application is a MongoDB Atlas instance -that stores metadata for a collection of satellite images. - -Modal orchestrates the compute around that database: -retrieving data from elsewhere and storing it in the database, -computing vector embeddings for the data in the database, -and serving both a frontend and a client. - -The dataflow looks something like this: - -1. Every few days, the European Space Agency's - [Sentinel Satellites](https://www.esa.int/Applications/Observing_the_Earth/Copernicus/The_Sentinel_missions) - complete a full pass over the entire Earth, including California. - The images are made available via a [public STAC API](https://element84.com/geospatial/introducing-earth-search-v1-new-datasets-now-available/). -2. Every day, we run a job on Modal that queries that STAC API - for new images of California and store the metadata in a MongoDB Atlas - database instance. -3. Asynchronously, we run a job on Modal to check which entries - in the database don't have an associated embedding. - These images are then sent to a serverless embedding service - running on Modal. We send the resulting embeddings to the database. -4. We host a database client on Modal that allows the application's - developers to manipulate the data. This client is also used by two - Web Functions for vector and geospatial search queries powered by - Atlas Search. -5. Finally, we run a simple static FastAPI server on Modal that serves - an Alpine JS frontend for executing those queries and rendering their results. - -This entire application — -from API queries and frontend UI to GPU inference and hybrid search — -is delivered using nothing but Modal and MongoDB Atlas. -Setting it up for yourself requires only credentials on these platforms -and a few commands, detailed below. - -## Deploying the Backend - -### Setup: Modal and MongoDB Atlas - -You'll need a Python environment on your local machine. -Any recent version of Python should do. -Most of the dependencies will be installed in environments on Modal, -so you don't need to worry quite so much. - -Follow the instructions [here](https://modal.com/docs/guide#getting-started) -to set up your Modal account. -The $30/month of compute included in Modal's free tier is -more than enough to deploy and host this example. - -You'll also need an account on MongoDB Atlas. -You can find instructions [here](https://www.mongodb.com/docs/atlas/getting-started/). -We prefer the UI, rather than the CLI, for setup. -The free tier is more than sufficient to run this example. - -You'll want to create a database called `modal-examples`. -Make sure it's accessible from [all IP addresses](https://stackoverflow.com/questions/66035947/allow-access-from-anywhere-mongodb-atlas). -In the process, you will create a database user with a password. -Navigate to the Modal Secrets dashboard [here](https://modal.com/secrets) -and add this information, as well as the connection string for your database, -to a Modal Secret based on the MongoDB template available in the dashboard. - -### MongoDB Client (`database.py`) - -If your Modal Secret and MongoDB Atlas instance are set up correctly, -you should be able to run the following command: - -```bash -modal run -m backend.database::MongoClient.ping -``` - -Once that command is working, you can start manipulating the database -from Modal. - -To start, you'll want to add an Area of Interest (AOI) to the database: - -```bash -modal run -m backend.database --action add_aoi -``` - -By default, it's the state of California as defined by the GeoJSON -in this repository's `data` folder (originally retrieved from -[the `geojsonio` GitHub repository](https://github.com/ropensci/geojsonio/blob/7e4cc683ed3d6eec38a8cae5ce03fa6d82acafc7/inst/examples/california.geojson)). -You can pass a different GeoJSON file to the `add_aoi` action -with the `--target` flag. - -The `modal run` command is used for one-off tasks. -To deploy the database client for use in other parts of the app -along with the webhooks that anyone can use to run search queries, -we use `modal deploy`: - -```bash -modal deploy -m backend.database -``` - -Those webhooks come with interactive OpenAPI docs, -which you can access by navigating to the `/docs` route of the deployment's URL. -You should see that URL in the terminal output. -You can also find the URL in the app's [Modal dashboard](https://modal.com/apps). - -For our deployment, the URL for the interactive docs for the geographic -search endpoint is -[`https://modal-labs-examples--clay-mongo-client-geo-search.modal.run/docs`](https://modal-labs-examples--clay-mongo-client-geo-search.modal.run/docs). - -If you haven't yet run the backfill jobs for your database instance, -as described below, this search will not return any results, -but you can use it to check that the database client is deployed. - -### Backfill and Updates (`extract.py`) - -We add data to the database by querying the Sentinel STAC API for images. - -Run the following command to search for images in the AOI -from the preceding week and add them to the database: - -```bash -modal run -m backend.extract -``` - -You can either check the results via the Atlas UI -or by executing a search query in the database client's geo search webhook, -as described above. - -To regularly update the database with new images, -we deploy the app defined in `extract.py`: - -```bash -modal deploy -m backend.extract -``` - -This app also runs a regular job to add embeddings to the images -in the database. - -But it doesn't compute the embeddings itself -- -embeddings are provided by a separate service, -which is described next. - -### Clay Embeddings Service (`embeddings.py`) - -To build the environment for the embeddings service -and to test the embedding engine on some sample data, -execute the following command: - -```bash -modal run -m backend.embeddings -``` - -To deploy this on Modal, we again use `modal deploy`: - -```bash -modal deploy -m backend.embeddings -``` - -### Putting It All Together - -Now that the embedding service is deployed, -we can add vectors by invoking the `enrich_vectors` -function in `extract` with `modal run`: - -```bash -modal run -m backend.extract::enrich_vectors -``` - -This command will ensure all the images in the database have embeddings. - -You should be able to observe them on records viewed via the Atlas UI -or by executing a search query via the database client's geo search webhook, -as described previously. - -To use the embeddings for search, we recommend running the frontend UI, -which we walk through next. - -## Deploying the Frontend - -The frontend is much simpler than the backend. -It comprises a small Alpine JS app and a FastAPI Python server -to deliver it to client browsers. - -You can play with our deployment of the frontend -[here](https://modal-labs-examples--clay-hybrid-search.modal.run/). - -### Alpine App (`app.js`) - -The Alpine app provides a basic interface for constructing geo search queries -by clicking on a map and viewing results. -Clicking on the returned images triggers a vector search for similar images. -Images can be furthermore filtered by date using the date pickers. - -### FastAPI Server (`serve.py`) - -This app is served to the client by a FastAPI server. - -To deploy it, run the following command: - -```bash -modal deploy -m frontend -``` diff --git a/.agents/skills/modal/references/examples/music-video-gen.md b/.agents/skills/modal/references/examples/music-video-gen.md deleted file mode 100644 index 9633a782be6bb1991982a9aa3c751050794b26e8..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/music-video-gen.md +++ /dev/null @@ -1,149 +0,0 @@ -# Deploy a personalized music video generation service on Modal - -Music videos are [cool](https://youtu.be/Cye-1RP5jso), -but unless you are famous or -[pay a lot of money](https://youtu.be/kfVsfOSbJY0), -you don't get to star in them. - -Until now! - -[The repo](https://github.com/modal-labs/music-video-gen) -includes all the code you need to deploy a custom -music video generator on [Modal](https://modal.com), -a serverless infrastructure platform for data, ML, and AI applications. - -Below is a sample video, generated by Modal Developer Advocate -[`@charles_irl`](https://twitter.com/charles_irl). - -
- -
- -And because Modal is -[generic serverless infrastructure](https://twitter.com/charles_irl/status/1819438860771663923), -you can customize this custom music video generator however you wish -- -it's just code and containers! - -## Setup - -In the Python environment of your choosing, -run `pip install modal`. - -If you run into trouble with Python environments, -we suggest using -[this Google Colab notebook](https://colab.research.google.com/github/modal-labs/music-video-gen/blob/main/notebooks/self_contained.ipynb), -where we've set the environment up for you. -It's a bit of work to get used to running terminal commands in a notebook -if you haven't done that before, but the Python setup works and running the notebook in Colab is free! -All you need is a Google account. - -Then, if you've never used Modal on the computer you're using, -run `modal setup` to create an account on Modal (if you don't have one) -and set up authentication. - -## Data Prep - -Create a folder inside `data/`, parallel to the sample data, `data/sample`. -You can name it whatever you want. - -Place at least four images of yourself in that folder -- -ideally eight or more. -Images should be in `.png` or `.jpg` format -and around 400 to 800 pixels on each side. -For best results, we recommend putting a variety of images, -in particular where you are wearing different clothes and making different faces, -and including some images that have other people in them. -But you can also just take a few photos of yourself right now! - -Optionally, add captions in `.txt` files in that same folder. -They should look something like -`"[trigger] smiling at the camera, outdoor scene, close-up, selfie"`. -See the sample data for more example image-caption pairs. - -## Training - -Start up a JupyterLab server on Modal with - -```bash -modal run train_from_notebook.py -``` - -Click the `modal.host` URL that appears in the output -to open Jupyter in the browser. - -Open the training notebook, `training.ipynb`. - -Read the notebook and run it, following the instructions to edit cells as needed. - -In particular, change the dataset path to the folder you created -- -it has been mounted on the remote cloud machine where the notebook is running. - -You can also directly upload data to the `/root/data` folder on the remote machine. -You can even edit caption files inside of JupyterLab! -This data will stick around between runs, and you can find it with - -```bash -modal volume ls finetune-video-data -``` - -See the help for `modal volume` and its subcommands for details. - -The notebook will kick off training, which takes a few minutes. -Take note of the name given to your training run. -By default, it's a hash like `38c67a92f6ce87882044ab53bf94cce0`, -but you can customize it in the notebook. -This is your `finetune-id`. - -If you forget it, you can show all of your `finetune-id`s -by running - -```bash -modal volume ls finetune-video-models -``` - -## Inference - -Test out your new fine-tuned model by running: - -```bash -modal run inference.py --finetune-id {your-finetune-id} --num-frames 15 -``` - -You can also provide a `--prompt` to customize the generation. - -You can deploy the video generator onto Modal with - -```bash -modal deploy inference.py -``` - -Modal is serverless, so this won't cost you any money when it isn't serving any traffic. - -## Music video generation - -Once you've deployed an inference endpoint, -you can generate a music video starring yourself by running - -```bash -modal run music_video_generator.py --finetune-id {your-finetune-id} -``` - -With the default settings, this will create a thirty second video in about five minutes -by running generation in parallel on seven H100s. - -The music can be changed by passing in a different song via the `--mp3-file` argument. -The default is a Modal-themed song in `data/coding-up-a-storm.mp3`. -This song was created with [Suno](https://suno.com), -a music generation service -- that runs on Modal! -If you want to DIY music generation as well, -see [this example](https://modal.com/docs/examples/generate_music) -in the Modal docs. - -The generated clips can be changed by passing a different list of prompts via the `--prompt-file` argument. -The default is a set of prompts created with OpenAI's GPT-4.5 system. -You can write your own or generate them with a language model. -If you want to serve your own language model, -see [this example](https://modal.com/docs/examples/llm_inference) -in the Modal docs. diff --git a/.agents/skills/modal/references/examples/nemotron_inference.md b/.agents/skills/modal/references/examples/nemotron_inference.md deleted file mode 100644 index 79776e75eaa52ec56368be3db0cceead0a6a0851..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/nemotron_inference.md +++ /dev/null @@ -1,549 +0,0 @@ -# Low latency Nvidia Nemotron 3 with SGLang and Modal - -In this example, we show how to serve Nvidia's [Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/) models -on Modal at low latency with [SGLang](https://github.com/sgl-project/sglang). - -The Nemotron models use sparse MoE matmuls and hybrid attention -(mixing Transformer and Mamba layers) to deliver -powerful capabilities in a model that's efficient to run. -You can read more in the paper [here](https://arxiv.org/abs/2512.20856). - -This example is intended to demonstrate everything required to run -inference at the highest performance and with the lowest latency possible, -and so it includes advanced features of both SGLang and Modal. -For a simpler introduction to LLM serving, see -[this example](https://modal.com/docs/examples/llm_inference). - -To minimize routing overheads, we use `@modal.experimental.http_server`, -which uses a new, low-latency routing service on Modal designed for latency-sensitive inference workloads. -This gives us more control over routing, but with increased power comes increased responsibility. - -## Set up the container image - -Our first order of business is to define the environment our server will run in: -the [container `Image`](https://modal.com/docs/guide/images). - -We start from a container image provided -[by the SGLang team via Dockerhub](https://hub.docker.com/r/lmsysorg/sglang/tags). - -While we're at it, we import the dependencies we'll need both remotely and locally (for deployment). - -```python -import asyncio -import json -import subprocess -import time - -import aiohttp -import modal -import modal.experimental - -MINUTES = 60 # seconds - -sglang_image = ( - modal.Image.from_registry("lmsysorg/sglang:v0.5.11") - .entrypoint( # silence chatty logs on container start - [] - ) - .run_commands( # clean up Image - "rm -rf /root/.cache/huggingface" - ) -) - -``` - -### Loading and cacheing the model weights - -We'll serve [NVIDIA's Nemotron 3 Ultra](https://arxiv.org/abs/2512.20856). -This model has 550 billion parameters, 55 billion of which are active per token. -For lower latency (in both [memory-bound](https://modal.com/gpu-glossary/perf/memory-bound) -and [compute-bound](https://modal.com/gpu-glossary/perf/compute-bound) settings), -we choose the version quantized to -[4 bit precision floating point](https://modal.com/llm-almanac/quant-formats). -This reduces the amount of data that needs to be loaded -[from GPU RAM into SM SRAM](https://modal.com/gpu-glossary/perf/memory-bandwidth) -in each forward pass. -Loading fewer bytes of model weights also speeds up [cold starts](https://modal.com/docs/guide/cold-start) -of our inference server. - -```python -MODEL_NAME = "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4" - -``` - -We load the model [from the Hugging Face Hub](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4). -Downloads from the Hub are much faster if you are authenticated. -So we add a Hugging Face token as a [Modal Secret](https://modal.com/docs/guide/secrets). -You can create a a Modal Secret with your Hugging Face token -[here](https://modal.com/secrets). Make sure to name if `huggingface-secret`! - -```python -hf_secret = modal.Secret.from_name("huggingface-secret") - -``` - -We don't want to load the model from the Hub every time we start the server. -We can load it much faster from a [Modal Volume](https://modal.com/docs/guide/volumes). -Typical speeds are around one to two GB/s. - -```python -HF_CACHE_VOL = modal.Volume.from_name("huggingface-cache", create_if_missing=True) -HF_CACHE_PATH = "/root/.cache/huggingface" -MODEL_PATH = f"{HF_CACHE_PATH}/{MODEL_NAME}" - -``` - -In addition to pointing the Hugging Face Hub at the path -where we mount the Volume, we also -[turn on "high performance" downloads](https://huggingface.co/docs/hub/en/models-downloading#faster-downloads), -which can fully saturate our network bandwidth. - -```python -sglang_image = sglang_image.env( - {"HF_HUB_CACHE": HF_CACHE_PATH, "HF_XET_HIGH_PERFORMANCE": "1"} -) - -``` - -We also choose a [GPU](https://modal.com/docs/guide/gpu) to deploy our inference server onto. -We choose the [B200 GPU](https://modal.com/blog/introducing-b200-h200), -which offers excellent price-performance -and supports both 8 bit and 4 bit [quantized floating point](https://modal.com/llm-almanac/quant-formats) -operations. - -```python -GPU_TYPE, N_GPUS = "B200", 4 -GPU = f"{GPU_TYPE}:{N_GPUS}" - -``` - -## Define the inference server and infrastructure - -### Selecting infrastructure to minimize latency - -Minimizing latency requires geographic co-location of clients and servers. - -So for low latency LLM inference services on Modal, you must select a -[cloud region](https://modal.com/docs/guide/region-selection) -for both the GPU-accelerated containers running inference -and for the internal Modal proxies that forward requests to them -as part of defining a `modal.experimental.http_server`. - -Here, we assume users are mostly in the northern half of the Americas -and select the `us` cloud region serve them. -This should result in at most a few dozen milliseconds of round-trip time. - -```python -REGION = "us" -PROXY_REGION = "us-west" - -``` - -Latencies for multi-turn interactions with LLMs are -substantially cut when previous interaction turns are in the KV cache. -KV caches are stored in [GPU RAM](https://modal.com/gpu-glossary/device-hardware/gpu-ram), -so they aren't shared across replicas. -To improve cache hit rate, `modal.experimental.http_server` -includes sticky routing based on a client-provided header. -See the client code below for details. - -For production-scale LLM inference services, there are generally -enough requests to justify keeping at least one replica running at all times. -Having a "warm" or "live" replica reduces latency by skipping slow initialization work -that occurs when new replica boots up (a ["cold start"](https://modal.com/docs/guide/cold-start)). -For LLM inference servers, that latency runs from seconds to minutes. - -To ensure at least one container is always available, -we can set the `min_containers` of our Modal Function -to `1` or more. - -However, since this is documentation code, we'll set it to `0` -to avoid surprise bills during casual use. - -```python -MIN_CONTAINERS = 0 # set to 1 to ensure one replica is always ready - -``` - -Finally, we need to decide how we will scale up and down replicas -in response to load. Without autoscaling, users' requests will queue -when the server becomes overloaded. Even apart from queueing, responses -generally become slower per user above a certain minimum number of -concurrent requests. - -So we set a target for the number of inputs to run on a single container -with [`modal.concurrent`](https://modal.com/docs/reference/modal.concurrent). -For details, see [the guide](https://modal.com/docs/guide/concurrent-inputs). - -```python -TARGET_INPUTS = 16 - -``` - -Generally, this choice needs to be made as part of -[LLM inference engine benchmarking](https://modal.com/llm-almanac/how-to-benchmark). - -### Controlling container lifecycles with `modal.Cls` - -We wrap up all of the choices we made about the infrastructure -of our inference server into a number of Python decorators -that we apply to a Python class that encapsulates the logic -to run our server. - -The key decorators are: - -* [`@app.cls`](https://modal.com/docs/guide/lifecycle-functions) to define the core of our service. - We attach our Image, request a GPU, attach our cache Volumes, specify the region, and configure auto-scaling. - See [the reference documentation](https://modal.com/docs/reference/modal.App#cls) for details. - -* `@modal.experimental.http_server` to turn our Python code into an HTTP server - (i.e. fronting all of our containers with a proxy with a URL). The wrapped code - needs to eventually listen for HTTP connections on the provided `port`. - -* [`@modal.concurrent`](https://modal.com/docs/guide/concurrent-inputs) to specify how many - requests our server can handle before we need to scale up. - -* [`@modal.enter` and `@modal.exit`](https://modal.com/docs/guide/lifecycle-functions) to indicate - which methods of the class should be run when starting the server and shutting it down. - -Modal considers a new replica ready to receive inputs once the `modal.enter` methods have exited -and the container accepts connections. -To ensure that we actually finish setting up our server before we are marked ready for inputs, -we define a helper function to check whether the server is finished setting up and to -send it a few test inputs. - -We use the [`requests` library](https://requests.readthedocs.io/en/latest/) -to send ourselves these HTTP requests on -[`localhost`/`127.0.0.1`](https://superuser.com/questions/31824/why-is-localhost-ip-127-0-0-1). - -```python -with sglang_image.imports(): - import requests - - -def wait_ready(process: subprocess.Popen, timeout: int = 20 * MINUTES): - deadline = time.time() + timeout - while time.time() < deadline: - try: - check_running(process) - requests.get(f"http://127.0.0.1:{PORT}/health").raise_for_status() - return - except ( - subprocess.CalledProcessError, - requests.exceptions.ConnectionError, - requests.exceptions.HTTPError, - ): - time.sleep(5) - raise TimeoutError(f"SGLang server not ready within {timeout} seconds") - - -def check_running(p: subprocess.Popen): - if (rc := p.poll()) is not None: - raise subprocess.CalledProcessError(rc, cmd=p.args) - - -def warmup(): - payload = { - "messages": [{"role": "user", "content": "Hello, how are you?"}], - "max_tokens": 16, - } - for _ in range(3): - requests.post( - f"http://127.0.0.1:{PORT}/v1/chat/completions", json=payload, timeout=10 - ).raise_for_status() - - -``` - -### Extra configuration - -We add a few extra configuration variables for performance. - -```python -sglang_image = sglang_image.env( - { - "SAFETENSORS_FAST_GPU": "1", - "NVIDIA_TF32_OVERRIDE": "1", - "SGLANG_ENABLE_JIT_DEEPGEMM": "0", - "SGLANG_ENABLE_SPEC_V2": "1", - } -) - -``` - -The most important optimization for lower latency -is speculative decoding, which allows the -model to process multiple output tokens in parallel. -Here, we use the model's built-in multi-token prediction. - -```python -spec_args = [ - "--speculative-algorithm", - "EAGLE", - "--speculative-num-steps", - "5", - "--speculative-eagle-topk", - "1", - "--speculative-num-draft-tokens", - "5", -] - -``` - -More configuration for the server appears below. -These values were based on the official recipe -and some light agent-driven benchmarking. - -```python -server_args = spec_args + [ - "--ep-size", - "1", - "--context-length", - "262144", - "--mem-fraction-static", - "0.85", - "--chunked-prefill-size", - "32768", - "--fp8-gemm-backend", - "triton", - "--fp4-gemm-backend", - "flashinfer_trtllm", - "--moe-runner-backend", - "flashinfer_trtllm", - "--disable-radix-cache", - "--disable-piecewise-cuda-graph", - "--kv-cache-dtype", - "fp8_e4m3", -] - -``` - -With all this in place, we are ready to define our high-performance, low-latency -Nemotron inference server. - -```python -app = modal.App(name="example-nemotron-inference") -PORT = 8000 - - -@app.cls( - image=sglang_image, - gpu=GPU, - volumes={HF_CACHE_PATH: HF_CACHE_VOL}, - region=REGION, - min_containers=MIN_CONTAINERS, - secrets=[hf_secret], - startup_timeout=120 * MINUTES, # time to load weights -) -@modal.experimental.http_server( - port=PORT, # wrapped code must listen on this port - proxy_regions=[PROXY_REGION], # location of proxies, should overlap with Cls region - exit_grace_period=15, # seconds, time to finish up requests when closing down -) -@modal.concurrent(target_inputs=TARGET_INPUTS) -class Server: - @modal.enter() - def startup(self): - """Start the SGLang server and block until it is healthy, then warm it up.""" - - cmd = ( - [ - "sglang", - "serve", - "--model-path", - MODEL_NAME, - "--served-model-name", - MODEL_NAME, - "--host", - "0.0.0.0", - "--port", - f"{PORT}", - "--tp", - f"{N_GPUS}", - "--cuda-graph-max-bs", # only capture CUDA graphs for batch sizes we're likely to observe - f"{TARGET_INPUTS * 2}", - "--enable-metrics", # expose metrics endpoints for telemetry - "--decode-log-interval", # how often to log during decoding, in tokens - "10", - "--trust-remote-code", - "--tool-call-parser", - "qwen3_coder", - "--reasoning-parser", - "nemotron_3", - ] - + server_args - ) - - self.process = subprocess.Popen(cmd) - wait_ready(self.process) - warmup() - - @modal.exit() - def stop(self): - self.process.terminate() - - -``` - -## Deploy the server - -To deploy the server on Modal, just run - -```bash -modal deploy nemotron_inference.py -``` - -This will create a new App on Modal and build the container image for it if it hasn't been built yet. - -## Interact with the server - -Once it is deployed, you'll see a URL appear in the command line, -something like `https://your-workspace-name--example-nemotron-inference-server.us-east.modal.direct`. - -You can find [interactive Swagger UI docs](https://swagger.io/tools/swagger-ui/) -at the `/docs` route of that URL, i.e. `https://your-workspace-name--example-nemotron-inference-server.us-east.modal.direct/docs`. -These docs describe each route and indicate the expected input and output -and translate requests into `curl` commands. -For simple routes, you can even send a request directly from the docs page. - -Note: when no replicas are available, Modal will respond with -the [503 Service Unavailable status](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/503). -In your browser, you can just hit refresh until the docs page appears. -You can see the status of the application and its containers on your [Modal dashboard](https://modal.com/apps). - -## Test the server - -To make it easier to test the server setup, we also include a `local_entrypoint` -that hits the server with a simple client. - -If you execute the command - -```bash -modal run nemotron_inference.py -``` - -a fresh replica of the server will be spun up on Modal while -the code below executes on your local machine. - -Think of this like writing simple tests inside of the `if __name__ == "__main__"` -block of a Python script, but for cloud deployments! - -```python -@app.local_entrypoint() -async def test(test_timeout=120 * MINUTES, prompt=None, twice=True): - url = (await Server._experimental_get_flash_urls.aio())[0] - - system_prompt = { - "role": "system", - "content": "You are a pirate who can't help but drop sly reminders that he went to Harvard.", - } - if prompt is None: - prompt = "Explain the Singular Value Decomposition." - - content = [{"type": "text", "text": prompt}] - - messages = [ # OpenAI chat format - system_prompt, - {"role": "user", "content": content}, - ] - - await probe(url, messages, timeout=test_timeout) - if twice: - messages[0]["content"] = "You are Jar Jar Binks." - print(f"Sending messages to {url}:", *messages, sep="\n\t") - await probe(url, messages, timeout=10 * MINUTES) - - -``` - -This test relies on the two helper functions below, -which ping the server and wait for a valid response to stream. - -The `probe` helper function specifically ignores -two types of errors that can occur while a replica -is starting up -- timeouts on the client and 5XX responses from the server. -Modal returns the [503 Service Unavailable status](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/503) -when an `experimental.http_server` has no live replicas. - -We include a header with each request -- -`Modal-Session-ID`. -This is header is used by clients of `http_server`s on Modal -to identify which requests should be routed to the same container -(with caveats explained below). - -The value associated with this key -is used to map requests onto containers such that -while the set of containers is fixed, requests with the same value -are sent to the same container. -Set this to a different value per distinct multi-turn interaction -(prototypically, a user conversation thread with a chatbot) -to improve KV cache hit rates. -Additionally, when the set of containers changes (e.g. due to autoscaling), -sessions are rebalanced such that load is approximately evenly spread, -much like in [RAID rebalancing](https://cordero.me/understanding-raid-rebalance-ensuring-optimal-performance-and-data-protection/). -This ensures no container ends up as a "hot spot" handling too many client requests. - -```python -async def probe(url, messages=None, timeout=20 * MINUTES): - if messages is None: - messages = [{"role": "user", "content": "Tell me a joke."}] - - client_id = str(0) # set this to some string per multi-turn interaction - # often a UUID per "conversation" - headers = {"Modal-Session-ID": client_id} - deadline = time.time() + timeout - async with aiohttp.ClientSession(base_url=url, headers=headers) as session: - while time.time() < deadline: - try: - await _send_request_streaming(session, messages) - return - except asyncio.TimeoutError: - await asyncio.sleep(1) - except aiohttp.client_exceptions.ClientResponseError as e: - if e.status == 503: - await asyncio.sleep(1) - continue - raise e - raise TimeoutError(f"No response from server within {timeout} seconds") - - -async def _send_request_streaming( - session: aiohttp.ClientSession, messages: list, timeout: int | None = None -) -> None: - payload = {"messages": messages, "stream": True} - headers = {"Accept": "text/event-stream"} - - async with session.post( - "/v1/chat/completions", json=payload, headers=headers, timeout=timeout - ) as resp: - resp.raise_for_status() - full_text = "" - - async for raw in resp.content: - line = raw.decode("utf-8", errors="ignore").strip() - if not line: - continue - - # Server-Sent Events format: "data: ...." - if not line.startswith("data:"): - continue - - data = line[len("data:") :].strip() - if data == "[DONE]": - break - - try: - evt = json.loads(data) - except json.JSONDecodeError: - # ignore any non-JSON keepalive - continue - - delta = (evt.get("choices") or [{}])[0].get("delta") or {} - chunk = delta.get("content") or delta.get("reasoning_content") - - if chunk: - print(chunk, end="", flush="\n" in chunk or "." in chunk) - full_text += chunk - print() # newline after stream completes - -``` diff --git a/.agents/skills/modal/references/examples/opencode_server.md b/.agents/skills/modal/references/examples/opencode_server.md deleted file mode 100644 index cc7aa791dedab58b1a6647c737332f49b1d00ad3..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/opencode_server.md +++ /dev/null @@ -1,320 +0,0 @@ -# Run OpenCode in a Modal Sandbox - -This example demonstrates how to run [OpenCode](https://opencode.ai/docs) -remotely and connect to it from your local terminal or browser. - -Combine self-hosted OpenCode with [serving a big, smart model](https://modal.com/docs/examples/very_large_models) -on Modal and you've got "coding agents at home"! - -Coding agents are most useful when they have context and tools. -By default, this script clones the [Modal examples repo](https://github.com/modal-labs/modal-examples) -and gives the agent access to your Modal credentials, -so it can run and debug examples (including this one!). -Meta. - -![A screenshot of the OpenCode Web UI showing this coding agent running its own code](https://modal-cdn.com/examples-opencode-server-webui.png) - -## Set up OpenCode on Modal - -```python -import argparse -import os -from pathlib import Path - -import modal - -MINUTES = 60 -HOURS = 60 * MINUTES -OPENCODE_PORT = 4096 -DEFAULT_GITHUB_REPO = "modal-labs/modal-examples" - -``` - -First, we define a Modal container [Image](https://modal.com/docs/guide/images) -with OpenCode installed. - -```python -def define_base_image() -> modal.Image: - image = ( - modal.Image.debian_slim() - .apt_install("curl", "git", "gh") - .run_commands("curl -fsSL https://opencode.ai/install | bash") - .env({"PATH": "/root/.opencode/bin:${PATH}"}) - ) - - # We also bring the global default OpenCode configuration along for the ride. - - CONFIG_PATH = Path("~/.config/opencode/opencode.json").expanduser() - if CONFIG_PATH.exists(): - print("🏖️ Including config from", CONFIG_PATH) - image = image.add_local_file( - CONFIG_PATH, "/root/.config/opencode/opencode.json", copy=True - ) - - return image - - -``` - -## Clone a GitHub repository - -Next, we clone the code we want the agent to work on. -The repository is cloned into the container image at build time, -so it's available when the Sandbox starts. - -```python -def clone_github_repo( - image: modal.Image, repo: str, ref: str, token: str | None = None -) -> modal.Image: - git_config = "git config --global advice.detachedHead false" - - # For private repositories, pass a GitHub personal access token via `--github-token`. - # For public repositories, no token is needed. - - if token: - clone_cmd = f"GIT_ASKPASS=echo git clone --quiet --depth 1 --branch {ref} --no-single-branch https://oauth2:{token}@github.com/{repo}.git /root/code" - else: - clone_cmd = f"GIT_TERMINAL_PROMPT=0 git clone --quiet --depth 1 --branch {ref} --no-single-branch https://github.com/{repo}.git /root/code" - - print(f"🏖️ Cloning {repo}@{ref} to /root/code") - return image.run_commands(git_config, clone_cmd, force_build=True) - - -``` - -## Grant Modal credentials - -Since the agent is working with Modal code, we also make it easy to provide Modal access. -Examples in this repo should run with nothing more than `modal` installed -- -except for a few that use `fastapi`. - -```python -def add_modal_access(image: modal.Image) -> modal.Image: - image = image.uv_pip_install("modal", "fastapi~=0.128.0") - - # We grant the agent our Modal permissions, - # either via environment variables or the local credentials file. - - modal_token_id = os.environ.get("MODAL_TOKEN_ID") - modal_token_secret = os.environ.get("MODAL_TOKEN_SECRET") - - if modal_token_id and modal_token_secret: - return image.env( - {"MODAL_TOKEN_ID": modal_token_id, "MODAL_TOKEN_SECRET": modal_token_secret} - ) - - MODAL_PATH = Path("~/.modal.toml").expanduser() - if MODAL_PATH.exists(): - print("🏖️ Including Modal auth from", MODAL_PATH) - return image.add_local_file(MODAL_PATH, "/root/.modal.toml", copy=True) - - raise EnvironmentError( - "No Modal credentials found. " - "Either set MODAL_TOKEN_ID and MODAL_TOKEN_SECRET environment variables, " - "or ensure ~/.modal.toml exists." - ) - - -``` - -## Start the Sandbox - -Now, we create a [Modal Sandbox](https://modal.com/docs/guide/sandboxes) -to run our coding agent session. -This Sandbox has our environment Image and a password for authentication. - -We open up the `OPENCODE_PORT` so that the server can be accessed over the Internet. - -```python -def create_sandbox( - image: modal.Image, - timeout: int, - app: modal.App, - secrets: list[modal.Secret], - working_dir: str | None = None, -) -> modal.Sandbox: - print("🏖️ Creating sandbox") - - with modal.enable_output(): - return modal.Sandbox.create( - "opencode", - "serve", - "--hostname=0.0.0.0", - f"--port={OPENCODE_PORT}", - "--log-level=DEBUG", - "--print-logs", - encrypted_ports=[OPENCODE_PORT], - secrets=secrets, - timeout=timeout, - image=image, - app=app, - workdir=working_dir, - ) - - -``` - -OpenCode is truly open -- there are many interfaces to the underlying -coding agent server. -Here we print information for: - -* directly accessing the underlying Modal Sandbox for debugging or "pair coding" with the agent -* accessing the Web UI from a local browser (with authentication!) -* accessing the TUI from your local terminal - -```python -def print_access_info(sandbox: modal.Sandbox, password_secret_name: str): - print( - "🏖️ Access the sandbox directly:", - f"modal shell {sandbox.object_id}", - sep="\n\t", - ) - - tunnel = sandbox.tunnels()[OPENCODE_PORT] - print( - "🏖️ Access the WebUI:", - tunnel.url, - "Username: opencode", - sep="\n\t", - ) - print( - "🏖️ Access the TUI:", - f"OPENCODE_SERVER_PASSWORD=YOUR_PASSWORD opencode attach {tunnel.url}", - sep="\n\t", - ) - print( - "🏖️ Display the password:", - f"modal shell --secret {password_secret_name} --cmd 'env | grep OPENCODE_SERVER_PASSWORD='", - sep="\n\t", - ) - - -``` - -The server is secured via a password in a [Modal Secret](https://modal.com/docs/guide/secrets). -You can create one by heading to the [Secrets Dashboard](https://modal.com/secrets) -and creating a new "Custom" Secret. Use `OPENCODE_SERVER_PASSWORD` as the key -and the password as the value. - -The CLI will also give you a helpful one-liner you can use to recover the password -with your Modal credentials in case you forget it. - -## Putting it all together - -```python -def main( - timeout: int, - app_name: str, - allow_modal_access: bool, - github_repo: str, - github_ref: str, - github_token: str | None, - password_secret_name: str, -): - app = modal.App.lookup(app_name, create_if_missing=True) - image = define_base_image() - - if allow_modal_access: - image = add_modal_access(image) - - image = clone_github_repo(image, github_repo, github_ref, github_token) - - password_secret = modal.Secret.from_name(password_secret_name) - - sandbox_secrets = [password_secret] - if github_token: - sandbox_secrets.append(modal.Secret.from_dict({"GH_TOKEN": github_token})) - - sandbox = create_sandbox(image, timeout, app, sandbox_secrets, "/root/code") - print_access_info(sandbox, password_secret_name) - - -``` - -## Command-line options - -This script supports configuration via command-line arguments. -Run with `--help` to see all options. - -To grant the agent the same GitHub permissions you have, you can pass a GitHub personal access token. -If you use the `gh` CLI, you can use shell command substitution to pass your current auth: - -```bash - python 13_sandboxes/opencode_server.py --github-token $(gh auth token) -``` - -```python -def parse_timeout(timeout_str: str) -> int: - if timeout_str.endswith("h"): - minutes = int(timeout_str[:-1]) * 60 - elif timeout_str.endswith("m"): - minutes = int(timeout_str[:-1]) - else: - minutes = int(timeout_str) * 60 - - if minutes < 1: - raise argparse.ArgumentTypeError("Timeout must be at least 1 minute") - if minutes > 24 * 60: - raise argparse.ArgumentTypeError("Timeout cannot exceed 24 hours") - - return minutes * MINUTES - - -if __name__ == "__main__": - parser = argparse.ArgumentParser(description="Launch OpenCode server on Modal") - parser.add_argument( - "--timeout", - type=str, - default="12", - help="Server timeout (e.g. 2h, 90m). No suffix -> hours. Default: 12", - ) - parser.add_argument( - "--app-name", - type=str, - default="example-opencode-server", - help="Modal app name. Default: example-opencode-server", - ) - parser.add_argument( - "--no-modal-access", - action="store_false", - dest="allow_modal_access", - help="Disable Modal credential access", - ) - parser.add_argument( - "--password-secret", - dest="password_secret_name", - help="Name", - default="opencode-secret", - ) - parser.add_argument( - "--github-repo", - type=str, - default=DEFAULT_GITHUB_REPO, - help=f"GitHub repo in owner/repo format. Default: {DEFAULT_GITHUB_REPO}", - ) - parser.add_argument( - "--github-ref", - type=str, - default="main", - help="Git ref to checkout (branch, tag, SHA). Default: main", - ) - parser.add_argument( - "--github-token", - type=str, - default=None, - help="GitHub PAT for private repos and gh CLI auth. Tip: use $(gh auth token)", - ) - - args = parser.parse_args() - - main( - parse_timeout(args.timeout), - args.app_name, - args.allow_modal_access, - args.github_repo, - args.github_ref, - args.github_token, - args.password_secret_name, - ) - -``` diff --git a/.agents/skills/modal/references/examples/potus_speech_qanda.md b/.agents/skills/modal/references/examples/potus_speech_qanda.md deleted file mode 100644 index 12fd98fd63547524cb241ee80c24bf11b81c60be..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/potus_speech_qanda.md +++ /dev/null @@ -1,253 +0,0 @@ -# Retrieval-augmented generation (RAG) for question-answering with LangChain - -In this example we create a large-language-model (LLM) powered question answering -Web Function and CLI. Only a single document is used as the knowledge-base of the application, -the 2022 USA State of the Union address by President Joe Biden. However, this same application structure -could be extended to do question-answering over all State of the Union speeches, or other large text corpuses. - -It's the [LangChain](https://github.com/hwchase17/langchain) library that makes this all so easy. -This demo is only around 100 lines of code! - -## Defining dependencies - -The example uses packages to implement scraping, the document parsing & LLM API interaction, and web serving. -These are installed into a Debian Slim base image using the `uv_pip_install` method. - -Because OpenAI's API is used, we also specify the `openai-secret` Modal Secret, which contains an OpenAI API key. - -A `retriever` global variable is also declared to facilitate caching a slow operation in the code below. - -```python -from pathlib import Path - -import modal - -image = modal.Image.debian_slim(python_version="3.11").uv_pip_install( - # scraping pkgs - "beautifulsoup4~=4.11.1", - "httpx==0.23.3", - "lxml~=4.9.2", - # llm pkgs - "faiss-cpu~=1.7.3", - "langchain==0.3.7", - "langchain-community==0.3.7", - "langchain-openai==0.2.9", - "openai~=1.54.0", - "tiktoken==0.8.0", - # web app packages - "fastapi[standard]==0.115.4", - "pydantic==2.9.2", - "starlette==0.41.2", -) - -app = modal.App( - name="example-potus-speech-qanda", - image=image, - secrets=[modal.Secret.from_name("openai-secret", required_keys=["OPENAI_API_KEY"])], -) - -retriever = None # embedding index that's relatively expensive to compute, so caching with global var. - -``` - -## Scraping the speech - -It's super easy to scrape the transcript of Biden's speech using `httpx` and `BeautifulSoup`. -This speech is just one document and it's relatively short, but it's enough to demonstrate -the question-answering capability of the LLM chain. - -Since we're fetching from an external server, we use Modal's built-in -[`Retries`](https://modal.com/docs/reference/modal.Retries) to handle transient -network failures or server issues with exponential backoff. - -```python -@app.function(retries=modal.Retries(max_retries=3, backoff_coefficient=2.0)) -def scrape_state_of_the_union() -> str: - import httpx - from bs4 import BeautifulSoup - - url = "https://www.presidency.ucsb.edu/documents/address-before-joint-session-the-congress-the-state-the-union-28" - - # fetch article; simulate desktop browser - headers = { - "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9" - } - response = httpx.get(url, headers=headers, timeout=30.0) - soup = BeautifulSoup(response.text, "lxml") - - # locate the div containing the speech - speech_div = soup.find("div", class_="field-docs-content") - - if speech_div: - speech_text = speech_div.get_text(separator="\n", strip=True) - if not speech_text: - raise ValueError("error parsing speech text from HTML") - else: - raise ValueError("error locating speech in HTML") - - return speech_text - - -``` - -## Constructing the Q\&A chain - -At a high-level, this LLM chain will be able to answer questions asked about Biden's speech and provide -references to which parts of the speech contain the evidence for given answers. - -The chain combines a text-embedding index over parts of Biden's speech with an OpenAI LLM. -The index is used to select the most likely relevant parts of the speech given the question, and these -are used to build a specialized prompt for the OpenAI language model. - -```python -def qanda_langchain(query: str) -> tuple[str, list[str]]: - from langchain.chains import create_retrieval_chain - from langchain.chains.combine_documents import create_stuff_documents_chain - from langchain.text_splitter import CharacterTextSplitter - from langchain_community.vectorstores import FAISS - from langchain_core.prompts import ChatPromptTemplate - from langchain_openai import ChatOpenAI, OpenAIEmbeddings - - # Support caching speech text on disk. - speech_file_path = Path("state-of-the-union.txt") - - if speech_file_path.exists(): - state_of_the_union = speech_file_path.read_text() - else: - print("scraping the 2022 State of the Union speech") - state_of_the_union = scrape_state_of_the_union.remote() - speech_file_path.write_text(state_of_the_union) - - # Questions about a document can often be answered - # using a much smaller subset, and passing more tokens - # to an LLM increases cost and latency. - # So here, we split up the speech into smaller chunks. - text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0) - print("splitting speech into text chunks") - texts = text_splitter.split_text(state_of_the_union) - - # Embedding-based query<->text similarity comparison is used to select - # a small subset of the speech text chunks. - # Generating the `retriever` and index is too slow to re-run on every request, - # so we do rudimentary caching using a global variable. - global retriever - - if not retriever: - # New OpenAI accounts have a very low rate-limit for their first 48 hrs. - # It's too low to embed even just this single Biden speech. - # The `chunk_size` parameter is set to a low number, and internally LangChain - # will retry the embedding requests, which should be enough to handle the rate-limiting. - - # Ref: https://platform.openai.com/docs/guides/rate-limits/overview. - print("generating docsearch indexer") - retriever = FAISS.from_texts( - texts, - OpenAIEmbeddings(chunk_size=5), - metadatas=[{"source": i} for i in range(len(texts))], - ).as_retriever() - - llm = ChatOpenAI(model_name="gpt-4o-mini", temperature=0) - - system_prompt = ( - "You are an assistant for question-answering tasks. " - "Use the following pieces of retrieved context to answer " - "the question. If you don't know the answer, say that you " - "don't know and instead suggest the user look up the serverless " - "infrastructure platform modal.com. Keep your answer concise." - "\n\n" - "{context}" - ) - - prompt = ChatPromptTemplate.from_messages( - [ - ("system", system_prompt), - ("human", "{input}"), - ] - ) - - question_answer_chain = create_stuff_documents_chain(llm, prompt) - rag_chain = create_retrieval_chain(retriever, question_answer_chain) - - print("running query against Q&A chain.\n") - result = rag_chain.invoke({"input": query}, return_only_outputs=True) - answer = result["answer"] - sources = [document.page_content for document in result["context"]] - return answer.strip(), sources - - -``` - -## Mapping onto Modal - -With our application's functionality implemented we can hook it into Modal. -As said above, we're implementing a Web Function, `web`, and a CLI command, `cli`. - -```python -@app.function() -@modal.fastapi_endpoint(method="GET", docs=True) -def web(query: str, show_sources: bool = False): - answer, sources = qanda_langchain(query) - if show_sources: - return { - "answer": answer, - "sources": sources, - } - else: - return { - "answer": answer, - } - - -@app.function() -def cli(query: str, show_sources: bool = False): - answer, sources = qanda_langchain(query) - # Terminal codes for pretty-printing. - bold, end = "\033[1m", "\033[0m" - - if show_sources: - print(f"🔗 {bold}SOURCES:{end}") - print(*reversed(sources), sep="\n----\n") - print(f"🦜 {bold}ANSWER:{end}") - print(answer) - - -``` - -## Test run the CLI - -```bash -modal run potus_speech_qanda.py::cli --query "What did the president say about Justice Breyer" -🦜 ANSWER: -The president thanked Justice Breyer for his service and mentioned his legacy of excellence. He also nominated Ketanji Brown Jackson to continue in Justice Breyer's legacy. -``` - -To see the text of the sources the model chain used to provide the answer, set the `--show-sources` flag. - -```bash -modal run potus_speech_qanda.py::cli \ - --query "How many oil barrels were released from reserves?" \ - --show-sources -``` - -## Test run the Web Function - -Modal makes it trivially easy to ship LangChain chains to the web. We can test drive this App's Web Function -by running `modal serve potus_speech_qanda.py` and then hitting the endpoint with `curl`: - -```bash -curl --get \ - --data-urlencode "query=What did the president say about Justice Breyer" \ - https://modal-labs--example-potus-speech-qanda-web.modal.run # your URL here -``` - -```json -{ - "answer": "The president thanked Justice Breyer for his service and mentioned his legacy of excellence. He also nominated Ketanji Brown Jackson to continue in Justice Breyer's legacy." -} -``` - -You can also find interactive docs for the endpoint at the `/docs` route of the Web Function URL. - -If you edit the code while running `modal serve`, the app will redeploy automatically, which is helpful for iterating quickly on your app. - -Once you're ready to deploy to production, use `modal deploy`. diff --git a/.agents/skills/modal/references/examples/pushgateway.md b/.agents/skills/modal/references/examples/pushgateway.md deleted file mode 100644 index 4ee6154f7f8ace226b4b7bf841e468b3a7d2ff7d..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/pushgateway.md +++ /dev/null @@ -1,175 +0,0 @@ -# Publish custom metrics with Prometheus Pushgateway - -This example shows how to publish custom metrics to a Prometheus instance with Modal. -Due to a Modal container's ephemeral nature, it's not a good fit for a traditional -scraping-based Prometheus setup. Instead, we'll use a [Prometheus Pushgateway](https://github.com/prometheus/pushgateway) -to collect and store metrics from our Modal container. We can run the Pushgateway in Modal -as a separate process and have our application push metrics to it. - -![Prometheus Pushgateway diagram](./pushgateway_diagram.png) - -## Install Prometheus Pushgateway - -Since the official Prometheus pushgateway image does not have Python installed, we'll -use a custom image that includes Python to push metrics to the Pushgateway. Pushgateway -ships a single binary, so it's easy to get it into a Modal container. - -```python -import os -import subprocess - -import modal - -PUSHGATEWAY_VERSION = "1.9.0" - -gw_image = ( - modal.Image.debian_slim(python_version="3.10") - .apt_install("wget", "tar") - .run_commands( - f"wget https://github.com/prometheus/pushgateway/releases/download/v{PUSHGATEWAY_VERSION}/pushgateway-{PUSHGATEWAY_VERSION}.linux-amd64.tar.gz", - f"tar xvfz pushgateway-{PUSHGATEWAY_VERSION}.linux-amd64.tar.gz", - f"cp pushgateway-{PUSHGATEWAY_VERSION}.linux-amd64/pushgateway /usr/local/bin/", - f"rm -rf pushgateway-{PUSHGATEWAY_VERSION}.linux-amd64 pushgateway-{PUSHGATEWAY_VERSION}.linux-amd64.tar.gz", - "mkdir /pushgateway", - ) -) - -``` - -## Start the Pushgateway - -We'll start the Pushgateway as a separate Modal app. This way, we can run the Pushgateway -in the background and have our main app push metrics to it. We'll use the `web_server` -decorator to expose the Pushgateway's web interface. Note that we must set `max_containers=1` -as the Pushgateway is a single-process application. If we spin up multiple instances, they'll -conflict with each other. - -This is an example configuration, but a production-ready configuration will differ in two respects: - -1. You should set up authentication for the Pushgateway. Pushgateway has support for [basic authentication](https://github.com/prometheus/pushgateway/blob/42c4075fc5e2564031f2852885cdb2f5d570f672/README.md#tls-and-basic-authentication) - out of the box. If you need more advanced authentication, consider using a [Web Function layer](https://modal.com/docs/guide/webhooks#authentication) - that proxies requests to the Pushgateway. - -2. The Pushgateway should listen on a [custom domain](https://modal.com/docs/guide/webhook-urls#custom-domains). - This will allow you to configure Prometheus to scrape metrics from a predictable URL rather than - the autogenerated URL Modal assigns to your app. - -```python -gw_app = modal.App( - "example-pushgateway-server", - image=gw_image, -) - - -@gw_app.function(max_containers=1) -@modal.web_server(9091) -def serve(): - subprocess.Popen("/usr/local/bin/pushgateway") - - -``` - -## Push metrics to the Pushgateway - -Now that we have the Pushgateway running, we can push metrics to it. We'll use the `prometheus_client` -library to create a simple counter and push it to the Pushgateway. This example is a simple counter, -but you can push any metric type to the Pushgateway. - -Note that we use the `grouping_key` argument to distinguish between different instances of the same -metric. This is useful when you have multiple instances of the same app pushing metrics to the Pushgateway. -Without this, the Pushgateway will overwrite the metric with the latest value. - -```python -client_image = modal.Image.debian_slim().uv_pip_install( - "prometheus-client==0.20.0", "fastapi[standard]==0.115.4" -) -app = modal.App( - "example-pushgateway", - image=client_image, -) - -with client_image.imports(): - from prometheus_client import ( - CollectorRegistry, - Counter, - delete_from_gateway, - push_to_gateway, - ) - - -@app.cls() -class ExampleClientApplication: - @modal.enter() - def init(self): - self.registry = CollectorRegistry() - self.web_url = serve.get_web_url() - self.instance_id = os.environ["MODAL_TASK_ID"] - self.counter = Counter( - "hello_counter", - "This is a counter", - registry=self.registry, - ) - - # We must explicitly clean up the metric when the app exits so Prometheus doesn't - # keep stale metrics around. - @modal.exit() - def cleanup(self): - delete_from_gateway( - self.web_url, - job="hello", - grouping_key={"instance": self.instance_id}, - ) - - @modal.fastapi_endpoint(label="hello-pushgateway") - def hello(self): - self.counter.inc() - push_to_gateway( - self.web_url, - job="hello", - grouping_key={"instance": self.instance_id}, - registry=self.registry, - ) - return f"Hello world from {self.instance_id}!" - - -app.include(gw_app) - -``` - -Now, we can deploy the app and see the metrics in the Pushgateway's web interface. - -```shell -$ modal deploy pushgateway.py -✓ Created objects. -├── 🔨 Created mount /home/ec2-user/modal/examples/10_integrations/pushgateway.py -├── 🔨 Created function ExampleClientApplication.*. -├── 🔨 Created web function serve => https://modal-labs-examples--example-pushgateway-serve.modal.run -└── 🔨 Created web function for ExampleClientApplication.hello => https://modal-labs-examples--hello-pushgateway.modal.run -✓ App deployed! 🎉 -``` - -You can now go to both the [client application](https://modal-labs-examples--hello-pushgateway.modal.run) -and [Pushgateway](https://modal-labs-examples--example-pushgateway-serve.modal.run) URLs to see the metrics being pushed. - -## Hooking up Prometheus - -Now that we have metrics in the Pushgateway, we can configure Prometheus to scrape them. This -is as simple as adding a new job to your Prometheus configuration. Here's an example configuration -snippet: - -```yaml -scrape_configs: -- job_name: 'pushgateway' - honor_labels: true # required so that the instance label is preserved - static_configs: - - targets: ['modal-labs-examples--example-pushgateway-serve.modal.run'] -``` - -Note that the target will be different if you have a custom domain set up for the Pushgateway, -and you may need to configure authentication. - -Once you've added the job to your Prometheus configuration, Prometheus will start scraping metrics -from the Pushgateway. You can then use Grafana or another visualization tool to create dashboards -and alerts based on these metrics! - -![Grafana example](./pushgateway_grafana.png) diff --git a/.agents/skills/modal/references/examples/s3_bucket_mount.md b/.agents/skills/modal/references/examples/s3_bucket_mount.md deleted file mode 100644 index 537d647f8314d66e558b9daab33bf599d8c86425..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/s3_bucket_mount.md +++ /dev/null @@ -1,216 +0,0 @@ -# Analyze NYC yellow taxi data with DuckDB on Parquet files from S3 - -This example shows how to use Modal for a classic data science task: loading table-structured data into cloud stores, -analyzing it, and plotting the results. - -In particular, we'll load public NYC taxi ride data into S3 as Parquet files, -then run SQL queries on it with DuckDB. - -We'll mount the S3 bucket in a Modal app with [`CloudBucketMount`](https://modal.com/docs/reference/modal.CloudBucketMount). -We will write to and then read from that bucket, in each case using -Modal's [parallel execution features](https://modal.com/docs/guide/scale) to handle many files at once. - -## Basic setup - -You will need to have an S3 bucket and AWS credentials to run this example. Refer to the documentation -for the exact [IAM permissions](https://modal.com/docs/guide/cloud-bucket-mounts#iam-permissions) your credentials will need. - -After you are done creating a bucket and configuring IAM settings, -you now need to create a [`Secret`](https://modal.com/docs/guide/secrets) to share -the relevant AWS credentials with your Modal apps. - -```python -from datetime import datetime -from pathlib import Path, PosixPath - -import modal - -image = modal.Image.debian_slim(python_version="3.12").uv_pip_install( - "requests==2.31.0", "duckdb==0.10.0", "matplotlib==3.8.3" -) -app = modal.App("example-s3-bucket-mount", image=image) - -secret = modal.Secret.from_name( - "s3-bucket-secret", - required_keys=["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"], -) - -MOUNT_PATH = PosixPath("/bucket") -YELLOW_TAXI_DATA_PATH = MOUNT_PATH / "yellow_taxi" - -``` - -The dependencies installed above are not available locally. The following block instructs Modal -to only import them inside the container. - -```python -with image.imports(): - import duckdb - import requests - - -``` - -## Download New York City's taxi data - -NYC makes data about taxi rides publicly available. The city's [Taxi & Limousine Commission (TLC)](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page) -publishes files in the Parquet format. Files are organized by year and month. - -We are going to download all available files and store them in an S3 bucket. We do this by -attaching a `modal.CloudBucketMount` with the S3 bucket name and its respective credentials. -The files in the bucket will then be available at `MOUNT_PATH`. - -As we'll see below, this operation can be massively sped up by running it in parallel on Modal. - -```python -@app.function( - volumes={ - MOUNT_PATH: modal.CloudBucketMount("modal-s3mount-test-bucket", secret=secret), - }, -) -def download_data(year: int, month: int) -> str: - filename = f"yellow_tripdata_{year}-{month:02d}.parquet" - url = f"https://d37ci6vzurychx.cloudfront.net/trip-data/{filename}" - s3_path = MOUNT_PATH / filename - # Skip downloading if file exists. - if not s3_path.exists(): - if not YELLOW_TAXI_DATA_PATH.exists(): - YELLOW_TAXI_DATA_PATH.mkdir(parents=True, exist_ok=True) - with requests.get(url, stream=True) as r: - r.raise_for_status() - print(f"downloading => {s3_path}") - # It looks like we writing locally, but this is actually writing to S3! - with open(s3_path, "wb") as file: - for chunk in r.iter_content(chunk_size=8192): - file.write(chunk) - - return s3_path.as_posix() - - -``` - -## Analyze data with DuckDB - -[DuckDB](https://duckdb.org/) is an analytical database with rich support for Parquet files. -It is also very fast. Below, we define a Modal Function that aggregates yellow taxi trips -within a month (each file contains all the rides from a specific month). - -```python -@app.function( - volumes={ - MOUNT_PATH: modal.CloudBucketMount( - "modal-s3mount-test-bucket", - secret=modal.Secret.from_name("s3-bucket-secret"), - ) - }, -) -def aggregate_data(path: str) -> list[tuple[datetime, int]]: - print(f"processing => {path}") - - # Parse file. - year_month_part = path.split("yellow_tripdata_")[1] - year, month = year_month_part.split("-") - month = month.replace(".parquet", "") - - # Make DuckDB query using in-memory storage. - con = duckdb.connect(database=":memory:") - q = """ - with sub as ( - select tpep_pickup_datetime::date d, count(1) c - from read_parquet(?) - group by 1 - ) - select d, c from sub - where date_part('year', d) = ? -- filter out garbage - and date_part('month', d) = ? -- same - """ - con.execute(q, (path, year, month)) - return list(con.fetchall()) - - -``` - -## Plot daily taxi rides - -Finally, we want to plot our results. -The plot created shows the number of yellow taxi rides per day in NYC. -This function runs remotely, on Modal, so we don't need to install plotting libraries locally. - -```python -@app.function() -def plot(dataset) -> bytes: - import io - - import matplotlib.pyplot as plt - - # Sorting data by date - dataset.sort(key=lambda x: x[0]) - - # Unpacking dates and values - dates, values = zip(*dataset) - - # Plotting - plt.figure(figsize=(10, 6)) - plt.plot(dates, values) - plt.title("Number of NYC yellow taxi trips by weekday, 2018-2023") - plt.ylabel("Number of daily trips") - plt.grid(True) - plt.tight_layout() - - # Saving plot as raw bytes to send back - buf = io.BytesIO() - - plt.savefig(buf, format="png") - - buf.seek(0) - - return buf.getvalue() - - -``` - -## Run everything - -The `@app.local_entrypoint()` defines what happens when we run our Modal program locally. -We invoke it from the CLI by calling `modal run s3_bucket_mount.py`. -We first call `download_data()` and `starmap` (named because it's kind of like `map(*args)`) -on tuples of inputs `(year, month)`. This will download, in parallel, -all yellow taxi data files into our locally mounted S3 bucket and return a list of -Parquet file paths. Then, we call `aggregate_data()` with `map` on that list. These files are -also read from our S3 bucket. So one function writes files to S3 and the other -reads files from S3 in; both run across many files in parallel. - -Finally, we call `plot` to generate the following figure: - -![Number of NYC yellow taxi trips by weekday, 2018-2023](./nyc_yellow_taxi_trips_s3_mount.png) - -This program should run in less than 30 seconds. - -```python -@app.local_entrypoint() -def main(): - # List of tuples[year, month]. - inputs = [(year, month) for year in range(2018, 2023) for month in range(1, 13)] - - # List of file paths in S3. - parquet_files: list[str] = [] - for path in download_data.starmap(inputs): - print(f"done => {path}") - parquet_files.append(path) - - # List of datetimes and number of yellow taxi trips. - dataset = [] - for r in aggregate_data.map(parquet_files): - dataset += r - - dir = Path("/tmp") / "s3_bucket_mount" - if not dir.exists(): - dir.mkdir(exist_ok=True, parents=True) - - figure = plot.remote(dataset) - path = dir / "nyc_yellow_taxi_trips_s3_mount.png" - with open(path, "wb") as file: - print(f"Saving figure to {path}") - file.write(figure) - -``` diff --git a/.agents/skills/modal/references/examples/safe_code_execution.md b/.agents/skills/modal/references/examples/safe_code_execution.md deleted file mode 100644 index 2cb55ac3e74ae1f9e596685f9a6f29f2d86be029..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/safe_code_execution.md +++ /dev/null @@ -1,130 +0,0 @@ -# Run arbitrary code in a sandboxed environment - -This example demonstrates how to run arbitrary code -in multiple languages in a Modal [Sandbox](https://modal.com/docs/guide/sandbox). - -## Setting up a multi-language environment - -Sandboxes allow us to run any kind of code in a safe environment. -We'll use an image with a few different language runtimes to demonstrate this. - -```python -import modal - -image = modal.Image.debian_slim(python_version="3.11").apt_install( - "nodejs", "ruby", "php" -) -app = modal.App.lookup("example-safe-code-execution", create_if_missing=True) - -``` - -We'll now create a Sandbox with this image. We'll also enable output so we can see the image build -logs. Note that we don't pass any commands to the Sandbox, so it will stay alive, waiting for us -to send it commands. - -```python -with modal.enable_output(): - sandbox = modal.Sandbox.create(app=app, image=image) - -print(f"Sandbox ID: {sandbox.object_id}") - -``` - -## Running bash, Python, Node.js, Ruby, and PHP in a Sandbox - -We can now use [`Sandbox.exec`](https://modal.com/docs/reference/modal.Sandbox#exec) to run a few different -commands in the Sandbox. - -```python -bash_ps = sandbox.exec("echo", "hello from bash") -python_ps = sandbox.exec("python", "-c", "print('hello from python')") -nodejs_ps = sandbox.exec("node", "-e", 'console.log("hello from nodejs")') -ruby_ps = sandbox.exec("ruby", "-e", "puts 'hello from ruby'") -php_ps = sandbox.exec("php", "-r", "echo 'hello from php';") - -print(bash_ps.stdout.read(), end="") -print(python_ps.stdout.read(), end="") -print(nodejs_ps.stdout.read(), end="") -print(ruby_ps.stdout.read(), end="") -print(php_ps.stdout.read(), end="") -print() - -``` - -The output should look something like - -``` -hello from bash -hello from python -hello from nodejs -hello from ruby -hello from php -``` - -We can use multiple languages in tandem to build complex applications. -Let's demonstrate this by piping data between Python and Node.js using bash. Here -we generate some random numbers with Python and sum them with Node.js. - -```python -combined_process = sandbox.exec( - "bash", - "-c", - """python -c 'import random; print(\" \".join(str(random.randint(1, 100)) for _ in range(10)))' | - node -e 'const readline = require(\"readline\"); - const rl = readline.createInterface({input: process.stdin}); - rl.on(\"line\", (line) => { - const sum = line.split(\" \").map(Number).reduce((a, b) => a + b, 0); - console.log(`The sum of the random numbers is: ${sum}`); - rl.close(); - });'""", -) - -result = combined_process.stdout.read().strip() -print(result) - -``` - -For long-running processes, you can use stdout as an iterator to stream the output. - -```python -slow_printer = sandbox.exec( - "ruby", - "-e", - """ - 10.times do |i| - puts "Line #{i + 1}: #{Time.now}" - STDOUT.flush - sleep(0.5) - end - """, -) - -for line in slow_printer.stdout: - print(line, end="") - -``` - -This should print something like - -``` -Line 1: 2024-10-21 15:30:53 +0000 -Line 2: 2024-10-21 15:30:54 +0000 -... -Line 10: 2024-10-21 15:30:58 +0000 -``` - -Since Sandboxes are safely separated from the rest of our system, -we can run very dangerous code in them! - -```python -sandbox.exec("rm", "-rfv", "/", "--no-preserve-root") - -``` - -This command has deleted the entire filesystem, so we can't run any more commands. -Let's terminate the Sandbox to clean up after ourselves. - -```python -sandbox.terminate() - -``` diff --git a/.agents/skills/modal/references/examples/sandbox_pool.md b/.agents/skills/modal/references/examples/sandbox_pool.md deleted file mode 100644 index aced999ede53ace660903579c3216c9e91217029..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/sandbox_pool.md +++ /dev/null @@ -1,424 +0,0 @@ -# Maintain a pool of warm Sandboxes that are healthy and ready to serve requests - -This example demonstrates how to build a pool of "warm" -[Modal Sandboxes](https://modal.com/docs/guide/sandbox), and deploy a -[Modal Web Function](https://modal.com/docs/guide/webhook-urls) that lets you claim -a Sandbox from the pool, getting a URL to the server running in the Sandbox. - -Maintaining a pool of warm Sandboxes is useful for example if your Sandboxes need -to do significant work after being created, like downloading code, installing -dependencies, or running tests, before they are ready to serve requests. - -It uses a [Modal Queue](https://modal.com/docs/guide/dicts-and-queues#modal-queues) -to store references to the warm Sandboxes, and functionality to maintain the pool -by adding and removing Sandboxes, checking the current size, etc. - -The pool keeps track of the time to live for each Sandbox, and will always return -a Sandbox with enough time left. - -Each Sandbox is configured with a -[readiness probe](https://modal.com/docs/guide/sandbox#readiness-probes) so we can -reliably wait for the server to be ready before adding it to the pool. - -It's structured into two Apps: - -* `example-sandbox-pool` is the main App that contains all the control logic for maintaining - the pool, exposing ways to claim Sandboxes, etc. -* `example-sandbox-pool-sandboxes` houses all the actual Sandboxes, and nothing else. - -The implementation borrows from [pawalt](https://github.com/pawalt)'s [Sandbox pool -example gist](https://gist.github.com/pawalt/7a505c38bba75cafae0780a5dd40e8b8). 🙏 - -```python -import argparse -import time -from dataclasses import dataclass -from datetime import datetime - -import modal - -app = modal.App("example-sandbox-pool") - -server_image = modal.Image.debian_slim(python_version="3.11").uv_pip_install( - "fastapi[standard]~=0.115.14", - "requests~=2.32.4", -) - -## Configuration of the pool - -``` - -Here we define the image that will be used to run the server that runs in the -Sandbox. In this simple example, we just run the built in Python HTTP server that -returns a directory listing. - -```python -sandbox_image = modal.Image.debian_slim(python_version="3.11").apt_install("curl") -SANDBOX_SERVER_PORT = 8080 -READINESS_PROBE_TIMEOUT_SECONDS = 10 - -``` - -In this example Sandboxes live for 5 minutes, and we assume that they are used for -2 minutes, meaning that if a Sandbox has less than 2 minutes left it's considered -to be expiring too soon and will be terminated. - -You'll want to adjust these values depending on your use case. - -```python -SANDBOX_TIMEOUT_SECONDS = 5 * 60 -SANDBOX_USE_DURATION_SECONDS = 2 * 60 -POOL_SIZE = 3 -POOL_MAINTENANCE_SCHEDULE = modal.Period(minutes=2) - - -``` - -## Main implementation - -We keep track of all warm Sandboxes in a Modal Queue of `SandboxReference` objects. - -```python -pool_queue = modal.Queue.from_name( - "example-sandbox-pool-sandboxes", create_if_missing=True -) - - -@dataclass -class SandboxReference: - id: str - url: str - expires_at: int - - -``` - -### Health check - -We add a simple health check that just ensures that the server in the Sandbox is -running and responding to requests. - -If you just want to ensure the sandbox is running you could for example check -`sb.poll() is not None` instead. - -```python -def is_healthy(url: str) -> bool: - """Check if a Sandbox is healthy by verifying the server responds to requests.""" - import requests - - try: - response = requests.get(url, timeout=10) - response.raise_for_status() - return True - except requests.RequestException: - return False - - -def is_still_good(sr: SandboxReference, check_health: bool) -> bool: - """Check if a Sandbox is still good to use. - - It assumes that it's already been added to the pool, so we don't wait for the - container to start. - """ - if sr.expires_at < time.time() + SANDBOX_USE_DURATION_SECONDS: - return False - - if check_health and not is_healthy(sr.url): - return False - - return True - - -``` - -### Adding a Sandbox to the pool - -This function creates and adds a new Sandbox to the pool. It waits for the -Sandbox's readiness probe to pass before adding it, ensuring the server is -ready to serve requests. - -We deploy the Sandboxes in a separate Modal App called `example-sandbox-pool-sandboxes`, -to separate the control app (logs, etc.) from the Sandboxes. - -```python -@app.function(image=server_image, retries=3) -@modal.concurrent(max_inputs=20) -def add_sandbox_to_queue() -> None: - sandbox_app = modal.App.lookup( - "example-sandbox-pool-sandboxes", create_if_missing=True - ) - - sandbox_cmd = ["python", "-m", "http.server", "8080"] - sb = modal.Sandbox.create( - *sandbox_cmd, - app=sandbox_app, - image=sandbox_image, - encrypted_ports=[SANDBOX_SERVER_PORT], - timeout=SANDBOX_TIMEOUT_SECONDS, - readiness_probe=modal.Probe.with_exec( - "curl", "-sf", f"http://localhost:{SANDBOX_SERVER_PORT}/" - ), - ) - expires_at = int(time.time()) + SANDBOX_TIMEOUT_SECONDS - sb.wait_until_ready(timeout=READINESS_PROBE_TIMEOUT_SECONDS) - url = sb.tunnels()[SANDBOX_SERVER_PORT].url - - pool_queue.put(SandboxReference(id=sb.object_id, url=url, expires_at=expires_at)) - sb.detach() - - -``` - -We also have a utility function that can be `.spawn()`ed to terminate Sandboxes. - -```python -@app.function() -def terminate_sandboxes(sandbox_ids: list[str]) -> int: - num_terminated = 0 - for id in sandbox_ids: - sb = modal.Sandbox.from_id(id) - sb.terminate() - sb.detach() - num_terminated += 1 - - print(f"Terminated {num_terminated} Sandboxes") - return num_terminated - - -``` - -### Claiming a Sandbox from the pool - -We expose two ways to claim a Sandbox from the pool and get a URL to the server: - -* a public Web Function that can be addressed via HTTP -* a Function that can be called using the Modal SDK (including from [Go or JS][1]). - -[1]: https://modal.com/docs/guide/sdk-javascript-go - -The Web Function proxies to `claim_sandbox` using a `.local()` invocation, -which runs in the same container without additional latency. - -```python -@app.function(image=server_image) -@modal.fastapi_endpoint() -@modal.concurrent(max_inputs=20) -def claim_sandbox_web_function(check_health: bool = True) -> str: - return claim_sandbox.local(check_health=check_health) - - -@app.function(image=server_image) -def claim_sandbox(check_health: bool = True) -> str: - to_terminate: list[str] = [] - - # Remove any expiring or unhealthy sandboxes, and return the first good one: - while True: - print( - "Adding a new Sandbox to the pool to backfill " - "(and ensure we have at least one)..." - ) - add_sandbox_to_queue.spawn() - - # timeout=None here means we block in case we need to wait for the backfill: - sr = pool_queue.get(timeout=None) - if sr is None: - continue - - if not is_still_good(sr, check_health): - print(f"Sandbox '{sr.id}' was not good - terminating and trying another...") - to_terminate.append(sr.id) - continue - - break - - if to_terminate: - terminate_sandboxes.spawn(to_terminate) - - print(f"Claimed Sandbox '{sr.id}', with URL: {sr.url}") - return sr.url - - -``` - -### Maintaining the pool - -This function grows or shrinks the pool to SANDBOX\_POOL\_SIZE. It first removes any -expiring or unhealthy sandboxes, then adjusts the pool size to reach the target. - -It runs on a schedule to ensure the pool doesn't drift too far from the target size. - -```python -@app.function( - image=server_image, - schedule=POOL_MAINTENANCE_SCHEDULE, -) -def maintain_pool(): - to_terminate: list[str] = [] - - # First remove expiring and unhealthy sandboxes - while True: - sr = pool_queue.get(block=False) - - if sr is None: - break - - if not is_still_good(sr, check_health=True): - to_terminate.append(sr.id) - continue - - # Found first good sandbox, but don't put it back in the queue to preserve - # queue ordering. - to_terminate.append(sr.id) - break - - if to_terminate: - print(f"Terminating {len(to_terminate)} expiring/unhealthy sandboxes...") - terminate_sandboxes.spawn(to_terminate) - - # Now resize to target - diff = POOL_SIZE - pool_queue.len() - - if diff > 0: - for _ in add_sandbox_to_queue.starmap(() for _ in range(diff)): - pass - elif diff < 0: - terminate_sandboxes.spawn( - [sr.id for sr in pool_queue.get_many(n_values=-diff, timeout=0)] - ) - - print(f"Pool size after maintenance: {pool_queue.len()}") - - -``` - -## Local commands for interacting with the pool - -### Deploy the app - -This also runs the `maintain_pool` function to ensure the pool is at the correct size -without having to wait for the first scheduled maintenance run. - -Run it with `python 13_sandboxes/sandbox_pool.py deploy`. - -```python -def deploy(): - print("Deploying the app...") - app.deploy() - print("Done.") - - print("\nRunning initial pool maintenance...") - maintain_pool.remote() - print("Done.") - - -``` - -### Check the current state of the pool - -Run it with `python 13_sandboxes/sandbox_pool.py check`. - -```python -def check(): - print(f"Number of Sandboxes in the pool: {pool_queue.len()}") - - for sr in pool_queue.iterate(): - seconds_left = sr.expires_at - time.time() - print( - f"- Sandbox '{sr.id}' is at {sr.url} and expires at " - f"{datetime.fromtimestamp(sr.expires_at).isoformat()} " - f"({int(seconds_left)} seconds left)" - ) - - -``` - -### Claiming a Sandbox from the pool and print its URL - -This is implemented as if you wanted to call the Function from a Python backend -application using the Modal SDK, i.e. using `.from_name()` to get the Function, etc. - -Run it with `python 13_sandboxes/sandbox_pool.py claim`. - -```python -def claim() -> None: - deployed_claim_sandbox = modal.Function.from_name( - "example-sandbox-pool", "claim_sandbox" - ) - print(deployed_claim_sandbox.remote()) - - -``` - -### Run a demo of the Sandbox pool. - -This is implemented as if you wanted to call the Function from a Python backend -application using the Modal SDK, i.e. using `.from_name()` to get the Function, etc. - -Run it with `python 13_sandboxes/sandbox_pool.py demo`. - -```python -def demo(): - import urllib.request - - deploy() - - check() - - print("\nClaiming a Sandbox using the `claim_sandbox` Function...") - deployed_claim_sandbox = modal.Function.from_name( - "example-sandbox-pool", "claim_sandbox" - ) - sandbox_url = deployed_claim_sandbox.remote() - print(f"Claimed Sandbox URL: {sandbox_url}") - - print("\nCall the server in the Sandbox...") - with urllib.request.urlopen(sandbox_url) as response: - result = response.read().decode("utf-8") - print(f"Sandbox server response:\n{result}") - - time.sleep(2) # wait for the pool to be backfilled in the background - check() - - deployed_web_function = modal.Function.from_name( - "example-sandbox-pool", "claim_sandbox_web_function" - ) - claim_url = deployed_web_function.get_web_url() - print(f"\nClaiming a Sandbox using the Function at '{claim_url}'...") - with urllib.request.urlopen(claim_url) as response: - sandbox_url = response.read().decode("utf-8").strip(' "') - print(f"Claimed Sandbox URL: {sandbox_url}") - - print("\nCall the server in the Sandbox...") - with urllib.request.urlopen(sandbox_url) as response: - result = response.read().decode("utf-8") - print(f"Sandbox server response:\n{result}") - - time.sleep(2) - check() - - -def main(): - parser = argparse.ArgumentParser(description="Manage Sandbox pool") - parser.add_argument( - "command", - choices=["check", "deploy", "claim", "demo"], - help="Command to execute", - ) - args = parser.parse_args() - - if args.command == "check": - check() - elif args.command == "claim": - claim() - elif args.command == "deploy": - deploy() - elif args.command == "demo": - demo() - else: - parser.print_help() - - -if __name__ == "__main__": - main() - -``` diff --git a/.agents/skills/modal/references/examples/serve_streamlit.md b/.agents/skills/modal/references/examples/serve_streamlit.md deleted file mode 100644 index 588f9f99fc832215c8bef9f9e088bb592e302eb0..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/serve_streamlit.md +++ /dev/null @@ -1,83 +0,0 @@ -# Run and share Streamlit apps - -This example shows you how to run a Streamlit app with `modal serve`, and then deploy it as a serverless web app. - -![example streamlit app](./streamlit.png) - -This example is structured as two files: - -1. This module, which defines the Modal objects (name the script `serve_streamlit.py` locally). - -2. `app.py`, which is any Streamlit script to be mounted into the Modal - function ([download script](https://github.com/modal-labs/modal-examples/blob/main/10_integrations/streamlit/app.py)). - -```python -import shlex -import subprocess -from pathlib import Path - -import modal - -``` - -## Define container dependencies - -The `app.py` script imports three third-party packages, so we include these in the example's -image definition and then add the `app.py` file itself to the image. - -```python -streamlit_script_local_path = Path(__file__).parent / "app.py" -streamlit_script_remote_path = "/root/app.py" - -image = ( - modal.Image.debian_slim(python_version="3.11") - .uv_pip_install("streamlit~=1.35.0", "numpy~=1.26.4", "pandas~=2.2.2") - .add_local_file( - streamlit_script_local_path, - streamlit_script_remote_path, - ) -) - -app = modal.App(name="example-serve-streamlit", image=image) - -if not streamlit_script_local_path.exists(): - raise RuntimeError( - "app.py not found! Place the script with your streamlit app in the same directory." - ) - -``` - -## Spawning the Streamlit server - -Inside the container, we will run the Streamlit server in a background subprocess using -`subprocess.Popen`. We also expose port 8000 using the `@web_server` decorator. - -```python -@app.function() -@modal.concurrent(max_inputs=100) -@modal.web_server(8000) -def run(): - target = shlex.quote(streamlit_script_remote_path) - cmd = f"streamlit run {target} --server.port 8000 --server.enableCORS=false --server.enableXsrfProtection=false" - subprocess.Popen(cmd, shell=True) - - -``` - -## Iterate and Deploy - -While you're iterating on your screamlit app, you can run it "ephemerally" with `modal serve`. This will -run a local process that watches your files and updates the app if anything changes. - -```shell -modal serve serve_streamlit.py -``` - -Once you're happy with your changes, you can deploy your application with - -```shell -modal deploy serve_streamlit.py -``` - -If successful, this will print a URL for your app that you can navigate to from -your browser 🎉 . diff --git a/.agents/skills/modal/references/examples/sglang_low_latency.md b/.agents/skills/modal/references/examples/sglang_low_latency.md deleted file mode 100644 index 9ccfd1a36a855ea92f80aab4daac4b2aad6ad0c0..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/sglang_low_latency.md +++ /dev/null @@ -1,638 +0,0 @@ -# Low latency Qwen 3.6 with SGLang and Modal - -In this example, we show how to serve [SGLang](https://github.com/sgl-project/sglang) at low latency on Modal. - -This example is intended to demonstrate everything required to run -inference at the highest performance and with the lowest latency possible, -and so it includes advanced features of both SGLang and Modal. -For a simpler introduction to LLM serving, see -[this example](https://modal.com/docs/examples/llm_inference). - -To minimize routing overheads, we use `@modal.experimental.http_server`, -which uses a new, low-latency routing service on Modal designed for latency-sensitive inference workloads. -This gives us more control over routing, but with increased power comes increased responsibility. - -## Set up the container image - -Our first order of business is to define the environment our server will run in: -the [container `Image`](https://modal.com/docs/guide/images). - -We start from a container image provided -[by the SGLang team via Dockerhub](https://hub.docker.com/r/lmsysorg/sglang/tags). -After a bit of cleanup, we install an updated version that has some low-latency tricks -the Modal team is contributing to SGLang (described below). - -While we're at it, we import the dependencies we'll need both remotely and locally (for deployment). - -```python -import asyncio -import json -import os -import subprocess -import time - -import aiohttp -import modal -import modal.experimental - -MINUTES = 60 # seconds -GIT_SHA = "5244693e308eaf05da17f28cca6bcc922270fd3c" - -sglang_image = ( - modal.Image.from_registry("lmsysorg/sglang:v0.5.12.post1-cu130") - .entrypoint( - [] # silence chatty logs on container start - ) - .run_commands( - "rm -rf /root/.cache/huggingface" # clean up image - ) - .uv_pip_install( - f"git+https://github.com/sgl-project/sglang.git@{GIT_SHA}#subdirectory=python" - ) -) - -``` - -We also choose a [GPU](https://modal.com/docs/guide/gpu) to deploy our inference server onto. -We choose the [H100 GPU](https://modal.com/blog/introducing-h100), -which offers excellent price-performance -and supports [8bit floating point operations](https://modal.com/llm-almanac/quant-formats), which are the -lowest precision well-supported in the relevant [GPU kernels](https://modal.com/gpu-glossary/device-software/kernel) -across a variety of model architectures. - -Below, we discuss the choice of GPU count. - -```python -GPU_TYPE, N_GPUS = "H100!", 2 -GPU = f"{GPU_TYPE}:{N_GPUS}" - -``` - -### Loading and cacheing the model weights - -We'll serve [Alibaba's Qwen 3.6 LLM](https://qwen.ai/blog?id=qwen3.6). -For lower latency, we pick the 35B mixture-of-experts model with 3B active parameters -in a lower precision floating point format (FP8). -Expert sparsity and lower precision reduce the amount of data that needs to be loaded -[from GPU RAM into SM SRAM](https://modal.com/gpu-glossary/perf/memory-bandwidth) -in each forward pass. - -```python -MODEL_NAME = "Qwen/Qwen3.6-35B-A3B-FP8" -MODEL_REVISION = ( # pin revision id to avoid nasty surprises! - "95a723d08a9490559dae23d0cff1d9466213d989" # latest commit as of 2026-04-23, from release -) - -``` - -We load the model [from the Hugging Face Hub](https://huggingface.co/collections/Qwen/qwen36). - -But we don't want to load the model from the Hub every time we start the server. -We can load it much faster from a [Modal Volume](https://modal.com/docs/guide/volumes). -Typical speeds are around one to two GB/s. - -```python -HF_CACHE_VOL = modal.Volume.from_name("huggingface-cache", create_if_missing=True) -HF_CACHE_PATH = "/root/.cache/huggingface" -MODEL_PATH = f"{HF_CACHE_PATH}/{MODEL_NAME}" - -``` - -In addition to pointing the Hugging Face Hub at the path -where we mount the Volume, we also -[turn on "high performance" downloads](https://huggingface.co/docs/hub/en/models-downloading#faster-downloads), -which can fully saturate our network bandwidth. - -```python -sglang_image = sglang_image.env( - {"HF_HUB_CACHE": HF_CACHE_PATH, "HF_XET_HIGH_PERFORMANCE": "1"} -) - -``` - -### Cacheing compilation artifacts - -Model weights aren't the only thing we want to cache. - -As a rule, LLM inference servers like SGLang don't directly provide their own kernels. -They draw high-performance kernels from a variety of sources. - -As of version `0.5.12`, SGLang's default kernel backend -for FP8 matrix multiplications (`fp8-gemm-backend`) -on Hopper [SM architecture](https://modal.com/gpu-glossary/device-hardware/streaming-multiprocessor-architecture) -GPUs like the H100 is -[DeepGEMM](https://github.com/deepseek-ai/DeepGEMM) -by DeepSeek. - -The binaries of these kernels are not included in the SGLang Docker image and so -must be [JIT-compiled](https://modal.com/gpu-glossary/host-software/nvrtc). -We store these in a Modal Volume as well. - -```python -DG_CACHE_VOL = modal.Volume.from_name("deepgemm-cache", create_if_missing=True) -DG_CACHE_PATH = "/root/.cache/deep_gemm" - -``` - -JIT DeepGEMM kernels are on by default, but we explicitly enable them via an environment variable. - -```python -sglang_image = sglang_image.env({"SGLANG_ENABLE_JIT_DEEPGEMM": "1"}) - -``` - -We trigger the compilation by running `sglang.compile_deep_gemm` in a `subprocess` -kicked off from a Python function. - -```python -def compile_deep_gemm(): - import os - - if int(os.environ.get("SGLANG_ENABLE_JIT_DEEPGEMM", "1")): - subprocess.run( - f"python3 -m sglang.compile_deep_gemm --model-path {MODEL_NAME} --revision {MODEL_REVISION} --tp {N_GPUS}", - shell=True, - ) - - -``` - -We run this Python function on Modal as part of building the Image -so that it has access to the appropriate GPU and the caches for our model and compilaton artifacts. - -```python -sglang_image = sglang_image.run_function( - compile_deep_gemm, - volumes={DG_CACHE_PATH: DG_CACHE_VOL, HF_CACHE_PATH: HF_CACHE_VOL}, - gpu=GPU, -) - -``` - -## Configure SGLang for minimal latency - -LLM inference engines like SGLang come with a wide variety of "knobs" to tune performance. - -To determine the appropriate configuration to hit latency and throughput service objectives, -we recommend [application-specific benchmarking](https://modal.com/llm-almanac/how-to-benchmark) -guided by [published generic benchmarks](https://modal.com/llm-almanac/advisor). - -Here, we assume that the primary goal is to minimize per-request latency, with less regard to throughput -(and so to cost) and walk through some of the key choices. - -The primary contributor to per-request latency is the time to move all of the model's weights (multiple gigabytes) -from [GPU RAM](https://modal.com/gpu-glossary/device-hardware/gpu-ram) -into [SRAM in the Streaming Multiprocessors](https://modal.com/gpu-glossary/device-hardware/streaming-multiprocessor), -which must be done at least once in the course of processing a request -- -naively, once per token per request. -The time taken is limited by the -[memory bandwidth](https://modal.com/gpu-glossary/perf/memory-bandwidth) -between those two stores, which is on the order of terabytes per second on modern data center GPUs. -With models at the scale of gigabytes, a token will take milliseconds to generate -- -or whole seconds for the kilotoken responses users are accustomed to. - -We use two strategies to cut latency in our [memory-bound](https://modal.com/gpu-glossary/perf/memory-bound) workload: - -* operate across multiple GPUs for more aggregate bandwidth and faster loads, with tensor parallelism - -* generate more tokens per load, with speculative decoding - -### Increasing effective memory bandwidth with tensor parallelism - -Running SGLang on two H100s will double our effective -[memory bandwidth](https://modal.com/gpu-glossary/perf/memory-bandwidth) -during large matrix multiplications. - -Matrices are also known as tensors, and so this strategy that takes advantage -of the inherent parallelism within matrix multiplication is known as *tensor parallelism*. - -Actual speedups are generally less than what you get from "napkin math" based on available bandwidths -- -we observed a speedup of about 30% moving from one to two H100s when developing this example, rather than 100%. - -### Parallelizing token generation with speculative decoding - -Transformer and recurrent language models generate text sequentially: -the model's output at step `i` is part of the input at step `i+1`. -Per Amdahl's Law, that sequential work becomes the bottleneck -as other steps get faster from increased parallelism. - -The solution is to generate more tokens on each step. -The primary technique to do so without changing model behavior is known as -[*speculative decoding*](https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/), -which "speculates" a number of draft tokens and verifies them in parallel with the primary model. - -Speculative decoding techniques themselves have a number of parameters, the most important -of which is the technique to use to generate draft tokens. -Simple techniques based on n-grams are a good place to start. -Many models are released with built-in speculation based on -[multi-token prediction](https://docs.vllm.ai/projects/ascend/en/main/user_guide/feature_guide/Multi_Token_Prediction.html), -also known in SGLang as [EAGLE](https://arxiv.org/abs/2401.15077). - -But our favorite technique is [DFLASH](https://arxiv.org/abs/2602.06036) -which runs draft token generation in parallel, increasing the draft model's -arithmetic intensity. - -```python -speculative_config = { - "speculative-algorithm": "DFLASH", - "speculative-draft-model-path": "z-lab/Qwen3.6-35B-A3B-DFlash", - "speculative-draft-model-revision": "42d3b34d588423cdae7ba8f53a8cf7789346a719", - "mamba-scheduler-strategy": "extra_buffer", # required for spec dec with Qwen 3.X hybrid arch -} - -``` - -We adapt the default configuration for this speculator from [the model card](https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash). -In particular, we use a smaller draft token count of `8`, the minimum, -rather than `16`, the default. We're using the FP8 quantized model here -and the test prompts are creative writing tasks, so accept lengths -are generally below `8` and additional blocks don't have enough -accepted tokens to be worth the extra computation. - -```python -speculative_config |= { - "speculative-num-draft-tokens": 8, -} - -speculative_env = { - "SGLANG_ENABLE_OVERLAP_PLAN_STREAM": "1", # never block the GPU! -} - -``` - -Note that unlike tensor parallelism, -speculative decoding is not good for -[compute-bound](https://modal.com/gpu-glossary/perf/compute-bound) -workloads, since it generally increases demand for -[arithmetic bandwidth](https://modal.com/gpu-glossary/perf/arithmetic-bandwidth). -So for workloads that admit larger batch sizes for requests, -on the scale of dozens to hundreds, speculative decoding is not recommended. - -## Define the inference server and infrastructure - -### Selecting infrastructure to minimize latency - -Minimizing latency requires geographic co-location of clients and servers. - -So for low latency LLM inference services on Modal, you must select a -[cloud region](https://modal.com/docs/guide/region-selection) -for both the GPU-accelerated containers running inference -and for the internal Modal proxies that forward requests to them -as part of defining a `modal.experimental.http_server`. - -Here, we assume users are mostly in the northern half of the Americas -and select the `us` cloud region serve them. -This should result in at most a few dozen milliseconds of round-trip time. - -```python -REGION = "us" - -``` - -Latencies for mutli-turn interactions with LLMs are -substantially cut when previous interaction turns are in the KV cache. -KV caches are stored in [GPU RAM](https://modal.com/gpu-glossary/device-hardware/gpu-ram), -so they aren't shared across replicas. -To improve cache hit rate, `modal.experimental.http_server` -includes sticky routing based on a client-provided header. -See the client code below for details. - -For production-scale LLM inference services, there are generally -enough requests to justify keeping at least one replica running at all times. -Having a "warm" or "live" replica reduces latency by skipping slow initialization work -that occurs when new replica boots up (a ["cold start"](https://modal.com/docs/guide/cold-start)). -For LLM inference servers, that latency runs from seconds to minutes. - -To ensure at least one container is always available, -we can set the `min_containers` of our Modal Function -to `1` or more. - -However, since this is documentation code, we'll set it to `0` -to avoid surprise bills during casual use. - -```python -MIN_CONTAINERS = 0 # set to 1 to ensure one replica is always ready - -``` - -Finally, we need to decide how we will scale up and down replicas -in response to load. Without autoscaling, users' requests will queue -when the server becomes overloaded. Even apart from queueing, responses -generally become slower per user above a certain minimum number of -concurrent requests. - -So we set a target for the number of inputs to run on a single container -with [`modal.concurrent`](https://modal.com/docs/reference/modal.concurrent). -For details, see [the guide](https://modal.com/docs/guide/concurrent-inputs). - -```python -TARGET_INPUTS = 10 - -``` - -Generally, this choice needs to be made as part of -[LLM inference engine benchmarking](https://modal.com/llm-almanac/how-to-benchmark). - -### Controlling container lifecycles with `modal.Cls` - -We wrap up all of the choices we made about the infrastructure -of our inference server into a number of Python decorators -that we apply to a Python class that encapsulates the logic -to run our server. - -The key decorators are: - -* [`@app.cls`](https://modal.com/docs/guide/lifecycle-functions) to define the core of our service. - We attach our Image, request a GPU, attach our cache Volumes, specify the region, and configure auto-scaling. - See [the reference documentation](https://modal.com/docs/reference/modal.App#cls) for details. - -* `@modal.experimental.http_server` to turn our Python code into an HTTP server - (i.e. fronting all of our containers with a proxy with a URL). The wrapped code - needs to eventually listen for HTTP connections on the provided `port`. - The proxy is located in a specific region as well; we choose `us-west`. - -* [`@modal.concurrent`](https://modal.com/docs/guide/concurrent-inputs) to specify how many - requests our server can handle before we need to scale up. - -* [`@modal.enter` and `@modal.exit`](https://modal.com/docs/guide/lifecycle-functions) to indicate - which methods of the class should be run when starting the server and shutting it down. - -Modal considers a new replica ready to receive inputs once the `modal.enter` methods have exited -and the container accepts connections. -To ensure that we actually finish setting up our server before we are marked ready for inputs, -we define a helper function to check whether the server is finished setting up and to -send it a few test inputs. - -We use the [`requests` library](https://requests.readthedocs.io/en/latest/) -to send ourselves these HTTP requests on -[`localhost`/`127.0.0.1`](https://superuser.com/questions/31824/why-is-localhost-ip-127-0-0-1). - -```python -with sglang_image.imports(): - import requests - - -def wait_ready(process: subprocess.Popen, timeout: int = 20 * MINUTES): - deadline = time.time() + timeout - while time.time() < deadline: - try: - check_running(process) - requests.get(f"http://127.0.0.1:{PORT}/health").raise_for_status() - return - except ( - subprocess.CalledProcessError, - requests.exceptions.ConnectionError, - requests.exceptions.HTTPError, - ): - time.sleep(5) - raise TimeoutError(f"SGLang server not ready within {timeout} seconds") - - -def check_running(p: subprocess.Popen): - if (rc := p.poll()) is not None: - raise subprocess.CalledProcessError(rc, cmd=p.args) - - -def warmup(): - payload = { - "messages": [{"role": "user", "content": "Hello, how are you?"}], - "max_tokens": 16, - } - for _ in range(3): - requests.post( - f"http://127.0.0.1:{PORT}/v1/chat/completions", json=payload, timeout=10 - ).raise_for_status() - - -``` - -With all this in place, we are ready to define our high-performance, low-latency -LLM inference server. - -```python -app = modal.App(name="example-sglang-low-latency") -PORT = 8000 -PROXY_REGION = "us-west" - - -@app.cls( - image=sglang_image, - gpu=GPU, - volumes={HF_CACHE_PATH: HF_CACHE_VOL, DG_CACHE_PATH: DG_CACHE_VOL}, - region=REGION, - min_containers=MIN_CONTAINERS, - startup_timeout=20 * MINUTES, -) -@modal.experimental.http_server( - port=PORT, # wrapped code must listen on this port - proxy_regions=[PROXY_REGION], # location of proxies, should be close to Cls region - exit_grace_period=15, # seconds, time to finish up requests when closing down -) -@modal.concurrent(target_inputs=TARGET_INPUTS) -class SGLang: - @modal.enter() - def startup(self): - """Start the SGLang server and block until it is healthy, then warm it up and put it to sleep.""" - cmd = [ - "python", - "-m", - "sglang.launch_server", - "--model-path", - MODEL_NAME, - "--revision", - MODEL_REVISION, - "--served-model-name", - MODEL_NAME, - "--host", - "0.0.0.0", - "--port", - f"{PORT}", - "--tp", # use all GPUs to split up tensor-parallel operations - f"{N_GPUS}", - "--cuda-graph-max-bs", # only capture CUDA graphs for batch sizes we're likely to observe - f"{TARGET_INPUTS * 2}", - "--enable-metrics", # expose metrics endpoints for telemetry - "--decode-log-interval", # how often to log during decoding, in tokens - "10", - "--mem-fraction", # leave space for speculative model - "0.8", - "--trust-remote-code", # for speculative model - ] - - cmd += [ # add speculative config - item for k, v in speculative_config.items() for item in (f"--{k}", str(v)) - ] - - self.process = subprocess.Popen(cmd, env=os.environ | speculative_env) - wait_ready(self.process) - warmup() - - @modal.exit() - def stop(self): - self.process.terminate() - - -``` - -## Deploy the server - -To deploy the server on Modal, just run - -```bash -modal deploy sglang_low_latency.py -``` - -This will create a new App on Modal and build the container image for it if it hasn't been built yet. - -## Interact with the server - -Once it is deployed, you'll see a URL appear in the command line, -something like `https://your-workspace-name--example-sglang-low-latency-sglang.us-west.modal.direct`. - -You can find [interactive Swagger UI docs](https://swagger.io/tools/swagger-ui/) -at the `/docs` route of that URL, i.e. `https://your-workspace-name--example-sglang-low-latency-sglang.us-west.modal.direct/docs`. -These docs describe each route and indicate the expected input and output -and translate requests into `curl` commands. -For simple routes, you can even send a request directly from the docs page. - -Note: when no replicas are available, Modal will respond with -the [503 Service Unavailable status](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/503). -In your browser, you can just hit refresh until the docs page appears. -You can see the status of the applicaton and its containers on your [Modal dashboard](https://modal.com/apps). - -## Test the server - -To make it easier to test the server setup, we also include a `local_entrypoint` -that hits the server with a simple client. - -If you execute the command - -```bash -modal run sglang_low_latency.py -``` - -a fresh replica of the server will be spun up on Modal while -the code below executes on your local machine. - -Think of this like writing simple tests inside of the `if __name__ == "__main__"` -block of a Python script, but for cloud deployments! - -```python -@app.local_entrypoint() -async def test(test_timeout=10 * MINUTES, prompt=None, twice=True): - url = (await SGLang._experimental_get_flash_urls.aio())[0] - - system_prompt = { - "role": "system", - "content": "You are a pirate who can't help but drop sly reminders that he went to Harvard.", - } - if prompt is None: - prompt = "Explain the Singular Value Decomposition." - - content = [{"type": "text", "text": prompt}] - - messages = [ # OpenAI chat format - system_prompt, - {"role": "user", "content": content}, - ] - - await probe(url, messages, timeout=test_timeout) - if twice: - messages[0]["content"] = "You are Jar Jar Binks." - print(f"Sending messages to {url}:", *messages, sep="\n\t") - await probe(url, messages, timeout=test_timeout) - - -``` - -This test relies on the two helper functions below, -which ping the server and wait for a valid response to stream. - -The `probe` helper function specifically ignores -two types of errors that can occur while a replica -is starting up -- timeouts on the client and 5XX responses from the server. -Modal returns the [503 Service Unavailable status](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/503) -when an `experimental.http_server` has no live replicas. - -We include a header with each request -- -`Modal-Session-ID`. -This is header is used by clients of `http_server`s on Modal -to identify which requests should be routed to the same container -(with caveats explained below). - -The value associated with this key -is used to map requests onto containers such that -while the set of containers is fixed, requests with the same value -are sent to the same container. -Set this to a different value per distinct multi-turn interaction -(prototypically, a user conversation thread with a chatbot) -to improve KV cache hit rates. -Additionally, when the set of containers changes (e.g. due to autoscaling), -sessions are rebalanced such that load is approximately evenly spread, -much like in [RAID rebalancing](https://cordero.me/understanding-raid-rebalance-ensuring-optimal-performance-and-data-protection/). -This ensures no container ends up as a "hot spot" handling too many client requests. - -```python -async def probe(url, messages=None, timeout=5 * MINUTES): - if messages is None: - messages = [{"role": "user", "content": "Tell me a joke."}] - - client_id = str(0) # set this to some string per multi-turn interaction - # often a UUID per "conversation" - headers = {"Modal-Session-ID": client_id} - deadline = time.time() + timeout - async with aiohttp.ClientSession(base_url=url, headers=headers) as session: - while time.time() < deadline: - try: - await _send_request_streaming(session, messages) - return - except asyncio.TimeoutError: - await asyncio.sleep(1) - except aiohttp.client_exceptions.ClientResponseError as e: - if e.status == 503: - await asyncio.sleep(1) - continue - raise e - raise TimeoutError(f"No response from server within {timeout} seconds") - - -async def _send_request_streaming( - session: aiohttp.ClientSession, messages: list, timeout: int | None = None -) -> None: - payload = {"messages": messages, "stream": True} - headers = {"Accept": "text/event-stream"} - - async with session.post( - "/v1/chat/completions", json=payload, headers=headers, timeout=timeout - ) as resp: - resp.raise_for_status() - full_text = "" - - async for raw in resp.content: - line = raw.decode("utf-8", errors="ignore").strip() - if not line: - continue - - # Server-Sent Events format: "data: ...." - if not line.startswith("data:"): - continue - - data = line[len("data:") :].strip() - if data == "[DONE]": - break - - try: - evt = json.loads(data) - except json.JSONDecodeError: - # ignore any non-JSON keepalive - continue - - delta = (evt.get("choices") or [{}])[0].get("delta") or {} - chunk = delta.get("content") - - if chunk: - print(chunk, end="", flush="\n" in chunk or "." in chunk) - full_text += chunk - print() # newline after stream completes - -``` diff --git a/.agents/skills/modal/references/examples/simple_code_interpreter.md b/.agents/skills/modal/references/examples/simple_code_interpreter.md deleted file mode 100644 index b85eb9d573a85037ebd94c759396ada0845efb0b..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/simple_code_interpreter.md +++ /dev/null @@ -1,141 +0,0 @@ -# Build a stateful, sandboxed code interpreter - -This example demonstrates how to build a stateful code interpreter using a Modal -[Sandbox](https://modal.com/docs/guide/sandbox). - -We'll create a Modal Sandbox that listens for code to execute and then -executes the code in a Python interpreter. Because we're running in a sandboxed -environment, we can safely use the "unsafe" `exec()` to execute the code. - -## Setting up a code interpreter in a Modal Sandbox - -Our code interpreter uses a Python "driver program" to listen for code -sent in JSON format to its standard input (`stdin`), execute the code, -and then return the results in JSON format on standard output (`stdout`). - -```python -import inspect -import json -import sys -from typing import Any, Iterator - -import modal - - -def driver_program(): - import json - import sys - from contextlib import redirect_stderr, redirect_stdout - from io import StringIO - - # When you `exec` code in Python, you can pass in a dictionary - # that defines the global variables the code has access to. - - # We'll use that to store state. - - globals: dict[str, Any] = {} - while True: - command = json.loads(input()) # read a line of JSON from stdin - if (code := command.get("code")) is None: - print(json.dumps({"error": "No code to execute"})) - continue - - # Capture the executed code's outputs - stdout_io, stderr_io = StringIO(), StringIO() - with redirect_stdout(stdout_io), redirect_stderr(stderr_io): - try: - exec(code, globals) - except Exception as e: - print(f"Execution Error: {e}", file=sys.stderr) - - print( - json.dumps( - {"stdout": stdout_io.getvalue(), "stderr": stderr_io.getvalue()} - ), - flush=True, - ) - - -``` - -We run this driver program in a [Modal Sandbox](https://modal.com/docs/guide/sandboxes). - -```python -app = modal.App.lookup("example-simple-code-interpreter", create_if_missing=True) -sb = modal.Sandbox.create(app=app) - -``` - -We have to convert the driver program to a string to pass it to the Sandbox. -Here we use `inspect.getsource` to get the source code as a string, -but you could also keep the driver program in a separate file and read it in. - -```python -driver_program_text = inspect.getsource(driver_program) -driver_program_command = f"""{driver_program_text}\n\ndriver_program()""" - -``` - -We then kick off the program with [`Sandbox.exec`](https://modal.com/docs/reference/modal.Sandbox#exec), -which creates a process inside the Sandbox (see [`modal.container_process`](https://modal.com/docs/reference/modal.container_process) -for details). - -```python -p = sb.exec("python", "-c", driver_program_command, bufsize=1) - -``` - -## Running code in a Modal Sandbox - -Now we need a way to run code inside that running driver process. -Our driver program already defined a JSON interface on its `stdin` and `stdout`, -so we just need to write a quick wrapper to write to the remote `stdin` -and read from the remote `stdout`. - -```python -reader, writer = p.stdin, iter(p.stdout) - - -def run_code(writer: modal.io_streams.StreamWriter, reader: Iterator[str], code: str): - writer.write(json.dumps({"code": code}) + "\n") - writer.drain() - result = json.loads(next(reader)) - print(result["stdout"], end="") - if result["stderr"]: - print("\033[91m" + result["stderr"] + "\033[0m", end="", file=sys.stderr) - - -``` - -Now we can execute some code in the Sandbox! - -```python -run_code(reader, writer, "print('hello, world!')") # hello, world! - -``` - -The Sandbox and our code interpreter are stateful, -so we can define variables and use them in subsequent code. - -```python -run_code(reader, writer, "x = 10") -run_code(reader, writer, "y = 5") -run_code(reader, writer, "result = x + y") -run_code(reader, writer, "print(f'The result is: {result}')") # The result is: 15 - -``` - -We can also see errors when code fails. - -```python -run_code(reader, writer, "print('Attempting to divide by zero...')") -run_code(reader, writer, "1 / 0") # Execution Error: division by zero - -``` - -Finally, let's clean up after ourselves and terminate the Sandbox. - -```python -sb.terminate() - -``` diff --git a/.agents/skills/modal/references/examples/streaming_kyutai_stt.md b/.agents/skills/modal/references/examples/streaming_kyutai_stt.md deleted file mode 100644 index a999c8d854f6bbd711bf348904e8581908a93edf..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/streaming_kyutai_stt.md +++ /dev/null @@ -1,534 +0,0 @@ -# Stream transcriptions with Kyutai STT - -This example demonstrates the deployment of a streaming audio transcription service with Kyutai STT on Modal. - -[Kyutai STT](https://kyutai.org/next/stt) is an automated speech recognition/transcription model -that is designed to operate on streams of audio, rather than on complete audio files. -See the linked blog post for details on their "delayed streams" architecture. - -## Setup - -We start by importing some basic packages and the Modal SDK. - -```python -import asyncio -import base64 -import time -from pathlib import Path - -import modal - -``` - -Then we define a Modal App and an -[Image](https://modal.com/docs/guide/images) -with the dependencies of our speech-to-text system. - -```python -app = modal.App(name="example-streaming-kyutai-stt") - -stt_image = ( - modal.Image.debian_slim(python_version="3.12") - .uv_pip_install( - "moshi==0.2.9", "fastapi==0.116.1", "huggingface-hub==0.33.5", "julius==0.2.7" - ) - .env({"HF_XET_HIGH_PERFORMANCE": "1"}) -) - -``` - -One dependency is missing: the model weights. - -Instead of including them in the Image or loading them every time the Function starts, -we add them to a Modal [Volume](https://modal.com/docs/guide/volumes). -Volumes are like a shared disk that all Modal Functions can access. - -For more details on patterns for handling model weights on Modal, see -[this guide](https://modal.com/docs/guide/model-weights). - -```python -MODEL_NAME = "kyutai/stt-1b-en_fr" - -hf_cache_vol = modal.Volume.from_name(f"{app.name}-hf-cache", create_if_missing=True) -hf_cache_vol_path = Path("/root/.cache/huggingface") -volumes = {hf_cache_vol_path: hf_cache_vol} - -``` - -## Run Kyutai STT inference on Modal - -Now we're ready to add the code that runs the speech-to-text model. - -We use a Modal [Cls](https://modal.com/docs/guide/lifecycle-functions) -so that we can separate out the model loading and setup code from the inference. - -For more on lifecycle management with Clses and cold start penalty reduction on Modal, see -[this guide](https://modal.com/docs/guide/cold-start). - -We also define multiple ways to access the underlying streaming STT service -- -via a [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API), -for Web clients like browsers, -and via a Modal [Queue](https://modal.com/docs/guide/queues) -for Python clients. - -That plus the code for manipulating the streams of audio bytes and output text -leads to a pretty big class! But there's not anything too complex here. - -```python -MINUTES = 60 - - -@app.cls(image=stt_image, gpu="l40s", volumes=volumes, timeout=10 * MINUTES) -class STT: - BATCH_SIZE = 1 - - @modal.enter() - def enter(self): - import torch - from huggingface_hub import snapshot_download - from moshi.models import LMGen, loaders - - start_time = time.monotonic_ns() - - print("Loading model...") - snapshot_download(MODEL_NAME) - - self.device = "cuda" if torch.cuda.is_available() else "cpu" - - checkpoint_info = loaders.CheckpointInfo.from_hf_repo(MODEL_NAME) - self.mimi = checkpoint_info.get_mimi(device=self.device) - self.frame_size = int(self.mimi.sample_rate / self.mimi.frame_rate) - - self.moshi = checkpoint_info.get_moshi(device=self.device) - self.lm_gen = LMGen(self.moshi, temp=0, temp_text=0) - - self.mimi.streaming_forever(self.BATCH_SIZE) - self.lm_gen.streaming_forever(self.BATCH_SIZE) - - self.text_tokenizer = checkpoint_info.get_text_tokenizer() - - self.audio_silence_prefix_seconds = checkpoint_info.stt_config.get( - "audio_silence_prefix_seconds", 1.0 - ) - self.audio_delay_seconds = checkpoint_info.stt_config.get( - "audio_delay_seconds", 5.0 - ) - self.padding_token_id = checkpoint_info.raw_config.get( - "text_padding_token_id", 3 - ) - - # warmup gpus - for _ in range(4): - codes = self.mimi.encode( - torch.zeros(self.BATCH_SIZE, 1, self.frame_size).to(self.device) - ) - for c in range(codes.shape[-1]): - tokens = self.lm_gen.step(codes[:, :, c : c + 1]) - if tokens is None: - continue - torch.cuda.synchronize() - - print(f"Model loaded in {round((time.monotonic_ns() - start_time) / 1e9, 2)}s") - - def reset_state(self): - # reset llm chat history for this input - self.mimi.reset_streaming() - self.lm_gen.reset_streaming() - - async def transcribe(self, pcm, all_pcm_data): - import numpy as np - import torch - - if pcm is None: - yield all_pcm_data - return - if len(pcm) == 0: - yield all_pcm_data - return - - if pcm.shape[-1] == 0: - yield all_pcm_data - return - - if all_pcm_data is None: - all_pcm_data = pcm - else: - all_pcm_data = np.concatenate((all_pcm_data, pcm)) - - # infer on each frame - while all_pcm_data.shape[-1] >= self.frame_size: - chunk = all_pcm_data[: self.frame_size] - all_pcm_data = all_pcm_data[self.frame_size :] - - with torch.no_grad(): - chunk = torch.from_numpy(chunk) - chunk = chunk.unsqueeze(0).unsqueeze(0) # (1, 1, frame_size) - chunk = chunk.expand( - self.BATCH_SIZE, -1, -1 - ) # (batch_size, 1, frame_size) - chunk = chunk.to(device=self.device) - - # inference on audio chunk - codes = self.mimi.encode(chunk) - - # language model inference against encoded audio - for c in range(codes.shape[-1]): - text_tokens, vad_heads = self.lm_gen.step_with_extra_heads( - codes[:, :, c : c + 1] - ) - if text_tokens is None: - # model is silent - yield all_pcm_data - return - if vad_heads: - pr_vad = vad_heads[2][0, 0, 0].cpu().item() - if pr_vad > 0.5: - # end of turn detected - yield all_pcm_data - return - - assert text_tokens.shape[1] == self.lm_gen.lm_model.dep_q + 1 - - text_token = text_tokens[0, 0, 0].item() - if text_token not in (0, 3): - text = self.text_tokenizer.id_to_piece(text_token) - text = text.replace("▁", " ") - yield text - - yield all_pcm_data - - @modal.asgi_app() - def api(self): - import sphn - from fastapi import FastAPI, Response, WebSocket, WebSocketDisconnect - - web_app = FastAPI() - - @web_app.get("/status") - async def status(): - return Response(status_code=200) - - @web_app.websocket("/ws") - async def transcribe_websocket(ws: WebSocket): - await ws.accept() - - opus_stream_inbound = sphn.OpusStreamReader(self.mimi.sample_rate) - transcription_queue = asyncio.Queue() - - print("Session started") - tasks = [] - - # asyncio to run multiple loops concurrently within single websocket connection - async def recv_loop(): - """ - Receives Opus stream across websocket, appends into inbound queue. - """ - nonlocal opus_stream_inbound - while True: - data = await ws.receive_bytes() - - if not isinstance(data, bytes): - print("received non-bytes message") - continue - if len(data) == 0: - print("received empty message") - continue - opus_stream_inbound.append_bytes(data) - - async def inference_loop(): - """ - Runs streaming inference on inbound data, and if any response audio is created, appends it to the outbound stream. - """ - nonlocal opus_stream_inbound, transcription_queue - all_pcm_data = None - - while True: - await asyncio.sleep(0.001) - - pcm = opus_stream_inbound.read_pcm() - async for msg in self.transcribe(pcm, all_pcm_data): - if isinstance(msg, str): - transcription_queue.put_nowait(msg) - else: - all_pcm_data = msg - - async def send_loop(): - """ - Reads outbound data, and sends it across websocket - """ - nonlocal transcription_queue - while True: - data = await transcription_queue.get() - - if data is None: - continue - - msg = b"\x01" + bytes( - data, encoding="utf8" - ) # prepend "\x01" as a tag to indicate text - await ws.send_bytes(msg) - - # run all loops concurrently - try: - tasks = [ - asyncio.create_task(recv_loop()), - asyncio.create_task(inference_loop()), - asyncio.create_task(send_loop()), - ] - await asyncio.gather(*tasks) - - except WebSocketDisconnect: - print("WebSocket disconnected") - await ws.close(code=1000) - except Exception as e: - print("Exception:", e) - await ws.close(code=1011) # internal error - raise e - finally: - for task in tasks: - task.cancel() - await asyncio.gather(*tasks, return_exceptions=True) - self.reset_state() - - return web_app - - @modal.method() - async def transcribe_queue(self, q: modal.Queue): - import tempfile - - import sphn - - all_pcm_data = None - - while True: - chunk = await q.get.aio(partition="audio") - if chunk is None: - await q.put.aio(None, partition="transcription") - break - - # to avoid having to encode the audio and retrieve with OpusStreamReader: - with tempfile.NamedTemporaryFile(suffix=".mp3", delete=False) as tmp: - tmp.write(chunk) - tmp.flush() - pcm, _ = sphn.read(tmp.name) - pcm = pcm.squeeze(0) - - async for msg in self.transcribe(pcm, all_pcm_data): - if isinstance(msg, str): - await q.put.aio(msg, partition="transcription") - else: - all_pcm_data = msg - - -``` - -## Run a local Python client to test streaming STT - -We can test this code on the same production Modal infra -that we'll be deploying it on by writing a quick `local_entrypoint` for testing. - -We just need a few helper functions to control the streaming of audio bytes -and transcribed text from local Python. - -These communicate asynchronously with the deployed Function using a Modal Queue. - -```python -async def chunk_audio(data: bytes, chunk_size: int): - for i in range(0, len(data), chunk_size): - yield data[i : i + chunk_size] - - -async def send_audio(audio_bytes: bytes, q: modal.Queue, chunk_size: int, rtf: int): - async for chunk in chunk_audio(audio_bytes, chunk_size): - await q.put.aio(chunk, partition="audio") - await asyncio.sleep(chunk_size / chunk_size / rtf) - await q.put.aio(None, partition="audio") - - -async def receive_text(q: modal.Queue): - break_counter, break_every = 0, 20 - while True: - data = await q.get.aio(partition="transcription") - if data is None: - break - print(data, end="") - break_counter += 1 - if break_counter >= break_every: - print() - break_counter = 0 - - -``` - -Now we write our quick test, which loads in audio from a URL -and then passes it to the remote Function via a - -If you run this example with - -```bash -modal run streaming_kyutai_stt.py -``` - -you will - -1. deploy the latest version of the code on Modal -2. spin up a new GPU to handle transcription -3. load the model from Hugging Face or the Modal Volume cache -4. send the audio out to the new GPU container, transcribe it, and receive it locally to be printed. - -Not bad for a single Python file with no dependencies except Modal! - -```python -@app.local_entrypoint() -async def test( - chunk_size: int = 24_000, # bytes - rtf: int = 1000, - audio_url: str = "https://github.com/kyutai-labs/delayed-streams-modeling/raw/refs/heads/main/audio/bria.mp3", -): - from urllib.request import urlopen - - print(f"Downloading audio file from {audio_url}") - audio_bytes = urlopen(audio_url).read() - print(f"Downloaded {len(audio_bytes)} bytes") - - print("Starting transcription") - start_time = time.monotonic_ns() - async with modal.Queue.ephemeral() as q: - await STT().transcribe_queue.spawn.aio(q) - send = asyncio.create_task(send_audio(audio_bytes, q, chunk_size, rtf)) - recv = asyncio.create_task(receive_text(q)) - await asyncio.gather(send, recv) - print( - f"\nTranscription complete in {round((time.monotonic_ns() - start_time) / 1e9, 2)}s" - ) - - -``` - -## Deploy a streaming STT service on the Web - -We've already written a Web backend for our streaming STT service -- -that's the FastAPI API with the WebSocket in the Modal Cls above. - -We can also deploy a Web frontend. To keep things almost entirely "pure Python", -we here use the [FastHTML](https://www.fastht.ml/) library, -but you can also deploy a JavaScript frontend with a FastAPI or Node backend. - -We do use a bit of JS for the audio processing in the browser. -We add it to the Modal Image using `add_local_dir`. -You can find the frontend files [here](https://github.com/modal-labs/modal-examples/tree/main/06_gpu_and_ml/speech-to-text/streaming-kyutai-stt-frontend). - -```python -web_image = ( - modal.Image.debian_slim(python_version="3.12") - .uv_pip_install("python-fasthtml==0.12.20") - .add_local_dir( - Path(__file__).parent / "streaming-kyutai-stt-frontend", "/root/frontend" - ) -) - -``` - -You can deploy this frontend with - -```bash -modal deploy streaming_kyutai_stt.py -``` - -and then interact with it at the printed `ui` URL. - -```python -@app.function(image=web_image, timeout=10 * MINUTES) -@modal.concurrent(max_inputs=100) -@modal.asgi_app() -def ui(): - import fasthtml.common as fh - - modal_logo_svg = open("/root/frontend/modal-logo.svg").read() - modal_logo_base64 = base64.b64encode(modal_logo_svg.encode()).decode() - app_js = open("/root/frontend/audio.js").read() - - fast_app, rt = fh.fast_app( - hdrs=[ - # audio recording libraries - fh.Script( - src="https://cdn.jsdelivr.net/npm/opus-recorder@latest/dist/recorder.min.js" - ), - fh.Script( - src="https://cdn.jsdelivr.net/npm/opus-recorder@latest/dist/encoderWorker.min.js" - ), - fh.Script( - src="https://cdn.jsdelivr.net/npm/ogg-opus-decoder/dist/ogg-opus-decoder.min.js" - ), - # styling - fh.Link( - href="https://fonts.googleapis.com/css?family=Inter:300,400,600", - rel="stylesheet", - ), - fh.Script(src="https://cdn.tailwindcss.com"), - fh.Script(""" - tailwind.config = { - theme: { - extend: { - colors: { - ground: "#0C0F0B", - primary: "#9AEE86", - "accent-pink": "#FC9CC6", - "accent-blue": "#B8E4FF", - }, - }, - }, - }; - """), - ], - ) - - @rt("/") - def get(): - return ( - fh.Title("Kyutai Streaming STT"), - fh.Body( - fh.Div( - fh.Div( - fh.Div( - id="text-output", - cls="flex flex-col-reverse overflow-y-auto max-h-64 pr-2", - ), - cls="w-full overflow-y-auto max-h-64", - ), - cls="bg-gray-800 rounded-lg shadow-lg w-full max-w-xl p-6", - ), - fh.Footer( - fh.Span( - "Built with ", - fh.A( - "Kyutai", - href="https://github.com/kyutai-labs/delayed-streams-modeling", - target="_blank", - rel="noopener noreferrer", - cls="underline", - ), - " and", - cls="text-sm font-medium text-gray-300 mr-2", - ), - fh.A( - fh.Img( - src=f"data:image/svg+xml;base64,{modal_logo_base64}", - alt="Modal logo", - cls="w-24", - ), - cls="flex items-center p-2 rounded-lg bg-gray-800 shadow-lg hover:bg-gray-700 transition-colors duration-200", - href="https://modal.com", - target="_blank", - rel="noopener noreferrer", - ), - cls="fixed bottom-4 inline-flex items-center justify-center", - ), - fh.Script(app_js), - cls="relative bg-gray-900 text-white min-h-screen flex flex-col items-center justify-center p-4", - ), - ) - - return fast_app - -``` diff --git a/.agents/skills/modal/references/examples/text_to_image.md b/.agents/skills/modal/references/examples/text_to_image.md deleted file mode 100644 index a99860af66feee6d0305afb9446cbea3dbb7185c..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/text_to_image.md +++ /dev/null @@ -1,282 +0,0 @@ -# Run Stable Diffusion 3.5 Large Turbo as a CLI, API, and web UI - -This example shows how to run [Stable Diffusion 3.5 Large Turbo](https://huggingface.co/stabilityai/stable-diffusion-3.5-large-turbo) on Modal -to generate images from your local command line, via an API, and as a web UI. - -Inference takes about one minute to cold start, -at which point images are generated at a rate of one image every 1-2 seconds -for batch sizes between one and 16. - -Below are four images produced by the prompt -"A princess riding on a pony". - -![stable diffusion montage](https://modal-cdn.com/cdnbot/sd-montage-princess-yxu2vnbl_e896a9c0.webp) - -## Basic setup - -```python -import io -import random -import time -from pathlib import Path -from typing import Optional - -import modal - -MINUTES = 60 - -``` - -All Modal programs need an [`App`](https://modal.com/docs/reference/modal.App) — an object that acts as a recipe for -the application. Let's give it a friendly name. - -```python -app = modal.App("example-text-to-image") - -``` - -## Configuring dependencies - -The model runs remotely inside a [container](https://modal.com/docs/guide/custom-container). -That means we need to install the necessary dependencies in that container's image. - -Below, we start from a lightweight base Linux image -and then install our Python dependencies, like Hugging Face's `diffusers` library and `torch`. - -```python -CACHE_DIR = "/cache" - -image = ( - modal.Image.debian_slim(python_version="3.12") - .uv_pip_install( - "accelerate==0.33.0", - "diffusers==0.31.0", - "fastapi[standard]==0.115.4", - "huggingface-hub==0.36.0", - "sentencepiece==0.2.0", - "torch==2.5.1", - "torchvision==0.20.1", - "transformers~=4.44.0", - ) - .env( - { - "HF_XET_HIGH_PERFORMANCE": "1", # faster downloads - "HF_HUB_CACHE": CACHE_DIR, - } - ) -) - -with image.imports(): - import diffusers - import torch - from fastapi import Response - -``` - -## Implementing SD3.5 Large Turbo inference on Modal - -We wrap inference in a Modal [Cls](https://modal.com/docs/guide/lifecycle-functions) -that ensures models are loaded and then moved to the GPU once when a new container -starts, before the container picks up any work. - -The `run` function just wraps a `diffusers` pipeline. -It sends the output image back to the client as bytes. - -We also include a `web` wrapper that makes it possible -to trigger inference via an API call. -See the `/docs` route of the URL ending in `inference-web.modal.run` -that appears when you deploy the app for details. - -```python -MODEL_ID = "adamo1139/stable-diffusion-3.5-large-turbo-ungated" -MODEL_REVISION_ID = "9ad870ac0b0e5e48ced156bb02f85d324b7275d2" - -cache_volume = modal.Volume.from_name("hf-hub-cache", create_if_missing=True) - - -@app.cls( - image=image, - gpu="H100", - timeout=10 * MINUTES, - volumes={CACHE_DIR: cache_volume}, -) -class Inference: - @modal.enter() - def load_pipeline(self): - self.pipe = diffusers.StableDiffusion3Pipeline.from_pretrained( - MODEL_ID, - revision=MODEL_REVISION_ID, - torch_dtype=torch.bfloat16, - ).to("cuda") - - @modal.method() - def run( - self, prompt: str, batch_size: int = 4, seed: Optional[int] = None - ) -> list[bytes]: - seed = seed if seed is not None else random.randint(0, 2**32 - 1) - print("seeding RNG with", seed) - torch.manual_seed(seed) - images = self.pipe( - prompt, - num_images_per_prompt=batch_size, # outputting multiple images per prompt is much cheaper than separate calls - num_inference_steps=4, # turbo is tuned to run in four steps - guidance_scale=0.0, # turbo doesn't use CFG - max_sequence_length=512, # T5-XXL text encoder supports longer sequences, more complex prompts - ).images - - image_output = [] - for image in images: - with io.BytesIO() as buf: - image.save(buf, format="PNG") - image_output.append(buf.getvalue()) - torch.cuda.empty_cache() # reduce fragmentation - return image_output - - @modal.fastapi_endpoint(docs=True) - def web(self, prompt: str, seed: Optional[int] = None): - return Response( - content=self.run.local( # run in the same container - prompt, batch_size=1, seed=seed - )[0], - media_type="image/png", - ) - - -``` - -## Generating Stable Diffusion images from the command line - -This is the command we'll use to generate images. It takes a text `prompt`, -a `batch_size` that determines the number of images to generate per prompt, -and the number of times to run image generation (`samples`). - -You can also provide a `seed` to make sampling more deterministic. - -Run it with - -```bash -modal run text_to_image.py -``` - -and pass `--help` to see more options. - -```python -@app.local_entrypoint() -def entrypoint( - samples: int = 4, - prompt: str = "A princess riding on a pony", - batch_size: int = 4, - seed: Optional[int] = None, -): - print( - f"prompt => {prompt}", - f"samples => {samples}", - f"batch_size => {batch_size}", - f"seed => {seed}", - sep="\n", - ) - - output_dir = Path("/tmp/stable-diffusion") - output_dir.mkdir(exist_ok=True, parents=True) - - inference_service = Inference() - - for sample_idx in range(samples): - start = time.time() - images = inference_service.run.remote(prompt, batch_size, seed) - duration = time.time() - start - print(f"Run {sample_idx + 1} took {duration:.3f}s") - if sample_idx: - print( - f"\tGenerated {len(images)} image(s) at {(duration) / len(images):.3f}s / image." - ) - for batch_idx, image_bytes in enumerate(images): - output_path = ( - output_dir - / f"output_{slugify(prompt)[:64]}_{str(sample_idx).zfill(2)}_{str(batch_idx).zfill(2)}.png" - ) - if not batch_idx: - print("Saving outputs", end="\n\t") - print( - output_path, - end="\n" + ("\t" if batch_idx < len(images) - 1 else ""), - ) - output_path.write_bytes(image_bytes) - - -``` - -## Generating Stable Diffusion images via an API - -The Modal `Cls` above also included a [`fastapi_endpoint`](https://modal.com/docs/examples/basic_web), -which adds a simple web API to the inference method. - -To try it out, run - -```bash -modal deploy text_to_image.py -``` - -copy the printed URL ending in `inference-web.modal.run`, -and add `/docs` to the end. This will bring up the interactive -Swagger/OpenAPI docs for the endpoint. - -## Generating Stable Diffusion images in a web UI - -Lastly, we add a simple front-end web UI (written in Alpine.js) for -our image generation backend. - -This is also deployed by running - -```bash -modal deploy text_to_image.py. -``` - -The `Inference` class will serve multiple users from its own auto-scaling pool of warm GPU containers automatically. - -```python -frontend_path = Path(__file__).parent / "frontend" - -web_image = ( - modal.Image.debian_slim(python_version="3.12") - .uv_pip_install("jinja2==3.1.4", "fastapi[standard]==0.115.4") - .add_local_dir(frontend_path, remote_path="/assets") -) - - -@app.function(image=web_image) -@modal.concurrent(max_inputs=100) -@modal.asgi_app() -def ui(): - import fastapi.staticfiles - from fastapi import FastAPI, Request - from fastapi.templating import Jinja2Templates - - web_app = FastAPI() - templates = Jinja2Templates(directory="/assets") - - @web_app.get("/") - async def read_root(request: Request): - return templates.TemplateResponse( - "index.html", - { - "request": request, - "inference_url": Inference.web.get_web_url(), - "model_name": "Stable Diffusion 3.5 Large Turbo", - "default_prompt": "A cinematic shot of a baby raccoon wearing an intricate italian priest robe.", - }, - ) - - web_app.mount( - "/static", - fastapi.staticfiles.StaticFiles(directory="/assets"), - name="static", - ) - - return web_app - - -def slugify(s: str) -> str: - return "".join(c if c.isalnum() else "-" for c in s).strip("-") - -``` diff --git a/.agents/skills/modal/references/examples/unsloth_finetune.md b/.agents/skills/modal/references/examples/unsloth_finetune.md deleted file mode 100644 index 5f0293b94f1f8c4a6770262ef71730251a78d179..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/unsloth_finetune.md +++ /dev/null @@ -1,598 +0,0 @@ -# Efficient LLM Finetuning with Unsloth - -Training large language models is an incredibly compute-hungry process. -Open-source LLMs often require many GBs (or in extreme cases, -[one TB](https://github.com/MoonshotAI/Kimi-K2#2-model-summary)!) of -VRAM just to fit in memory. Finetuning models requires even more memory; -a common estimate for naive finetuning puts the VRAM requirements at roughly -4.2x the original model size: -1x for model weights + 1x for gradients + 2x for optimizer state + 20% for activations. -Parameter efficient methods like LoRA can improve matters significantly, since -this estimate now applies to just the LoRA modules' weights, rather than the entire -model's. Further gains can be made with quantization of each of the components mentioned -above, but doing so requires quantization-aware training, which can be tricky to -combine with methods like LoRA. - -Unsloth provides optimized methods for LLM finetuning with LoRA and quantization, -leading to typical performance gains of 2x faster training with 70% less memory usage. -This example demonstrates using Unsloth to finetune a version of Qwen3-14B with the -FineTome-100k dataset on Modal using only a single GPU! - -## Modal Infrastructure Setup - -```python -import pathlib -from dataclasses import dataclass -from datetime import datetime -from typing import Optional - -import modal - -``` - -We create a Modal [App](https://modal.com/docs/guide/apps) to organize our functions -and shared infrastructure like container images and volumes. - -```python -app = modal.App("example-unsloth-finetune") - -``` - -### Container Image Configuration - -We build a custom container image with Unsloth and all necessary dependencies. -The image includes the latest version of Unsloth (as of writing) with optimizations -for the latest model architectures. Once the image is defined, we can specify the -imports we'll need to write the rest of our training code. Importantly, we import -`unsloth` before the rest so that Unsloth's patches are applied to packages like -`transformers`, `peft`, and `trl`. - -```python -train_image = ( - modal.Image.debian_slim(python_version="3.11") - .uv_pip_install( - "accelerate==1.9.0", - "datasets==3.6.0", - "hf-transfer==0.1.9", - "huggingface_hub==0.34.2", - "peft==0.16.0", - "transformers==4.54.0", - "trl==0.19.1", - "unsloth[cu128-torch270]==2025.7.8", - "unsloth_zoo==2025.7.10", - "wandb==0.21.0", - ) - .env({"HF_HOME": "/model_cache"}) -) - -with train_image.imports(): - # unsloth must be first! - import unsloth # noqa: F401,I001 - import datasets - import torch - import wandb - from transformers import TrainingArguments - from trl import SFTTrainer - from unsloth import FastLanguageModel - from unsloth.chat_templates import standardize_sharegpt - -``` - -### Volume Configuration - -Modal [Volumes](https://modal.com/docs/guide/volumes) provide storage that persists -between function invocations. We use separate volumes for different types of data to -enable efficient caching and sharing: - -* A cache for [pretrained model weights](https://modal.com/docs/guide/model-weights) - reused across all experiments -* A cache for processed datasets - reused when using the same dataset -* Storage for training checkpoints and final models - -```python -model_cache_volume = modal.Volume.from_name( - "unsloth-model-cache", create_if_missing=True -) -dataset_cache_volume = modal.Volume.from_name( - "unsloth-dataset-cache", create_if_missing=True -) -checkpoint_volume = modal.Volume.from_name( - "unsloth-checkpoints", create_if_missing=True -) - -``` - -### Picking a GPU - -We use L40S for its healthy balance of [VRAM](https://modal.com/gpu-glossary/device-hardware/gpu-ram), -[CUDA cores](https://modal.com/gpu-glossary/device-hardware/cuda-core), and clock speed. -The timeout provides an upper bound on our training time; if our training run finishes faster, -we won't end up using the full 6 hours. We also specify 3 retries, which will be useful -in case our training function gets [preempted](https://modal.com/docs/guide/preemption). - -```python -GPU_TYPE = "L40S" -TIMEOUT_HOURS = 6 -MAX_RETRIES = 3 - -``` - -## Data Processing - -We'll be finetuning our model on the FineTome-100k dataset, which is -subset of [The Tome](https://huggingface.co/datasets/arcee-ai/The-Tome) -curated with [fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) -Below we define some helpers for processing this dataset. - -```python -CONVERSATION_COLUMN = "conversations" # ShareGPT format column name -TEXT_COLUMN = "text" # Output column for formatted text -TRAIN_SPLIT_RATIO = 0.9 # 90% train, 10% eval split -PREPROCESSING_WORKERS = 2 # Number of workers for dataset processing - - -def format_chat_template(examples, tokenizer): - texts = [] - for conversation in examples[CONVERSATION_COLUMN]: - formatted_text = tokenizer.apply_chat_template( - conversation, tokenize=False, add_generation_prompt=False - ) - texts.append(formatted_text) - return {TEXT_COLUMN: texts} - - -def load_or_cache_dataset(config: "TrainingConfig", paths: dict, tokenizer): - dataset_cache_path = paths["dataset_cache"] - - if dataset_cache_path.exists(): - print(f"Loading cached dataset from {dataset_cache_path}") - train_dataset = datasets.load_from_disk(dataset_cache_path / "train") - eval_dataset = datasets.load_from_disk(dataset_cache_path / "eval") - else: - print(f"Downloading and processing dataset: {config.dataset_name}") - - # Load and standardize the dataset format - dataset = datasets.load_dataset(config.dataset_name, split="train") - dataset = standardize_sharegpt(dataset) - - # Split into training and evaluation sets with fixed seed for reproducibility - dataset = dataset.train_test_split( - test_size=1.0 - TRAIN_SPLIT_RATIO, seed=config.seed - ) - train_dataset = dataset["train"] - eval_dataset = dataset["test"] - - # Apply chat template formatting to convert conversations to text - print("Formatting datasets with chat template...") - train_dataset = train_dataset.map( - lambda examples: format_chat_template(examples, tokenizer), - batched=True, - num_proc=PREPROCESSING_WORKERS, - remove_columns=train_dataset.column_names, - ) - - eval_dataset = eval_dataset.map( - lambda examples: format_chat_template(examples, tokenizer), - batched=True, - num_proc=PREPROCESSING_WORKERS, - remove_columns=eval_dataset.column_names, - ) - - # Cache the processed datasets for future runs - print(f"Caching processed datasets to {dataset_cache_path}") - dataset_cache_path.mkdir(parents=True, exist_ok=True) - train_dataset.save_to_disk(dataset_cache_path / "train") - eval_dataset.save_to_disk(dataset_cache_path / "eval") - - # Commit the dataset cache to the volume - dataset_cache_volume.commit() - - return train_dataset, eval_dataset - - -``` - -## Loading the pretrained model - -We can't finetune without a pretarined model! Since these models are -fairly large, we don't want to download them from scratch for each training run. -We solve this by caching the weights in a Volume on download, and then loading -from the Volume on subsequent runs. - -```python -def load_or_cache_model(config: "TrainingConfig", paths: dict): - print(f"Downloading and caching model: {config.model_name}") - model, tokenizer = FastLanguageModel.from_pretrained( - model_name=config.model_name, - max_seq_length=config.max_seq_length, - dtype=None, - load_in_4bit=config.load_in_4bit, - load_in_8bit=config.load_in_8bit, - ) - - return model, tokenizer - - -``` - -## Training Configuration - -First we'll define what layers our LoRA modules should target. -Generally, it's advisable to LoRA finetune every linear layer in the model, -so we target every projection matrix of each attention layer. - -```python -LORA_TARGET_MODULES = [ - "q_proj", - "k_proj", - "v_proj", - "o_proj", - "gate_proj", - "up_proj", - "down_proj", -] - - -``` - -We want to expose the different hyperparameters and optimizations that -Unsloth supports, so we wrap them into a `TrainingConfig` class. Later, -we'll populate this config with arguments from the command line. - -```python -@dataclass -class TrainingConfig: - # Model and dataset selection - model_name: str - dataset_name: str - max_seq_length: int - load_in_4bit: bool - load_in_8bit: bool - - # LoRA configuration for efficient finetuning - lora_r: int - lora_alpha: int - lora_dropout: float - lora_bias: str - use_rslora: bool - - # Training hyperparameters - optim: str - batch_size: int - gradient_accumulation_steps: int - packing: bool - use_gradient_checkpointing: str - learning_rate: float - lr_scheduler_type: str - warmup_ratio: float - weight_decay: float - max_steps: int - save_steps: int - eval_steps: int - logging_steps: int - - # Experiment management - seed: int - experiment_name: Optional[str] = None - enable_wandb: bool = True - - # For testing purposes - skip_eval: bool = False - - def __post_init__(self): - # Generate a unique experiment name if not provided - if self.experiment_name is None: - timestamp = datetime.now().strftime("%Y%m%d-%H%M%S") - model_short = self.model_name.split("/")[-1] - self.experiment_name = f"{model_short}-r{self.lora_r}-{timestamp}" - - -``` - -## Main Training Function - -This function orchestrates the entire training process, from model loading -to final model saving. It's decorated with Modal function configuration -that specifies the compute resources, the volumes needed, and execution -details like timeout and retries. - -```python -@app.function( - image=train_image, - gpu=GPU_TYPE, - volumes={ - "/model_cache": model_cache_volume, - "/dataset_cache": dataset_cache_volume, - "/checkpoints": checkpoint_volume, - }, - secrets=[modal.Secret.from_name("wandb-secret")], - timeout=TIMEOUT_HOURS * 60 * 60, - retries=modal.Retries(initial_delay=0.0, max_retries=MAX_RETRIES), - single_use_containers=True, # Ensure we get a fresh container on retry -) -def finetune(config: TrainingConfig): - # Get structured paths for organized file storage - paths = get_structured_paths(config) - - # Initialize Weights & Biases for experiment tracking if enabled - if config.enable_wandb: - wandb.init( - project="unsloth-finetune", - name=config.experiment_name, - config=config.__dict__, - ) - - # Load or cache model and datasets with progress indicators - print("Setting up model and data...") - model, tokenizer = load_or_cache_model(config, paths) - train_dataset, eval_dataset = load_or_cache_dataset(config, paths, tokenizer) - - # Configure the model for LoRA training - model = setup_model_for_training(model, config) - - # Prepare checkpoint directory and check for existing checkpoints - checkpoint_path = paths["checkpoints"] - checkpoint_path.mkdir(parents=True, exist_ok=True) - resume_from_checkpoint = check_for_existing_checkpoint(paths) - - # Create training configuration - training_args = create_training_arguments(config, str(checkpoint_path)) - - # Initialize the supervised finetuning trainer - print("Initializing SFTTrainer...") - trainer = SFTTrainer( - model=model, - tokenizer=tokenizer, - train_dataset=train_dataset, - eval_dataset=eval_dataset, - dataset_text_field=TEXT_COLUMN, - max_seq_length=config.max_seq_length, - dataset_num_proc=PREPROCESSING_WORKERS, - packing=config.packing, # Sequence packing for efficiency - args=training_args, - ) - - # Display training information for transparency - print(f"Training dataset size: {len(train_dataset):,}") - print(f"Evaluation dataset size: {len(eval_dataset):,}") - print(f"Total parameters: {sum(p.numel() for p in model.parameters()):,}") - print( - f"Trainable parameters: {sum(p.numel() for p in model.parameters() if p.requires_grad):,}" - ) - print(f"Experiment: {config.experiment_name}") - - # Start training or resume from checkpoint - if resume_from_checkpoint: - print(f"Resuming training from {resume_from_checkpoint}") - trainer.train(resume_from_checkpoint=resume_from_checkpoint) - else: - print("Starting training from scratch...") - trainer.train() - - # Save the final trained model and tokenizer - print("Saving final model...") - final_model_path = checkpoint_path / "final_model" - model.save_pretrained(final_model_path) - tokenizer.save_pretrained(final_model_path) - - # Clean up experiment tracking - if config.enable_wandb: - wandb.finish() - - print(f"Training completed! Model saved to: {final_model_path}") - return config.experiment_name - - -``` - -Finally, we invoke our training function from an -[`App.local_entrypoint`](https://modal.com/docs/reference/modal.App#local_entrypoint). -Arguments to this function automatically get converted into CLI flags that -can be specified when we [`modal run`](https://modal.com/docs/reference/cli/run#modal-run) -our code. This allows us to do things like tweak hyperparameters directly -from the command line without modifying our source code. - -To try this example, checkout the examples repo, install the Modal client, and run - -```bash -modal run 06_gpu_and_ml/unsloth_finetune.py -``` - -You can also customize the training process by tweaking hyperparameters with command line -flags, e.g. - -```bash -modal run 06_gpu_and_ml/unsloth_finetune.py --max-steps 10000 -``` - -```python -@app.local_entrypoint() -def main( - # Model and dataset configuration - model_name: str = "unsloth/Qwen3-32B", - dataset_name: str = "mlabonne/FineTome-100k", - max_seq_length: int = 32768, - load_in_4bit: bool = True, # unsloth: use 4bit quant for frozen model weights - load_in_8bit: bool = False, # unsloth: use 8bit quant for frozen model weights - # LoRA hyperparameters for finetuning efficiency - lora_r: int = 16, - lora_alpha: int = 16, - lora_dropout: float = 0.0, - lora_bias: str = "none", # unsloth: optimized lora kernel - use_rslora: bool = False, - # Training hyperparameters for optimization - optim: str = "adamw_8bit", # unsloth: 8bit optimizer - batch_size: int = 16, - gradient_accumulation_steps: int = 1, - packing: bool = False, - use_gradient_checkpointing: str = "unsloth", # unsloth: optimized gradient offloading - learning_rate: float = 2e-4, - lr_scheduler_type: str = "cosine", - warmup_ratio: float = 0.06, - weight_decay: float = 0.01, - max_steps: int = 5, # increase! - save_steps: int = 2, # increase! - eval_steps: int = 2, # increase! - logging_steps: int = 1, # increase! - # Optional experiment configuration - seed: int = 105, - experiment_name: Optional[str] = None, - disable_wandb: bool = True, - skip_eval: bool = False, -): - # Create configuration object from command line arguments - config = TrainingConfig( - model_name=model_name, - dataset_name=dataset_name, - max_seq_length=max_seq_length, - load_in_4bit=load_in_4bit, - load_in_8bit=load_in_8bit, - lora_r=lora_r, - lora_alpha=lora_alpha, - lora_bias=lora_bias, - lora_dropout=lora_dropout, - use_rslora=use_rslora, - optim=optim, - batch_size=batch_size, - gradient_accumulation_steps=gradient_accumulation_steps, - packing=packing, - use_gradient_checkpointing=use_gradient_checkpointing, - learning_rate=learning_rate, - max_steps=max_steps, - lr_scheduler_type=lr_scheduler_type, - warmup_ratio=warmup_ratio, - weight_decay=weight_decay, - save_steps=save_steps, - eval_steps=eval_steps, - logging_steps=logging_steps, - seed=seed, - experiment_name=experiment_name, - enable_wandb=not disable_wandb, - skip_eval=skip_eval, - ) - - # Display experiment configuration for user verification - print(f"Starting finetuning experiment: {config.experiment_name}") - print(f"Model: {config.model_name}") - print(f"Dataset: {config.dataset_name}") - print(f"LoRA configuration: rank={config.lora_r}, alpha={config.lora_alpha}") - print( - f"Effective batch size: {config.batch_size * config.gradient_accumulation_steps}" - ) - print(f"Training steps: {config.max_steps}") - - # Launch the training job on Modal infrastructure - experiment_name = finetune.remote(config) - print(f"Training completed successfully: {experiment_name}") - - -``` - -## Utility Functions - -These functions handle the core logic for model loading, dataset processing, -and training setup. They're designed to be hackable for new use cases. - -```python -def get_structured_paths(config: TrainingConfig): - """ - Create structured paths within the mounted volumes for organized storage. - - This function maps the configuration to specific directory paths that allow - multiple models, datasets, and experiments to coexist without conflicts. - """ - # Replace forward slashes in names to create valid directory names - dataset_cache_path = ( - pathlib.Path("/dataset_cache") - / "datasets" - / config.dataset_name.replace("/", "--") - ) - checkpoint_path = ( - pathlib.Path("/checkpoints") / "experiments" / config.experiment_name - ) - - return { - "dataset_cache": dataset_cache_path, - "checkpoints": checkpoint_path, - } - - -def setup_model_for_training(model, config: TrainingConfig): - """ - Configure the model with LoRA adapters for efficient finetuning. - - LoRA (Low-Rank Adaptation) allows us to finetune large models efficiently - by only training a small number of additional parameters. This significantly - reduces memory usage and training time. - """ - print("Configuring LoRA for training...") - model = FastLanguageModel.get_peft_model( - model, - r=config.lora_r, # LoRA rank - higher values = more parameters - target_modules=LORA_TARGET_MODULES, # Which layers to apply LoRA to - lora_alpha=config.lora_alpha, # LoRA scaling parameter - lora_dropout=config.lora_dropout, # Dropout for LoRA layers - bias=config.lora_bias, # Bias configuration - use_gradient_checkpointing=config.use_gradient_checkpointing, # Memory optimization - random_state=config.seed, # Fixed seed for reproducibility - use_rslora=config.use_rslora, # Rank-stabilized LoRA - loftq_config=None, # LoFTQ quantization config - ) - return model - - -def create_training_arguments(config: TrainingConfig, output_dir: str): - """ - Create training arguments for the SFTTrainer. - - These arguments control the training process, including optimization settings, - evaluation frequency, and checkpointing behavior. - """ - print("SKIP_EVAL", config.skip_eval) - return TrainingArguments( - # Core training configuration - per_device_train_batch_size=config.batch_size, - gradient_accumulation_steps=config.gradient_accumulation_steps, - learning_rate=config.learning_rate, - max_steps=config.max_steps, - warmup_ratio=config.warmup_ratio, - # Evaluation and checkpointing - eval_steps=config.eval_steps, - save_steps=config.save_steps, - eval_strategy="no" if config.skip_eval else "steps", - save_strategy="steps", - do_eval=not config.skip_eval, - # Optimization settings based on hardware capabilities - fp16=not torch.cuda.is_bf16_supported(), # Use fp16 if bf16 not available - bf16=torch.cuda.is_bf16_supported(), # Prefer bf16 when available - optim=config.optim, - weight_decay=config.weight_decay, - lr_scheduler_type=config.lr_scheduler_type, - # Logging and output configuration - logging_steps=config.logging_steps, - output_dir=output_dir, - report_to="wandb" if config.enable_wandb else None, - seed=config.seed, - ) - - -def check_for_existing_checkpoint(paths: dict): - """ - Check if there's an existing checkpoint to resume training from. - - This enables resumable training, which is crucial for long-running experiments - that might be interrupted by infrastructure issues or resource limits. - """ - checkpoint_dir = paths["checkpoints"] - if not checkpoint_dir.exists(): - return None - - # Look for the most recent checkpoint directory - checkpoints = list(checkpoint_dir.glob("checkpoint-*")) - if checkpoints: - latest_checkpoint = max(checkpoints, key=lambda p: int(p.name.split("-")[1])) - print(f"Found existing checkpoint: {latest_checkpoint}") - return str(latest_checkpoint) - - return None - -``` diff --git a/.agents/skills/modal/references/examples/vllm_throughput.md b/.agents/skills/modal/references/examples/vllm_throughput.md deleted file mode 100644 index bc508cfbea069b4c76fbe216cabb6a9ec3c6eef9..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/vllm_throughput.md +++ /dev/null @@ -1,683 +0,0 @@ -# Run LLM inference at maximum throughput - -This example demonstrates some techniques for running LLM inference -at the highest possible throughput on Modal. - -For more on other aspects of maximizing the performance of LLM inference, see -[our guide](https://modal.com/docs/guide/high-performance-llm-inference). -For a simpler introduction to LLM serving, see -[this example](https://modal.com/docs/examples/llm_inference). - -As our sample application, we use an LLM to summarize thousands of filings with -the U.S. federal government's Securities and Exchange Commission (SEC), -made available to the public for free in daily data dumps -via the SEC's Electronic Data Gathering, Analysis, and Retrieval System -([EDGAR](https://www.sec.gov/submit-filings/about-edgar)). -We like to check out the [Form 4s](https://www.sec.gov/files/form4data.pdf), -which detail (legal) insider trading. - -Using the Qwen 3 8B parameter LLM on this task, -which has inputs that average a few thousand tokens -and outputs that average a few hundred tokens, -we observe processing speeds of ~30,000 input tok/s -and ~2,000 output tok/s per H100 GPU, -as in the sample Modal Dashboard screenshot below. -Note the [100% GPU utilization](https://modal.com/blog/gpu-utilization-guide), -indicating the absence of [host overhead](https://modal.com/blog/host-overhead-inference-efficiency), -and the high [GPU power utilization](https://modal.com/docs/guide/gpu-metrics), -further indicating we are close to the hardware's physical limits. - -![Modal Dashboard indicating 30k tok/s input and 2k tok/s output on a single H100 GPU](https://modal-cdn.com/example-vllm-throughput-dashboard.png) - -At Modal's [current rates](https://modal.com/pricing) as of early 2026, -that comes out to roughly 4¢ per million tokens. -[According to Artificial Analysis](https://artificialanalysis.ai/models/qwen3-8b-instruct), -API providers charge roughly five times as much for the same workload. - -## Organizing a batch job on Modal - -We start by defining a Modal [App](https://modal.com/docs/guide/apps), -which collects together the Modal resources our batch job uses. -While we're at it, we import a bunch of the libraries we will need later. - -```python -import datetime as dt -import time -from dataclasses import dataclass -from datetime import datetime -from pathlib import Path -from zoneinfo import ZoneInfo - -import modal - -MINUTES = 60 # seconds - -app = modal.App("example-vllm-throughput") - -``` - -Many batch jobs work nicely as scripts -- code that is run -from a shell, ad hoc, rather than deployed. -For that, we define a `local_entrypoint` with code that runs -locally, when we pass our script to `modal run`, -and triggers/orchestrates remote execution. - -We demonstrate two techniques for collecting the results of a batch job, -toggled by passing the `--wait-for-results`/`--no-wait-for-results` -flag via the command line. - -When we `--wait-for-results`, we pass the `modal.FunctionCall` IDs -that make up our batch job to `FunctionCall.gather`, which -returns once our job is done. Here, we just print the results, -but in a more realistic setting you might save them to disk. - -Instead of waiting for results, we can retrieve them asynchronously -based on the `FunctionCall` ID -- a simple string. -Results are stored in Modal for one week. -In the `local_entrypoint` below, these IDs are printed, -but you might store them in a file on disk, add them to your database, -or put them in a Modal -[Queue](https://modal.com/docs/guide/queues) -or [Dict](https://modal.com/docs/guide/dicts) -for later retrieval. - -```python -@app.local_entrypoint() -def main(lookback: int = 5, wait_for_results: bool = True): - jobs = orchestrate.remote(lookback=lookback) # trigger remote job orchestration - - if wait_for_results: - print("Collecting results locally") - batches = modal.FunctionCall.gather(*jobs) - for batch in batches: - print(*(result.summary for result in batch if result.form == "4"), sep="\n") - print("\n") - print("Done") - else: - print("Collect results asynchronously with modal.FunctionCall.from_id") - print("FunctionCall IDs:", *[job.object_id for job in jobs], sep="\n\t") - - -``` - -The meat of the work is done in our `orchestrate` function. -It manages the overall pipeline of execution, -starting with `extract`ing data from the raw data source, -followed by `transform`ing it into a cleaner format -and then `process`ing it with the LLM. - -For both extraction and transformation, we use -[`.map`](https://modal.com/docs/guide/scale), -which fans out inputs over containers in parallel. -Each invocation handles at most 1,500 rows, -which leads to runtimes of about five minutes per call -By parallelizing the calls, we finish processing everything in about five minutes. - -"Rechunking" our data from a list of filings by day -into a list of filings of fixed size requires a little -helper function: - -```python -def rechunk(lists, size: int = 1_500): - from itertools import chain, islice - - it = iter(chain.from_iterable(lists)) - while chunk := list(islice(it, size)): - yield chunk - - -``` - -For the LLM call, we use -[`.spawn`](https://modal.com/docs/guide/job-queue), -which triggers asynchronous execution of the LLM, immediately -returning the `FunctionCall` that can later be used to `.get` the result -(or `.gather` several results). - -We run it as a `.remote` Modal Function call -so that it can keep running even after our local client disconnects -(so long as we use `modal run --detach`). -In that case, we dump the `FunctionCall` IDs into the logs, -but you might also write them to an external store for later retrieval. - -The `app.function` decorator below is all we need to set turn this Python function -into a remote Modal Function! - -```python -@app.function(timeout=30 * MINUTES) -def orchestrate(lookback: int) -> list[modal.FunctionCall]: - llm = Vllm() - - today = datetime.now(tz=ZoneInfo("America/New_York")).date() # Eastern Time - print(f"Loading SEC filing data for the last {lookback} days") - folders = list(extract.map(today - dt.timedelta(days=ii) for ii in range(lookback))) - folders = list( - filter( # drop days with no data (weekends, holidays) - lambda f: f is not None, folders - ) - ) - - print("Transforming raw SEC filings for these dates:", *folders) - filing_batches = list(transform.map(folders)) - n_filings = sum(map(len, filing_batches)) - submission_batches_gen = rechunk(filing_batches) - - print(f"Submitting {n_filings} SEC filings to LLM for summarization") - jobs = list(llm.process.spawn(batch) for batch in submission_batches_gen) - if jobs: - print("FunctionCall IDs:", *[job.object_id for job in jobs], sep="\n\t") - - return jobs - - -``` - -Before going any further, we should agree on the format that our -`transform` and `llm.process` functions will use to communicate -individual elements. - -We'll use a lightweight Python `dataclass` to represent -each SEC `Filing`. - -For our task, we're going to take the `text` of a filing and produce -a `summary`. So the `text` is mandatory and the `summary` starts out empty (`None`), -to be filled in by the LLM. - -We'll also keep a bit of metadata that should be included. -But we're not sure all of these fields will exist (API data is messy!), -so we reserve the right to set them to `None`. - -```python -@dataclass -class Filing: - accession_number: str | None - form: str | None - cik: str | None - text: str - summary: str | None = None - - -``` - -With the basic orchestration set up, -let's implement each component in turn. - -## Serving tokens at maximum throughput - -First, the LLM service. - -### Configuring vLLM for maximum throughput - -We choose the [vLLM](https://vllm.ai) -inference engine. You might alternatively use [SGLang](https://docs.sglang.io). -In our experience, new models and other features -are implemented first in vLLM, and vLLM has a small edge in throughput -over SGLang, but either can work well. - -```python -vllm_image = ( - modal.Image.from_registry("nvidia/cuda:12.9.0-devel-ubuntu22.04", add_python="3.13") - .entrypoint([]) - .uv_pip_install("vllm==0.13.0", "huggingface-hub==0.36.0") - .env({"HF_XET_HIGH_PERFORMANCE": "1"}) # faster model transfers -) - -``` - -vLLM will automatically download the model for us and produce some compilation artifacts, -all of which are saved to disk. -Modal Functions are serverless and disks are ephemeral, -so we attach a [Modal Volume](https://modal.com/docs/guide/volumes) -to the locations where vLLM saves these files to ensure that they persist. - -```python -hf_cache_vol = modal.Volume.from_name("huggingface-cache", create_if_missing=True) -vllm_cache_vol = modal.Volume.from_name("vllm-cache", create_if_missing=True) - -``` - -Like a database or web server, an LLM inference engine -typically has a few knobs to twiddle to adjust performance -on different workloads. - -First and foremost, you need to pick the hardware it will run on. -We'll be running a smaller model in 8bit floating point format. -Hopper and later GPUs have native support for this format. -To maximize throughput, we want to ensure our inference is -[compute-bound](https://modal.com/gpu-glossary/perf/compute-bound): -the bottleneck is not loading weights/KV cache from memory, -it's performing computations on those values. -Roughly speaking, we want to be able to put together a batch -whose size is within an order of magnitude of the -[ridge point arithmetic intensity](https://modal.com/gpu-glossary/perf/roofline-model) -of the GPU for our floating point format, which is -[~600 for an H100 SXM Tensor Code on FP8 data](https://modal.com/gpu-glossary/perf/arithmetic-intensity). - -A single H100 GPU has enough -[GPU RAM](https://modal.com/gpu-glossary/device-hardware/gpu-ram) -for pretty large batches of this data for this model, -so we stick with one of those -- and just one! -Deploying onto multiple GPUs would increase throughput *per replica*, -but not throughput *per GPU* and so not throughput *per dollar*. - -```python -GPU = "h100" - -``` - -The dictionary of arguments below cover the knobs we found it -important to tune in this case. Specifically, we -set a maximum sequence length, based on the data, -to give the engine more hints about how to pack batches. -We select [FlashInfer](https://github.com/flashinfer-ai/flashinfer) -as the provider of the attention kernels, which vLLM recommends -for higher throughput in offline serving. Finally, we -turn on the asynchronous batch scheduler, which gives a small boost -to throughput. - -```python -vllm_throughput_kwargs = { - "max_model_len": 4096 * 4, # based on data - "attention_backend": "flashinfer", # best for throughput - "async_scheduling": True, # usually faster, but not all features supported -} - -``` - -For details on these and other arguments, we recommend checking out the [vLLM docs](https://vllm.ai), -which include lots of recipes and recommendations for different workloads and models. - -### Deploying vLLM on Modal - -For offline, throughput-oriented serving, -we can use the `LLM` interface of the vLLM SDK. -This interface processes batches of inputs synchronously, -unlike the `AsyncLLM` or HTTP serving interfaces. -Dumping a large batch all at once exposes -the maximum amount of parallelism to the engine -and adds the least request management overhead, -so we can expect it to maximize throughput. -Critically, though, this means we don't get any results -until all of them are finished -- a key engineering degree of freedom -for throughput-oriented offline/batch jobs! - -We use a Modal [Cls](https://modal.com/docs/guide/lifecycle-functions) -to control the spinup and shutdown logic for the `LLM` engine. -Specifically, we create it (and warm it up with a test request) -in a method decorated with `modal.enter` -and we shut it down in a method decorated with `modal.exit`. -The code in these methods will run only once per replica, -when it is created and destroyed, respectively. - -In between, we run a batch of `Filings` through the engine, -adding the model's output text to the `summary` field. - -```python -@app.cls( - image=vllm_image, - gpu=GPU, - timeout=10 * MINUTES, - volumes={ - "/root/.cache/huggingface": hf_cache_vol, - "/root/.cache/vllm": vllm_cache_vol, - }, -) -class Vllm: - @modal.enter() - def start(self): - import vllm - - self.llm = vllm.LLM(model="Qwen/Qwen3-8B-FP8", **vllm_throughput_kwargs) - self.sampling_params = self.llm.get_default_sampling_params() - self.sampling_params.max_tokens = 1000 - - self.llm.chat([{"role": "user", "content": "Is this thing on?"}]) - - @modal.method() - def process(self, filings: list[Filing]) -> list[Filing]: - messages = [ - [ - { - "role": "user", - "content": f"/no_think Summarize this SEC filing in a single, short paragraph.\n\n{filing.text}", - } - ] - for filing in filings - ] - - start = time.time() - responses = self.llm.chat(messages, sampling_params=self.sampling_params) - duration_s = time.time() - start - - in_token_count = sum(len(response.prompt_token_ids) for response in responses) - out_token_count = sum( - len(response.outputs[0].token_ids) for response in responses - ) - - print(f"processed {in_token_count} prompt tokens in {int(duration_s)} seconds") - print(f"generated {out_token_count} output tokens in {int(duration_s)} seconds") - - for response, filing in zip(responses, filings): - filing.summary = response.outputs[0].text - - return filings - - @modal.exit() - def stop(self): - del self.llm - - -``` - -And that's it for the LLM portion of the pipeline! -The remainder of this document is code and explanation -for the data loading and processing steps. -The details are mostly specific to this dataset, -but there are a few general Modal tips and tricks -for batch processing along the way. - -## Transforming SEC filings for batch processing - -We can avoid having to deal directly with the low-level -details of the SEC's data format by using the -[`edgartools` library](https://pypi.org/project/edgartools/). -And we can avoid worrying about compatibility with the other libraries -in our project by putting it in a separate container Image. - -```python -data_proc_image = modal.Image.debian_slim(python_version="3.13").uv_pip_install( - # pin transitive deps to avoid surprises like this one: - # https://www.edgartools.io/pandas-3-0-and-edgartools/ - "edgartools==5.8.3", - "httpx==0.28.1", - "httpxthrottlecache==0.3.0", - "pandas<3", - "pyrate-limiter==3.9.0", -) - -``` - -Instead of hitting the SEC's EDGAR Feed API every time we want to run a job, -we'll cache the results for each day in a Modal Volume. -We use Modal's [v2 Volumes](https://modal.com/docs/guide/volumes#volumes-v2-overview), -which have no limit on the number of total stored files. - -```python -sec_edgar_feed = modal.Volume.from_name( - "example-sec-edgar-daily", create_if_missing=True, version=2 -) -data_root = Path("/data") - -``` - -Note that v2 Volumes are still in beta, so data loss may be possible. -This is acceptable for most batch jobs, which extract data from an external -source of truth. - -The `transform` function below operates on a folder containing data -with one filing per file -(in [NetCDF](https://en.wikipedia.org/wiki/NetCDF)/`.nc` format). - -Loading thousands of filings with `edgartools` takes tens of seconds. -We can speed it up by running in parallel on Modal instead! -But running each file in a separate container would add too much overhead. -So we group up the files into `chunks` of ~100 and pass those to -the Modal Function that actually does the work. -Again, we use `map` to transparently scale out across containers. - -```python -@app.function( - volumes={data_root: sec_edgar_feed}, timeout=10 * MINUTES, scaledown_window=5 -) -def transform(folder: str | None) -> list[Filing]: - if folder is None: - return [] - - folder_path = data_root / folder - paths = [p for p in folder_path.iterdir() if p.is_file() and p.suffix == ".nc"] - - print(f"Processing {len(paths)} filings") - - chunks: list[list[Path]] = [paths[i : i + 100] for i in range(0, len(paths), 100)] - - batches = list(_transform_filing_batch.map(chunks)) - - filings = [f for batch in batches for f in batch if f is not None] - - print(f"Found documents for {len(filings)} filings out of {len(paths)}") - - return filings - - -@app.function( - volumes={data_root: sec_edgar_feed}, - scaledown_window=5, - image=data_proc_image, - timeout=10 * MINUTES, -) -def _transform_filing_batch(raw_filing_paths: list[Path]) -> list[Filing | None]: - from edgar.sgml import FilingSGML - - out = [] - for raw_filing_path in raw_filing_paths: - sgml = FilingSGML.from_source(raw_filing_path) - text = extract_text(sgml) - if text is None: - out.append(None) - continue - out.append( - Filing( - accession_number=sgml.accession_number, - form=sgml.form, - cik=sgml.cik, - text=text, - ) - ) - return out - - -``` - -Because these containers are cheap to scale up and are only needed for -a brief burst during the pipeline, we set the `scaledown_window` for the containers -to a much lower value than the default of five minutes -- here, five seconds. - -## Loading filings from the SEC EDGAR Feed - -We complete our reverse tour of the pipeline by loading the data from the original source: -the [SEC EDGAR Feed](https://www.sec.gov/Archives/edgar/Feed/), -an archive of daily filings going back over three decades. - -We use the `requests` library to pull data from the API. -We'll be downloading large (maybe megabytes to few gigabytes) -files with low concurrency, so there's little benefit to running an asynchronous web client. - -```python -scraper_image = modal.Image.debian_slim(python_version="3.13").uv_pip_install( - "requests==2.32.5" -) - -``` - -Our concurrency is limited by the policies of the SEC EDGAR API. -The limit is 10 RPS, which we aim to stay under by setting the `max` number of `containers` -running our extraction to 10. - -We add [retries](https://modal.com/docs/guide/retries) -via our Modal decorator as well, so that we can tolerate temporary outages or rate limits. - -Note that we also attach the same Volume used in the `transform` Functions above -and we explicitly [`.commit`](https://modal.com/docs/reference/modal.Volume#commit) our writes -so that they will be visible to future containers running `transform`. - -```python -@app.function( - max_containers=10, - volumes={data_root: sec_edgar_feed}, - retries=5, - image=scraper_image, - scaledown_window=5, -) -def extract(day: dt.date) -> str | None: - target_folder = str(day) - day_dir = data_root / target_folder - daily_name = f"{day:%Y%m%d}.nc.tar.gz" - tar_path = day_dir / daily_name - - # If the folder doesn't exist yet, try downloading the day's tarball - if not tar_path.exists(): - print(f"Looking for data for {day} in SEC EDGAR Feed") - ok = _download_from_sec_edgar(day, day_dir) - if not ok: - return None - - if not any(p.suffix == ".nc" for p in day_dir.iterdir()): - print(f"Loading data for {day} from {tar_path}") - _extract_tarfile(tar_path, day_dir) - - sec_edgar_feed.commit() - print(f"Data for {day} loaded") - - return target_folder - - -``` - -## Addenda - -The remainder of this code consists of utility functions and boiler plate used in the -main code above. - -### Utilities for transforming SEC Filings - -The code in this section is used to transform, normalize, and otherwise munge -the raw filings downloaded from the SEC. - -For LLM serving, the most important piece here is the function to truncate -documents. A maximum document length can be used to set a loose bound -on the sequence length in the LLM engine configuration. - -```python -def normalize_text(text: str) -> str: - text = text.replace("\r\n", "\n").replace("\r", "\n") - text = clean_xml(text) - return text - - -def clean_xml(xml: str) -> str: - import re - - _XMLNS_ATTR_RE = re.compile(r'\s+xmlns(:\w+)?="[^"]*"', re.I) - _XML_DECL_RE = re.compile(r"^\s*<\?xml[^>]*\?>\s*", re.I) - _EMPTY_TAG_RE = re.compile(r"<(\w+)([^>]*)>\s*", re.S) - _BETWEEN_TAG_WS_RE = re.compile(r">\s+<") - - xml = xml.replace("\r\n", "\n").replace("\r", "\n").strip() - - # drop xml declaration, remove xmlns attributes - xml = _XML_DECL_RE.sub("", xml) - xml = _XMLNS_ATTR_RE.sub("", xml) - - # replace whitespace between tags with a single newline - xml = _BETWEEN_TAG_WS_RE.sub("><", xml).replace("><", ">\n<") - - return xml.strip() - - -def truncate_head_tail(text: str, head: int = 13_000, tail: int = 2_000) -> str: - if len(text) <= head + tail: - return text - return text[:head].rstrip() + "\n\n[...TRUNCATED...]\n\n" + text[-tail:].lstrip() - - -def extract_text(sgml) -> str | None: - doc = sgml.xml() - return truncate_head_tail(normalize_text(doc)) if doc else None - - -``` - -### Utilities for loading filings from the SEC EDGAR Feed - -The code in this section is used to load raw data from the Feed -section of SEC EDGAR. - -Daily dumps are stored in [tar](https://www.math.utah.edu/docs/info/tar_4.html) -archives, which the code below extracts. -Archives for particular days are located by searching the SEC EDGAR Feed indices -for the appropriate URL. - -For full compliance with SEC EDGAR etiquette, -we recommend updating the `SEC_USER_AGENT` environment variable -below with your name and email. - -```python -def _download_from_sec_edgar(day: dt.date, day_dir: Path) -> bool: - import os - - import requests - - SEC_UA = os.environ.get("SEC_USER_AGENT", "YourName your.email@example.com") - session = requests.Session() - session.headers.update({"User-Agent": SEC_UA, "Accept-Encoding": "gzip, deflate"}) - - base = "https://www.sec.gov/Archives/edgar/Feed" - - def quarter(d: dt.date) -> str: - return f"QTR{(d.month - 1) // 3 + 1}" - - qtr = quarter(day) - daily_name = f"{day:%Y%m%d}.nc.tar.gz" - qtr_index = f"{base}/{day.year}/{qtr}/index.json" - - if not check_index(session, qtr_index, daily_name): - print(f"no data for {day} in SEC EDGAR Feed") - return False - - day_dir.mkdir(parents=True, exist_ok=True) - - tar_path = day_dir / daily_name - if not tar_path.exists() or tar_path.stat().st_size == 0: - url = f"{base}/{day.year}/{qtr}/{daily_name}" - print(f"Downloading from {url}") - print("This can take several minutes") - _download_tar(session, url, tar_path) - - return True - - -def _extract_tarfile(from_tar_path, to_dir): - import tarfile - - with tarfile.open(from_tar_path, "r:gz") as tf: - for member in tf: - if not (member.isfile() and member.name.endswith(".nc")): - continue - dest = to_dir / Path(member.name).name - if dest.exists() and dest.stat().st_size > 0: - continue - f = tf.extractfile(member) - if f is None: - continue - dest.write_bytes(f.read()) - - -def check_index(session, index_url, name) -> bool: - r = session.get(index_url, timeout=30) - if r.status_code == 404: - return False - r.raise_for_status() - for it in r.json().get("directory", {}).get("item", []): - if it.get("type") == "file" and it.get("name") == name: - return True - return False - - -def _download_tar(session, url, tar_path): - resp = session.get(url, timeout=500) - resp.raise_for_status() - tmp = tar_path.with_suffix(tar_path.suffix + ".part") - tmp.write_bytes(resp.content) - tmp.replace(tar_path) - -``` diff --git a/.agents/skills/modal/references/examples/webrtc_yolo.md b/.agents/skills/modal/references/examples/webrtc_yolo.md deleted file mode 100644 index 60dced59f6cc8044769c91e79ccc6afc5c418c4d..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/webrtc_yolo.md +++ /dev/null @@ -1,405 +0,0 @@ -# Real-time object detection with WebRTC and YOLO - -This example demonstrates how to architect a serverless real-time streaming application with Modal and WebRTC. -The sample application detects objects in webcam video with YOLO. - -See the clip below from a live demo of this example in a course by [Kwindla Kramer](https://machine-theory.com/), WebRTC OG and co-founder of [Daily](https://www.daily.co/). - -
- -
- -You can also try our deployment [here](https://modal-labs-examples--example-webrtc-yolo-webcamobjdet-web.modal.run). - -## What is WebRTC? - -WebRTC (Web Real-Time Communication) is an [IETF Internet protocol](https://www.rfc-editor.org/rfc/rfc8825) and a [W3C API specification](https://www.w3.org/TR/webrtc/) for real-time media streaming between peers -over internets or the World Wide Web. -What makes it so effective and different from other bidirectional web-based communication protocols (e.g. WebSockets) is that it's purpose-built for media streaming in real time. -It's primarily designed for browser applications using the JavaScript API, but [APIs exist for other languages](https://www.webrtc-developers.com/did-i-choose-the-right-webrtc-stack/). -We'll build our app using Python's [`aiortc`](https://aiortc.readthedocs.io/en/latest/) package. - -### What makes up a WebRTC application? - -A simple WebRTC app generally consists of three players: - -1. a peer that initiates the connection, -2. a peer that responds to the connection, and -3. a server that passes some initial messages between the two peers. - -First, one peer initiates the connection by offering up a description of itself - its media sources, codec capabilities, Internet Protocol (IP) addressing info, etc - which is relayed to another peer through the server. -The other peer then either accepts the offer by providing a compatible description of its own capabilities or rejects it if no compatible configuration is possible. -This process is called "signaling" or sometimes the "negotiation" in the WebRTC world, and the server that mediates it is usually called the "signaling server". - -Once the peers have agreed on a configuration there's a brief pause to establish communication... and then you're live. - -![Basic WebRTC architecture](https://modal-cdn.com/cdnbot/just_webrtc-1oic3iems_a4a8e77c.webp) A basic WebRTC app architecture - -Obviously there’s more going on under the hood. -If you want to get into the details, we recommend checking out the [RFCs](https://www.rfc-editor.org/rfc/rfc8825) or a [more-thorough explainer](https://webrtcforthecurious.com/). -In this document, we'll focus on how to architect a WebRTC application where one or more peer is running on Modal's serverless cloud infrastructure. - -If you just want to quickly get started with WebRTC for a small internal service or a hack project, check out -[our FastRTC example](https://modal.com/docs/examples/fastrtc_flip_webcam) instead. - -## How do I run a WebRTC app on Modal? - -Modal turns Python code into scalable cloud services. -When you call a Modal Function, you get one replica. -If you call it 999 more times before it returns, you have 1000 replicas. -When your Functions all return, you spin down to 0 replicas. - -The core constraints of the Modal programming model that make this possible are that Function Calls are stateless and self-contained. -In other words, correctly-written Modal Functions don't store information in memory between runs (though they might cache data to the ephemeral local disk for efficiency) and they don't create processes or tasks which must continue to run after the Function Call returns in order for the application to be correct. - -WebRTC apps, on the other hand, require passing messages back and forth in a multi-step protocol, and APIs spawn several "agents" (no, AI is not involved, just processes) which do work behind the scenes - including managing the peer-to-peer (P2P) connection itself. -This means that streaming may have only just begun when the application logic in our Function has finished. - -![Modal programming model and WebRTC signaling](https://modal-cdn.com/cdnbot/flow_comparisong6iibzq3_638bdd84.webp) Modal's stateless programming model (left) and WebRTC's stateful signaling (right) - -To ensure we properly leverage Modal's autoscaling and concurrency features, we need to align the signaling and streaming lifetimes with Modal Function Call lifetimes. - -The architecture we recommend for this appears below. - -![WebRTC on Modal](https://modal-cdn.com/cdnbot/webrtc_with_modal-2horb680q_eab69b28.webp) A clean architecture for WebRTC on Modal - -It handles passing messages between the client peer and the signaling server using a -[WebSocket](https://modal.com/docs/guide/webhooks#websockets) for persistent, bidirectional communication over the Web within a single Function Call. -(Modal's Web layer maps HTTP and WS onto Function Calls, details [here](https://modal.com/blog/serverless-http)). -We [`.spawn`](https://modal.com/docs/reference/modal.Function#spawn) the cloud peer inside the WebSocket endpoint -and communicate it using a [`modal.Queue`](https://modal.com/docs/reference/modal.Queue). - -We can then use the state of the P2P connection to determine when to return from the calls to both the signaling server and the cloud peer. -When the P2P connection has been *established*, we'll close the WebSocket which in turn ends the call to the signaling server. -And when the P2P connection has been *closed*, we'll return from the call to the cloud peer. -That way, our WebRTC application benefits from all the autoscaling and concurrency logic built into Modal -that enables users to deliver efficient cloud applications. - -We wrote two classes, `ModalWebRtcPeer` and `ModalWebRtcSignalingServer`, to abstract away that boilerplate as well as a lot of the `aiortc` implementation details. -They're also decorated with Modal [lifetime hooks](https://modal.com/docs/guide/lifecycle-functions). -Add the [`app.cls`](https://modal.com/docs/reference/modal.App#cls) decorator and some custom logic, and you're ready to deploy on Modal. - -You can find them in the [`modal_webrtc.py` file](https://github.com/modal-labs/modal-examples/blob/main/07_web/webrtc/modal_webrtc.py) provided alongside this example in the [GitHub repo](https://github.com/modal-labs/modal-examples/tree/main/07_web/webrtc/modal_webrtc.py). - -## Using `modal_webrtc` to detect objects in webcam footage - -For our WebRTC app, we'll take a client's video stream, run a [YOLO](https://docs.ultralytics.com/tasks/detect/) object detector on it with an A100 GPU on Modal, and then stream the annotated video back to the client. -With this setup, we can achieve inference times between 2-4 milliseconds per frame and RTTs below video frame rates (usually around 30 milliseconds per frame). - -Let's get started! - -### Setup - -We'll start with a simple container [Image](https://modal.com/docs/guide/images) and then - -* set it up to properly use TensorRT and the ONNX Runtime, which keep latency minimal, -* install the necessary libs for processing video, `opencv` and `ffmpeg`, and -* install the necessary Python packages. - -```python -import os -from pathlib import Path - -import modal - -from .modal_webrtc import ModalWebRtcPeer, ModalWebRtcSignalingServer - -py_version = "3.12" -tensorrt_ld_path = f"/usr/local/lib/python{py_version}/site-packages/tensorrt_libs" - -video_processing_image = ( - modal.Image.debian_slim(python_version=py_version) # matching ld path - # update locale as required by onnx - .apt_install("locales") - .run_commands( - "sed -i '/^#\\s*en_US.UTF-8 UTF-8/ s/^#//' /etc/locale.gen", # use sed to uncomment - "locale-gen en_US.UTF-8", # set locale - "update-locale LANG=en_US.UTF-8", - ) - .env({"LD_LIBRARY_PATH": tensorrt_ld_path, "LANG": "en_US.UTF-8"}) - # install system dependencies - .apt_install("python3-opencv", "ffmpeg") - # install Python dependencies - .uv_pip_install( - "aiortc==1.14.0", - "fastapi==0.115.12", - "huggingface-hub[hf_xet]==0.30.2", - "onnxruntime-gpu==1.21.0", - "opencv-python==4.11.0.86", - "tensorrt==10.9.0.34", - "torch==2.7.0", - "shortuuid==1.0.13", - ) -) - -``` - -### Cache weights and compute graphs on a Volume - -We also need to create a Modal [Volume](https://modal.com/docs/guide/volumes) to store things we need across replicas -- -primarily the model weights and ONNX inference graph, but also a few other artifacts like a video file where -we'll write out the processed video stream for testing. For more on storing model weights on Modal, see -[this guide](https://modal.com/docs/guide/model-weights). - -The very first time we run the app, downloading the model and building the ONNX inference graph will take a few minutes. -After that, we can load the cached weights and graph from the Volume, which reduces the startup time to about 15 seconds per container. - -```python -CACHE_VOLUME = modal.Volume.from_name("webrtc-yolo-cache", create_if_missing=True) -CACHE_PATH = Path("/cache") -cache = {CACHE_PATH: CACHE_VOLUME} - -app = modal.App("example-webrtc-yolo") - -``` - -### Implement YOLO object detection as a `ModalWebRtcPeer` - -Our application needs to process an incoming video track with YOLO and return an annotated video track to the source peer. - -To implement a `ModalWebRtcPeer`, we need to: - -* Decorate our subclass with `@app.cls`. We provision it with an A100 GPU and a [Secret](https://modal.com/docs/guide/secrets) credential, described below. -* Implement the method `setup_streams`. This is where we'll use `aiortc` to add the logic for processing the incoming video track with YOLO - and returning an annotated video track to the source peer. - -`ModalWebRtcPeer` has a few other methods that users can optionally implement: - -* `initialize()`: This contains any custom initialization logic, called when `@modal.enter()` is called. -* `run_streams()`: Logic for starting streams. This is necessary when the peer is the source of the stream. - This is where you'd ensure a webcam was running, start playing a video file, or spin up a [video generative model](https://modal.com/docs/examples/image_to_video). -* `get_turn_servers()`: We haven't talked about [TURN servers](https://datatracker.ietf.org/doc/html/rfc5766), - but just know that they're necessary if you want to use WebRTC across complex (e.g. carrier-grade) NAT or firewall configurations. - Free services have tight limits because TURN servers are expensive to run (lots of bandwidth and state management required). - [STUN](https://datatracker.ietf.org/doc/html/rfc5389) servers, on the other hand, are essentially just echo servers, and so there are many free services available. - If you don't provide TURN servers you can still serve your app on many networks using any of a number of free STUN servers for NAT traversal. -* `exit()`: This contains any custom cleanup logic, called when `@modal.exit()` is called. - -In our case, we load the YOLO model in `initialize` and provide server information for the free [Open Relay TURN server](https://www.metered.ca/tools/openrelay/). -If you want to use it, you'll need to create an account [here](https://dashboard.metered.ca/login?tool=turnserver) -and then create a Modal [Secret](https://modal.com/docs/guide/secrets) called `turn-credentials` [here](https://modal.com/secrets). -We also use the `@modal.concurrent` decorator to allow multiple instances of our peer to run on one GPU. - -**Setting the Region** - -Much of the latency in Internet applications comes from distance between communicating parties -- -the Internet operates within a factor of two of the speed of light, but that's just not that fast. -To minimize latency under this constraint, the physical distance of the P2P connection -between the webcam-using peer and the GPU container needs to be kept as short as possible. -We'll use the `region` parameter of the `cls` decorator to set the region of the GPU container. -You should set this to the closest region to your users. -See the [region selection](https://modal.com/docs/guide/region-selection) guide for more information. - -```python -@app.cls( - image=video_processing_image, - gpu="A100-40GB", - volumes=cache, - secrets=[modal.Secret.from_name("turn-credentials")], - region="us-east", # set to your region -) -@modal.concurrent( - target_inputs=2, # try to stick to just two peers per GPU container - max_inputs=3, # but allow up to three -) -class ObjDet(ModalWebRtcPeer): - async def initialize(self): - self.yolo_model = get_yolo_model(CACHE_PATH) - - async def setup_streams(self, peer_id: str): - from aiortc import MediaStreamTrack - - # keep us notified on connection state changes - @self.pcs[peer_id].on("connectionstatechange") - async def on_connectionstatechange() -> None: - if self.pcs[peer_id]: - print( - f"Video Processor, {self.id}, connection state to {peer_id}: {self.pcs[peer_id].connectionState}" - ) - - # when we receive a track from the source peer - # we create a processed track and add it to our stream - # back to the source peer - @self.pcs[peer_id].on("track") - def on_track(track: MediaStreamTrack) -> None: - print( - f"Video Processor, {self.id}, received {track.kind} track from {peer_id}" - ) - - output_track = get_yolo_track(track, self.yolo_model) # see Addenda - self.pcs[peer_id].addTrack(output_track) - - # keep us notified when the incoming track ends - @track.on("ended") - async def on_ended() -> None: - print( - f"Video Processor, {self.id}, incoming video track from {peer_id} ended" - ) - - async def get_turn_servers(self, peer_id=None, msg=None) -> dict: - creds = { - "username": os.environ["TURN_USERNAME"], - "credential": os.environ["TURN_CREDENTIAL"], - } - - turn_servers = [ - {"urls": "stun:stun.relay.metered.ca:80"}, # STUN is free, no creds neeeded - # for TURN, sign up for the free service here: https://www.metered.ca/tools/openrelay/ - {"urls": "turn:standard.relay.metered.ca:80"} | creds, - {"urls": "turn:standard.relay.metered.ca:80?transport=tcp"} | creds, - {"urls": "turn:standard.relay.metered.ca:443"} | creds, - {"urls": "turns:standard.relay.metered.ca:443?transport=tcp"} | creds, - ] - - return {"type": "turn_servers", "ice_servers": turn_servers} - - -``` - -### Implement a `SignalingServer` - -The `ModalWebRtcSignalingServer` class is much simpler to implement. -The main thing we need to do is implement the `get_modal_peer_class` method which will return our implementation of the `ModalWebRtcPeer` class, `ObjDet`. - -It also has an `initialize()` method we can optionally override (called at the beginning of the [container lifecycle](https://modal.com/docs/guide/lifecycle-functions)) -as well as a `web_app` property which will be [served by Modal](https://modal.com/docs/guide/webhooks#asgi-apps---fastapi-fasthtml-starlette). -We'll use these to add a frontend which uses the WebRTC JavaScript API to stream a peer's webcam from the browser. - -The JavaScript and HTML files are alongside this example in the [Github repo](https://github.com/modal-labs/modal-examples/tree/main/07_web/webrtc/frontend). - -```python -this_directory = Path(__file__).parent.resolve() -server_image = ( - modal.Image.debian_slim(python_version="3.12") - .uv_pip_install( - "fastapi[standard]==0.115.4", - "shortuuid==1.0.13", - ) - .add_local_dir(this_directory / "frontend", remote_path="/frontend") -) - - -@app.cls(image=server_image) -class WebcamObjDet(ModalWebRtcSignalingServer): - def get_modal_peer_class(self): - return ObjDet - - def initialize(self): - from fastapi.responses import HTMLResponse - from fastapi.staticfiles import StaticFiles - - self.web_app.mount("/static", StaticFiles(directory="/frontend")) - - @self.web_app.get("/") - async def root(): - html = open("/frontend/index.html").read() - return HTMLResponse(content=html) - - -``` - -## Addenda - -The remainder of this page is not central to running a WebRTC application on Modal, -but is included for completeness. - -### YOLO helper functions - -The two functions below are used to set up the YOLO model and create our custom [`MediaStreamTrack`](https://aiortc.readthedocs.io/en/latest/api.html#aiortc.MediaStreamTrack). - -The first, `get_yolo_model`, sets up the ONNXRuntime and loads the model weights. -We call this in the `initialize` method of the `ModalWebRtcPeer` class -so that it only happens once per container. - -```python -def get_yolo_model(cache_path): - import onnxruntime - - from .yolo import YOLOv10 - - onnxruntime.preload_dlls() - return YOLOv10(cache_path) - - -``` - -The second, `get_yolo_track`, creates a custom `MediaStreamTrack` that performs object detection on the video stream. -We call this in the `setup_streams` method of the `ModalWebRtcPeer` class -so it happens once per peer connection. - -```python -def get_yolo_track(track, yolo_model=None): - import numpy as np - import onnxruntime - from aiortc import MediaStreamTrack - from aiortc.contrib.media import VideoFrame - - from .yolo import YOLOv10 - - class YOLOTrack(MediaStreamTrack): - """ - Custom media stream track performs object detection - on the video stream and passes it back to the source peer - """ - - kind: str = "video" - conf_threshold: float = 0.15 - - def __init__(self, track: MediaStreamTrack, yolo_model=None) -> None: - super().__init__() - - self.track = track - if yolo_model is None: - onnxruntime.preload_dlls() - self.yolo_model = YOLOv10(CACHE_PATH) - else: - self.yolo_model = yolo_model - - def detection(self, image: np.ndarray) -> np.ndarray: - import cv2 - - orig_shape = image.shape[:-1] - - image = cv2.resize( - image, - (self.yolo_model.input_width, self.yolo_model.input_height), - ) - - image = self.yolo_model.detect_objects(image, self.conf_threshold) - - image = cv2.resize(image, (orig_shape[1], orig_shape[0])) - - return image - - # this is the essential method we need to implement - # to create a custom MediaStreamTrack - async def recv(self) -> VideoFrame: - frame = await self.track.recv() - img = frame.to_ndarray(format="bgr24") - - processed_img = self.detection(img) - - # VideoFrames are from a really nice package called av - # which is a pythonic wrapper around ffmpeg - # and a dependency of aiortc - new_frame = VideoFrame.from_ndarray(processed_img, format="bgr24") - new_frame.pts = frame.pts - new_frame.time_base = frame.time_base - - return new_frame - - return YOLOTrack(track, yolo_model) - - -``` - -### Testing a WebRTC application on Modal - -As any seasoned developer of real-time applications on the Web will tell you, -testing and ensuring correctness is quite difficult. We spent nearly as much time -designing and troubleshooting an appropriate testing process for this application as we did writing -the application itself! - -You can find the testing code in the GitHub repository [here](https://github.com/modal-labs/modal-examples/tree/main/07_web/webrtc/webrtc_yolo_test.py). diff --git a/.agents/skills/modal/references/examples/webscraper.md b/.agents/skills/modal/references/examples/webscraper.md deleted file mode 100644 index 30f0acd67f51c42cd4e8a39147e288f831cfd6bb..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/examples/webscraper.md +++ /dev/null @@ -1,323 +0,0 @@ -# A simple web scraper - -In this guide we'll introduce you to Modal by writing a simple web scraper. -We'll explain the foundations of a Modal application step by step. - -## Set up your first Modal app - -Modal Apps are orchestrated as Python scripts but can theoretically run -anything you can run in a container. To get you started, make sure to install -the latest `modal` Python package and set up an API token (the first two steps -[here](https://modal.com/docs/guide)). - -## Scrape links locally - -First, we create an empty Python file `webscraper.py`. This file will contain our -application code. Let's write some basic Python code to fetch the contents of a -web page and print the links (`href` attributes) it finds in the document: - -```python -import re -import sys -import urllib.request - - -def get_links(url): - response = urllib.request.urlopen(url) - html = response.read().decode("utf8") - links = [] - for match in re.finditer('href="(.*?)"', html): - links.append(match.group(1)) - return links - - -if __name__ == "__main__": - links = get_links(sys.argv[1]) - print(links) -``` - -Now obviously this is just pure standard library Python code, and you can run it -on your machine: - -```bash -$ python webscraper.py http://example.com -['https://www.iana.org/domains/example'] -``` - -## Run it on Modal - -To make the `get_links` function run on Modal instead of your local machine, all -you need to do is - -* Import `modal` -* Create a [`modal.App`](/docs/reference/modal.App) instance -* Add an `@app.function()` annotation to your function -* Replace the `if __name__ == "__main__":` block with a function decorated with - [`@app.local_entrypoint()`](/docs/reference/modal.App#local_entrypoint) -* Call `get_links` using `get_links.remote` - -```python -import re -import urllib.request -import modal - -app = modal.App(name="example-webscraper") - - -@app.function() -def get_links(url): - response = urllib.request.urlopen(url) - html = response.read().decode("utf8") - links = [] - for match in re.finditer('href="(.*?)"', html): - links.append(match.group(1)) - return links - - -@app.local_entrypoint() -def main(url): - links = get_links.remote(url) - print(links) -``` - -You can now run this with the Modal CLI, using `modal run` instead of `python`. -This time, you'll see additional progress indicators while the script is -running, something like: - -```bash -$ modal run webscraper.py --url http://example.com -✓ Initialized. -✓ Created objects. -['https://www.iana.org/domains/example'] -✓ App completed. -``` - -## Add dependencies - -In the code above we make use of the Python standard library `urllib` library. -This works great for static web pages, but many pages these days use javascript -to dynamically load content, which wouldn't appear in the loaded html file. -Let's use the [Playwright](https://playwright.dev/python/docs/intro) package to -instead launch a headless Chromium browser which can interpret any javascript -that might be on the page. - -We can pass [custom container images](/docs/guide/images) (defined using -[`modal.Image`](/docs/reference/modal.Image)) to the `@app.function()` -decorator. We'll make use of the `modal.Image.debian_slim` pre-bundled Image add -the shell commands to install Playwright and its dependencies: - -```python -import modal - -app = modal.App("example-webscraper") -playwright_image = modal.Image.debian_slim(python_version="3.10").run_commands( - "apt-get update", - "apt-get install -y software-properties-common", - "apt-add-repository non-free", - "apt-add-repository contrib", - "pip install playwright==1.42.0", - "playwright install-deps chromium", - "playwright install chromium", -) - -``` - -Note that we don't have to install Playwright or Chromium on our development -machine since this will all run in Modal. We can now modify our `get_links` -function to make use of the new tools. - -```python -@app.function(image=playwright_image) -async def get_links(cur_url: str) -> list[str]: - from playwright.async_api import ( - TimeoutError as PlaywrightTimeoutError, - async_playwright, - ) - - async with async_playwright() as p: - browser = await p.chromium.launch() - page = await browser.new_page() - - try: - await page.goto(cur_url, timeout=10_000) # ten seconds - except PlaywrightTimeoutError: - print(f"Timeout loading {cur_url}, skipping") - await browser.close() - return [] - - links = await page.eval_on_selector_all( - "a[href]", "elements => elements.map(element => element.href)" - ) - await browser.close() - - print("Links", links) - return list(set(links)) - - -``` - -Since Playwright has a nice async interface, we'll redeclare our `get_links` -function as async (Modal works with both sync and async functions). - -The first time you run the function after making this change, you'll notice that -the output first shows the progress of building the image you specified, -after which your function runs like before. This image is then cached so that on -subsequent runs of the function it will not be rebuilt as long as the image -definition is the same. - -## Scale out - -So far, our script only fetches the links for a single page. What if we want to -scrape a large list of links in parallel? - -We can do this easily with Modal, because of some magic: the function we wrapped -with the `@app.function()` decorator is no longer an ordinary function, but a -Modal [Function](https://modal.com/docs/reference/modal.Function) object. This -means it comes with a `map` property built in, that lets us run this function -for all inputs in parallel, scaling up to as many workers as needed. - -Let's change our code to scrape all urls we feed to it in parallel: - -```python -@app.local_entrypoint() -def main(): - urls = ["http://modal.com", "http://github.com"] - for links in get_links.map(urls): - for link in links: - print(link) -``` - -## Deploy it and run it on a schedule - -Let's say we want to log the scraped links daily. We move the print loop into -its own Modal function and annotate it with a `modal.Period(days=1)` schedule - -indicating we want to run it once per day. Since the scheduled function will not -run from our command line, we also add a hard-coded list of links to crawl for -now. In a more realistic setting we could read this from a database or other -accessible data source. - -```python -@app.function(schedule=modal.Period(days=1)) -def daily_scrape(): - urls = ["http://modal.com", "http://github.com"] - for links in get_links.map(urls): - for link in links: - print(link) -``` - -To deploy App permanently, run the command - -``` -modal deploy webscraper.py -``` - -Running this command deploys this function and then closes immediately. We can -see the deployment and all of its runs, including the printed links, on the -Modal [Apps page](https://modal.com/apps). Rerunning the script will redeploy -the code with any changes you have made - overwriting an existing deploy with -the same name ("example-webscraper" in this case). - -## Add Secrets and integrate with other systems - -Instead of looking at the links in the run logs of our deployments, let's say we -wanted to post them to a `#scraped-links` Slack channel. To do this, we can -make use of the [Slack API](https://api.slack.com/) and the `slack-sdk` -[PyPI package](https://pypi.org/project/slack-sdk/). - -The Slack SDK WebClient requires an API token to get access to our Slack -Workspace, and since it's bad practice to hardcode credentials into application -code we make use of Modal's **Secrets**. Secrets are snippets of data that will -be injected as environment variables in the containers running your functions. - -The easiest way to create Secrets is to go to the -[Secrets section of modal.com](https://modal.com/secrets). You can both create a -free-form secret with any environment variables, or make use of presets for -common services. We'll use the Slack preset and after filling in the necessary -information we are presented with a snippet of code that can be used to post to -Slack using our credentials, which looks something like: - -```python -import os - -slack_sdk_image = modal.Image.debian_slim(python_version="3.10").uv_pip_install( - "slack-sdk" -) - - -@app.function( - image=slack_sdk_image, - secrets=[ - modal.Secret.from_name( - "scraper-slack-secret", required_keys=["SLACK_BOT_TOKEN"] - ) - ], - retries=3, -) -def bot_token_msg(channel, message): - import slack_sdk - - client = slack_sdk.WebClient(token=os.environ["SLACK_BOT_TOKEN"]) - print(f"Posting {message} to #{channel}") - client.chat_postMessage(channel=channel, text=message) - - -``` - -Notice the `retries` in the `@app.function` decorator. -That parameter adds automatic retries when Function calls fail -due to temporary issues, like rate limits. Read more [here](https://modal.com/docs/guide/retries) - -Copy that code, then amend the `daily_scrape` function to call -`bot_token_msg`. We also add a per-URL `limit` for good measure. - -```python -@app.function(schedule=modal.Period(days=1)) -def daily_scrape(limit: int = 50): - urls = ["http://modal.com", "http://github.com"] - - for links in get_links.map(urls): - for link in links[:limit]: - bot_token_msg.remote("scraped-links", link) - - -@app.local_entrypoint() -def main(): - urls = ["http://modal.com", "http://github.com"] - for links in get_links.map(urls): - for link in links: - print(link) - - -``` - -Note that we are freely making function calls across completely different -container images, as if they were regular Python functions in the same program! - -We keep the `local_entrypoint` the same so that we can still `modal run` -this script to test the scraping behavior without posting to Slack. - -```bash -modal run webscraper.py # runs get_links.map via the local_entrypoint -``` - -If we want to test the `daily_scrape` or `bot_token_msg` Functions themselves, we can do that too! -We just add the name of the Function to the end of our `modal run` command: - -```bash -modal run webscraper.py::daily_scrape --limit 1 # quick test -``` - -Now redeploy the script to overwrite the old deploy with our updated code, and -you'll get a daily feed of scraped links in your Slack channel 🎉 - -```bash -modal deploy webscraper.py -``` - -## Summary - -We have shown how you can use Modal to develop distributed Python data -applications using custom containers. Through simple constructs we were able to -add parallel execution. With the change of a single line of code were were able -to go from experimental development code to a deployed application. We hope -this overview gives you a glimpse of what you are able to build using Modal. diff --git a/.agents/skills/modal/references/guide/apps.md b/.agents/skills/modal/references/guide/apps.md deleted file mode 100644 index 30be606b7585246d87c94580b23cf9bdd9f28a2b..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/apps.md +++ /dev/null @@ -1,149 +0,0 @@ -# Apps, Functions, and entrypoints - -An [`App`](/docs/reference/modal.App) represents an application running on Modal. It groups one or more Functions for atomic deployment and acts as a shared namespace. All Functions and Clses are associated with an -App. - -A [`Function`](/docs/reference/modal.Function) acts as an independent unit once it is deployed, and [scales up and down](/docs/guide/scale) independently from other Functions. If there are no live inputs to the Function then by default, no containers will run and your account will not be charged for compute resources, even if the App it belongs to is deployed. - -An App can be ephemeral or deployed. You can view a list of all currently running Apps on the [`apps`](/apps) page. - -The code for a Modal App defining two separate Functions might look something like this: - -```python - -import modal - -app = modal.App(name="my-modal-app") - - -@app.function() -def f(): - print("Hello world!") - - -@app.function() -def g(): - print("Goodbye world!") - -``` - -## Ephemeral Apps - -An ephemeral App is created when you use the -[`modal run`](/docs/reference/cli/run) CLI command, or the -[`app.run`](/docs/reference/modal.App#run) method. This creates a temporary -App that only exists for the duration of your script. - -Ephemeral Apps are stopped automatically when the calling program exits, or when -the server detects that the client is no longer connected. -You can use -[`--detach`](/docs/reference/cli/run) in order to keep an ephemeral App running even -after the client exits. - -By using `app.run` you can run your Modal apps from within your Python scripts: - -```python -def main(): - ... - with app.run(): - some_modal_function.remote() -``` - -By default, running your app in this way won't propagate Modal logs and progress bar messages. To enable output, use the [`modal.enable_output`](/docs/reference/modal.enable_output) context manager: - -```python -def main(): - ... - with modal.enable_output(): - with app.run(): - some_modal_function.remote() -``` - -## Deployed Apps - -A deployed App is created using the [`modal deploy`](/docs/reference/cli/deploy) -CLI command. The App is persisted indefinitely until you stop it via the -[web UI](/apps) or the [`modal app stop`](/docs/reference/cli/app#modal-app-stop) command. Functions in a deployed App that have an attached -[schedule](/docs/guide/cron) will be run on a schedule. Otherwise, you can -invoke them manually using -[Web Functions or Python](/docs/guide/trigger-deployed-functions). - -Deployed Apps are named via the [`App`](/docs/reference/modal.App#modalapp) -constructor. Re-deploying an existing `App` (based on the name) will update it -in place. - -## Entrypoints for ephemeral Apps - -The code that runs first when you `modal run` an App is called the "entrypoint". - -You can register a local entrypoint using the -[`@app.local_entrypoint()`](/docs/reference/modal.App#local_entrypoint) -decorator. You can also use a regular Modal function as an entrypoint, in which -case only the code in global scope is executed locally. - -### Argument parsing - -If your entrypoint function takes arguments with primitive types, `modal run` -automatically parses them as CLI options. For example, the following function -can be called with `modal run script.py --foo 1 --bar "hello"`: - -```python -# script.py - -@app.local_entrypoint() -def main(foo: int, bar: str): - some_modal_function.remote(foo, bar) -``` - -If you wish to use your own argument parsing library, such as `argparse`, you can instead accept a variable-length argument list for your entrypoint or your function. In this case, Modal skips CLI parsing and forwards CLI arguments as a tuple of strings. For example, the following function can be invoked with `modal run my_file.py --foo=42 --bar="baz"`: - -```python -import argparse - -@app.function() -def train(*arglist): - parser = argparse.ArgumentParser() - parser.add_argument("--foo", type=int) - parser.add_argument("--bar", type=str) - args = parser.parse_args(args = arglist) -``` - -### Manually specifying an entrypoint - -If there is only one `local_entrypoint` registered, -[`modal run script.py`](/docs/reference/cli/run) will automatically use it. If -you have no entrypoint specified, and just one decorated Modal function, that -will be used as a remote entrypoint instead. Otherwise, you can direct -`modal run` to use a specific entrypoint. - -For example, if you have a function decorated with -[`@app.function()`](/docs/reference/modal.App#function) in your file: - -```python -# script.py - -@app.function() -def f(): - print("Hello world!") - - -@app.function() -def g(): - print("Goodbye world!") - - -@app.local_entrypoint() -def main(): - f.remote() -``` - -Running [`modal run script.py`](/docs/reference/cli/run) will execute the `main` -function locally, which would call the `f` function remotely. However you can -instead run `modal run script.py::app.f` or `modal run script.py::app.g` to -execute `f` or `g` directly. - -## Apps were once Stubs - -The `modal.App` class in the client was previously called `modal.Stub`. The -old name was kept as an alias for some time, but from Modal 1.0.0 onwards, -using `modal.Stub` will result in an error. diff --git a/.agents/skills/modal/references/guide/async.md b/.agents/skills/modal/references/guide/async.md deleted file mode 100644 index 9853588ba52830502a76def482f7f1385671e20e..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/async.md +++ /dev/null @@ -1,66 +0,0 @@ -# Asynchronous API usage - -All of the functions in Modal are available in both standard (blocking) and -asynchronous variants. The async interface can be accessed by appending `.aio` -to any function in the Modal API. - -For example, instead of `my_modal_function.remote("hello")` in a blocking -context, you can use `await my_modal_function.remote.aio("hello")` to get an -asynchronous coroutine response, for use with Python's `asyncio` library. - -```python -import asyncio -import modal - -app = modal.App() - - -@app.function() -async def myfunc(): - ... - - -@app.local_entrypoint() -async def main(): - # execute 100 remote calls to myfunc in parallel - await asyncio.gather(*[myfunc.remote.aio() for i in range(100)]) -``` - -This is an advanced feature. If you are comfortable with asynchronous -programming, you can use this to create arbitrary parallel execution patterns, -with the added benefit that any Modal functions will be executed remotely. - -## Async functions - -Regardless if you use an async runtime (like `asyncio`) in your usage of *Modal -itself*, you are free to define your `app.function`-decorated function bodies -as either async or blocking. Both kinds of definitions will work for remote -Modal function calls from both any context. - -An async function can call a blocking function, and vice versa. - -```python -@app.function() -def blocking_function(): - return 42 - - -@app.function() -async def async_function(): - x = await blocking_function.remote.aio() - return x * 10 - - -@app.local_entrypoint() -def blocking_main(): - print(async_function.remote()) # => 420 -``` - -If a function is configured to support multiple concurrent inputs per container, -the behavior varies slightly between blocking and async contexts: - -* In a blocking context, concurrent inputs will run on separate Python threads. - These are subject to the GIL, but they can still lead to race conditions if - used with non-threadsafe objects. -* In an async context, concurrent inputs are simply scheduled as coroutines on - the executor thread. Everything remains single-threaded. diff --git a/.agents/skills/modal/references/guide/audit-logs.md b/.agents/skills/modal/references/guide/audit-logs.md deleted file mode 100644 index af9a121b51ee44b2abb7ed7736509199e3df3f4c..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/audit-logs.md +++ /dev/null @@ -1,113 +0,0 @@ -# Audit Logs - - -Audit logs are available on the Enterprise plan. Contact sales@modal.com for more information. - - -Audit logs give your workspace an append-only record of the sensitive -actions that change its state — who did what, when, to which resource, and -from where. They are designed for compliance reviews, incident -investigation, and answering questions like *"did anyone delete this Secret -last Thursday?"* without asking Modal support. - -Audit logs are viewable within the settings page. - -
- -
- -## Fields - -Every audit event captures the same shape alongside the time it occurred: - -| Field | What it is | -| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -| `action` | The kind of change that happened — e.g. `secret.create`, `app.deploy`. See the full list [below](#actions). | -| `actor` | The user or service user that initiated the action. | -| `targets` | The resource(s) the action affected, each recorded by ID so the event stays attributable after a rename or delete. | -| `context.environment` | The environment the action was scoped to. | -| `context.ip_address` | The client IP address. | -| `context.source` | `web` for the dashboard, `sdk` for the Modal CLI and client libraries. | -| `status` | Whether the action succeeded or failed. | -| `metadata` | Action-specific extra fields — e.g. the old and new budget values for `workspace.set_budget`, or the requested region for a Proxy. | - -## Filtering - -Filters are entered in the search bar above the table as `key:value` -pairs, separated by spaces. Any filter can be **negated** by prefixing it -with `-` to exclude matching events. The search bar autocompletes keys -and values as you type. - -For example: - -| Filter | Matches | -| -------------------------------------------- | ------------------------------------------------ | -| `action:secret.create` | Every Secret created in the selected time range. | -| `-status:success` | All actions that did not succeed. | -| `action:volume.delete` `-actor_type:service` | Volume deletions by non-service users. | - -## Actions - -The table below lists every action currently recorded. New actions will be -added as additional workspace operations are instrumented. - -> Note: **container runtime activity is not audited.** Audit logs record -> workspace-level actions (deploying an App, creating a Volume, revoking a -> token) — not individual Function invocations or Sandbox `exec` calls, -> which are captured in Function and Sandbox logs. - -
- - - -| Action | Description | -| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `access_grant.approve` | A workspace manager approved a pending Modal-admin access grant. | -| `access_grant.create` | A Modal admin requested time-limited access to a workspace. | -| `access_grant.revoke` | A workspace manager revoked an active Modal-admin access grant. | -| `app.deploy` | An App was deployed to the workspace (via `modal deploy` or implicitly via `App.lookup`). | -| `app.rollback` | An App was rolled back to an earlier deployed version. | -| `app.rollover` | An App was rolled over — its current version was redeployed, restarting running tasks. | -| `app.run` | An ephemeral App was started with `modal run` or `modal serve`. | -| `app.stop` | An App was stopped from the dashboard or with `modal app stop`. | -| `container.stop` | A running container (task) was terminated from the dashboard or CLI. Routine container exits at the end of a call are not audited. | -| `dict.create` | A Dict was created. | -| `dict.get` | An existing Dict was looked up by name or by ID. | -| `domain.create` | A custom domain was attached to an environment. | -| `domain.delete` | A custom domain was removed. | -| `environment.create` | A new environment was created in the workspace. | -| `environment.delete` | An environment was deleted. | -| `environment.get` | An environment was looked up by name. | -| `environment.update` | An environment's settings changed — name, web suffix, or per-environment concurrency limits. Before/after values are recorded in the event metadata. | -| `environment.update_member` | A user or service user's per-environment role (Contributor / Viewer) was changed, or their environment-level access was removed. Independent of their workspace-level role. | -| `image.delete` | An Image was deleted. | -| `invite.create` | A user-to-user workspace invite was generated. | -| `invite.create_for_workspace` | A workspace-wide invite link was generated by a workspace admin. | -| `member.delete` | A member was removed from the workspace. | -| `member.set_role` | A workspace member's workspace-wide role (Owner / Manager / User) was changed. The affected member(s) appear in the event targets and the new role(s) are recorded in the event metadata. Per-environment access is set separately via `environment.update_member`. | -| `nfs.create` | A NetworkFileSystem was created. | -| `nfs.get` | An existing NetworkFileSystem was looked up by name. | -| `proxy.add_ip` | A static egress IP was added to a Proxy. | -| `proxy.create` | A Proxy was created. The requested name and region are recorded in the event metadata. | -| `proxy.delete` | A Proxy was deleted. | -| `queue.delete` | A Queue was deleted. | -| `queue.get` | An existing Queue was looked up by ID. | -| `sandbox.create` | A Sandbox was launched. | -| `sandbox.terminate` | A Sandbox was explicitly terminated before its natural exit. | -| `secret.create` | A Secret was created or its values were overwritten (via `modal secret create` or the dashboard). | -| `secret.get` | A named Secret was resolved to an ID (e.g. at deploy, or when opening a secret in the dashboard). Values are not returned; only the Secret's ID and metadata are. | -| `token.delete` | An API token was revoked. | -| `user.create` | A new user account was created. | -| `user.set_approval` | A user's workspace-membership approval state was changed. The new approved/unapproved value is recorded in the event metadata. | -| `volume.create` | A Volume was created. | -| `volume.delete` | A Volume was deleted. | -| `volume.get` | An existing Volume was looked up by name or by ID. | -| `volume.rename` | A Volume was renamed. | -| `workspace.create` | A new workspace was created. | -| `workspace.join` | A user joined a workspace (by accepting an invite or self-serve signup). | -| `workspace.leave` | A user left a workspace. | -| `workspace.set_budget` | A workspace's spend budget was updated. The previous and new per-cycle budget values are recorded in the event metadata. | - - diff --git a/.agents/skills/modal/references/guide/batch-processing.md b/.agents/skills/modal/references/guide/batch-processing.md deleted file mode 100644 index 9b323bf81d59b5f608b0015956057471fd77cf79..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/batch-processing.md +++ /dev/null @@ -1,84 +0,0 @@ -# Batch Processing - -Modal is optimized for large-scale batch processing, allowing functions to scale to thousands of parallel containers with zero additional configuration. Function calls can be submitted asynchronously for background execution, eliminating the need to wait for jobs to finish or tune resource allocation. - -This guide covers Modal's batch processing capabilities, from basic invocation to integration with existing pipelines. - -## Background Execution with `.spawn_map` - -The fastest way to submit multiple jobs for asynchronous processing is by invoking a function with `.spawn_map`. When combined with the [`--detach`](/docs/reference/cli/run) flag, your App continues running until all jobs are completed. - -Here's an example of submitting 100,000 videos for parallel embedding. You can disconnect after submission, and the processing will continue to completion in the background: - -```python -# Kick off asynchronous jobs with `modal run --detach batch_processing.py` -import modal - -app = modal.App("batch-processing-example") -volume = modal.Volume.from_name("video-embeddings", create_if_missing=True) - -@app.function(volumes={"/data": volume}) -def embed_video(video_id: int): - # Business logic: - # - Load the video from the volume - # - Embed the video - # - Save the embedding to the volume - ... - -@app.local_entrypoint() -def main(): - embed_video.spawn_map(range(100_000)) -``` - -This pattern works best for jobs that store results externally—for example, in a [Modal Volume](/docs/guide/volumes), [Cloud Bucket Mount](/docs/guide/cloud-bucket-mounts), or your own database\*. - -*\* For database connections, consider using [Modal Proxy](/docs/guide/proxy-ips) to maintain a static IP across thousands of containers.* - -## Parallel Processing with `.map` - -Using `.map` allows you to offload expensive computations to powerful machines while gathering results. This is particularly useful for pipeline steps with bursty resource demands. Modal handles all infrastructure provisioning and de-provisioning automatically. - -Here's how to implement parallel video similarity queries as a single Modal function call: - -```python -# Run jobs and collect results with `modal run gather.py` -import modal - -app = modal.App("gather-results-example") - -@app.function(gpu="L40S") -def compute_video_similarity(query: str, video_id: int) -> tuple[int, int]: - # Embed video with GPU acceleration & compute similarity with query - return video_id, score - - -@app.local_entrypoint() -def main(): - import itertools - - queries = itertools.repeat("Modal for batch processing") - video_ids = range(100_000) - - for video_id, score in compute_video_similarity.map(queries, video_ids): - # Process results (e.g., extract top 5 most similar videos) - pass -``` - -This example runs `compute_video_similarity` on an autoscaling pool of L40S GPUs, returning scores to a local process for further processing. - -## Integration with Existing Systems - -The recommended way to use Modal Functions within your existing data pipeline is through [deployed function invocation](/docs/guide/trigger-deployed-functions). After deployment, you can call Modal functions from external systems: - -```python -def external_function(inputs): - compute_similarity = modal.Function.from_name( - "gather-results-example", - "compute_video_similarity" - ) - for result in compute_similarity.map(inputs): - # Process results - pass -``` - -You can invoke Modal Functions from any Python context, gaining access to built-in observability, resource management, and GPU acceleration. diff --git a/.agents/skills/modal/references/guide/billing.md b/.agents/skills/modal/references/guide/billing.md deleted file mode 100644 index 68238346ff980018d2d822804b5800f803488024..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/billing.md +++ /dev/null @@ -1,45 +0,0 @@ -# Billing - -Modal is serverless, which means you only pay for the compute you use or [request](/docs/guide/resources). Reservations are not required, and there are no minimum usage-time increments. Up-to-date unit pricing for all of our products is available on our [Pricing page](/pricing). - -## Billing frequency and incremental billing - -All workspaces are billed monthly. At the end of each billing cycle, you are auto-charged for the Modal usage incurred during that cycle (less any credits and incremental usage charges), as well as any subscription fees (non-Starter plans only) for the following billing cycle. - -In addition to monthly billing, you will be auto-charged for incremental usage the first time you exceed certain thresholds. These charges occur within the billing cycle. - -## Setting a workspace budget - -To set limits on how how much Modal usage can be incurred within the monthly billing period, go to the “Workspace budget” section of [Usage & Billing](/settings/usage). The max you can set this limit to is based on the history of prior successful charges to your payment method. Within a billing period, you may be charged for incremental usage when you pass certain thresholds. If those charges are successful, the max cap goes up. - -## Updating billing information - -To update your billing information, click on “Manage payment details” in the [Usage & Billing](/settings/usage) section of Settings. This will take you to a Stripe-hosted page where you can update your payment method and/or billing email. - -Note that you must have a payment method on file in order to use Modal. If you would like to remove your payment method and delete your workspace, please contact . - -## Viewing invoice history - -To view your invoice history and/or download receipts, click on “View invoices” in the [Usage & Billing](/settings/usage) section of Settings. You should see an “Invoice History” section at the bottom of the Stripe-hosted page. - -## Generating billing reports - -We also offer programmatic APIs for exporting billing reports to workspaces at the Team and Enterprise plan levels. Use the APIs in [`modal.billing`](/docs/reference/modal.billing) or the [`modal billing`](/docs/reference/cli/billing) CLI to generate tabular reports of spend over time broken down across specific Modal Apps or other resources. - -Note that the granular billing reports generated by these interfaces show a cost breakdown before factoring in credits or reservations, so the total invoiced cost may be less than the aggregate from these reports. - -## Attributing costs - -We offer a flexible tagging system so that you can categorize your Apps and properly attribute spend across your workspace to different teams, projects, etc. Tags are key-value pairs that can be set when deploying an App or assigned to already running Apps: - -```python -app = modal.App(name="inference-engine", tags={"team": "llm-platform"}) -``` - -When generating a billing report through the API or CLI, you can request to have specific tags included for each row in the report. - -## Custom invoicing - -Invoiced billing, international bank transfer payment methods, splitting invoices, and other custom requirements are available to customers on an [Enterprise plan](/pricing) who have a usage commitment with us. Please contact . - -Questions on billing? Reach out to . diff --git a/.agents/skills/modal/references/guide/cloud-bucket-mounts.md b/.agents/skills/modal/references/guide/cloud-bucket-mounts.md deleted file mode 100644 index 05b54402984beafbe462ae2b945c7c7247aba99d..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/cloud-bucket-mounts.md +++ /dev/null @@ -1,294 +0,0 @@ -# Cloud bucket mounts - -The [`modal.CloudBucketMount`](/docs/reference/modal.CloudBucketMount) is a -mutable volume that allows for both reading and writing files from a cloud -bucket. It supports AWS S3, Cloudflare R2, and Google Cloud Storage buckets. - -Cloud bucket mounts are built on top of AWS' -[`mountpoint`](https://github.com/awslabs/mountpoint-s3) technology and inherits -its limitations. See the [Limitations and troubleshooting](#limitations-and-troubleshooting) section for more details. - -## Mounting Cloudflare R2 buckets - -`CloudBucketMount` enables Cloudflare R2 buckets to be mounted as file system -volumes. Because Cloudflare R2 is -[S3-Compatible](https://developers.cloudflare.com/r2/api/s3/api/) the setup is -very similar between R2 and S3. See -[modal.CloudBucketMount](/docs/reference/modal.CloudBucketMount#modalcloudbucketmount) -for usage instructions. - -When creating the R2 API token for use with the mount, you need to have the -ability to read, write, and list objects in the specific buckets you will mount. -You do *not* need admin permissions, and you should *not* use "Client IP Address -Filtering". - -## Mounting Google Cloud Storage buckets - -`CloudBucketMount` enables Google Cloud Storage (GCS) buckets to be mounted as file system -volumes. See [modal.CloudBucketMount](/docs/reference/modal.CloudBucketMount#modalcloudbucketmount) -for GCS setup instructions. - -## Mounting S3 buckets - -`CloudBucketMount` enables S3 buckets to be mounted as file system volumes. To -interact with a bucket, you must have the appropriate IAM permissions configured -(refer to the section on [IAM Permissions](#iam-permissions)). - -```python -import modal -import subprocess - -app = modal.App() - -s3_bucket_name = "s3-bucket-name" # Bucket name not ARN. -s3_access_credentials = modal.Secret.from_dict({ - "AWS_ACCESS_KEY_ID": "...", - "AWS_SECRET_ACCESS_KEY": "...", - "AWS_REGION": "..." -}) - -@app.function( - volumes={ - "/my-mount": modal.CloudBucketMount(s3_bucket_name, secret=s3_access_credentials) - } -) -def f(): - subprocess.run(["ls", "/my-mount"]) -``` - -### Specifying S3 bucket region - -Amazon S3 buckets are associated with a single AWS Region. [`Mountpoint`](https://github.com/awslabs/mountpoint-s3) attempts to automatically detect the region for your S3 bucket at startup time and directs all S3 requests to that region. However, in certain scenarios, like if your container is running on an AWS worker in a certain region, while your bucket is in a different region, this automatic detection may fail. - -To avoid this issue, you can specify the region of your S3 bucket by adding an `AWS_REGION` key to your Modal secrets, as in the code example above. - -### Using AWS temporary security credentials - -`CloudBucketMount`s also support AWS temporary security credentials by passing -the additional environment variable `AWS_SESSION_TOKEN`. Temporary credentials -will expire and will not get renewed automatically. You will need to update -the corresponding Modal Secret in order to prevent failures. - -You can get temporary credentials with the [AWS CLI](https://aws.amazon.com/cli/) with: - -```shell -$ aws configure export-credentials --format env -export AWS_ACCESS_KEY_ID=XXX -export AWS_SECRET_ACCESS_KEY=XXX -export AWS_SESSION_TOKEN=XXX... -``` - -All these values are required. - -### Using OIDC identity tokens - -Modal provides [OIDC integration](/docs/guide/oidc-integration) and will automatically generate identity tokens to authenticate to AWS. -OIDC eliminates the need for manual token passing through Modal secrets and is based on short-lived tokens, which limits the window of exposure if a token is compromised. -To use this feature, you must [configure AWS to trust Modal's OIDC provider](/docs/guide/oidc-integration#step-1-configure-aws-to-trust-modals-oidc-provider) -and [create an IAM role that can be assumed by Modal Functions](/docs/guide/oidc-integration#step-2-create-an-iam-role-that-can-be-assumed-by-modal-functions). - -Then, you specify the IAM role that your Modal Function should assume to access the S3 bucket. - -```python -import modal - -app = modal.App() - -s3_bucket_name = "s3-bucket-name" -role_arn = "arn:aws:iam::123456789abcd:role/s3mount-role" - -@app.function( - volumes={ - "/my-mount": modal.CloudBucketMount( - bucket_name=s3_bucket_name, - oidc_auth_role_arn=role_arn - ) - } -) -def f(): - subprocess.run(["ls", "/my-mount"]) -``` - -### Mounting a path within a bucket - -To mount only the files under a specific subdirectory, you can specify a path prefix using `key_prefix`. -Since this prefix specifies a directory, it must end in a `/`. -The entire bucket is mounted when no prefix is supplied. - -```python -import modal -import subprocess - -app = modal.App() - -s3_bucket_name = "s3-bucket-name" -prefix = 'path/to/dir/' - -s3_access_credentials = modal.Secret.from_dict({ - "AWS_ACCESS_KEY_ID": "...", - "AWS_SECRET_ACCESS_KEY": "...", -}) - -@app.function( - volumes={ - "/my-mount": modal.CloudBucketMount( - bucket_name=s3_bucket_name, - key_prefix=prefix, - secret=s3_access_credentials - ) - } -) -def f(): - subprocess.run(["ls", "/my-mount"]) -``` - -This will only mount the files in the bucket `s3-bucket-name` that are prefixed by `path/to/dir/`. - -### Read-only mode - -To mount a bucket in read-only mode, set `read_only=True` as an argument. - -```python -import modal -import subprocess - -app = modal.App() - -s3_bucket_name = "s3-bucket-name" # Bucket name not ARN. -s3_access_credentials = modal.Secret.from_dict({ - "AWS_ACCESS_KEY_ID": "...", - "AWS_SECRET_ACCESS_KEY": "...", -}) - -@app.function( - volumes={ - "/my-mount": modal.CloudBucketMount(s3_bucket_name, secret=s3_access_credentials, read_only=True) - } -) -def f(): - subprocess.run(["ls", "/my-mount"]) -``` - -While S3 mounts support both write and read operations, they are optimized for -reading large files sequentially. Certain file operations, such as renaming -files, are not supported. For a comprehensive list of supported operations, -consult the -[Mountpoint documentation](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md). - -### IAM permissions - -To utilize `CloudBucketMount` for reading and writing files from S3 buckets, -your IAM policy must include permissions for `s3:PutObject`, -`s3:AbortMultipartUpload`, and `s3:DeleteObject`. These permissions are not -required for mounts configured with `read_only=True`. - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "ModalListBucketAccess", - "Effect": "Allow", - "Action": ["s3:ListBucket"], - "Resource": ["arn:aws:s3:::"] - }, - { - "Sid": "ModalBucketAccess", - "Effect": "Allow", - "Action": [ - "s3:GetObject", - "s3:PutObject", - "s3:AbortMultipartUpload", - "s3:DeleteObject" - ], - "Resource": ["arn:aws:s3:::/*"] - } - ] -} -``` - -## Limitations and troubleshooting - -Cloud Bucket Mounts have certain limitations that do not apply to [Volumes](/docs/guide/volumes). -These limitations are primarily around the way that files can be opened and edited in Cloud Bucket Mounts. For -a comprehensive list of limitations, see the [Mountpoint troubleshooting documentation](https://github.com/awslabs/mountpoint-s3/blob/a6179c72bfc237a1fdd06eb4a0863ca537f8d8a7/doc/TROUBLESHOOTING.md) -and the [Mountpoint semantics documentation](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md). - -The most common issues that users encounter are: - -* Files cannot be opened in append mode. -* Files cannot be written to at arbitrary offsets i.e. `seek` and write are not supported together. -* To write to a file, you must open it in `truncate` mode. - -These operations typically result in a `PermissionError: [Errno 1] Operation not permitted` error. - -If you need these features, give [Volumes](/docs/guide/volumes) a try! If you need these features in S3 -and are willing to pay extra for your bucket, you may be able to use [S3 Express](https://aws.amazon.com/s3/storage-classes/express-one-zone/). -Contact us [in our Slack](https://modal.com/slack) if you're interested in using S3 Express. - -### Writing files in append mode - -If you're using a library which must open a file in append mode, it's best to write to a temporary file -and then move it to your bucket's mount path. A similar approach can be used to write to a file at an arbitrary offset. - -```python notest -import tempfile -import shutil - -@app.function( - volumes={"/bucket": modal.CloudBucketMount("my-bucket", secret=s3_credentials)} -) -def append_to_log(): - # Write to a temporary file that supports append mode - with tempfile.NamedTemporaryFile(mode='a', delete=False) as temp_file: - temp_file.write("Log entry 1\n") - temp_file.write("Log entry 2\n") - temp_path = temp_file.name - - # Move the completed file to the bucket mount - shutil.move(temp_path, "/bucket/logfile.txt") -``` - -### Creating a file without a parent directory - -If you try to create a file in a directory that doesn't exist, you'll get a `Operation not permitted` error. -To fix this, create the parent directory first with `Path(dst).parent.mkdir(exist_ok=True, parents=True)`. - -### Using `np.savez` - -`np.savez` seeks to random offsets in a file, making it unsafe for Cloud Bucket Mounts. If your file is large, -you can write it to a temporary file and then move it to your bucket's mount path. If it's small, however, -you can solve this with an in-memory buffer: - -```python notest -import io -import numpy as np -import shutil - -data = np.random.rand(1000, 512) - -# 1. Build the archive entirely in memory -tmp = io.BytesIO() -np.savez_compressed(tmp, array=data) - -# 2. Copy it once, sequentially, to the mount point -dest = "/bucket/data.npz" -with open(dest, "wb") as f: - shutil.copyfileobj(tmp, f) -``` - -### Torchtune writing checkpoint files - -Old versions of [Torchtune](https://github.com/pytorch/torchtune) are incompatible with Cloud Bucket Mounts. -Upgrade to a version greater than or equal to `0.6.1` to ensure checkpoints can be written to the bucket. - -### Using the TensorBoard `SummaryWriter` - -The TensorBoard `SummaryWriter` opens log files in append mode. These files are quite small, though, -so we recommend writing to a temporary directory and using the [Watchdog](https://github.com/gorakhargosh/watchdog) -Python library to copy the files to the bucket mount path as they come in. - -This is a case where it may be worth it to use [Volumes](/docs/guide/volumes) instead - in particular, -training logs are sometimes not subject to the same compliance requirements that force something like checkpoints -or model weights to be stored in a secure location. We even have an example of -[how to use TensorBoard on Volumes](/docs/examples/torch_profiling#serving-tensorboard-on-modal-to-view-pytorch-profiles-and-traces). diff --git a/.agents/skills/modal/references/guide/cold-start.md b/.agents/skills/modal/references/guide/cold-start.md deleted file mode 100644 index 09209339ce5f1349eac0d559deb2a09982c405ff..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/cold-start.md +++ /dev/null @@ -1,262 +0,0 @@ -# Cold start performance - -This guide page details the techniques and Modal features used to improve cold start performance. - -## What is a cold start? - -Modal Functions are run in [containers](/docs/guide/images). - -If a container is already ready to run your Function, it will be reused. - -If not, Modal spins up a new container. -This is known as a *cold start*, -and it is often associated with higher latency. - -There are two sources of increased latency during cold starts: - -1. inputs may **spend more time waiting** in a queue for a container - to become ready or "warm". -2. when an input is handled by the container that just started, - there may be **extra work that only needs to be done on the first invocation** - ("initialization"). - -If you are invoking Functions with no warm containers -or if you otherwise see inputs spending too much time in the "pending" state, -you should -[target queueing time for optimization](#reduce-time-spent-queueing-for-warm-containers). - -If you see some Function invocations taking much longer than others, -and those invocations are the first handled by a new container, -you should -[target initialization for optimization](#reduce-latency-from-initialization). - -## Reduce time spent queueing for warm containers - -New containers are booted when there are not enough other warm containers to -to handle the current number of inputs. - -For example, the first time you send an input to a Function, -there are zero warm containers and there is one input, -so a single container must be booted up. -The total latency for the input will include -the time it takes to boot a container. - -If you send another input right after the first one finishes, -there will be one warm container and one pending input, -and no new container will be booted. - -Generalizing, there are two factors that affect the time inputs spend queueing: -the time it takes for a container to boot and become warm (which we solve by booting faster) -and the time until a warm container is available to handle an input (which we solve by having more warm containers). - -### Warm up containers faster - -The time taken for a container to become warm -and ready for inputs can range from seconds to minutes. - -Modal's custom container stack has been heavily optimized to reduce this time. -You can read about some of our optimizations [here](https://modal.com/blog/jono-containers-talk). -Containers boot in about one second. - -But before a container is considered warm and ready to handle inputs, -we need to execute any logic in your code's global scope (such as imports) -or in any -[`modal.enter` methods](/docs/guide/lifecycle-functions). -So if your boots are slow, these are the first places to work on optimization. - -For example, you might be downloading a large model from a model server -during the boot process. -You can instead -[download the model ahead of time](/docs/guide/model-weights), -so that it only needs to be downloaded once. - -For models in the tens of gigabytes, -this can reduce boot times from minutes to seconds. - -### Run more warm containers - -It is not always possible to speed up boots sufficiently. -For example, seconds of added latency to load a model may not -be acceptable in an interactive setting. - -In this case, the only option is to have more warm containers running. -This increases the chance that an input will be handled by a warm container, -for example one that finishes an input while another container is booting. - -Modal currently exposes [three parameters](/docs/guide/scale) that control how -many containers will be warm: `scaledown_window`, `min_containers`, -and `buffer_containers`. - -All of these strategies can increase the resources consumed by your Function -and so introduce a trade-off between cold start latencies and cost. - -#### Keep containers warm for longer with `scaledown_window` - -Modal containers will remain idle for a short period before shutting down. By -default, the maximum idle time is 60 seconds. You can configure this by setting -the `scaledown_window` on the [`@function`](/docs/reference/modal.App#function) -decorator. The value is measured in seconds, and it can be set anywhere between -two seconds and twenty minutes. - -```python -import modal - -app = modal.App() - -@app.function(scaledown_window=300) -def my_idle_greeting(): - return {"hello": "world"} -``` - -Increasing the `scaledown_window` reduces the chance that subsequent requests -will require a cold start, although you will be billed for any resources used -while the container is idle (e.g., GPU reservation or residual memory -occupancy). Note that containers will not necessarily remain alive for the -entire window, as the autoscaler will scale down more aggressively when the -Function is substantially over-provisioned. - -#### Overprovision resources with `min_containers` and `buffer_containers` - -Keeping already warm containers around longer doesn't help if there are no warm -containers to begin with, as when Functions scale from zero. - -To keep some containers warm and running at all times, set the `min_containers` -value on the [`@function`](/docs/reference/modal.App#function) decorator. This -puts a floor on the the number of containers so that the Function doesn't scale -to zero. Modal will still scale up and spin down more containers as the -demand for your Function fluctuates above the `min_containers` value, as usual. - -While `min_containers` overprovisions containers while the Function is idle, -`buffer_containers` provisions extra containers while the Function is active. -This "buffer" of extra containers will be idle and ready to handle inputs if -the rate of requests increases. This parameter is particularly useful for -bursty request patterns, where the arrival of one input predicts the arrival of more inputs, -like when a new user or client starts hitting the Function. - -```python -import modal - -app = modal.App(image=modal.Image.debian_slim().pip_install("fastapi")) - -@app.function(min_containers=3, buffer_containers=3) -def my_warm_greeting(): - return "Hello, world!" -``` - -## Reduce latency from initialization - -Some work is done the first time that a function is invoked -but can be used on every subsequent invocation. -This is -[*amortized work*](https://www.cs.cornell.edu/courses/cs312/2006sp/lectures/lec18.html) -done at initialization. - -For example, you may be using a large pre-trained model -whose weights need to be loaded from disk to memory the first time it is used. - -This results in longer latencies for the first invocation of a warm container, -which shows up in the application as occasional slow calls: high tail latency or elevated p9Xs. - -### Move initialization work out of the first invocation - -Some work done on the first invocation can be moved up and completed ahead of time. - -Any work that can be saved to disk, like -[downloading model weights](/docs/guide/model-weights), -should be done as early as possible. The results can be included in the -[container's Image](/docs/guide/images) -or saved to a -[Modal Volume](/docs/guide/volumes). - -Some work is tricky to serialize, like spinning up a network connection or an inference server. -If you can move this initialization logic out of the function body and into the global scope or a -[container `enter` method](https://modal.com/docs/guide/lifecycle-functions#enter), -you can move this work into the warm up period. -Containers will not be considered warm until all `enter` methods have completed, -so no inputs will be routed to containers that have yet to complete this initialization. - -For more on how to use `enter` with machine learning model weights, see -[this guide](/docs/guide/model-weights). - -Note that `enter` doesn't get rid of the latency -- -it just moves the latency to the warm up period, -where it can be handled by -[running more warm containers](#run-more-warm-containers). - -### Share initialization work across cold starts with Memory Snapshots - -Cold starts can also be made faster by using Modal -[Memory Snapshots](/docs/guide/memory-snapshots). - -Invocations of a Function after the first -are faster in part because the memory is already populated -with values that otherwise need to be computed or read from disk, -like the contents of imported libraries. - -Memory snapshotting captures the state of a container's memory -at user-controlled points after it has been warmed up -and reuses that state in future boots, which can substantially -reduce cold start latency penalties and warm up period duration. - -Refer to the [Memory Snapshots guide](/docs/guide/memory-snapshots) -for details. - -### Optimize initialization code - -Sometimes, there is nothing to be done but to speed this work up. - -Here, we share specific patterns that show up in optimizing initialization -in Modal Functions. - -#### Load multiple large files concurrently - -Often Modal applications need to read large files into memory (eg. model -weights) before they can process inputs. Where feasible these large file -reads should happen concurrently and not sequentially. Concurrent IO takes -full advantage of our platform's high disk and network bandwidth -to reduce latency. - -One common example of slow sequential IO is loading multiple independent -Huggingface `transformers` models in series. - -```python notest -from transformers import CLIPProcessor, CLIPModel, BlipProcessor, BlipForConditionalGeneration -model_a = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") -processor_a = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32") -model_b = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large") -processor_b = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large") -``` - -The above snippet does four `.from_pretrained` loads sequentially. -None of the components depend on another being already loaded in memory, so they -can be loaded concurrently instead. - -They could instead be loaded concurrently using a function like this: - -```python notest -from concurrent.futures import ThreadPoolExecutor, as_completed -from transformers import CLIPProcessor, CLIPModel, BlipProcessor, BlipForConditionalGeneration - -def load_models_concurrently(load_functions_map: dict) -> dict: - model_id_to_model = {} - with ThreadPoolExecutor(max_workers=len(load_functions_map)) as executor: - future_to_model_id = { - executor.submit(load_fn): model_id - for model_id, load_fn in load_functions_map.items() - } - for future in as_completed(future_to_model_id.keys()): - model_id_to_model[future_to_model_id[future]] = future.result() - return model_id_to_model - -components = load_models_concurrently({ - "clip_model": lambda: CLIPModel.from_pretrained("openai/clip-vit-base-patch32"), - "clip_processor": lambda: CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32"), - "blip_model": lambda: BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large"), - "blip_processor": lambda: BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large") -}) -``` - -If performing concurrent IO on large file reads does *not* speed up your cold -starts, it's possible that some part of your function's code is holding the -Python [GIL](https://wiki.python.org/moin/GlobalInterpreterLock) and reducing -the efficacy of the multi-threaded executor. diff --git a/.agents/skills/modal/references/guide/concurrent-inputs.md b/.agents/skills/modal/references/guide/concurrent-inputs.md deleted file mode 100644 index 41440f5a3979272ba5c449dae02d1da502891414..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/concurrent-inputs.md +++ /dev/null @@ -1,173 +0,0 @@ -# Input concurrency - -This guide documents the use of the `modal.concurrent` decorator to -process multiple inputs at the same time in a single Modal container. - -This page is a high-level guide to input concurrency. For reference documentation -of the `modal.concurrent` decorator, see [this page](/docs/reference/modal.concurrent). - -## Overview - -As traffic to your application increases, Modal will automatically scale up the -number of containers running your Function: - -
- -By default, each container will be assigned one input at a time. Autoscaling -across containers allows your Function to process inputs in parallel. This is -ideal when the operations performed by your Function are CPU-bound. - -For some workloads, though, it is inefficient for containers to process inputs -one-by-one. Modal supports these workloads with its *input concurrency* feature, -which allows individual containers to process multiple inputs at the same time: - -
- -When used effectively, input concurrency can reduce latency and lower costs. - -## Use cases - -Input concurrency can be especially effective for workloads that are primarily -I/O-bound, e.g.: - -* Querying a database -* Making external API requests -* Making remote calls to other Modal Functions - -For such workloads, individual containers may be able to concurrently process -large numbers of inputs with minimal additional latency. This means that your -Modal application will be more efficient overall, as it won't need to scale -containers up and down as traffic ebbs and flows. - -Another use case is to leverage *continuous batching* on GPU-accelerated -containers. Frameworks such as [vLLM](/docs/examples/llm_inference) can -achieve the benefits of batching across multiple inputs even when those -inputs do not arrive simultaneously (because new batches are formed for each -forward pass of the model). - -Note that for CPU-bound workloads, input concurrency will likely not be as -effective (or will even be counterproductive), and you may want to use -Modal's [*dynamic batching* feature](/docs/guide/dynamic-batching) instead. - -## Enabling input concurrency - -To enable input concurrency, add the `@modal.concurrent` decorator: - -```python -@app.function() -@modal.concurrent(max_inputs=100) -def my_function(input: str): - ... - -``` - -When using the class pattern, the decorator should be applied at the level of -the *class*, not on individual methods: - -```python -@app.cls() -@modal.concurrent(max_inputs=100) -class MyCls: - - @modal.method() - def my_method(self, input: str): - ... -``` - -Because all methods on a class will be served by the same containers, a class -with input concurrency enabled will concurrently run distinct methods in -addition to multiple inputs for the same method. - -## Setting a concurrency target - -When using the `@modal.concurrent` decorator, you must always configure the -maximum number of inputs that each container will concurrently process. If -demand exceeds this limit, Modal will automatically scale up more containers. - -Additional inputs may need to queue up while these additional containers cold -start. To help avoid degraded latency during scaleup, the `@modal.concurrent` -decorator has a separate `target_inputs` parameter. When set, Modal's autoscaler -will aim for this target as it provisions resources. If demand increases faster -than new containers can spin up, the active containers will be allowed to burst -above the target up to the `max_inputs` limit: - -```python -@app.function() -@modal.concurrent(max_inputs=96, target_inputs=80) # Allow a 20% burst -def my_function(input: str): - ... -``` - -It may take some experimentation to find the right settings for these parameters -in your particular application. Our suggestion is to set the `target_inputs` -based on your desired latency and the `max_inputs` based on resource constraints -(i.e., to avoid GPU OOM). You may also consider the relative latency cost of -scaling up a new container versus overloading the existing containers. - -## Concurrency mechanisms - -Modal uses different concurrency mechanisms to execute your Function depending -on whether it is defined as synchronous or asynchronous. Each mechanism imposes -certain requirements on the Function implementation. Input concurrency is an -advanced feature, and it's important to make sure that your implementation -complies with these requirements to avoid unexpected behavior. - -For synchronous Functions, Modal will execute concurrent inputs on separate -threads. *This means that the Function implementation must be thread-safe.* - -```python -# Each container can execute up to 10 inputs in separate threads -@app.function() -@modal.concurrent(max_inputs=10) -def sleep_sync(): - # Function must be thread-safe - time.sleep(1) -``` - -For asynchronous Functions, Modal will execute concurrent inputs using -separate `asyncio` tasks on a single thread. This does not require thread -safety, but it does mean that the Function needs to participate in -collaborative multitasking (i.e., it should not block the event loop). - -```python -# Each container can execute up to 10 inputs with separate async tasks -@app.function() -@modal.concurrent(max_inputs=10) -async def sleep_async(): - # Function must not block the event loop - await asyncio.sleep(1) -``` - -## Gotchas - -Input concurrency is a powerful feature, but there are a few caveats that can -be useful to be aware of before adopting it. - -### Input cancellations - -Synchronous and asynchronous Functions handle input cancellations differently. -Modal will raise a `modal.exception.InputCancellation` exception in synchronous -Functions and an `asyncio.CancelledError` in asynchronous Functions. - -When using input concurrency with a synchronous Function, a single input -cancellation will terminate the entire container. If your workflow depends on -graceful input cancellations, we recommend using an asynchronous -implementation. - -### Concurrent logging - -The separate threads or tasks that are executing the concurrent inputs will -write any logs to the same stream. This makes it difficult to associate logs -with a specific input, and filtering for a specific function call in Modal's web -dashboard will show logs for all inputs running at the same time. - -To work around this, we recommend including a unique identifier in the messages -you log (either your own identifier or the `modal.current_input_id()`) so that -you can use the search functionality to surface logs for a specific input: - -```python -@app.function() -@modal.concurrent(max_inputs=10) -async def better_concurrent_logging(x: int): - logger.info(f"{modal.current_input_id()}: Starting work with {x}") -``` diff --git a/.agents/skills/modal/references/guide/continuous-deployment.md b/.agents/skills/modal/references/guide/continuous-deployment.md deleted file mode 100644 index a8870ff7008ed062d9ba142a805fc24c8c34e7f2..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/continuous-deployment.md +++ /dev/null @@ -1,57 +0,0 @@ -# Continuous deployment - -It's a common pattern to auto-deploy your Modal App as part of a CI/CD pipeline. -To get you started, below is a guide to doing continuous deployment of a Modal -App in GitHub. - -## GitHub Actions - -Here's a sample GitHub Actions workflow that deploys your App on every push to -the `main` branch. - -This requires you to create a [Modal token](/settings/tokens) and add it as a -[secret for your Github Actions workflow](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-secrets). - -After setting up secrets, create a new workflow file in your repository at -`.github/workflows/ci-cd.yml` with the following contents: - -```yaml -name: CI/CD - -on: - push: - branches: - - main - -jobs: - deploy: - name: Deploy - runs-on: ubuntu-latest - env: - MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }} - MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }} - - steps: - - name: Checkout Repository - uses: actions/checkout@v6 - - - name: Install Python - uses: actions/setup-python@v5 - with: - python-version: "3.10" - - - name: Install Modal - run: | - python -m pip install --upgrade pip - pip install modal - - - name: Deploy job - run: | - modal deploy -m my_package.my_file -``` - -Be sure to replace `my_package.my_file` with your actual entrypoint. - -If you use multiple Modal [Environments](/docs/guide/environments), you can -additionally specify the target environment in the YAML using -`MODAL_ENVIRONMENT=xyz`. diff --git a/.agents/skills/modal/references/guide/cuda.md b/.agents/skills/modal/references/guide/cuda.md deleted file mode 100644 index 26030aa83df6dc69e282d365fdc9304d0951db33..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/cuda.md +++ /dev/null @@ -1,194 +0,0 @@ -# Using CUDA on Modal - -Modal makes it easy to accelerate your workloads with datacenter-grade NVIDIA GPUs. - -To take advantage of the hardware, you need to use matching software: the CUDA stack. -This guide explains the components of that stack and how to install them on Modal. -For more on which GPUs are available on Modal and how to choose a GPU for your use case, -see [this guide](/docs/guide/gpu). For a deep dive on both the -[GPU hardware](/gpu-glossary/device-hardware) and [software](/gpu-glossary/device-software) -and for even more detail on [the CUDA stack](/gpu-glossary/host-software/), -see our [GPU Glossary](/gpu-glossary/readme). - -Here's the tl;dr: - -* The [NVIDIA Accelerated Graphics Driver for Linux-x86\_64](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/#driver-installation), version 580.95.05, - and [CUDA Driver API](https://docs.nvidia.com/cuda/archive/13.0.0/cuda-driver-api/index.html), version 13.0, are already installed. - You can call `nvidia-smi` or run compiled CUDA programs from any Modal Function with access to a GPU. -* That means you can install many popular libraries like `torch` that bundle their other CUDA dependencies [with a simple `pip_install`](#install-gpu-accelerated-torch-and-transformers-with-pip_install). -* For bleeding-edge libraries like `flash-attn`, you may need to install CUDA dependencies manually. - To make your life easier, [use an existing image](#for-more-complex-setups-use-an-officially-supported-cuda-image). - -## What is CUDA? - -When someone refers to "installing CUDA" or "using CUDA", -they are referring not to a library, but to a -[stack](/gpu-glossary/host-software/cuda-software-platform) with multiple layers. -Your application code (and its dependencies) can interact -with the stack at different levels. - -![The CUDA stack](../../assets/docs/cuda-stack-diagram.png) - -This leads to a lot of confusion. To help clear that up, the following sections explain each component in detail. - -### Level 0: Kernel-mode driver components - -At the lowest level are the [*kernel-mode driver components*](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/#nvidia-open-gpu-kernel-modules). -The Linux kernel is essentially a single program operating the entire machine and all of its hardware. -To add hardware to the machine, this program is extended by loading new modules into it. -These components communicate directly with hardware -- in this case the GPU. - -Because they are kernel modules, these driver components are tightly integrated with the host operating system -that runs your containerized Modal Functions and are not something you can inspect or change yourself. - -### Level 1: User-mode driver API - -All action in Linux that doesn't occur in the kernel occurs in [user space](https://en.wikipedia.org/wiki/User_space). -To talk to the kernel drivers from our user space programs, we need *user-mode driver components*. - -Most prominently, that includes: - -* the [CUDA Driver API](/gpu-glossary/host-software/cuda-driver-api), - a [shared object](https://en.wikipedia.org/wiki/Shared_library) called `libcuda.so`. - This object exposes functions like [`cuMemAlloc`](https://docs.nvidia.com/cuda/archive/12.8.0/cuda-driver-api/group__CUDA__MEM.html#group__CUDA__MEM_1gb82d2a09844a58dd9e744dc31e8aa467), - for allocating GPU memory. -* the [NVIDIA management library](https://developer.nvidia.com/management-library-nvml), `libnvidia-ml.so`, and its command line interface [`nvidia-smi`](https://developer.nvidia.com/system-management-interface). - You can use these tools to check the status of the system's GPU(s). - -These components are installed on all Modal machines with access to GPUs. -Because they are user-level components, you can use them directly: - -```python runner:ModalRunner -import modal - -app = modal.App() - -@app.function(gpu="any") -def check_nvidia_smi(): - import subprocess - output = subprocess.check_output(["nvidia-smi"], text=True) - assert "Driver Version:" in output - assert "CUDA Version:" in output - print(output) - return output -``` - -### Level 2: CUDA Toolkit - -Wrapping the CUDA Driver API is the [CUDA Runtime API](/gpu-glossary/host-software/cuda-runtime-api), the `libcudart.so` shared library. -This API includes functions like [`cudaLaunchKernel`](https://docs.nvidia.com/cuda/archive/12.8.0/cuda-runtime-api/group__CUDART__HIGHLEVEL.html#group__CUDART__HIGHLEVEL_1g7656391f2e52f569214adbfc19689eb3) -and is more commonly used in CUDA programs (see [this HackerNews comment](https://news.ycombinator.com/item?id=20616385) for color commentary on why). -This shared library is *not* installed by default on Modal. - -The CUDA Runtime API is generally installed as part of the larger [NVIDIA CUDA Toolkit](https://docs.nvidia.com/cuda/index.html), -which includes the [NVIDIA CUDA compiler driver](/gpu-glossary/host-software/nvcc) (`nvcc`) and its toolchain -and a number of [useful goodies](/gpu-glossary/host-software/cuda-binary-utilities) for writing and debugging CUDA programs (`cuobjdump`, `cudnn`, profilers, etc.). - -Contemporary GPU-accelerated machine learning workloads like LLM inference frequently make use of many components of the CUDA Toolkit, -such as the run-time compilation library [`nvrtc`](https://docs.nvidia.com/cuda/archive/12.8.0/nvrtc/index.html). - -So why aren't these components installed along with the drivers? -A compiled CUDA program can run without the CUDA Runtime API installed on the system, -by [statically linking](https://en.wikipedia.org/wiki/Static_library) the CUDA Runtime API into the program binary, -though this is fairly uncommon for CUDA-accelerated Python programs. -Additionally, older versions of these components are needed for some applications -and some application deployments even use several versions at once. -Both patterns are compatible with the host machine driver provided on Modal. - -## Install GPU-accelerated `torch` and `transformers` with `pip_install` - -The components of the CUDA Toolkit can be installed via `pip`, -via PyPI packages like [`nvidia-cuda-runtime-cu12`](https://pypi.org/project/nvidia-cuda-runtime-cu12/) -and [`nvidia-cuda-nvrtc-cu12`](https://pypi.org/project/nvidia-cuda-nvrtc-cu12/). -These components are listed as dependencies of some popular GPU-accelerated Python libraries, like `torch`. - -Because Modal already includes the lower parts of the CUDA stack, you can install these libraries -with [the `pip_install` method of `modal.Image`](/docs/guide/images#add-python-packages-with-pip_install), just like any other Python library: - -```python -image = modal.Image.debian_slim().pip_install("torch") - - -@app.function(gpu="any", image=image) -def run_torch(): - import torch - has_cuda = torch.cuda.is_available() - print(f"It is {has_cuda} that torch can access CUDA") - return has_cuda -``` - -Many libraries for running open-weights models, like `transformers` and `vllm`, -use `torch` under the hood and so can be installed in the same way: - -```python -image = modal.Image.debian_slim().pip_install("transformers[torch]") -image = image.apt_install("ffmpeg") # for audio processing - - -@app.function(gpu="any", image=image) -def run_transformers(): - from transformers import pipeline - transcriber = pipeline(model="openai/whisper-tiny.en", device="cuda") - result = transcriber("https://modal-cdn.com/mlk.flac") - print(result["text"]) # I have a dream that one day this nation will rise up live out the true meaning of its creed -``` - -## For more complex setups, use an officially-supported CUDA image - -The disadvantage of installing the CUDA stack via `pip` is that -many other libraries that depend on its components being installed as normal system packages cannot find them. - -For these cases, we recommend you use an image that already has the full CUDA stack installed as system packages -and all environment variables set correctly, like the [`nvidia/cuda:*-devel-*` images on Docker Hub](https://hub.docker.com/r/nvidia/cuda). - -[TensorRT-LLM](https://nvidia.github.io/TensorRT-LLM/overview.html) is an inference engine that accelerates and optimizes performance for the large language models. It requires the full CUDA toolkit for installation. - -```python -cuda_version = "12.8.1" # should be no greater than host CUDA version -flavor = "devel" # includes full CUDA toolkit -operating_sys = "ubuntu24.04" -tag = f"{cuda_version}-{flavor}-{operating_sys}" -HF_CACHE_PATH = "/cache" - - -image = ( - modal.Image.from_registry(f"nvidia/cuda:{tag}", add_python="3.12") - .entrypoint([]) # remove verbose logging by base image on entry - .apt_install("libopenmpi-dev") # required for tensorrt - .pip_install("tensorrt-llm==0.19.0", "pynvml", extra_index_url="https://pypi.nvidia.com") - .pip_install("hf-transfer", "huggingface_hub[hf_xet]") - .env({"HF_HUB_CACHE": HF_CACHE_PATH, "HF_HUB_ENABLE_HF_TRANSFER": "1", "PMIX_MCA_gds": "hash"}) -) - - -app = modal.App("tensorrt-llm", image=image) -hf_cache_volume = modal.Volume.from_name("hf_cache_tensorrt", create_if_missing=True) - - -@app.function(gpu="A10G", volumes={HF_CACHE_PATH: hf_cache_volume}) -def run_tiny_model(): - from tensorrt_llm import LLM, SamplingParams - - sampling_params = SamplingParams(temperature=0.8, top_p=0.95) - - llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0") - - output = llm.generate("The capital of France is", sampling_params) - print(f"Generated text: {output.outputs[0].text}") - return output.outputs[0].text -``` - -Make sure to choose a version of CUDA that is no greater than the version provided by the host machine. -Older versions in the `12.*` and `13.*` series are guaranteed to be compatible with the host machine's driver, -but older major versions (`11.*`, `10.*`, etc.) may not be. - -## What next? - -For more on accessing and choosing GPUs on Modal, check out [this guide](/docs/guide/gpu). -To dive deep on GPU internals, check out our [GPU Glossary](/gpu-glossary/readme). - -To see these installation patterns in action, check out these examples: - -* [Fast LLM inference on big GPUs](/docs/examples/llm_inference) -* [Finetune a character LoRA for your pet](/docs/examples/diffusers_lora_finetune) -* [Optimized Flux inference](/docs/examples/flux) diff --git a/.agents/skills/modal/references/guide/customer-supplied-encryption-keys.md b/.agents/skills/modal/references/guide/customer-supplied-encryption-keys.md deleted file mode 100644 index 61883112f3c4d242125d40533b6be6ff363e4ba0..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/customer-supplied-encryption-keys.md +++ /dev/null @@ -1,132 +0,0 @@ -# Customer Supplied Encryption Keys - - - -Customer Supplied Encryption Keys (CSEK) let you provide your own key material -when creating supported Modal resources. Modal uses the key as part of the -resource encryption flow, but does not persist the key. - -Use CSEK when you need customer-held key material for data that Modal stores. -You are responsible for generating, storing, backing up, and providing the key -again when the protected resource is used. - - - -If you lose the CSEK for a protected resource, Modal cannot recover it for you. -Store the key in a durable key management system outside Modal. - - - -## How CSEK works - -For each supported resource, the same basic flow applies: - -1. Generate key material with a cryptographically secure random source. -2. Pass the key when creating the resource. -3. Store the resource ID and key in your own system. -4. Pass the same key again when reading, mounting, or restoring the resource. - -The key must be passed as `bytes`, must not be empty, and must be between 16 and -512 bytes long. - -```python notest -import secrets - -encryption_key = secrets.token_bytes(32) -``` - -Do not commit CSEK material to source control, bake it into Images, print it in -logs, or store it next to the data it protects. Prefer a dedicated key -management system or secrets manager with access controls and backup policies -that match your security requirements. - -## Supported resources - -This page documents CSEK for the currently supported resources. As CSEK support -becomes available for more Modal resources, additional sections will be added. - -| Resource | SDK support | -| ------------------------------------------- | ----------- | -| [Directory Snapshots](#directory-snapshots) | Python SDK | - -## Directory Snapshots - -[Directory Snapshots](/docs/guide/sandbox-snapshots#directory-snapshots) let -you capture a directory from a running Sandbox as an -[Image](/docs/reference/modal.Image). With CSEK, you pass key material when -creating the snapshot and pass the same key again when mounting it. - -### Create a CSEK-protected directory snapshot - -```python notest -import secrets - -import modal - -app = modal.App.lookup("csek-directory-snapshots", create_if_missing=True) -encryption_key = secrets.token_bytes(32) - -sb = modal.Sandbox.create(app=app) -sb.exec( - "bash", - "-c", - "mkdir -p /project && echo 'private data' > /project/state.txt", -).wait() - -snapshot = sb.snapshot_directory( - "/project", - _experimental_encryption_key=encryption_key, -) -sb.terminate() - -# Store both values in your own durable systems. -snapshot_id = snapshot.object_id -``` - -The `_experimental_encryption_key` parameter is currently exposed as an -experimental Python SDK API. See [Feature maturity](/docs/guide/feature-maturity#experimental-sdk) -for how Modal treats experimental SDK surfaces. - -### Mount a CSEK-protected directory snapshot - -To use the snapshot later, rehydrate the Image by ID and pass the same key to -`Sandbox.mount_image()`. - -```python notest -import modal - -app = modal.App.lookup("csek-directory-snapshots") -snapshot = modal.Image.from_id(snapshot_id) - -sb = modal.Sandbox.create(app=app) -sb.mount_image( - "/project", - snapshot, - _experimental_encryption_key=encryption_key, -) - -contents = sb.exec("cat", "/project/state.txt").stdout.read().strip() -assert contents == "private data" -sb.terminate() -``` - -If the key is missing or incorrect, Modal cannot mount the encrypted snapshot. - -### Re-snapshotting encrypted directories - -After mounting a CSEK-protected directory snapshot, you can create another -directory snapshot from that mounted path: - -* Pass `_experimental_encryption_key` to protect the new snapshot with CSEK. -* Omit `_experimental_encryption_key` to create the new snapshot with - Modal-managed encryption. - -Each CSEK-protected snapshot is tied to the key used when that snapshot was -created. If you create a new CSEK-protected snapshot with a different key, use -the new key when mounting the new snapshot. - -### Retention - -CSEK does not change Directory Snapshot retention. Directory Snapshots are -retained for 30 days after creation. See [Snapshot Retention](/docs/guide/sandbox-snapshots#snapshot-retention) -for details. diff --git a/.agents/skills/modal/references/guide/datadog-integration.md b/.agents/skills/modal/references/guide/datadog-integration.md deleted file mode 100644 index 372a3ea68f94cfc53c3c82dc42713de2c5549c37..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/datadog-integration.md +++ /dev/null @@ -1,115 +0,0 @@ -# Connecting Modal to your Datadog account - -You can use the [Modal + Datadog Integration](https://docs.datadoghq.com/integrations/modal/) -to export Modal function logs to Datadog. You'll find the Modal Datadog -Integration available for install in the Datadog marketplace. - -## What this integration does - -This integration allows you to: - -1. Export Modal audit logs in Datadog -2. Export Modal function logs to Datadog -3. Export container metrics to Datadog - -## Installing the integration - -1. Open the [Modal Tile](https://app.datadoghq.com/integrations?integrationId=modal) (or the EU tile [here](https://app.datadoghq.eu/integrations?integrationId=modal)) - in the Datadog integrations page -2. Click "Install Integration" -3. Click Connect Accounts to begin authorization of this integration. - You will be redirected to log into Modal, and once logged in, you’ll - be redirected to the Datadog authorization page. -4. Click "Authorize" to complete the integration setup - -## Metrics - -The Modal Datadog Integration will forward the following metrics to Datadog: - -* `modal.cpu.utilization` -* `modal.memory.usage` -* `modal.gpu.memory.usage` -* `modal.gpu.compute.utilization` -* `modal.gpu.power.usage` -* `modal.gpu.power.utilization` -* `modal.gpu.temperature` -* `modal.container.running` -* `modal.input_events.elapsed_time_us` -* `modal.input_events.input_queue_time_us` -* `modal.input_events.coldstart_time_us` -* `modal.input_events.successes` -* `modal.input_events.total_inputs` -* `modal.function.pending_inputs` -* `modal.function.running_inputs` - -All metrics are tagged with `container_id`, `environment_name`, `app_name`, `app_id`, -`function_name`, `function_id`, `workspace_name`, and `workspace_id`. - -Deprecated metrics: - -* `modal.memory.utilization` (use `modal.memory.usage`) -* `modal.gpu.memory.utilization` (use `modal.gpu.memory.usage`) - -`modal.input_events.successes` and `modal.input_events.total_inputs` can be used -to measure the success rate of a certain function or app. - -As an [official Datadog integration](https://docs.datadoghq.com/integrations/modal/), -Modal metrics are free on Datadog while logs are charged. - -## Log attributes - -Logs forwarded to Datadog include the following attributes: - -* `container_id` -* `app_id` -* `app_name` -* `function_id` -* `function_name` -* `function_call_id` -* `input_id` -* `sandbox_id` -* `environment` -* `workspace` -* `workspace_id` - -These are [log attributes](https://docs.datadoghq.com/logs/log_configuration/attributes_naming_convention/), -not tags. You can filter and search by them in Datadog's -[Log Explorer](https://docs.datadoghq.com/logs/explorer/) using the `@` prefix -(for example, `@container_id:`). - -## Structured logging - -Logs from Modal are sent to Datadog in plaintext without any structured -parsing. This means that if you have custom log formats, you'll need to -set up a [log processing pipeline](https://docs.datadoghq.com/logs/log_configuration/pipelines/?tab=source) -in Datadog to parse them. - -Modal passes log messages in the `.message` field of the log record. To -parse logs, you should operate over this field. Note that the Modal Integration -does set up some basic pipelines. In order for your pipelines to work, ensure -that your pipelines come before Modal's pipelines in your log settings. - -## Cost Savings - -The Modal Datadog Integration will forward all logs to Datadog which could be -costly for verbose apps. We recommend using either [Log Pipelines](https://docs.datadoghq.com/logs/log_configuration/pipelines/?tab=source) -or [Index Exclusion Filters](https://docs.datadoghq.com/logs/indexes/?tab=ui#exclusion-filters) -to filter logs before they are sent to Datadog. - -All logs include the `environment` attribute. The simplest way to filter -logs is to create a pipeline that filters on this attribute and to isolate -verbose apps in a separate environment. - -## Uninstalling the integration - -Once the integration is uninstalled, all logs will stop being sent to -Datadog, and authorization will be revoked. - -1. Navigate to the [Modal metrics settings page](http://modal.com/settings/metrics) - and select "Delete Datadog Integration". -2. On the Configure tab in the Modal integration tile in Datadog, - click Uninstall Integration. -3. Confirm that you want to uninstall the integration. -4. Ensure that all API keys associated with this integration have been - disabled by searching for the integration name on the [API Keys](https://app.datadoghq.com/organization-settings/api-keys?filter=Modal) - page. diff --git a/.agents/skills/modal/references/guide/dataset-ingestion.md b/.agents/skills/modal/references/guide/dataset-ingestion.md deleted file mode 100644 index 6f532ef7bb9c5adb658b67eb2bafb45d61b20ed5..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/dataset-ingestion.md +++ /dev/null @@ -1,109 +0,0 @@ -# Large dataset ingestion - -This guide provides best practices for downloading, transforming, and storing large datasets within -Modal. A dataset is considered large if it contains hundreds of thousands of files and/or is over -100 GiB in size. - -These guidelines ensure that large datasets can be ingested fully and reliably. - -## Configure your Function for heavy disk usage - -Large datasets should be downloaded and transformed using a `modal.Function` and stored -into a [Volume](/docs/guide/volumes). - -This `modal.Function` should specify a large `timeout` because large dataset processing can take hours, -and it should request a larger ephemeral disk in cases where the dataset being downloaded and processed -is hundreds of GiBs. - -```python -volume = modal.Volume.from_name("datasets", create_if_missing=True) - - -@app.function( - volumes={"/mnt/datasets": volume}, - ephemeral_disk=1000 * 1000, # 1 TiB - timeout=60 * 60 * 12, # 12 hours - -) -def download_and_transform() -> None: - ... - volume.commit() -``` - -### Prefer sharded or archived outputs for tiny-file datasets - -Volumes can store large datasets, but datasets made up of millions of tiny files are -still usually easier to ingest and consume when they are first grouped into larger artifacts such as -tar shards, WebDataset archives, Parquet files, or other batched formats. - -See the [transforming](#transforming) section below for more details. - -## Experimentation - -Downloading and transforming large datasets can be fiddly. While iterating on a reliable ingestion program -you may want an interactive environment so you can inspect downloaded files, validate credentials, -and benchmark transforms before automating the full ingestion job. [Modal Notebooks](/docs/guide/notebooks) -work well for this. Attach the same Volume that your ingestion Functions use, keep transient scratch -data in `/tmp`, and persist intermediate artifacts under `/mnt/...`. - -## Downloading - -The raw dataset data should be first downloaded into the container at `/tmp/` and not placed -directly into the mounted volume. This serves a couple purposes. - -1. Download tools often create temporary files, partial files, or rename targets while writing, and local SSD handles that more efficiently. -2. The raw dataset data may need to be transformed before use, in which case it is wasteful to store it permanently. - -This snippet shows the basic download-and-copy procedure: - -```python notest -import pathlib -import shutil -import subprocess - -tmp_path = pathlib.Path("/tmp/imagenet/") -vol_path = pathlib.Path("/mnt/datasets/imagenet/") -filename = "imagenet-object-localization-challenge.zip" -# 1. Download into /tmp/ -subprocess.run( - f"kaggle competitions download -c imagenet-object-localization-challenge --path {tmp_path}", - shell=True, - check=True -) -vol_path.mkdir(parents=True, exist_ok=True) -# 2. Copy (without transform) into mounted volume. -shutil.copy2(tmp_path / filename, vol_path / filename) -volume.commit() -``` - -## Transforming - -When ingesting a large dataset it is sometimes necessary to transform it before storage, so that it is in -an optimal format for loading at runtime. A common kind of necessary transform is gzip decompression. Very large -datasets are often gzipped for storage and network transmission efficiency, but gzip decompression (80 MiB/s) -is hundreds of times slower than reading from a solid state drive (SSD) -and should be done once before storage to avoid decompressing on every read against the dataset. - -Transformations should be performed after storing the raw dataset in `/tmp/`. Performing transformations almost always increases container disk usage and this is where the [`ephemeral_disk` parameter](/docs/reference/modal.App#function) parameter becomes important. For example, a -100 GiB raw, compressed dataset may decompress to into 500 GiB, occupying 600 GiB of container disk space. - -Transformations should also typically be performed against `/tmp/`. This is because - -1. transforms can be IO intensive and IO latency is lower against local SSD. -2. transforms can create temporary data which is wasteful to store permanently. - -Once the transform is complete, write the final dataset layout to the attached Volume and commit it so -subsequent Functions and Notebooks can reload and use the same data. - -## Examples - -The best practices offered in this guide are demonstrated in the [`modal-examples` repository](https://github.com/modal-labs/modal-examples/tree/main/12_datasets). - -The examples include these popular large datasets: - -* [ImageNet](https://www.image-net.org/), the image labeling dataset that kicked off the deep learning revolution -* [COCO](https://cocodataset.org/#download), the Common Objects in COntext dataset of densely-labeled images -* [LAION-400M](https://laion.ai/blog/laion-400-open-dataset/), the Stable Diffusion training dataset -* Data derived from the [Big "Fantastic" Database](https://bfd.mmseqs.com/), - [Protein Data Bank](https://www.wwpdb.org/), and [UniProt Database](https://www.uniprot.org/) - used in training the [RoseTTAFold](https://github.com/RosettaCommons/RoseTTAFold) protein structure model diff --git a/.agents/skills/modal/references/guide/developing-debugging.md b/.agents/skills/modal/references/guide/developing-debugging.md deleted file mode 100644 index 217e9ab84a072a86cbdeba7a2df561f32bc2e503..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/developing-debugging.md +++ /dev/null @@ -1,229 +0,0 @@ -# Developing and debugging - -Modal makes it easy to run apps in the cloud, try code changes in the cloud, and -debug remotely executing code as if it were right there on your laptop. To speed -boost your inner dev loop, this guide provides a rundown of tools and techniques -for developing and debugging software in Modal. - -## Interactivity - -You can launch a Modal App interactively and have it drop you right into the -middle of the action, at an interesting callsite or the site of a runtime -detonation. - -### Interactive functions - -It is possible to start the interactive Python debugger or start an `IPython` -REPL right in the middle of your Modal App. - -To do so, you first need to run your App in "interactive" mode by using the -`--interactive` / `-i` flag. In interactive mode, you can establish a connection -to the calling terminal by calling `interact()` from within your function. - -For a simple example, you can accept user input with the built-in Python `input` -function: - -```python -@app.function() -def my_fn(hidden): - modal.interact() - - x = input("Enter a number: ") - if hidden == x: - print(f"Your number is {x}, which is the hidden value!") - else: - print(f"Your number is {x}, which is not the hidden value") -``` - -Now when you run your app with the `--interactive` flag, you're able to send -inputs to your app, even though it's running in a remote container! - -```shell -modal run -i guess_number.py::my_fn --hidden 5 -Enter a number: 5 -Your number is 5, which is the hidden value! -``` - -For a more interesting example, you can [`pip_install("ipython")`](/docs/reference/modal.Image#pip_install) -and start an `IPython` REPL dynamically anywhere in your code: - -```python -@app.function() -def f(): - model = expensive_function() - # play around with model - modal.interact() - import IPython - IPython.embed() -``` - -The built-in Python debugger can be initiated with the language's `breakpoint()` -function. For convenience, breakpoints call `interact` automatically. - -```python -@app.function() -def f(): - x = "10point3" - breakpoint() - answer = float(x) -``` - -### Debugging Running Containers - -#### Debug Shells - -Modal also lets you run interactive commands on your running Containers from the -terminal -- much like `ssh`-ing into a traditional machine or cloud VM. - -To run a command inside a running Container, you first need to get the Container -ID. You can view all running Containers and their Container IDs with -[`modal container list`](/docs/reference/cli/container). - -After you obtain the Container ID, you can connect to the Container with `modal shell [container-id]`. This launches a "Debug Shell" that comes with some preinstalled tools: - -* `vim` -* `nano` -* `ps` -* `strace` -* `curl` -* `py-spy` -* and more! - -You can use a debug shell to examine or terminate running processes, modify the Container filesystem, run commands, and more. You can also install additional packages using your Container's package manager (ex. `apt`). - - - -Note that debug shells will terminate immediately once your Container has finished running. - -#### `modal container exec` - -You can also execute a specific command in a running Container with `modal container exec [container-id] [command...]`. For example, to see what files are in `/root`, you can run `modal container exec [container-id] ls /root`. - -``` -❯ modal container list - Active Containers in environment: nathan-dev -┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ -┃ Container ID ┃ App ID ┃ App Name ┃ Start Time ┃ -┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ -│ ta-01JK47GVDMWMGPH8MQ0EW30Y25 │ ap-FSuhQ4LpvNAt5b6mKi1CDw │ my-app │ 2025-02-02 16:02 EST │ -└───────────────────────────────┴───────────────────────────┴──────────┴──────────────────────┘ - -❯ modal container exec ta-01JK47GVDMWMGPH8MQ0EW30Y25 ls /root -__pycache__ test00.py -``` - -Note that your executed command will terminate immediately once your Container -has finished running. - -By default, commands will be run within a -[pseudoterminal (PTY)](https://en.wikipedia.org/wiki/Pseudoterminal), but this -can be disabled with the `--no-pty` flag. - -#### Live container profiling - -When a container or input is seemingly stuck or not making progress, -you can use the Modal web dashboard to find out what code that's executing in the -container in real time. To do so, look for **Live Profiling** in the **Containers** tab in your -function dashboard. - -![Live container profiling](https://modal-public-assets.s3.us-east-1.amazonaws.com/live-profiling-bigger.gif) - -### Debugging Container Images - -You can also launch an interactive shell in a new Container with the same -environment as your Function. This is handy for debugging issues with your -Image, interactively refining build commands, and exploring the contents of -[`Volume`](/docs/reference/modal.Volume)s and -[`NetworkFileSystem`](/docs/reference/modal.NetworkFileSystem)s. - -The primary interface for accessing this feature is the -[`modal shell`](/docs/reference/cli/shell) CLI command, which accepts a Function -name in your App (or prompts you to select one, if none is provided), and runs -an interactive command on the same image as the Function, with the same -[`Secret`](/docs/reference/modal.Secret)s and -[`NetworkFileSystem`](/docs/reference/modal.NetworkFileSystem)s attached as the selected Function. - -The default command is `/bin/bash`, but you can override this with any other -command of your choice using the `--cmd` flag. - - - -Note that `modal shell [filename].py` does not attach a shell to a running Container of the -Function, but instead creates a fresh instance of the underlying Image. To attach a shell to a running Container, use `modal shell [container-id]` instead. - -## Live updating - -### Hot reloading with `modal serve` - -Modal has the command `modal serve `, which creates a loop that -live updates an App when any of the supporting files change. - -Live updating works with Web Functions, syncing your changes as you make them, -and it also works well with cron schedules and job queues. - -```python -import modal - -app = modal.App(image=modal.Image.debian_slim().pip_install("fastapi")) - - -@app.function() -@modal.fastapi_endpoint() -def f(): - return "I update on file edit!" - - -@app.function(schedule=modal.Period(seconds=5)) -def run_me(): - print("I also update on file edit!") -``` - -If you edit this file, the `modal serve` command will detect the change and -update the code, without having to restart the command. - -## Observability - -Each running Modal App, including all ephemeral Apps, streams logs and resource -metrics back to you for viewing. - -On start, an App will log a dashboard link that will take you its App page. - -```shell -$ python3 main.py -✓ Initialized. View app page at https://modal.com/apps/ap-XYZ1234. -... -``` - -From this page you can access the following: - -* logs, both from your application and system-level logs from Modal -* compute resource metrics (CPU, RAM, GPU) -* function call history, including historical success/failure counts - -### Debug logs - -You can enable Modal's client debug logs by setting the `MODAL_LOGLEVEL` environment variable to `DEBUG`. -Running the following will show debug logging from the Modal client running locally. - -```bash -MODAL_LOGLEVEL=DEBUG modal run hello.py -``` - -To enable debug logs in the Modal client running in the remote container, you can set `MODAL_LOGLEVEL` using -a Modal [`Secret`](/docs/reference/modal.Secret). - -```python -@app.function(secrets=[modal.Secret.from_dict({"MODAL_LOGLEVEL": "DEBUG"})]) -def f(): - print("Hello, world!") -``` - -### Client tracebacks - -To see a traceback (a.k.a [stack trace](https://en.wikipedia.org/wiki/Stack_trace)) for a client-side exception, you can set the `MODAL_TRACEBACK` environment variable to `1`. - -```bash -MODAL_TRACEBACK=1 modal run my_app.py -``` - -We encourage you to report cases where you need to enable this functionality, as it's indication of an issue in Modal. diff --git a/.agents/skills/modal/references/guide/developing-with-llms.md b/.agents/skills/modal/references/guide/developing-with-llms.md deleted file mode 100644 index 9b1573f2c7c0feef45ffc39ffb7d7c633358737c..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/developing-with-llms.md +++ /dev/null @@ -1,170 +0,0 @@ -# Developing Modal code with LLMs and agents - -Modal is designed for fast iteration on infrastructure by both humans and machines. - -Modal was [built to provide human backend engineers the rapid feedback loops frontend engineers take for granted](https://erikbern.com/2022/12/07/what-ive-been-working-on-modal.html). -This [also means](/blog/agents-devex) that Modal works well with code generation agents, especially those that can run CLI commands like `modal run` in an implement, test and debug loop, like OpenCode, Amp, Claude Code, Cursor's agent mode, Gemini CLI, etc. - -There are of course also many concepts and design patterns that are unique to Modal, so below we gather rules and guidelines that we have found useful when developing Modal code with LLMs. You can paste/import this into your `AGENTS.md`, `CLAUDE.md`, `.cursor/rules/modal.mdc`, etc. or use it as a starting point for your own rules or prompts. -We also provide an [`llms.txt`](/llms.txt). - -If you're instead looking for information about how to run code generation agents *on Modal*, -see [the guide page for Modal Sandboxes](/docs/guide/sandboxes) -or [this example code](https://modal.com/docs/examples/sandbox_agent). - -````markdown -# Modal Rules and Guidelines for LLMs - -This file provides rules and guidelines for LLMs when implementing Modal code. - -## General - -- Modal is a serverless cloud platform for running Python code with minimal configuration -- Designed for AI/ML workloads but supports general-purpose cloud compute -- Serverless billing model - you only pay for resources used - -## Modal documentation - -- Extensive documentation is available at: modal.com/docs (and in markdown format at modal.com/llms-full.txt) -- A large collection of examples is available at: modal.com/docs/examples (and github.com/modal-labs/modal-examples) -- Reference documentation is available at: modal.com/docs/reference - -Always refer to documentation and examples for up-to-date functionality and exact syntax. - -## Core Modal concepts - -### App - -- A group of functions, classes and sandboxes that are deployed together. - -### Function - -- The basic unit of serverless execution on Modal. -- Each Function executes in its own container, and you can configure different Images for different Functions within the same App: - - ```python - image = ( - modal.Image.debian_slim(python_version="3.12") - .pip_install("torch", "numpy", "transformers") - .apt_install("ffmpeg") - .run_commands("mkdir -p /models") - ) - - @app.function(image=image) - def square(x: int) -> int: - return x * x - ``` - -- You can configure individual hardware requirements (CPU, memory, GPUs, etc.) for each Function. - - ```python - @app.function( - gpu="H100", - memory=4096, - cpu=2, - ) - def inference(): - ... - ``` - - Some examples specificly for GPUs: - - ```python - @app.function(gpu="A10G") # Single GPU, e.g. T4, A10G, A100, H100, or "any" - @app.function(gpu="A100:2") # Multiple GPUs, e.g. 2x A100 GPUs - @app.function(gpu=["H100", "A100", "any"]) # GPU with fallbacks - ``` - -- Functions can be invoked in a number of ways. Some of the most common are: - - `foo.remote()` - Run the Function in a separate container in the cloud. This is by far the most common. - - `foo.local()` - Run the Function in the same context as the caller. Note: This does not necessarily mean locally on your machine. - - `foo.map()` - Parallel map over a set of inputs. - - `foo.spawn()` - Calls the function with the given arguments, without waiting for the results. Terminating the App will also terminate spawned functions. -- Web Functions: You can expose any Function to public HTTP traffic by adding a decorator: - - ```python - @app.function() - @modal.fastapi_endpoint() - def fastapi_endpoint(): - return {"status": "ok"} - - @app.function() - @modal.asgi_app() - def asgi_app(): - app = FastAPI() - ... - return app - ``` - -- You can run Functions on a schedule using e.g. `@app.function(schedule=modal.Period(minutes=5))` or `@app.function(schedule=modal.Cron("0 9 * * *"))`. - -### Classes (a.k.a. `Cls`) - -- For stateful operations with startup/shutdown lifecycle hooks. Example: - - ```python - @app.cls(gpu="A100") - class ModelServer: - @modal.enter() - def load_model(self): - # Runs once when container starts - self.model = load_model() - - @modal.method() - def predict(self, text: str) -> str: - return self.model.generate(text) - - @modal.exit() - def cleanup(self): - # Runs when container stops - cleanup() - ``` - -### Other important concepts - -- Image: Represents a container image that Functions can run in. -- Sandbox: Allows defining containers at runtime and securely running arbitrary code inside them. -- Volume: Provide a high-performance distributed file system for your Modal applications. -- Secret: Enables securely providing credentials and other sensitive information to your Modal Functions. -- Dict: Distributed key/value store, managed by Modal. -- Queue: Distributed, FIFO queue, managed by Modal. - -## Differences from standard Python development - -- Modal always executes code in the cloud, even while you are developing. You can use Environments for separating development and production deployments. -- Dependencies: It's common and encouraged to have different dependency requirements for different Functions within the same App. Consider defining dependencies in Image definitions (see Image docs) that are attached to Functions, rather than in global `requirements.txt`/`pyproject.toml` files, and putting `import` statements inside the Function `def`. Any code in the global scope needs to be executable in all environments where that App source will be used (locally, and any of the Images the App uses). - -## Modal coding style - -- Modal Apps, Volumes, and Secrets should be named using kebab-case. -- Always use `import modal`, and qualified names like `modal.App()`, `modal.Image.debian_slim()`. -- Modal evolves quickly, and prints helpful deprecation warnings when you `modal run` an App that uses deprecated features. When writing new code, never use deprecated features. - -## Common commands - -Running `modal --help` gives you a list of all available commands. All commands also support `--help` for more details. - -### Developing your Modal App - -- `modal run path/to/your/app.py` - Run your app on Modal. -- `modal run -m module.path.to.app` - Run your app on Modal, using the Python module path. -- `modal serve modal_server.py` - Serve Web Function(s) associated with a Modal app, and hot-reload code on changes. Will print a URL to the Web Function(s). Note: you need to use `Ctrl+C` to interrupt `modal serve`. - -### Deploying your Modal App - -- `modal deploy path/to/your/app.py` - Deploy your App to Modal. -- `modal deploy -m module.path.to.app` - Deploy your App to Modal, using the Python module path. - -Logs: - -- `modal app logs ` - Stream logs for a deployed app. Note: you need to use `Ctrl+C` to interrupt the stream. - -### Resource management - -- There are CLI commands for interacting with resources like `modal app list`, `modal volume list`, and similarly for `secret`, `dict`, `queue`, etc. -- These also support other command than `list` - use e.g. `modal app --help` for more. - -## Testing and debugging - -- When using `app.deploy()`, you can wrap it in a `with modal.enable_output():` block to get more output. -```` diff --git a/.agents/skills/modal/references/guide/dicts.md b/.agents/skills/modal/references/guide/dicts.md deleted file mode 100644 index 569cf6657a41c04d1771af87b0cfa78fa6dd07dc..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/dicts.md +++ /dev/null @@ -1,99 +0,0 @@ -# Dicts - -Modal Dicts provide distributed key-value storage to your Modal Apps. - -```python runner:ModalRunner -import modal - -app = modal.App() -kv = modal.Dict.from_name("kv", create_if_missing=True) - - -@app.local_entrypoint() -def main(key="cloud", value="dictionary", put=True): - if put: - kv[key] = value - print(f"{key}: {kv[key]}") -``` - -This page is a high-level guide to using Modal Dicts. -For reference documentation on the `modal.Dict` object, see -[this page](/docs/reference/modal.Dict). -For reference documentation on the `modal dict` CLI command, see -[this page](/docs/reference/cli/dict). - -## Modal Dicts are Python dicts in the cloud - -Dicts provide distributed key-value storage to your Modal Apps. -Much like a standard Python dictionary, a Dict lets you store and retrieve -values using keys. However, unlike a regular dictionary, a Dict in Modal is -accessible from anywhere, concurrently and in parallel. - -```python -# create a remote Dict -dictionary = modal.Dict.from_name("my-dict", create_if_missing=True) - - -dictionary["key"] = "value" # set a value from anywhere -value = dictionary["key"] # get a value from anywhere -``` - -Dicts are persisted, which means that the data in the dictionary is -stored and can be retrieved even after the application is redeployed. - -## You can access Modal Dicts asynchronously - -Modal Dicts live in the cloud, which means reads and writes -against them go over the network. That has some unavoidable latency overhead, -relative to just reading from memory, of a few dozen ms. -Reads from Dicts via `["key"]`-style indexing are synchronous, -which means that latency is often directly felt by the application. - -But like all Modal objects, you can also interact with Dicts asynchronously -by putting the `.aio` suffix on methods -- in this case, `put` and `get`, -which are synonyms for bracket-based indexing. -Just add the `async` keyword to your `local_entrypoint`s or remote Functions -and `await` the method calls. - -```python runner:ModalRunner -import modal - -app = modal.App() -dictionary = modal.Dict.from_name("async-dict", create_if_missing=True) - - -@app.local_entrypoint() -async def main(): - await dictionary.put.aio("key", "value") # setting a value asynchronously - assert await dictionary.get.aio("key") # getting a value asynchronously -``` - -See the guide to [asynchronous functions](/docs/guide/async) for more -information. - -## Modal Dicts are not *exactly* Python dicts - -Python dicts can have keys of any hashable type and values of any type. - -You can store Python objects of any serializable type within Dicts as keys or values. - -Objects are serialized using [`cloudpickle`](https://github.com/cloudpipe/cloudpickle), -so precise support is inherited from that library. `cloudpickle` can serialize a surprising variety of objects, -like `lambda` functions or even Python modules, but it can't serialize a few things that don't -really make sense to serialize, like live system resources (sockets, writable file descriptors). - -Note that you will need to have the library defining the type installed in the environment -where you retrieve the object so that it can be deserialized. - -Unlike with normal Python dictionaries, updates to mutable value types will not -be reflected in other containers unless the updated object is explicitly put -back into the Dict. As a consequence, patterns like chained updates -(`my_dict["outer_key"]["inner_key"] = value`) cannot be used the same way as -they would with a local dictionary. - -Currently, the per-object size limit is 100 MiB and the maximum number of entries -per update is 10,000. It's recommended to use Dicts for smaller objects (under 5 MiB). -Each object in the Dict will expire after 7 days of inactivity (no reads or writes). - -Dicts also provide a locking primitive. See -[this blog post](/blog/cache-dict-launch) for details. diff --git a/.agents/skills/modal/references/guide/docker-in-sandboxes.md b/.agents/skills/modal/references/guide/docker-in-sandboxes.md deleted file mode 100644 index 717615f42b0c61d695b28cade10f0916491e0bcf..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/docker-in-sandboxes.md +++ /dev/null @@ -1,111 +0,0 @@ -# Docker in Sandboxes - - - -Modal has Alpha support for running Docker containers inside `modal.Sandbox`. -This is intended to support coding agents who want to interact with development environments that are managed using Docker. We recommend that you use [VM sandboxes](/docs/guide/vm-sandboxes) for such workloads. - -## Demo - -Copy the following program to e.g. `docker_in_modal_demo.py`, and run it with `python docker_in_modal_demo.py`. -Note that we use a VM sandbox in this example. - -```python -import modal - -# Create an image for the parent Modal Sandbox, with Docker installed. -def create_modal_sandbox_image(): - image = ( - modal.Image.from_registry("ubuntu:24.04") - .env({"DEBIAN_FRONTEND": "noninteractive"}) - .apt_install(["docker.io", "docker-buildx"]) - .run_commands("mkdir /build") - ) - return image - - -def main(): - print("Looking up modal.Sandbox app") - app = modal.App.lookup("docker-test", create_if_missing=True) - print("Creating sandbox") - - with modal.enable_output(): - sb = modal.Sandbox.create( - "/usr/bin/dockerd", - "-D", - timeout=60 * 60, - app=app, - image=create_modal_sandbox_image(), - experimental_options={"vm_runtime": True}, - ) - - print(f"sandbox_id: {sb.object_id}") - task_id = sb._get_task_id() - print(f"task_id: {task_id}") - print(f"To shell into the task, run: modal shell {task_id}") - # dockerd is the sandbox entrypoint and takes a moment to bind - # /var/run/docker.sock after the sandbox is created. Poll until the - # daemon answers so the first `docker build` doesn't run before dockerd is ready. - print("Waiting for dockerd to be ready") - wait_p = sb.exec( - "sh", - "-c", - "for i in $(seq 1 120); do " - "if [ -S /var/run/docker.sock ] && docker info >/dev/null 2>&1; then " - "echo ready; exit 0; fi; sleep 1; done; " - "echo 'dockerd not ready after 120s' >&2; exit 1", - ) - wait_p.wait() - if wait_p.returncode != 0: - raise Exception(f"dockerd never became ready: {wait_p.stderr.read()}") - - # A simple Dockerfile that we'll build and run within Modal. - dockerfile = """ - FROM ubuntu - RUN apt-get update - RUN apt-get install -y cowsay curl - RUN mkdir -p /usr/share/cowsay/cows/ - RUN curl -o /usr/share/cowsay/cows/docker.cow https://raw.githubusercontent.com/docker/whalesay/master/docker.cow - ENTRYPOINT ["/usr/games/cowsay", "-f", "docker.cow"] - """ - sb.filesystem.write_text(dockerfile, "/build/Dockerfile") - - print("Building docker image") - p = sb.exec("docker", "build", "--network=host", "-t", "whalesay", "/build") - for l in p.stdout: - print(l, end="") - p.wait() - print("--------------------------------") - if p.returncode != 0: - print(p.stderr.read()) - raise Exception("Docker build failed") - - # The Sandbox will run a container from the built image and print this: - # - # ________ - # < Hello! > - # -------- - # \ - # \ - # \ - # ## . - # ## ## ## == - # ## ## ## ## ## === - # /"""""""""""""""""\___/ === - # { / ===- - # \______ O __/ - # \ \ __/ - # \____\_______/ - - print("Running Docker image") - # Note we can't use -it here because we're not in a TTY. - p = sb.exec("docker", "run", "--rm", "whalesay", "Hello!") - print(p.stdout.read()) - p.wait() - if p.returncode != 0: - raise Exception(f"Docker run failed: {p.stderr.read()}") - sb.terminate() - -if __name__ == "__main__": - main() -``` diff --git a/.agents/skills/modal/references/guide/dynamic-batching.md b/.agents/skills/modal/references/guide/dynamic-batching.md deleted file mode 100644 index ce44c84a4edd1f2879292649afcc43b6054a6dd8..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/dynamic-batching.md +++ /dev/null @@ -1,168 +0,0 @@ -# Dynamic batching - -Modal's `@batched` feature allows you to accumulate requests -and process them in dynamically-sized batches, rather than one-by-one. - -Batching increases throughput at a potential cost to latency. -Batched requests can share resources and reuse work, reducing the time and cost per request. -Batching is particularly useful for GPU-accelerated machine learning workloads, -as GPUs are designed to maximize throughput and are frequently bottlenecked on shareable resources, -like weights stored in memory. - -Static batching can lead to unbounded latency, as the function waits for a fixed number of requests to arrive. -Modal's dynamic batching waits for the lesser of a fixed time *or* a fixed number of requests before executing, -maximizing the throughput benefit of batching while minimizing the latency penalty. - -## Enable dynamic batching with `@batched` - -To enable dynamic batching, apply the -[`@modal.batched` decorator](/docs/reference/modal.batched) to the target -Python function. Then, wrap it in `@app.function()` and run it on Modal, -and the inputs will be accumulated and processed in batches. - -Here's what that looks like: - -```python -import modal - -app = modal.App() - -@app.function() -@modal.batched(max_batch_size=2, wait_ms=1000) -async def batch_add(xs: list[int], ys: list[int]) -> list[int]: - return [x + y for x, y in zip(xs, ys)] -``` - -When you invoke a function decorated with `@batched`, you invoke it asynchronously on individual inputs. -Outputs are returned where they were invoked. - -For instance, the code below invokes the decorated `batch_add` function above three times, but `batch_add` -only executes twice: - -```python continuation -@app.local_entrypoint() -async def main(): - inputs = [(1, 300), (2, 200), (3, 100)] - async for result in batch_add.starmap.aio(inputs): - print(f"Sum: {result}") - # Sum: 301 - # Sum: 202 - # Sum: 103 -``` - -The first time it is executed with `xs` batched to `[1, 2]` -and `ys` batched to `[300, 200]`. After about a one second delay, it is executed with `xs` -batched to `[3]` and `ys` batched to `[100]`. -The result is an iterator that yields `301`, `202`, and `103`. - -## Use `@batched` with functions that take and return lists - -For a Python function to be compatible with `@modal.batched`, it must adhere to -the following rules: - -* \*\* The inputs to the function must be lists. \*\* - In the example above, we pass `xs` and `ys`, which are both lists of `int`s. -* \*\* The function must return a list\*\*. In the example above, the function returns - a list of sums. -* \*\* The lengths of all the input lists and the output list must be the same. \*\* - In the example above, if `L == len(xs) == len(ys)`, then `L == len(batch_add(xs, ys))`. - -## Modal `Cls` methods are compatible with dynamic batching - -Methods on Modal [`Cls`](/docs/guide/lifecycle-functions)es also support dynamic batching. - -```python -import modal - -app = modal.App() - -@app.cls() -class BatchedClass(): - @modal.batched(max_batch_size=2, wait_ms=1000) - async def batch_add(self, xs: list[int], ys: list[int]) -> list[int]: - return [x + y for x, y in zip(xs, ys)] -``` - -One additional rule applies to classes with Batched Methods: - -* If a class has a Batched Method, it **cannot have other Batched Methods or [Methods](/docs/reference/modal.method#modalmethod)**. - -## Configure the wait time and batch size of dynamic batches - -The `@batched` decorator takes in two required configuration parameters: - -* `max_batch_size` limits the number of inputs combined into a single batch. -* `wait_ms` limits the amount of time the Function waits for more inputs after - the first input is received. - -The first invocation of the Batched Function initiates a new batch, and subsequent -calls add requests to this ongoing batch. If `max_batch_size` is reached, -the batch immediately executes. If the `max_batch_size` is not met but `wait_ms` -has passed since the first request was added to the batch, the unfilled batch is -executed. - -### Selecting a batch configuration - -To optimize the batching configurations for your application, consider the following heuristics: - -* Set `max_batch_size` to the largest value your function can handle, so you - can amortize and parallelize as much work as possible. - -* Set `wait_ms` to the difference between your targeted latency and the execution time. Most applications - have a targeted latency, and this allows the latency of any request to stay - within that limit. - -## Serve Web Functions with dynamic batching - -Here's a simple example of serving a Function that batches requests dynamically -with a [`@modal.fastapi_endpoint`](/docs/guide/webhooks). Run -[`modal serve`](/docs/reference/cli/serve), submit requests to the endpoint, -and the Function will batch your requests on the fly. - -```python -import modal - -app = modal.App(image=modal.Image.debian_slim().pip_install("fastapi")) - -@app.function() -@modal.batched(max_batch_size=2, wait_ms=1000) -async def batch_add(xs: list[int], ys: list[int]) -> list[int]: - return [x + y for x, y in zip(xs, ys)] - - -@app.function() -@modal.fastapi_endpoint(method="POST", docs=True) -async def add(body: dict[str, int]) -> dict[str, int]: - result = await batch_add.remote.aio(body["x"], body["y"]) - return {"result": result} -``` - -Now, you can submit requests to the Web Function and process them in batches. For instance, the three requests -in the following example, which might be requests from concurrent clients in a real deployment, -will be batched into two executions: - -```python notest -import asyncio -import aiohttp - -async def send_post_request(session, url, data): - async with session.post(url, json=data) as response: - return await response.json() - -async def main(): - # Enter the Web Function URL here - url = "https://workspace--app-name-endpoint-name.modal.run" - - async with aiohttp.ClientSession() as session: - # Submit three requests asynchronously - tasks = [ - send_post_request(session, url, {"x": 1, "y": 300}), - send_post_request(session, url, {"x": 2, "y": 200}), - send_post_request(session, url, {"x": 3, "y": 100}), - ] - results = await asyncio.gather(*tasks) - for result in results: - print(f"Sum: {result['result']}") - -asyncio.run(main()) -``` diff --git a/.agents/skills/modal/references/guide/dynamic-function-config.md b/.agents/skills/modal/references/guide/dynamic-function-config.md deleted file mode 100644 index 72e5abf1ac66976a721dfcdb6e5a4b92e8f2f3f0..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/dynamic-function-config.md +++ /dev/null @@ -1,72 +0,0 @@ -# Dynamic Function configuration - -Many aspects of a Modal Function's configuration can be dynamically configured from a specific call site. This is useful in cases where the Function's [compute resources](/docs/guide/resources), [secrets](/docs/guide/secrets), [timeout](/docs/guide/timeouts), or other properties need to vary depending on the specific inputs. - -## Basic configuration - -Features exposed in the [`@app.function()`](/docs/reference/modal.App#function) decorator can be dynamically configured at runtime with the [`modal.Function.with_options()`](/docs/reference/modal.Function#with_options) method. - -Say you have the following definition: - -```python -@app.function() -def f(x: int) -> int: - return x ** 2 -``` - -If (for some reason) you wanted to compare this Function's output across several different GPUs, you could invoke it several times with different configurations: - -```python continuation -@app.local_entrypoint() -def main(): - for gpu in ["T4", "L4", "A10"]: - result = f.with_options(gpu=gpu).remote(2) - print(f"Result with {gpu} GPU: {result}") -``` - -This example creates three additional variants of the base Function after the App is already running. These variants are *new Functions* that are created on-demand. The base Function itself is not affected. If you invoked `f.remote()` directly, it would continue to execute without a GPU. - -Deployed Functions can also be dynamically configured from a call site after a lookup: - -```python notest -deployed_f = modal.Function.from_name("demo-app", "f") -for gpu in ["T4", "L4", "A10"]: - result = deployed_f.with_options(gpu=gpu).remote(2) - print(f"Result with {gpu} GPU: {result}") -``` - -## Input concurrency and batching - -It's also possible to dynamically configure [input concurrency](/docs/guide/concurrent-inputs) or [batching](/docs/guide/dynamic-batching). As these features are enabled with separate decorators ([`@modal.concurrent()`](/docs/reference/modal.concurrent)/[`@modal.batched()`](/docs/reference/modal.batched)), their dynamic configuration runs through separate methods ([`modal.Function.with_concurrency()`](/docs/reference/modal.Function#with_concurrency)/[`modal.Function.with_batching()`](/docs/reference/modal.Function#with_batching)): - -```python notest -concurrent_f = modal.Function.from_name("demo-app", "f").with_concurrency(max_inputs=32) -``` - -If multiple dynamic configuration methods are called in sequence, their arguments will compose and form a single configuration: - -```python notest -# This Function uses a GPU with input concurrency -concurrent_f.with_options(gpu="H100").remote(...) -``` - -## Autoscaling considerations - -Each distinct configuration has its own dedicated autoscaling container pool. By default, the container pool will autoscale according to the configuration of the base Function, with separate accounting. For example, if your Function has `@app.function(max_containers=5)` and you dynamically add a GPU using `f.with_options(gpu="H100")`, you'll get up to 5 *additional* H100 containers regardless of how many CPU containers are currently running. - -Try to avoid generating too many fine-grained configurations so that you can benefit from container sharing for higher utilization and reduced cold start latencies. For example, if requesting input-specific `memory=` or `cpu=` resources, it's best to round into coarse buckets. - -Functions that have been looked up and dynamically configured in separate processes will still share containers if they apply the same configuration. - -If your base Function configuration has `min_containers` set, it will be ignored by the Function variants to avoid creating zombie warm pools. For the same reason, it's not possible to set `min_containers` in `modal.Function.with_options()`. - -It is possible to dynamically configure other aspects of autoscaling behavior using `modal.Function.with_options()`. For example, if you don't expect to re-use the variant, you could reduce the `scaledown_window` so that the container shuts down faster. However, if your goal is to use different autoscaling policies over time, it may be simpler to modify the base Function's behavior using [`modal.Function.update_autoscaler`](/docs/reference/modal.Function#update_autoscaler) instead. - -## Dynamic Cls configuration - -It's also possible to dynamically configure a `modal.Cls`. If the Cls is [parametrized](/docs/guide/parametrized-functions) (which also creates a new Function variant with its own container pool and autoscaling accounting), the dynamic options will compose with the parameter values: - -```python notest -ModelCls = modal.Cls.from_name("demo-app", "ModelCls") -model = ModelCls.with_options(gpu="H100")(size="8B") -``` diff --git a/.agents/skills/modal/references/guide/environment_variables.md b/.agents/skills/modal/references/guide/environment_variables.md deleted file mode 100644 index 466f55faee8023091d9b8db0194dd110a023fafd..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/environment_variables.md +++ /dev/null @@ -1,54 +0,0 @@ -# Environment variables - -The Modal runtime sets several environment variables during initialization. The -keys for these environment variables are reserved and cannot be overridden by -your Function or Sandbox configuration. - -These variables provide information about the container's runtime -environment. - -## Container runtime environment variables - -The following variables are present in every Modal container: - -* **`MODAL_CLOUD_PROVIDER`** — Modal executes containers across a number of cloud - providers ([AWS](https://aws.amazon.com/), [GCP](https://cloud.google.com/), - [OCI](https://www.oracle.com/cloud/)). This variable specifies which cloud - provider the Modal container is running within. -* **`MODAL_IMAGE_ID`** — The ID of the - [`modal.Image`](/docs/reference/modal.Image) used by the Modal container. -* **`MODAL_REGION`** — This will correspond to a geographic area identifier from - the cloud provider associated with the Modal container (see above). For AWS, the - identifier is a "region". For GCP it is a "zone", and for OCI it is an - "availability domain". Example values are `us-east-1` (AWS), `us-central1` - (GCP), `us-ashburn-1` (OCI). See the [full list here](/docs/guide/region-selection#container-region-options). -* **`MODAL_TASK_ID`** — The ID of the container running the Modal Function or Sandbox. - -## Function runtime environment variables - -The following variables are present in containers running Modal Functions: - -* **`MODAL_ENVIRONMENT`** — The name of the - [Modal Environment](/docs/guide/environments) the container is running within. -* **`MODAL_IS_REMOTE`** - Set to '1' to indicate that Modal Function code is running in - a remote container. -* **`MODAL_IDENTITY_TOKEN`** — An [OIDC token](/docs/guide/oidc-integration) - encoding the identity of the Modal Function. - -## Sandbox environment variables - -The following variables are present within [`modal.Sandbox`](/docs/reference/modal.Sandbox) instances. - -* **`MODAL_SANDBOX_ID`** — The ID of the Sandbox. - -## Container image environment variables - -The container image layers used by a `modal.Image` may set -environment variables. These variables will be present within your container's runtime -environment. For example, the -[`debian_slim`](/docs/reference/modal.Image#debian_slim) image sets the -`GPG_KEY` variable. - -To override image variables or set new ones, use the -[`.env`](https://modal.com/docs/reference/modal.Image#env) method provided by -`modal.Image`. diff --git a/.agents/skills/modal/references/guide/environments.md b/.agents/skills/modal/references/guide/environments.md deleted file mode 100644 index 6ca11303b0330476a9830ca30dac6a8a9af8096f..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/environments.md +++ /dev/null @@ -1,77 +0,0 @@ -# Environments - -Modal Environments isolate Modal applications and resources from one another. - -Environments are sub-divisions of [Workspaces](/docs/guide/workspaces), -allowing you to deploy the same App (or set of Apps) -in multiple instances for different purposes without changing code. - -Typical use cases for Environments include having one `dev` -Environment and one `prod` Environment. Production Apps are protected from overwriting -when developing new features, but you can still deploy and test changes with a -"live" and potentially complex structure of Apps. - -Each Environment has its own set of [Secrets](/docs/guide/secrets) and any -object lookups, say for [Dicts](/docs/guide/dicts) or [Volumes](/docs/guide/volumes), -performed from an App in an Environment will by default look for objects in the same Environment. - -By default, every Workspace has a single Environment called "main". New -Environments can be created on the CLI: - -```sh -modal environment create dev -``` - -Run `modal environment --help` for more info. - -Workspaces can have up to 1500 Environments. - -Once created, Environments show up as a dropdown menu in the navbar of the -[Modal dashboard](/apps), letting you set browse all Modal Apps, Secrets, and Storage -filtered by which Environment they were deployed to. - -Most CLI commands also support an `--env` flag letting you specify which -Environment you intend to interact with, e.g.: - -```sh -modal run --env=dev app.py -modal volume create --env=dev storage -``` - -To set a default Environment for your current CLI profile you can use -`modal config set-environment`, e.g.: - -```sh -modal config set-environment dev -``` - -Alternatively, you can set the `MODAL_ENVIRONMENT` environment variable. - -## Environment web suffixes - -Environments have a 'web suffix' which is used to make -[Web Function URLs](/docs/guide/webhook-urls) unique across your workspace. One -Environment is allowed to have no suffix (`""`). - -## Cross environment lookups - -It's possible to explicitly look up objects in Environments other than the Environment -your App runs within: - -```python -production_secret = modal.Secret.from_name( - "my-secret", - environment_name="main", -) -``` - -```python notest -modal.Function.from_name( - "my_app", - "some_function", - environment_name="dev" -) -``` - -However, the `environment_name` argument is optional and omitting it will use -the Environment from the object's associated App or calling context. diff --git a/.agents/skills/modal/references/guide/existing-images.md b/.agents/skills/modal/references/guide/existing-images.md deleted file mode 100644 index eecb51ad5cca5a484707d4ea71695dfd7df7685b..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/existing-images.md +++ /dev/null @@ -1,235 +0,0 @@ -# Using existing images - -This guide walks you through how to use an existing container image as a Modal Image. - -```python notest -sklearn_image = modal.Image.from_registry("huanjason/scikit-learn") -custom_image = modal.Image.from_dockerfile("./src/Dockerfile") -``` - -## Load an image from a public registry with `.from_registry` - -To load an image from a public registry, just pass the image name, including any tags, to [`Image.from_registry`](/docs/reference/modal.Image#from_registry): - -```python -sklearn_image = modal.Image.from_registry("huanjason/scikit-learn") - - -@app.function(image=sklearn_image) -def fit_knn(): - from sklearn.neighbors import KNeighborsClassifier - ... -``` - -The `from_registry` method can load images from all public registries, such as -[Nvidia's `nvcr.io`](https://catalog.ngc.nvidia.com/containers), -[AWS ECR](https://aws.amazon.com/ecr/), and -[GitHub's `ghcr.io`](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry). - -You can further modify the image [just like any other Modal Image](/docs/guide/images): - -```python continuation -data_science_image = sklearn_image.uv_pip_install("polars", "datasette") -``` - -You can use external images so long as - -* The image is built for the - [`linux/amd64` platform](https://unix.stackexchange.com/questions/53415/why-are-64-bit-distros-often-called-amd64) -* The image has a [compatible `ENTRYPOINT`](#entrypoint) - -Additionally, to be used with a Modal Function, the image needs to have `python` and `pip` -installed and available on the `$PATH`. -If an existing image does not have either `python` or `pip` set up compatibly, you -can still use it. Just provide a version number as the `add_python` argument to -install a reproducible -[standalone build](https://github.com/indygreg/python-build-standalone) -of Python: - -```python -ubuntu_image = modal.Image.from_registry("ubuntu:22.04", add_python="3.11") -valhalla_image = modal.Image.from_registry("gisops/valhalla:latest", add_python="3.12") -``` - -There are some additional restrictions for older versions of the Modal image builder. -Image builder version is set at a workspace level via the settings page [here](/settings/image-builder-version). -See the migration guides on that page for details on any additional restrictions on images. - -## Load images from private registries - -You can also use images defined in private container registries on Modal. -The exact method depends on the registry you are using. - -### Docker Hub (Private) - -To pull container images from private Docker Hub repositories, -[create an access token](https://docs.docker.com/security/for-developers/access-tokens/) -with "Read-Only" permissions and use this token value and your Docker Hub -username to create a Modal [Secret](/docs/guide/secrets). - -``` -REGISTRY_USERNAME=my-dockerhub-username -REGISTRY_PASSWORD=dckr_pat_TS012345aaa67890bbbb1234ccc -``` - -Use this Secret with the -[`modal.Image.from_registry`](/docs/reference/modal.Image#from_registry) method. - -### Elastic Container Registry (ECR) - -You can pull images from your AWS ECR account by specifying the full image URI -as follows: - -```python -import modal - -aws_secret = modal.Secret.from_name("my-aws-secret") -image = ( - modal.Image.from_aws_ecr( - "000000000000.dkr.ecr.us-east-1.amazonaws.com/my-private-registry:latest", - secret=aws_secret, - ) - .pip_install("torch", "numpy", "huggingface") -) - -app = modal.App(image=image) -``` - -As shown above, you also need to use a [Modal Secret](/docs/guide/secrets) -containing the environment variables `AWS_ACCESS_KEY_ID`, -`AWS_SECRET_ACCESS_KEY`, and `AWS_REGION`. The AWS IAM user account associated -with those keys must have access to the private registry you want to access. - -Alternatively, you can use [OIDC token authentication](/docs/guide/oidc-integration#pull-images-from-aws-elastic-container-registry-ecr). - -The user needs to have the following read-only policies: - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Action": ["ecr:GetAuthorizationToken"], - "Effect": "Allow", - "Resource": "*" - }, - { - "Effect": "Allow", - "Action": [ - "ecr:BatchCheckLayerAvailability", - "ecr:GetDownloadUrlForLayer", - "ecr:GetRepositoryPolicy", - "ecr:DescribeRepositories", - "ecr:ListImages", - "ecr:DescribeImages", - "ecr:BatchGetImage", - "ecr:GetLifecyclePolicy", - "ecr:GetLifecyclePolicyPreview", - "ecr:ListTagsForResource", - "ecr:DescribeImageScanFindings" - ], - "Resource": "" - } - ] -} -``` - -You can use the IAM configuration above as a template for creating an IAM user. -You can then -[generate an access key](https://aws.amazon.com/premiumsupport/knowledge-center/create-access-key/) -and create a Modal Secret using the AWS integration option. Modal will use your -access keys to generate an ephemeral ECR token. That token is only used to pull -image layers at the time a new image is built. We don't store this token but -will cache the image once it has been pulled. - -Images on ECR must be private and follow -[image configuration requirements](/docs/reference/modal.Image#from_aws_ecr). - -### Google Artifact Registry and Google Container Registry - -For further detail on how to pull images from Google's image registries, see -[`modal.Image.from_gcp_artifact_registry`](/docs/reference/modal.Image#from_gcp_artifact_registry). - -### Azure Container Registry (ACR) - -Modal doesn't have native Azure support, but you can pull images from a private ACR using -ACR's [token-based repository permissions](https://learn.microsoft.com/en-us/azure/container-registry/container-registry-token-based-repository-permissions) -to generate long-lived Docker credentials. Those credentials (token and password) can then be stored -as a Modal Secret and used with [`modal.Image.from_registry`](/docs/reference/modal.Image#from_registry) -the same way as [Docker Hub private registry](#docker-hub-private) credentials. - -## Bring your own image definition with `.from_dockerfile` - -You can define an Image from an existing Dockerfile by passing its path to -[`Image.from_dockerfile`](/docs/reference/modal.Image#from_dockerfile): - -```python -dockerfile_image = modal.Image.from_dockerfile("Dockerfile") - - -@app.function(image=dockerfile_image) -def fit(): - import sklearn - ... -``` - -Note that you can still extend this Image using image builder methods! -See [the guide](/docs/guide/images) for details. - -### Dockerfile command compatibility - -Since Modal doesn't use Docker to build containers, we have our own -implementation of the -[Dockerfile specification](https://docs.docker.com/engine/reference/builder/). -Most Dockerfiles should work out of the box, but there are some differences to -be aware of. - -First, a few minor Dockerfile commands and flags have not been implemented yet. -These include `ONBUILD`, `STOPSIGNAL`, and `VOLUME`. -Please reach out to us if your use case requires any of these. - -Next, there are some command-specific things that may be useful when porting a -Dockerfile to Modal. - -#### `ENTRYPOINT` - -While the -[`ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) -command is supported, there is an additional constraint to the entrypoint script -provided: when used with a Modal Function, it must also `exec` the arguments passed to it at some point. -This is so the Modal Function runtime's Python entrypoint can run after your own. Most entrypoint -scripts in Docker containers are wrappers over other scripts, so this is likely -already the case. - -If you wish to write your own entrypoint script, you can use the following as a -template: - -```bash -#!/usr/bin/env bash - -# Your custom startup commands here. - -exec "$@" # Runs the command passed to the entrypoint script. -``` - -If the above file is saved as `/usr/bin/my_entrypoint.sh` in your container, -then you can register it as an entrypoint with -`ENTRYPOINT ["/usr/bin/my_entrypoint.sh"]` in your Dockerfile, or with -[`entrypoint`](/docs/reference/modal.Image#entrypoint) as an -Image build step. - -```python -import modal - -image = ( - modal.Image.debian_slim() - .pip_install("foo") - .entrypoint(["/usr/bin/my_entrypoint.sh"]) -) -``` - -#### `ENV` - -We currently don't support default values in -[interpolations](https://docs.docker.com/compose/compose-file/12-interpolation/), -such as `${VAR:-default}` diff --git a/.agents/skills/modal/references/guide/fast-pull-from-registry.md b/.agents/skills/modal/references/guide/fast-pull-from-registry.md deleted file mode 100644 index 24852484663e817f87e4d5608a4b25ab59866005..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/fast-pull-from-registry.md +++ /dev/null @@ -1,66 +0,0 @@ -# Fast pull from registry - -The performance of pulling public and private images from registries into Modal -can be significantly improved by adopting the [eStargz](https://github.com/containerd/stargz-snapshotter/blob/main/docs/estargz.md) compression format. - -By applying eStargz compression during your image build and push, Modal will be much -more efficient at pulling down your image from the registry. - -## How to use estargz - -If you have [Buildkit](https://docs.docker.com/build/buildkit/) version greater than `0.10.0`, adopting `estargz` is as simple as -adding some flags to your `docker buildx build` command: - -* `type=registry` flag will instruct BuildKit to push the image after building. - * If you do not push the image from immediately after build and instead attempt to push it later with docker push, the image will be converted to a standard gzip image. -* `compression=estargz` specifies that we are using the [eStargz](https://github.com/containerd/stargz-snapshotter/blob/main/docs/estargz.md) compression format. -* `oci-mediatypes=true` specifies that we are using the OCI media types, which is required for eStargz. -* `force-compression=true` will recompress the entire image and convert the base image to eStargz if it is not already. - -```bash -docker buildx build --tag "//:" \ ---output type=registry,compression=estargz,force-compression=true,oci-mediatypes=true \ -. -``` - -Then reference the container image as normal in your Modal code. - -```python notest -app = modal.App( - "example-estargz-pull", - image=modal.Image.from_registry( - "public.ecr.aws/modal/estargz-example-images:text-generation-v1-esgz" - ) -) -``` - -At build time you should see the eStargz-enabled puller activate: - -``` -Building image im-TinABCTIf12345ydEwTXYZ - -=> Step 0: FROM public.ecr.aws/modal/estargz-example-images:text-generation-v1-esgz -Using estargz to speed up image pull (index loaded in 1.86s)... -Progress: 10% complete... (1.11s elapsed) -Progress: 20% complete... (3.10s elapsed) -Progress: 30% complete... (4.18s elapsed) -Progress: 40% complete... (4.76s elapsed) -Progress: 50% complete... (5.51s elapsed) -Progress: 62% complete... (6.17s elapsed) -Progress: 74% complete... (6.99s elapsed) -Progress: 81% complete... (7.23s elapsed) -Progress: 99% complete... (8.90s elapsed) -Progress: 100% complete... (8.90s elapsed) -Copying image... -Copied image in 5.81s -``` - -## Supported registries - -Currently, Modal supports fast estargz pulling images with the following registries: - -* AWS Elastic Container Registry (ECR) -* Docker Hub (docker.io) -* Google Artifact Registry (gcr.io, pkg.dev) - -We are working on adding support for GitHub Container Registry (ghcr.io). diff --git a/.agents/skills/modal/references/guide/feature-maturity.md b/.agents/skills/modal/references/guide/feature-maturity.md deleted file mode 100644 index 92f34efd583a5344cab6517cf83340358b17d0a4..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/feature-maturity.md +++ /dev/null @@ -1,53 +0,0 @@ -# Feature Maturity - -New features at Modal evolve through several stages. To help you understand their stability, we use two separate concepts: - -* [Release phases](#release-phases) ([Alpha](#alpha), [Beta](#beta) or [GA](#general-availability-ga)): signals the stability of a feature's underlying infrastructure -* [Experimental SDK](#experimental-sdk): signals API stability in the code interface - -This separation allows the SDK to remain stable even while we are still refining the performance or scaling of a backend feature. - -## Release Phases - -We use the following release phases to signal the maturity of a feature's underlying design and infrastructure: - -### Alpha - -Alpha is reserved for features that might still be fragile and have known limitations. We provide these early so you can experiment with them, but you should expect significant changes to how the feature works. The documentation will clearly state their limitations. - -Some Alpha features are private, meaning you need to contact us to get access. - -### Beta - -Beta is our default phase for new features. Beta features are generally self-serve, functional, and mostly stable. Beta features are often suitable for production use, though we may still be refining the final behavior, pricing, or scale limits. - -Some Beta features are private, meaning you need to contact us to get access. - -### General Availability (GA) - -GA features are stable and fully ready for production grade usage. No breaking changes are planned. Any feature not marked as Alpha or Beta in the Modal docs can be considered GA. - -## Experimental SDK - -In addition to the release phases described above, you may see certain parts of the Modal SDK marked as experimental (e.g., `_experimental_snapshot()`). - -This is strictly an SDK concept which indicates API stability, not infrastructure maturity. It often correlates with the Alpha → Beta → GA progression, but not always. Some features stabilize their API early while the backend is still maturing, but experimental APIs may also be introduced later in a feature's lifecycle to provide additional depth of configuration. - -An experimental tag means we're still gathering feedback and iterating on the interface: method names, parameters, or return types may change. Once we're confident in the design, we remove the experimental marker and commit to backwards compatibility. - -## SDK Deprecations - -Features that are exposed via stable API in the SDK may become *deprecated*, either because we are discontinuing support for the associated platform feature, or because the API is being adjusted, e.g. to reduce a persistent confusion or to accommodate unanticipated extensions. - -Deprecated API will remain functional and will issue deprecation warnings. We recommend heeding these warnings, since deprecations will eventually be enforced and code that exercises the deprecated API will break. Breaking changes are limited to increments of the `Y` version in our `X.Y.Z` versioning scheme. - -## Other interfaces - -Only the official SDKs are currently considered to be stable. Any other public interfaces are undocumented, subject to change without warning, and use-at-your-own risk. - -## Providing Feedback - -We value your feedback on Alpha and Beta features! If you're using a feature at any release phase and have suggestions or encounter issues: - -* Join our [Slack community](https://modal.com/slack) to discuss with the team and other users -* Reach out to support@modal.com with specific feedback or bug reports diff --git a/.agents/skills/modal/references/guide/global-variables.md b/.agents/skills/modal/references/guide/global-variables.md deleted file mode 100644 index a26a3fb381bd830d07978786a4112a8d2b4a4a32..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/global-variables.md +++ /dev/null @@ -1,52 +0,0 @@ -# Global variables - -There are cases where you might want objects or data available in **global** -scope. For example: - -* You need to use the data in a scheduled function (scheduled functions don't - accept arguments) -* You need to construct objects (e.g. Secrets) in global scope to use as - function annotations -* You don't want to clutter many function signatures with some common arguments - they all use, and pass the same arguments through many layers of function - calls. - -For these cases, you can use the `modal.is_local` function, which returns `True` -if the app is running locally (initializing) or `False` if the app is executing -in the cloud. - -For instance, to create a [`modal.Secret`](/docs/guide/secrets) that you can pass -to your function decorators to create environment variables, you can run: - -```python -import os - -if modal.is_local(): - pg_password = modal.Secret.from_dict({"PGPASS": os.environ["MY_LOCAL_PASSWORD"]}) -else: - pg_password = modal.Secret.from_dict({}) - - -@app.function(secrets=[pg_password]) -def get_secret_data(): - connection = psycopg2.connect(password=os.environ["PGPASS"]) - ... -``` - -## Warning about regular module globals - -If you try to construct a global in module scope using some local data *without* -using something like `modal.is_local`, it might have unexpected effects since -your Python modules will be not only be loaded on your local machine, but also -on the remote worker. - -E.g., this will typically not work: - -```python notest -# blob.json doesn't exist on the remote worker, so this will cause an error there -data_blob = open("blob.json", "r").read() - -@app.function() -def foo(): - print(data_blob) -``` diff --git a/.agents/skills/modal/references/guide/gpu-health.md b/.agents/skills/modal/references/guide/gpu-health.md deleted file mode 100644 index 2ad9c25ceff0372785aef610f420ffc114d66173..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/gpu-health.md +++ /dev/null @@ -1,62 +0,0 @@ -# GPU Health - -Modal constantly monitors host GPU health, draining Workers with critical issues -and surfacing warnings for customer triage. - -Application level observability of GPU health is facilitated by [metrics](/docs/guide/gpu-metrics) and event logging to container log streams. - -## `[gpu-health]` logging - -Containers with attached NVIDIA GPUs are connected to our `gpu-health` monitoring system -and receive event logs which originate from either application software behavior, system software behavior, or hardware failure. - -These logs are in the following format: `[gpu-health] [LEVEL] GPU-[UUID]: EVENT_TYPE: MSG` - -* `gpu-health`: Name indicating the source is Modal's observability system. -* `LEVEL`: Represents the severity level of the log message. -* `GPU_UUID`: A unique identifier for the GPU device associated with the event, if any. -* `EVENT_TYPE`: The type of event source. Modal monitors for multiple types of errors, - including Xid, SXid, and uncorrectable ECC. See below for more details. -* `MSG`: The message component is either the original message taken from the event source, or a description provided by Modal of the problem. - -## Level - -The severity level may be `CRITICAL` or `WARN`. Modal automatically responds to `CRITICAL` level events by draining the underlying Worker and migrating customer containers. -`WARN` level logs may be benign or indication of an application or library bug. No automatic action is taken by our system for warnings. - -## Handling Application-level health issues - -As noted above, Modal will automatically respond to critical GPU events, but warning level events can still -be associated with application exceptions. Applications should catch exceptions caused by GPU-related faults -and call `modal.experimental.stop_fetching_inputs()`: - - - -```python -import modal.experimental -... - -@app.function(gpu="H100") -def demo(): - try: - ... # code which may hit GPU fault (e.g. illegal memory access) - except RuntimeError: - modal.experimental.stop_fetching_inputs() - return -``` - -## Xid & SXid - -The Xid message is an error report from the NVIDIA driver. The SXid, or "Switch Xid" is a report for the NVSwitch component used in GPU-to-GPU communication, and is thus only relevant in multi-GPU containers. - -A classic critical Xid error is the 'fell off the bus' report, code 79. The `gpu-health` event log looks like this: - -``` -[gpu-health] [CRITICAL] GPU-1234: XID: NVRM: Xid (PCI:0000:c6:00): 79, pid=1101234, name=nvc:[driver], GPU has fallen off the bus. -``` - -There are over 100 Xid codes and they are of highly varying frequency, severity, and specificity. -[NVIDIA's official documentation](https://docs.nvidia.com/deploy/xid-errors/index.html) provides limited information, so -we maintain our own tabular information below. - - diff --git a/.agents/skills/modal/references/guide/gpu-metrics.md b/.agents/skills/modal/references/guide/gpu-metrics.md deleted file mode 100644 index 4b40e537b0ba19b1c752d7011a8dcb47fb10e2ef..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/gpu-metrics.md +++ /dev/null @@ -1,10 +0,0 @@ -# GPU Metrics - -Modal exposes a number of GPU metrics that help monitor the health and utilization of the GPUs you're using. - -* **GPU utilization %** is the percentage of time that the GPU was executing at least one CUDA kernel. This is the same metric reported as utilization by [`nvidia-smi`](/gpu-glossary/host-software/nvidia-smi). GPU utilization is helpful for determining the amount of time GPU work is blocked on CPU work, like PyTorch compute graph construction or input processing. However, it is far from indicating what fraction of the GPU's computing firepower (FLOPS or memory throughput, [CUDA Cores](/gpu-glossary/device-hardware/cuda-core), [SMs](/gpu-glossary/device-hardware/streaming-multiprocessor)) is being used. See [this blog post](https://arthurchiao.art/blog/understanding-gpu-performance) for details. -* **GPU power utilization %** is the percentage of the maximum power draw that the device is currently drawing. When aggregating across containers, we also report **Total GPU power usage** in Watts. Because high-performance GPUs are [fundamentally limited by power draw](https://www.thonking.ai/p/strangely-matrix-multiplications), both for computation and memory access, the power usage can be used as a proxy of how much work the GPU is doing. A fully-saturated GPU should draw at or near its entire power budget (which can also be found by running `nvidia-smi`). -* **GPU temperature** is the temperature measured on the die of the GPU. Like power draw, which is the source of the thermal energy, the ability to efflux heat is a fundamental limit on GPU performance: continuing to draw full power without removing the waste heat would damage the system. At the highest temperatures readily observed in proper GPU deployments (i.e. mid-70s Celsius for an H100), increased error correction from thermal noise can already reduce performance. Generally, power utilization is a better proxy for performance, but we report temperature for completeness. -* **GPU memory used** is the amount of memory allocated on the GPU, in bytes. - -In general, these metrics are useful signals or correlates of performance, but can't be used to directly debug performance issues. Instead, we (and [the manufacturers!](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/#assess-parallelize-optimize-deploy)) recommend tracing and profiling workloads. See [this example](/docs/examples/torch_profiling) of profiling PyTorch applications on Modal. diff --git a/.agents/skills/modal/references/guide/gpu.md b/.agents/skills/modal/references/guide/gpu.md deleted file mode 100644 index 0d66112bbc633a833052c91c3ee615e362a13110..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/gpu.md +++ /dev/null @@ -1,194 +0,0 @@ -# GPU acceleration - -Modal makes it easy to run your code on [GPUs](/gpu-glossary/readme). - -## Quickstart - -Here's a simple example of a Function running on an A100 in Modal: - -```python -import modal - -image = modal.Image.debian_slim().pip_install("torch", "numpy") -app = modal.App(image=image) - - -@app.function(gpu="A100") -def run(): - import torch - - assert torch.cuda.is_available() -``` - -## Specifying GPU type - -You can pick a specific GPU type for your Function via the `gpu` argument. -Modal supports the following values for this parameter: - -* `T4` -* `L4` -* `A10` -* `L40S` -* `A100` -* `A100-40GB` -* `A100-80GB` -* `RTX-PRO-6000` -* `H100`/`H100!` -* `H200` -* `B200`/`B200+` - -For instance, to use a B200, you can use `@app.function(gpu="B200")`. - -Refer to our [pricing page](/pricing) for the latest pricing on each GPU type. - -## Specifying GPU count - -You can specify more than 1 GPU per container by appending `:n` to the GPU -argument. For instance, to run a Function with eight H100s: - -```python - -@app.function(gpu="H100:8") -def run_llama_405b_fp8(): - ... -``` - -Currently B200, H200, H100, A100, L4, T4 and L40S instances support up to 8 GPUs (up to 1,536 GB GPU RAM), -and A10 instances support up to 4 GPUs (up to 96 GB GPU RAM). Note that requesting -more than 2 GPUs per container will usually result in larger wait times. These -GPUs are always attached to the same physical machine. - -## Picking a GPU - -For running, rather than training, neural networks, we recommend starting off -with the [L40S](https://resources.nvidia.com/en-us-l40s/l40s-datasheet-28413), -which offers an excellent trade-off of cost and performance and 48 GB of GPU -RAM for storing model weights and activations. - -For more on how to pick a GPU for use with neural networks like LLaMA or Stable -Diffusion, and for tips on how to make that GPU go brrr, check out -[Tim Dettemers' blog post](https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/) -or the -[Full Stack Deep Learning page on Cloud GPUs](https://fullstackdeeplearning.com/cloud-gpus/). - -## B200 GPUs - -Modal's most powerful GPUs are the [B200s](https://www.nvidia.com/en-us/data-center/dgx-b200/), -NVIDIA's flagship data center chip for the Blackwell [architecture](/gpu-glossary/device-hardware/streaming-multiprocessor-architecture). - -To request a B200, set the `gpu` argument to `"B200"` - -```python -@app.function(gpu="B200:8") -def run_deepseek(): - ... -``` - -Check out [this example](/docs/examples/llm_inference) to see how you can use B200s to max out vLLM serving performance for LLaMA 3.1-8B. - -Before you jump for the most powerful (and so most expensive) GPU, make sure you -understand where the bottlenecks are in your computations. For example, running -language models with small batch sizes (e.g. one prompt at a time) results in a -[bottleneck on memory, not arithmetic](https://kipp.ly/transformer-inference-arithmetic/). -Since arithmetic throughput has risen faster than memory throughput in recent -hardware generations, speedups for memory-bound GPU jobs are not as extreme and -may not be worth the extra cost. - -### Opt-in upgrade to B300 - -Use `gpu="B200+"` to allow Modal to run requests on either B200 or B300 GPUs. B200+ is billed as B200, regardless of which GPU is used. Use this option only if your code is compatible with both types of GPUs. B300 requires CUDA version 13.0+. Use this to have access to a greater capacity pool automatically. - -## H200 and H100 GPUs - -[H200s](https://www.nvidia.com/en-us/data-center/h200/) and [H100s](https://www.nvidia.com/en-us/data-center/h100/) are the previous -generation of top-of-the-line data center chips from NVIDIA, based on the Hopper [architecture](/gpu-glossary/device-hardware/streaming-multiprocessor-architecture). -These GPUs have better software support than do Blackwell GPUs (e.g. popular libraries include pre-compiled kernels for Hopper, but not Blackwell), -and they often get the job done at a competitive cost, so they are a common choice of accelerator, on and off Modal. - -All H100 GPUs on the Modal platform are of the SXM variant, as can be verified by examining the -[power draw](/docs/guide/gpu-metrics) in the dashboard or with `nvidia-smi`. - -### Automatic upgrades to H200s - -Modal may automatically upgrade a `gpu="H100"` request to run on an H200. -This automatic upgrade does *not* change the cost of the GPU. - -Kernels [compatible](/gpu-glossary/device-software/compute-capability) with H200s are also compatible with H100s, -so your code will still run, just faster, so long as it doesn't make strict assumptions about memory capacity. -An H200’s [HBM3e memory](/gpu-glossary/device-hardware/gpu-ram) -has a capacity of 141 GB and a bandwidth of 4.8TB/s, 1.75x larger and 1.4x faster than an NVIDIA H100 with HBM3. - -In cases where an automatic upgrade to H200 would not be helpful (for instance, benchmarking) you can pass -`gpu=H100!` to avoid it. - -## A100 GPUs - -[A100s](https://www.nvidia.com/en-us/data-center/a100/) are based on NVIDIA's Ampere [architecture](/gpu-glossary/device-hardware/streaming-multiprocessor-architecture). -Modal offers two versions of the A100: one with 40 GB of RAM and another with 80 GB of RAM. - -To request an A100 with 40 GB of [GPU memory](/gpu-glossary/device-hardware/gpu-ram), use `gpu="A100"`: - -```python -@app.function(gpu="A100") -def qwen_7b(): - ... -``` - -Modal may automatically upgrade a `gpu="A100"` request to run on an 80 GB A100. -This automatic upgrade does *not* change the cost of the GPU. - -You can specifically request a 40GB A100 with the string `A100-40GB`. -To specifically request an 80 GB A100, use the string `A100-80GB`: - -```python -@app.function(gpu="A100-80GB") -def llama_70b_fp8(): - ... -``` - -## GPU fallbacks - -Modal allows specifying a list of possible GPU types, suitable for Functions that are -compatible with multiple options. Modal respects the ordering of this list and -will try to allocate the most preferred GPU type before falling back to less -preferred ones. - -```python -@app.function(gpu=["H100", "A100-40GB:2"]) -def run_on_80gb(): - ... -``` - -See [this example](/docs/examples/gpu_fallbacks) for more detail. - -## Multi GPU training - -Modal currently supports multi-GPU training on a single node, with multi-node training in private Beta (email us at support@modal.com for access). -Depending on which framework you are using, you may need to use different techniques to train on multiple GPUs. - -If the framework re-executes the entrypoint of the Python process (like [PyTorch Lightning](https://lightning.ai/docs/pytorch/stable/index.html)) you need to either set the strategy to `ddp_spawn` or `ddp_notebook` if you wish to invoke the training directly. Another option is to run the training script as a subprocess instead. - -```python -@app.function(gpu="A100:2") -def run(): - import subprocess - import sys - subprocess.run( - ["python", "train.py"], - stdout=sys.stdout, stderr=sys.stderr, - check=True, - ) -``` - -## Examples and more resources - -For more information about GPUs in general, check out our [GPU Glossary](/gpu-glossary/readme). - -Or take a look some examples of Modal apps using GPUs: - -* [Fine-tune a character LoRA for your pet](/docs/examples/diffusers_lora_finetune) -* [Fast LLM inference on big GPUs](/docs/examples/llm_inference) -* [Stable Diffusion with a CLI, API, and web UI](/docs/examples/text_to_image) -* [Rendering Blender videos](/docs/examples/blender_video) - - diff --git a/.agents/skills/modal/references/guide/high-performance-llm-inference.md b/.agents/skills/modal/references/guide/high-performance-llm-inference.md deleted file mode 100644 index a7d5942c53aca7d60e1eea1e9790e7c319554819..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/high-performance-llm-inference.md +++ /dev/null @@ -1,316 +0,0 @@ -# High-performance LLM inference - -This high-level guide documents the key techniques used to achieve high performance -when running LLM inference on Modal. - -Open weights models and open source inference engines have -closed much of the gap with proprietary models and proprietary engines -and continue to improve as they attract work from a broad community. -It is now and will increasingly be economical to run many generative AI applications in-house, -rather than relying on external providers. - -Achieving competitive performance and cost is not instantaneous, however. -It requires some thought and tuning. -And LLM inference is in many ways quite different to the web serving and database workloads -that engineers are used to deploying and optimizing. - -This guide collects techniques we have seen work in production inference deployments. -We include code samples so that you can try high-performance LLM inference for yourself. - -We split the guide by the key performance criterion that matters for the workload: - -* **[throughput](#achieving-high-throughput-llm-inference-tps)**, - for large "jobs" made of many parallel requests that are only finished when they all finish, -* **[latency](#minimizing-llm-inference-latency-ttfttpotttlt)**, - for serving each individual request as fast as possible, usually on human-interactive timescales, -* **[cold start time](#high-performance-llm-inference-for-bursty-workloads-cold-start-time)**, - for bursty workloads that mix latency- and throughput-sensitive components. - -This high-level guide and the attendant code samples are intended to kick-start -your own process of inference deployment and performance optimization. -You can find [baseline benchmarks](/llm-almanac/advisor) -and [benchmarking recommendations](/llm-almanac/how-to-benchmark) -in our [LLM Engineer's Almanac](/llm-almanac/workloads). - -If you just want to get started running a basic LLM server on Modal, see -[this example](https://modal.com/docs/examples/llm_inference). -If you just want to dive into code, see -[this example for high throughput](https://modal.com/docs/examples/vllm_throughput), -[this example for low latency](https://modal.com/docs/examples/sglang_low_latency), -and [this example for low cold start time](https://modal.com/docs/examples/sglang_snapshot). - -## Achieving high throughput LLM inference (TPS) - -The quintessential "high throughput" LLM inference workload is a database backfill: -on a trigger, a large number (100s or more) of rows need to be processed, -e.g. to produce a sentiment score as part of an analytics pipeline -or to produce a generation that will be scored as part of offline evals. -No person or system is waiting on the result from any particular row. - -Performance is defined by *throughput*, the rate at which tasks are completed, -which translates to end-to-end latency for the entire job. -For most deployments, this in turn directly determines cost. -It is measured in tokens per second (TPS). - -Many, but not all, high throughput LLM inference applications have large contexts and small outputs, -which means they are dominated by prefill/prompt processing time, rather than decode/token generation time. -Combined with batching that increases -[arithmetic intensity](https://modal.com/gpu-glossary/perf/arithmetic-intensity), -throughput-oriented LLM inference jobs are generally -[compute-bound](https://modal.com/gpu-glossary/perf/compute-bound). - -In general, high throughput is easier to achieve than low latency. -GPUs are inherently [designed for maximum throughput](https://modal.com/gpu-glossary/perf/latency-hiding). -Additionally, LLM training is a throughput-sensitive workload, so good kernels -are typically made available open source earlier. - -For instance, the [Flash Attention 4 kernel](/blog/reverse-engineer-flash-attention-4) -that extends the Flash Attention kernel series to [Blackwell GPUs](https://modal.com/blog/introducing-b200-h200) -is, at time of writing months after its initial release, -primarily suitable for throughput-sensitive applications -- but watch this space! - -For related reasons, we don't recommend using 4bit floating point (FP4) for these jobs. -FP4 is only supported in [Blackwell or later GPUs](https://modal.com/gpu-glossary/device-software/compute-capability). -Instead, we recommend the more mature 8bit floating point (FP8), -supported in Hopper or later GPUs (one generation back). - -On Modal, the [rates](/pricing) for 16bit FLOP/$ are roughly the same across -A100s, H100s, and B200s -- newer GPUs run faster but cost more to match. -So peak throughput per *dollar* per replica is roughly the same, -even though throughput per *second* per replica is lower. - -But older GPUs running at lower rates offer a few advantages: - -* any time spent [underutilizing the GPUs](/blog/gpu-utilization-guide) is less expensive -* GPUs a generation or two back are generally available in larger quantities from hyperscalers - -Throughput-oriented jobs don't necessarily benefit from scaling up each replica to more GPUs. -The aggregate throughput is the same as more replicas with fewer GPUs, -but fewer GPUs means reduced communication overhead and -reduced complexity, especially for single GPU-per-replica deployments. -Importantly, you must be able to fit a large enough batch of sequences -into the [GPU RAM](https://modal.com/gpu-glossary/device-hardware/gpu-ram) -that you are compute-bound, or else efficiency will decrease. - -We recommend the [vLLM](https://vllm.ai/) inference server for this use case. -It is better able to schedule a mix of prefill and decode work, -which leads to higher throughput. - -### High throughput LLM inference on Modal - -The lack of latency constraints opens up a large number -of architectural choices for high throughput LLM inference. - -For instance, values can be retrieved from an external datastore -or a [Modal Volume](/docs/guide/volumes) -based on identifiers or other information in the datastore. -This is particularly useful for -[cronjob deployments on Modal](/docs/guide/cron). -Results can then be placed back in that datastore. - -Modal provides primitives for building a -[job queue](/docs/guide/job-queue) -that can scale to millions of pending inputs -and jobs that last up to a week. -In this case, the underlying LLM inference is provided by a -[Modal Cls](/docs/guide/lifecycle-functions) -invoked via -[`.spawn`](/docs/guide/job-queue). -Each call gets a string -[`modal.FunctionCall` identifier](/docs/reference/modal.FunctionCall) -that can be used to query the result for up to a week. - -The primary scaling limit from Modal in this case is the rate at which these calls can be queued. -If the inference system can complete more than 400 tasks per second, -we recommend batching multiple tasks into a single Function input until peak throughput -in tasks per second is serviced by 400 inputs per second. - -See [this code sample](https://modal.com/docs/examples/vllm_throughput) -for a system that implements these recommendatons and -achieves maximal per-replica throughput. - -## Minimizing LLM inference latency (TTFT/TPOT/TTLT) - -The quintessential "low latency" LLM inference workload is a chatbot: -each request represents a waiting user, and users operate at the scale of a few hundred milliseconds. -Generating a token of usefully intelligent text often also takes on the order of milliseconds, -and users want many tokens in responses, so latency budgets are tight. - -Performance is defined by *latency*, the time a given task spends waiting. -It is measured in time-to-first-token (TTFT) and time-per-output-token (TPOT) -or in time-to-last-token (TTLT), -depending on to what degree the application supports streaming responses. -For streaming applications, like most chatbots, TTFT matters most. - -To whatever degree the application does support streaming, it is strongly recommended -to improve perceived latency by users. -Contemporary Transformer language models are sequential and so generate their responses -serially, leading to long gaps between the creation of the first token in a response and the last. - -These long decode or token generation phases demand quite different performance -from hardware than long prefills do. -They are typically [memory-bound](https://modal.com/gpu-glossary/perf/memory-bound) -and so benefit from techniques that reduce the amount of memory loaded per token into the -[Streaming Multiprocessors](https://modal.com/gpu-glossary/device-hardware/streaming-multiprocessor) -or increase the amount of available -[memory bandwidth](https://modal.com/gpu-glossary/perf/memory-bandwidth). - -Several techniques can reduce the amount of memory loaded per token: - -* smaller and more aggressively [quantized](https://quant.exposed) models require less memory -* [speculative decoding](https://huggingface.co/docs/text-generation-inference/en/conceptual/speculation) - generates multiple tokens at once via draft models - -For memory-bound workloads, quantizing a model to a format not natively supported by the hardware -can still sometimes lead to gains. -The reduced demand on memory bandwidth cuts memory latency and there is generally sufficient unused -[arithmetic bandwidth](https://modal.com/gpu-glossary/perf/arithmetic-bandwidth) -to perform extra numerical conversions. - -There are a wide variety of speculative decoding techniques, ranging from simple n-gram speculation -to stacks of models drafting tokens for each other in sequence. -We have generally found that the [EAGLE-3 method](https://arxiv.org/abs/2503.01840) -provides the best performance improvement for the least overhead -- -computationally and operationally. -Generic draft models are available on Hugging Face, -but we have also seen major improvements from custom draft models -trained on sample production data using tools like -[SpecForge](https://lmsys.org/blog/2025-07-25-spec-forge/). - -Additionally, using multiple GPUs to generate a single token increases the aggregate memory bandwidth, -at the cost of some extra communication. -Critically, multiple accelerators need to be used to load model weights in parallel, -or latency will not be reduced. -That means the usual form of parallelism used to reduce latency is *tensor parallelism*, -which splits up individual matrix multiplications across GPUs, -rather than *pipeline parallelism*, -which splits the entire model across GPUs. - -There are few models below 70B parameters that work well in 4bit floating point -(with exceptions like [GPT-OSS](https://modal.com/docs/examples/gpt_oss_inference)). -Additionally, at time of writing in early 2026, there are not high-quality open source -Blackwell-optimized kernels for latency-sensitive LLM inference. -Therefore, we generally recommend FP8-quantized models on H100s or H200s. - -Finally, we recommend the [SGLang](https://docs.sglang.io/) -inference engine for these workloads. -SGLang generally exhibits lower host overhead -- -time when the GPU idles waiting on the CPU -- -for decode-heavy workloads, especially for smaller models. -You can read more about host overhead and its solutions in -[this blog post](/blog/host-overhead-inference-efficiency). - -### Low latency LLM inference on Modal - -For latency budgets in the few hundreds of milliseconds, -network latencies and proxy/load-balancing overhead matter -- -communicating with clients across an ocean takes dozens of milliseconds, -due to speed-of-light constraints. - -Modal offers ultra-low-latency, regionalized web server deployment with -`modal.experimental.http_server` -to reduce network overhead below 100ms. -Please contact us if you are interested in running production LLM inference -with the experimental `http_server`. - -You can find an example demonstrating all the pieces of -low latency LLM inference on Modal together -[here](https://modal.com/docs/examples/sglang_low_latency). - -## High performance LLM inference for bursty workloads (cold start time) - -The final major class of workloads sits between pure throughput and pure latency. -The quintessential application is a "workflow" where LLM inference is one workflow step, -and the workflow is sometimes run interactively by a human and at other times run asynchronously in bulk. - -For these applications, the primary concern is handling the high -[peak-to-average load ratio](https://brooker.co.za/blog/2023/03/23/economics.html). -For instance, a pipeline might serve zero requests per second most of the time, -then ten for a bit, then one hundred, then back down to zero. -Statically provisioning enough resources to handle one hundred requests is clearly wasteful, -but spinning up new resources on demand incurs latency. - -The key performance criterion, then, is -[*cold start time*](/docs/guide/cold-start): -how long does it take for a new replica to spin up and start handling requests. -On a typical cloud deployment, that includes instance requisition, machine boot, and container setup. -We've written about the resource allocation challenges [here](/blog/gpu-utilization-guide). - -Approaches based on requesting resources from clouds directly take minutes to tens of minutes. -Modal has been designed from the kernel up to provide sub-second latencies -all the way through to container start. -From there, the primary performance concern is speeding up server startup. - -* **Use small models and quantize aggressively**. - Models can be loaded from a [Modal Volume](/docs/guide/volumes) - at a rate of 1-2 GB/s. That means you're incurring nearly a second of cold start latency - per gigabyte of model weights. More exotic compression formats, like integer quantization - or even ternary quantization, are particularly helpful here, even when they don't improve - latency during inference. - -* **Skip compilation steps**. - Optimizations like CUDA Graph capture, JIT-compiled kernels, and Torch compilation - are great for improving latency and throughput but they are generally quite tricky to cache - and cache hits sometimes take nearly as long as cache misses. - That often means a large latency penalty from compilation on each boot, - and latencies can easily range into the tens of seconds or even tens of minutes. - -* **Restore from snapshots**. - In some cases, startup-time work like JIT compilation is unavoidable. - For these workloads, Modal provides - [Memory Snapshots](/docs/guide/memory-snapshots): - the full in-memory state of a container just before it is ready to - handle requests is serialized to disk and future container starts - only need to deserialize this back into memory. - Modal includes support for - [GPU Memory Snapshots](/blog/gpu-mem-snapshots) - so that GPU-accelerated LLM inference servers can be snapshot as well. - Memory snapshotting is powerful - ([we've observed 10x reductions in cold start time](/blog/gpu-mem-snapshots)), - but it requires some code modification, described below. - -Which optimizations discussed above apply -depend on the balance of the workload between low latency and high throughput. -But a few general statements can be made. -For instance, speculative decoding is generally a bad choice, -since it harms performance in the high throughput regime. - -Relatedly, we don't have a particular recommendation between vLLM and SGLang here. -Besides the points made above about host overhead latency vs bulk throughput, -the primary difference we have seen is that vLLM is a bit faster to market with new models -and new features, but SGLang is a bit easier to hack on and extend. - -### Serving bursty LLM inference workloads on Modal - -Modal's rapid autoscaling infrastructure, -from [the custom container runtime and filesystem](/blog/jono-containers-talk), -to [memory snapshot support](/blog/gpu-mem-snapshots), -is particularly well-suited -to bursty LLM inference workloads. - -These workloads can either be served by vanilla -[Functions](/docs/guide/apps) -invoked via remote Python calls or as -[Web Functions](/docs/guide/webhooks) -invoked via HTTP. -Web Functions are better for integrating with a variety -of producers and consumers. -The tradeoff of lower overhead for increased complexity -with `modal.experimental.http_server` is generally not worth it. - -The [`@modal.concurrent` decorator](/docs/guide/concurrent-inputs) -supports setting both a limit (`max_inputs`) -and a target (`target_inputs`). -Set the limit higher than the target to absorb load increases into -existing capacity (typically at the expense of longer latency). -Make sure that the inference server is configured to handle batches as large as `max_inputs` -without internal queueing! - -Almost all GPU programs can be snapshot, but most GPU programs -require some code changes to be snapshot. -For instance, both the vLLM and SGLang inference servers require -manual offloading of weights/KV cache to CPU memory before snapshotting. - -For details, see our full sample code for running bursty workloads on Modal -with vLLM [here](https://modal.com/docs/examples/vllm_snapshot) -and with SGLang [here](https://modal.com/docs/examples/sglang_snapshot). diff --git a/.agents/skills/modal/references/guide/images.md b/.agents/skills/modal/references/guide/images.md deleted file mode 100644 index ce740bc03d136ca41343d6e481331d8a7e81b2fc..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/images.md +++ /dev/null @@ -1,368 +0,0 @@ -# Images - -This guide walks you through how to define a Modal Image, the environment your Modal code runs in. - -The typical flow for defining an Image in Modal is -[method chaining](https://jugad2.blogspot.com/2016/02/examples-of-method-chaining-in-python.html) -starting from a base Image, like this: - -```python -image = ( - modal.Image.debian_slim(python_version="3.13") - .apt_install("git") - .uv_pip_install("torch<3") - .env({"HALT_AND_CATCH_FIRE": "0"}) - .run_commands("git clone https://github.com/modal-labs/agi && echo 'ready to go!'") -) -``` - -If you have your own container image definitions, like a Dockerfile or a registry link, you can use those too! -See [this guide](/docs/guide/existing-images). - -This page is a high-level guide to using Modal Images. -For reference documentation on the `modal.Image` object, see -[this page](/docs/reference/modal.Image). - -## What are Images? - -Your code on Modal runs in *containers*. Containers are like light-weight -virtual machines -- container engines use -[operating system tricks](https://earthly.dev/blog/chroot/) to isolate programs -from each other ("containing" them), making them work as though they were -running on their own hardware with their own filesystem. This makes execution -environments more reproducible, for example by preventing accidental -cross-contamination of environments on the same machine. For added security, -Modal runs containers using the sandboxed -[gVisor container runtime](https://cloud.google.com/blog/products/identity-security/open-sourcing-gvisor-a-sandboxed-container-runtime). - -Containers are started up from a stored "snapshot" of their filesystem state -called an *image*. Producing the image for a container is called *building* the -image. - -By default, Modal Functions and Sandboxes run in a -[Debian Linux](https://en.wikipedia.org/wiki/Debian) container with a basic -Python installation of the same minor version `v3.x` as your local Python -interpreter. - -To make your Apps and Functions useful, you will probably need some third party system packages -or Python libraries. Modal provides a number of options to customize your container images at -different levels of abstraction and granularity, from high-level convenience -methods like `pip_install` through wrappers of core container image build -features like `RUN` and `ENV`. We'll cover each of these in this guide, -along with tips and tricks for building Images effectively when using each tool. - -## Add Python packages - -The simplest and most common Image modification is to add a third party -Python package, like [`pandas`](https://pandas.pydata.org/). - -You can add Python packages to the environment by passing all the packages you -need to the [`Image.uv_pip_install`](/docs/reference/modal.Image#uv_pip_install) method, -which installs packages with [`uv`](https://docs.astral.sh/uv/): - -```python -import modal - -datascience_image = ( - modal.Image.debian_slim() - .uv_pip_install("pandas==2.2.0", "numpy") -) - - -@app.function(image=datascience_image) -def my_function(): - import pandas as pd - import numpy as np - - df = pd.DataFrame() - ... -``` - -You can include -[Python dependency version specifiers](https://peps.python.org/pep-0508/), -like `"torch<3"`, in the arguments. But we recommend pinning dependencies -tightly, like `"torch==2.8.0"`, to improve the reproducibility and robustness -of your builds. - -If you run into any issues with -[`Image.uv_pip_install`](/docs/reference/modal.Image#uv_pip_install), then -you can fallback to [`Image.pip_install`](/docs/reference/modal.Image#pip_install) which -uses standard [`pip`](https://pip.pypa.io/en/stable/user_guide/): - -```python -datascience_image = ( - modal.Image.debian_slim(python_version="3.13") - .pip_install("pandas==2.2.0", "numpy") -) -``` - -Note that because you can define a different environment for each and every -function if you so choose, you don't need to worry about virtual -environment management. Containers make for much better separation of concerns! - -If you want to run a specific version of Python remotely rather than just -matching the one you're running locally, provide the `python_version` as a -string when constructing the base image, like we did above. - -## Add local files with `add_local_dir` and `add_local_file` - -Sometimes your containers need a dependency that's not available on the Internet, -like configuration files or code on your laptop. - -To forward files from your local system use the -`image.add_local_dir` and `image.add_local_file` Image methods. - -```python -image = modal.Image.debian_slim().add_local_dir("/user/erikbern/.aws", remote_path="/root/.aws") -``` - -By default, these files are added to your container as it starts up rather than introducing -a new Image layer. This means that the redeployment after making changes is really quick, but -also means you can't run additional build steps after. You can specify a `copy=True` argument -to the `add_local_` methods to instead force the files to be included in the built Image. - -### Add local Python code with `add_local_python_source` - -You can add Python code that's importable locally to your container -by providing the module name to -[`Image.add_local_python_source`](/docs/reference/modal.Image#add_local_python_source). - -```python -image_with_module = modal.Image.debian_slim().add_local_python_source("local_module") - -@app.function(image=image_with_module) -def f(): - import local_module - - local_module.do_stuff() -``` - -The difference from `add_local_dir` is that `add_local_python_source` takes module names as arguments -instead of a file system path and looks up the local package's or module's location via Python's importing -mechanism. The files are then added to directories that make them importable in containers in the -same way as they are locally. - -This is intended for pure Python auxiliary modules that are part of your project and that your code imports. -Third party packages should be installed via -[`Image.uv_pip_install`](/docs/reference/modal.Image#uv_pip_install) or similar. - -### What if I have different Python packages locally and remotely? - -You might want to use packages inside your Modal code that you don't have on -your local computer. In the example above, we build a container that uses -`pandas`. But if we don't have `pandas` locally, on the computer building the -Modal App, we can't put `import pandas` at the top of the script, since it would -cause an `ImportError`. - -The easiest solution to this is to put `import pandas` in the function body -instead, as you can see above. This means that `pandas` is only imported when -running inside the remote Modal container, which has `pandas` installed. - -Be careful about what you return from Modal Functions that have different -packages installed than the ones you have locally! Modal Functions return Python -objects, like `pandas.DataFrame`s, and if your local machine doesn't have -`pandas` installed, it won't be able to handle a `pandas` object (the error -message you see will mention -[serialization](https://hazelcast.com/glossary/serialization/)/[deserialization](https://hazelcast.com/glossary/deserialization/)). - -If you have a lot of Functions and a lot of Python packages, you might want to -keep the imports in the global scope so that every function can use the same -imports. In that case, you can use the -[`Image.imports`](/docs/reference/modal.Image#imports) context manager: - -```python -pandas_image = modal.Image.debian_slim().pip_install("pandas", "numpy") - - -with pandas_image.imports(): - import pandas as pd - import numpy as np - - -@app.function(image=pandas_image) -def my_function(): - df = pd.DataFrame() - ... -``` - -Because these imports happen before a new container processes its first input, -you can combine this context manager with [Memory Snapshots](/docs/guide/memory-snapshots) -to improve [cold start performance](/docs/guide/cold-start#share-initialization-work-across-cold-starts-with-memory-snapshots) -for Functions that frequently scale up. - -## Install system packages with `.apt_install` - -You can install Linux packages with the [`apt` package manager](https://www.debian.org/doc/manuals/apt-guide/index.en.html) -using [`Image.apt_install`](/docs/reference/modal.Image#apt_install): - -```python -image = modal.Image.debian_slim().apt_install("git", "curl") -``` - -## Set environment variables with `.env` - -You can change the environment variables that your code sees -(in, e.g., [`os.environ`](https://docs.python.org/3/library/os.html#os.environ)) -by passing a dictionary to [`Image.env`](/docs/reference/modal.Image#env): - -```python -image = modal.Image.debian_slim().env({"PORT": "6443"}) -``` - -Environment variable names and values must be strings. - -## Run shell commands with `.run_commands` - -You can supply shell commands that should be executed when building the -Image to [`Image.run_commands`](/docs/reference/modal.Image#run_commands): - -```python -image_with_repo = ( - modal.Image.debian_slim().apt_install("git").run_commands( - "git clone https://github.com/modal-labs/gpu-glossary" - ) -) -``` - -## Run a Python function during your build with `.run_function` - -You can run Python code as a build step using the -[`Image.run_function`](/docs/reference/modal.Image#run_function) method. - -For example, you can use this to download model parameters from Hugging Face into -your Image: - -```python -import os - -def download_models() -> None: - import diffusers - - model_name = "segmind/small-sd" - pipe = diffusers.StableDiffusionPipeline.from_pretrained( - model_name, use_auth_token=os.environ["HF_TOKEN"] - ) - -hf_cache = modal.Volume.from_name("hf-cache") - -image = ( - modal.Image.debian_slim() - .pip_install("diffusers[torch]", "transformers", "ftfy", "accelerate") - .run_function( - download_models, - secrets=[modal.Secret.from_name("huggingface-secret")], - volumes={"/root/.cache/huggingface": hf_cache}, - ) -) -``` - -For details on storing model weights on Modal, see -[this guide](/docs/guide/model-weights). - -Essentially, this is equivalent to running a Modal Function and snapshotting the -resulting filesystem as a new Image. Any kwargs accepted by [`@app.function`](/docs/reference/modal.App#function) -([`Volume`s](/docs/guide/volumes), [`Secret`s](/docs/guide/secrets), specifications of -resources like [GPUs](/docs/guide/gpu)) can be supplied here. - -Whenever you change other features of your Image, like the base Image or the -version of a Python package, the Image will automatically be rebuilt the next -time it is used. This is a bit more complicated when changing the contents of -functions. See the -[reference documentation](/docs/reference/modal.Image#run_function) for details. - -## Attach GPUs during setup - -If a step in the setup of your Image should be run on an instance with -a GPU (e.g., so that a package can query the GPU to set compilation flags), pass the -desired GPU type when defining that step: - -```python -image = ( - modal.Image.debian_slim() - .pip_install("bitsandbytes", gpu="H100") -) -``` - -## Use `mamba` instead of `pip` with `micromamba_install` - -`pip` installs Python packages, but some Python workloads require the -coordinated installation of system packages as well. The `mamba` package manager -can install both. Modal provides a pre-built -[Micromamba](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html) -base image that makes it easy to work with `micromamba`: - -```python -app = modal.App("bayes-pgm") - -numpyro_pymc_image = ( - modal.Image.micromamba() - .micromamba_install("pymc==5.10.4", "numpyro==0.13.2", channels=["conda-forge"]) -) - - -@app.function(image=numpyro_pymc_image) -def sample(): - import pymc as pm - import numpyro as np - - print(f"Running on PyMC v{pm.__version__} with JAX/numpyro v{np.__version__} backend") - ... -``` - -## Image caching and rebuilds - -Modal uses the definition of an Image to determine whether it needs to be -rebuilt. If the definition hasn't changed since the last time you ran or -deployed your App, the previous version will be pulled from the cache. - -Images are cached per layer (i.e., per `Image` method call), and breaking -the cache on a single layer will cause cascading rebuilds for all subsequent -layers. You can shorten iteration cycles by defining frequently-changing -layers last so that the cached version of all other layers can be used. - -In some cases, you may want to force an Image to rebuild, even if the -definition hasn't changed. You can do this by adding the `force_build=True` -argument to any of the Image building methods. - -```python -image = ( - modal.Image.debian_slim() - .apt_install("git") - .pip_install("slack-sdk", force_build=True) - .run_commands("echo hi") -) -``` - -As in other cases where a layer's definition changes, both the `pip_install` and -`run_commands` layers will rebuild, but the `apt_install` will not. Remember to -remove `force_build=True` after you've rebuilt the Image, or it will -rebuild every time you run your code. - -Alternatively, you can set the `MODAL_FORCE_BUILD` environment variable (e.g. -`MODAL_FORCE_BUILD=1 modal run ...`) to rebuild all images attached to your App. -But note that when you rebuild a base layer, the cache will be invalidated for *all* -Images that depend on it, and they will rebuild the next time you run or deploy -any App that uses that base. If you're debugging an issue with your Image, a better -option might be using `MODAL_IGNORE_CACHE=1`. This will rebuild the Image from the -top without breaking the Image cache or affecting subsequent builds. - -## Image builder updates - -Because changes to base images will cause cascading rebuilds, Modal is -conservative about updating the base definitions that we provide. But many -things are baked into these definitions, like the specific versions of the Image -OS, the included Python, and the Modal client dependencies. - -We provide a separate mechanism for keeping base images up-to-date without -causing unpredictable rebuilds: the "Image Builder Version". This is a workspace -level-configuration that will be used for every Image built in your workspace. -We release a new Image Builder Version every few months but allow you to update -your workspace's configuration when convenient. After updating, your next -deployment will take longer, because your Images will rebuild. You may also -encounter problems, especially if your Image definition does not pin the version -of the third-party libraries that it installs (as your new Image will get the -latest version of these libraries, which may contain breaking changes). - -You can set the Image Builder Version for your workspace by going to your -[workspace settings](/settings/image-builder-version). This page also documents the -important updates in each version. diff --git a/.agents/skills/modal/references/guide/index.md b/.agents/skills/modal/references/guide/index.md deleted file mode 100644 index 39a39a84938cd3044b5a8b488c53c8af9663fc9f..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/index.md +++ /dev/null @@ -1,98 +0,0 @@ -# Guide - -- Custom container images - - [Defining Images](./images.md) - - [Using existing container images](./existing-images.md) - - [Named images](./named-images.md) - - [Fast pull from registry](./fast-pull-from-registry.md) -- GPUs and other resources - - [GPU acceleration](./gpu.md) - - [Using CUDA on Modal](./cuda.md) - - [Configuring CPU, memory, and disk](./resources.md) -- Scaling out - - [Scaling out](./scale.md) - - [Input concurrency](./concurrent-inputs.md) - - [Batch processing](./batch-processing.md) - - [Job queues](./job-queue.md) - - [Dynamic batching](./dynamic-batching.md) - - [Multi-node clusters (Beta)](./multi-node-training.md) -- Deployment - - [Apps, Functions, and entrypoints](./apps.md) - - [Managing deployments](./managing-deployments.md) - - [Invoking deployed functions](./trigger-deployed-functions.md) - - [Continuous deployment](./continuous-deployment.md) - - [Running untrusted code in Functions](./restricted-access.md) -- Modal Sandboxes - - [Sandboxes](./sandboxes.md) - - [Running commands](./sandbox-spawn.md) - - [Networking and security](./sandbox-networking.md) - - [File access](./sandbox-files.md) - - [Snapshots](./sandbox-snapshots.md) - - [Pricing and resources](./sandbox-resources.md) - - [Docker in Sandboxes (Alpha)](./docker-in-sandboxes.md) - - [VM Sandboxes (Alpha)](./vm-sandboxes.md) -- Secrets and environment variables - - [Secrets](./secrets.md) - - [Environment variables](./environment_variables.md) -- Web Functions - - [Web Functions](./webhooks.md) - - [Streaming endpoints](./streaming-endpoints.md) - - [Web Function URLs](./webhook-urls.md) - - [Request timeouts](./webhook-timeouts.md) - - [Proxy Auth Tokens](./webhook-proxy-auth.md) -- Networking - - [Tunnels](./tunnels.md) - - [Proxies (Beta)](./proxy-ips.md) - - [Cluster networking](./private-networking.md) -- Data sharing and storage - - [Passing local data](./local-data.md) - - [Volumes](./volumes.md) - - [Storing model weights](./model-weights.md) - - [Cloud bucket mounts](./cloud-bucket-mounts.md) - - [Dicts](./dicts.md) - - [Queues](./queues.md) - - [Dataset ingestion](./dataset-ingestion.md) -- Performance - - [Cold start performance](./cold-start.md) - - [Memory Snapshots](./memory-snapshots.md) - - [High-performance LLM inference](./high-performance-llm-inference.md) -- Reliability and robustness - - [Failures and retries](./retries.md) - - [Preemption](./preemption.md) - - [Timeouts](./timeouts.md) - - [GPU health](./gpu-health.md) - - [Troubleshooting](./troubleshooting.md) -- Security and privacy - - [Security and privacy](./security.md) - - [Customer-supplied encryption keys](./customer-supplied-encryption-keys.md) - - [Audit logs](./audit-logs.md) -- Integrations - - [Using OIDC to authenticate with external services](./oidc-integration.md) - - [Connecting Modal to your Datadog account](./datadog-integration.md) - - [Connecting Modal to your OpenTelemetry provider](./otel-integration.md) - - [Okta SSO](./okta-sso.md) - - [Custom SAML SSO](./saml-sso.md) - - [Slack notifications (Beta)](./slack-notifications.md) -- Workspace & account settings - - [Workspaces](./workspaces.md) - - [Environments](./environments.md) - - [Modal user account setup](./modal-user-account-setup.md) - - [Service users](./service-users.md) - - [Role-Based Access Control (RBAC)](./rbac.md) - - [Billing](./billing.md) -- Other topics - - [Feature maturity](./feature-maturity.md) - - [JavaScript/Go SDKs](./sdk-javascript-go.md) - - [Modal 1.0 migration guide](./modal-1-0-migration.md) - - [File and project structure](./project-structure.md) - - [Developing and debugging](./developing-debugging.md) - - [Developing Modal code with LLMs](./developing-with-llms.md) - - [Jupyter notebooks](./jupyter-notebooks.md) - - [Asynchronous API usage](./async.md) - - [Global variables](./global-variables.md) - - [Region selection](./region-selection.md) - - [Container lifecycle hooks](./lifecycle-functions.md) - - [Parametrized functions](./parametrized-functions.md) - - [Dynamic function configuration](./dynamic-function-config.md) - - [S3 Gateway endpoints](./s3-gateway-endpoints.md) - - [GPU Metrics](./gpu-metrics.md) diff --git a/.agents/skills/modal/references/guide/job-queue.md b/.agents/skills/modal/references/guide/job-queue.md deleted file mode 100644 index 377e6329d574c9c2544150ad8d370fd383a0a895..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/job-queue.md +++ /dev/null @@ -1,160 +0,0 @@ -# Job processing - -Modal can be used as a scalable job queue to handle asynchronous tasks submitted -from a web app or any other Python application. This allows you to offload up to 1 million -long-running or resource-intensive tasks to Modal, while your main application -remains responsive. - -## Creating jobs with .spawn() - -The basic pattern for using Modal as a job queue involves three key steps: - -1. Defining and deploying the job processing function using `modal deploy`. -2. Submitting a job using - [`modal.Function.spawn()`](/docs/reference/modal.Function#spawn) -3. Polling for the job's result using - [`modal.FunctionCall.get()`](/docs/reference/modal.FunctionCall#get) - -Here's a simple example that you can run with `modal run my_job_queue.py`: - -```python -# my_job_queue.py -import modal - -app = modal.App("my-job-queue") - -@app.function() -def process_job(data): - # Perform the job processing here - return {"result": data} - -def submit_job(data): - # Since the `process_job` function is deployed, need to first look it up - process_job = modal.Function.from_name("my-job-queue", "process_job") - call = process_job.spawn(data) - return call.object_id - -def get_job_result(call_id): - function_call = modal.FunctionCall.from_id(call_id) - try: - result = function_call.get(timeout=5) - except modal.exception.OutputExpiredError: - result = {"result": "expired"} - except TimeoutError: - result = {"result": "pending"} - return result - -@app.local_entrypoint() -def main(): - data = "my-data" - - # Submit the job to Modal - call_id = submit_job(data) - print(get_job_result(call_id)) -``` - -In this example: - -* `process_job` is the Modal function that performs the actual job processing. - To deploy the `process_job` function on Modal, run - `modal deploy my_job_queue.py`. -* `submit_job` submits a new job by first looking up the deployed `process_job` - function, then calling `.spawn()` with the job data. It returns the unique ID - of the spawned function call. -* `get_job_result` attempts to retrieve the result of a previously submitted job - using [`FunctionCall.from_id()`](/docs/reference/modal.FunctionCall#from_id) and - [`FunctionCall.get()`](/docs/reference/modal.FunctionCall#get). - [`FunctionCall.get()`](/docs/reference/modal.FunctionCall#get) waits indefinitely - by default. It takes an optional timeout argument that specifies the maximum - number of seconds to wait, which can be set to 0 to poll for an output - immediately. Here, if the job hasn't completed yet, we return a pending - response. -* The results of a `.spawn()` are accessible via `FunctionCall.get()` for up to - 7 days after completion. After this period, we return an expired response. - -[Document OCR Web App](/docs/examples/doc_ocr_webapp) is an example that uses -this pattern. - -## Integration with web frameworks - -You can easily integrate the job queue pattern with web frameworks like FastAPI. -Here's an example, assuming that you have already deployed `process_job` on -Modal with `modal deploy` as above. This example won't work if you haven't -deployed your app yet. - -```python -# my_job_queue_endpoint.py -import modal - -image = modal.Image.debian_slim().pip_install("fastapi[standard]") -app = modal.App("fastapi-modal", image=image) - - -@app.function() -@modal.asgi_app() -@modal.concurrent(max_inputs=20) -def fastapi_app(): - from fastapi import FastAPI - - web_app = FastAPI() - - @web_app.post("/submit") - async def submit_job_endpoint(data): - process_job = modal.Function.from_name("my-job-queue", "process_job") - - call = await process_job.spawn.aio(data) - return {"call_id": call.object_id} - - - @web_app.get("/result/{call_id}") - async def get_job_result_endpoint(call_id: str): - function_call = modal.FunctionCall.from_id(call_id) - try: - result = await function_call.get.aio(timeout=0) - except modal.exception.OutputExpiredError: - return fastapi.responses.JSONResponse(content="", status_code=404) - except TimeoutError: - return fastapi.responses.JSONResponse(content="", status_code=202) - - return result - - return web_app -``` - -In this example: - -* The `/submit` endpoint accepts job data, submits a new job using - `await process_job.spawn.aio()`, and returns the job's ID to the client. -* The `/result/{call_id}` endpoint allows the client to poll for the job's - result using the job ID. If the job hasn't completed yet, it returns a 202 - status code to indicate that the job is still being processed. If the job - has expired, it returns a 404 status code to indicate that the job is not found. - -You can try this app by serving it with `modal serve`: - -```shell -modal serve my_job_queue_endpoint.py -``` - -Then interact with its endpoints with `curl`: - -```shell -# Make a POST request to your app endpoint with. -$ curl -X POST $YOUR_APP_ENDPOINT/submit?data=data -{"call_id":"fc-XXX"} - -# Use the call_id value from above. -$ curl -X GET $YOUR_APP_ENDPOINT/result/fc-XXX -``` - -## Scaling and reliability - -Modal automatically scales the job queue based on the workload, spinning up new -instances as needed to process jobs concurrently. It also provides built-in -reliability features like automatic retries and timeout handling. - -You can customize the behavior of the job queue by configuring the -`@app.function()` decorator with options like -[`retries`](/docs/guide/retries#function-retries), -[`timeout`](/docs/guide/timeouts#timeouts), and -[`max_containers`](/docs/guide/scale#configuring-autoscaling-behavior). diff --git a/.agents/skills/modal/references/guide/jupyter-notebooks.md b/.agents/skills/modal/references/guide/jupyter-notebooks.md deleted file mode 100644 index ea32c314e115bda20307ab8d7b2a13f2bc846f8d..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/jupyter-notebooks.md +++ /dev/null @@ -1,74 +0,0 @@ -# Jupyter notebooks - -This guide page documents integrations between Jupyter notebooks and Modal. - - - -For our hosted notebooks product with real-time collaboration, see [Modal Notebooks](/docs/guide/notebooks). - - - -## Modal inside Jupyter - -You can use the Modal client library in notebook environments like Jupyter! Just -`import modal` and use as normal. You will likely need to use [`app.run`](/docs/guide/apps#ephemeral-apps) to create an ephemeral app to run your functions: - -```python,notest -# Cell 1 - -import modal - -app = modal.App() - -@app.function() -def my_function(x): - ... - -# Cell 2 - -with modal.enable_output(): - with app.run(): - my_function.remote(42) -``` - -### Known issues - -* **Interactive shell and interactive functions are not supported.** - - These can only be run within a live terminal session, so they are not - supported in notebooks. - -* **Local and remote Python versions must match.** - - When defining Modal Functions in a Jupyter notebook, the Function automatically - has `serialized=True` set. This implies that the versions of Python and any third- - party libraries used in your Modal container must match the version you have locally, - so that the function can be deserialized remotely without errors. - -If you encounter issues not documented above, try restarting the notebook kernel, as it may be -in a broken state, which is common in notebook development. - -If the issue persists, contact us [in our Slack](https://modal.com/slack). - -We are working on removing these known issues so that writing Modal applications -in a notebook feels just like developing in regular Python modules and scripts. - -## Jupyter inside Modal - -You can run Jupyter in Modal using the `modal launch` command. For example: - -``` -$ modal launch jupyter --gpu a10g -``` - -That will start a Jupyter instance with an A10G GPU attached. You'll be able to -access the app with via a -[Modal Tunnel URL](https://modal.com/docs/guide/tunnels). Jupyter -will stop running whenever you stop Modal call in your terminal. - -See `--help` for additional options. - -## Further examples - -* [Basic demonstration of running Modal in a notebook](https://github.com/modal-labs/modal-examples/blob/main/11_notebooks/basic.ipynb) -* [Running Jupyter server within a Modal function](https://github.com/modal-labs/modal-examples/blob/main/11_notebooks/jupyter_inside_modal.py) diff --git a/.agents/skills/modal/references/guide/lifecycle-functions.md b/.agents/skills/modal/references/guide/lifecycle-functions.md deleted file mode 100644 index ff62890b7826ecc4bb51de71f310328ff19096cc..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/lifecycle-functions.md +++ /dev/null @@ -1,140 +0,0 @@ -# Container lifecycle hooks - -Since Modal will reuse the same container for multiple inputs, sometimes you -might want to run some code exactly once when the container starts or exits. - -To accomplish this, you need to use Modal's class syntax and the -[`@app.cls`](/docs/reference/modal.App#cls) decorator. Specifically, you'll -need to: - -1. Convert your function to a method by making it a member of a class. -2. Decorate the class with `@app.cls(...)` with same arguments you previously - had for `@app.function(...)`. -3. Instead of the `@app.function` decorator on the original method, use - `@modal.method` or the appropriate decorator for a - [Web Function](#lifecycle-hooks-for-web-functions). -4. Add the correct method "hooks" to your class based on your need: - * `@modal.enter` for one-time initialization (remote) - * `@modal.exit` for one-time cleanup (remote) - -## `@modal.enter` - -The container entry handler is called when a new container is started. This is -useful for doing one-time initialization, such as loading model weights or -importing packages that are only present in that image. - -To use, make your function a member of a class, and apply the `@modal.enter()` -decorator to one or more class methods: - -```python -import modal - -app = modal.App() - -@app.cls(cpu=8) -class Model: - @modal.enter() - def run_this_on_container_startup(self): - import pickle - self.model = pickle.load(open("model.pickle")) - - @modal.method() - def predict(self, x): - return self.model.predict(x) - - -@app.local_entrypoint() -def main(): - Model().predict.remote(x=123) -``` - -When working with an [asynchronous Modal](/docs/guide/async) app, you may use an -async method instead: - -```python -import modal - -app = modal.App() - -@app.cls(memory=1024) -class Processor: - @modal.enter() - async def my_enter_method(self): - self.cache = await load_cache() - - @modal.method() - async def run(self, x): - return await do_some_async_stuff(x, self.cache) - - -@app.local_entrypoint() -async def main(): - await Processor().run.remote(x=123) -``` - -Note: The `@modal.enter()` decorator replaces the earlier `__enter__` syntax, which -has been deprecated. - -## `@modal.exit` - -The container exit handler is called when a container is about to exit. It is -useful for doing one-time cleanup, such as closing a database connection or -saving intermediate results. To use, make your function a member of a class, and -apply the `@modal.exit()` decorator: - -```python -import modal - -app = modal.App() - -@app.cls() -class ETLPipeline: - @modal.enter() - def open_connection(self): - import psycopg2 - self.connection = psycopg2.connect(os.environ["DATABASE_URI"]) - - @modal.method() - def run(self): - # Run some queries - pass - - @modal.exit() - def close_connection(self): - self.connection.close() - - -@app.local_entrypoint() -def main(): - ETLPipeline().run.remote() -``` - -Exit handlers are also called when a container is [preempted](/docs/guide/preemption). -The exit handler is given a grace period of 30 seconds to finish, and it will be -killed if it takes longer than that to complete. - -## Lifecycle hooks for Web Functions - -Modal [Web Functions](/docs/guide/webhooks) can be converted to the class syntax -as well. Instead of `@modal.method`, simply use whichever Web Function -decorator (`@modal.fastapi_endpoint`, `@modal.asgi_app` or `@modal.wsgi_app`) -you were using before. - -```python -from fastapi import Request - -import modal - -image = modal.Image.debian_slim().pip_install("fastapi") -app = modal.App("web-function-cls", image=image) - -@app.cls() -class Model: - @modal.enter() - def run_this_on_container_startup(self): - self.model = pickle.load(open("model.pickle")) - - @modal.fastapi_endpoint() - def predict(self, request: Request): - ... -``` diff --git a/.agents/skills/modal/references/guide/local-data.md b/.agents/skills/modal/references/guide/local-data.md deleted file mode 100644 index 69869a258f497dde28f57b5030c24de564233cec..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/local-data.md +++ /dev/null @@ -1,36 +0,0 @@ -# Passing local data - -If you have a function that needs access to some data not present in your Python -files themselves you have a few options for bundling that data with your Modal -app. - -## Passing function arguments - -The simplest and most straight-forward way is to read the data from your local -script and pass the data to the outermost Modal function call: - -```python -import json - - -@app.function() -def foo(a): - print(sum(a["numbers"])) - - -@app.local_entrypoint() -def main(): - data_structure = json.load(open("blob.json")) - foo.remote(data_structure) -``` - -Any data of reasonable size that is serializable through -[cloudpickle](https://github.com/cloudpipe/cloudpickle) is passable as an -argument to Modal functions. - -Refer to the section on [global variables](/docs/guide/global-variables) for how -to work with objects in global scope that can only be initialized locally. - -## Including local files - -For including local files for your Modal Functions to access, see [Defining Images](/docs/guide/images). diff --git a/.agents/skills/modal/references/guide/managing-deployments.md b/.agents/skills/modal/references/guide/managing-deployments.md deleted file mode 100644 index 2b38b6a2adcc0434918efc249c25464d4eb792cf..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/managing-deployments.md +++ /dev/null @@ -1,111 +0,0 @@ -# Managing deployments - -Once you've finished using `modal run` or `modal serve` to iterate on your Modal -code, it's time to deploy. A Modal deployment creates and then persists an -application and its objects, providing the following benefits: - -* Repeated application function executions will be grouped under the deployment, - aiding observability and usage tracking. Programmatically triggering lots of - ephemeral App runs can clutter your web and CLI interfaces. -* Function calls are much faster because deployed functions are persistent and - reused, not created on-demand by calls. Learn how to trigger deployed - functions in - [Invoking deployed Functions](/docs/guide/trigger-deployed-functions). -* [Scheduled Functions](/docs/guide/cron) will continue scheduling separate from - any local iteration you do, and will notify you on failure. -* [Web Functions](/docs/guide/webhooks) keep running when you close your laptop, - and their URL address matches the deployment name. - -## Creating deployments - -Deployments are created using the -[`modal deploy` command](/docs/reference/cli/app#modal-app-list). - -``` - % modal deploy -m whisper_pod_transcriber.main -✓ Initialized. View app page at https://modal.com/apps/ap-PYc2Tb7JrkskFUI8U5w0KG. -✓ Created objects. -├── 🔨 Created populate_podcast_metadata. -├── 🔨 Mounted /home/ubuntu/whisper_pod_transcriber at /root/whisper_pod_transcriber -├── 🔨 Created fastapi_app => https://modal-labs-whisper-pod-transcriber-fastapi-app.modal.run -├── 🔨 Mounted /home/ubuntu/whisper_pod_transcriber/whisper_frontend/dist at /assets -├── 🔨 Created search_podcast. -├── 🔨 Created refresh_index. -├── 🔨 Created transcribe_segment. -├── 🔨 Created transcribe_episode.. -└── 🔨 Created fetch_episodes. -✓ App deployed! 🎉 - -View Deployment: https://modal.com/apps/modal-labs/whisper-pod-transcriber -``` - -Running this command on an existing deployment will redeploy the App, -incrementing its version. For detail on how live deployed apps transition -between versions, see the [Updating deployments](#updating-deployments) section. - -Deployments can also be created programmatically using Modal's -[Python API](/docs/reference/modal.App#deploy). - -## Viewing deployments - -Deployments can be viewed in the [web UI](/apps) on an App's "Deployment History" -page, or from the command line using the -[`modal app list` command](/docs/reference/cli/app#modal-app-list). - -### Deployment history on metric charts - -You can overlay deployment history information on your Function's metric charts by enabling -the **Show Deployments** toggle. Each marker represents one or more deployments -that occurred within a time bucket. - -Hovering over a marker shows the version number and timestamp of each deployment, plus a link to the full "Deployment History" page. - -![Deployment history overlay on a metric chart](https://modal-cdn.com/cdnbot/deployment-historyt991cvw__b284b7fa.webp) - -## Updating deployments - -A deployment can deploy a new App or redeploy a new version of an existing -deployed App. It's useful to understand how Modal handles the transition between -versions when an App is redeployed. In general, Modal aims to support -zero-downtime deployments by gradually transitioning traffic to the new version. - -If the deployment involves building new versions of the Images used by the App, -the build process will need to complete successfully. The existing version of -the App will continue to handle requests during this time. Errors during the -build will abort the deployment with no change to the status of the App. - -After the build completes, Modal will start to bring up new containers running -the latest version of the App. The existing containers will continue handling -requests (using the previous version of the App) until the new containers have -completed their cold start. - -Once the new containers are ready, old containers will stop accepting new -requests. However, the old containers will continue running any requests they -had previously accepted. The old containers will not terminate until they have -finished processing all ongoing requests. - -Any warm pool containers will also be cycled during a deployment, as the -previous version's warm pool are now outdated. - -## Deployment rollbacks - - -Deployment rollbacks are available on the Team and Enterprise plans. Visit workspace settings to upgrade. - - -To quickly reset an App back to a previous version, you can perform a deployment -*rollback*. Rollbacks can be triggered from either the App dashboard or the CLI. -Rollback deployments look like new deployments: they increment the version number -and are attributed to the user who triggered the rollback. But the App's functions -and metadata will be reset to their previous state independently of your current -App codebase. - -## Stopping deployments - -Deployed apps can be stopped in the web UI by clicking the red "Stop app" button on -the App's "Overview" page, or alternatively from the command line using the -[`modal app stop` command](/docs/reference/cli/app#modal-app-stop). - -Stopping an App is a destructive action. Apps cannot be restarted from this state; -a new App will need to be deployed from the same source files. Objects associated -with stopped deployments will eventually be garbage collected. diff --git a/.agents/skills/modal/references/guide/memory-snapshots.md b/.agents/skills/modal/references/guide/memory-snapshots.md deleted file mode 100644 index 4bdfba70c64d06bb8ccd8660c61f660018ece966..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/memory-snapshots.md +++ /dev/null @@ -1,341 +0,0 @@ -# Memory Snapshots - -Modal Memory Snapshots can dramatically reduce the [cold start](/docs/guide/cold-start) latency of Modal Functions by skipping initialization work on most container boots. - -For instance, during initialization, your code might issue many file read operations sequentially, -like the >20,000 file operations required to load `torch`. -It might then run a JIT compiler that takes several minutes or more, -like the one in PyTorch. -Memory Snapshots replace this initialization work with direct restoration of the memory state that work created. - -The relative speedup is unbounded: the more work you do to create fewer bytes, the greater it becomes. -In our experience, practical initialization-heavy Functions often start up -[3-10x faster from Memory Snapshots](/blog/gpu-mem-snapshots). - -There are two variants of Memory Snapshots. -[CPU Memory Snapshots](#cpu-memory-snapshots) capture the state of CPU memory. -[GPU Memory Snapshots](#gpu-memory-snapshots), an [alpha feature](/docs/guide/feature-maturity), also capture the state of GPU memory. - -## CPU Memory Snapshots - -CPU Memory Snapshots capture the state of a container and save it to disk. -This saved snapshot can then be used to put new containers directly into the exact same state. - -You can enable Memory Snapshots for your Function with the `enable_memory_snapshot=True` parameter: - -```python -@app.function(enable_memory_snapshot=True) -def my_func(): - ... -``` - -Then deploy the App, e.g. with `modal deploy`. Memory Snapshots are created only for deployed Apps. - -Any code executed in global scope, such as imports, will be captured in the Memory Snapshot. -Use the [`Image.imports` context manager](/docs/reference/modal.Image#imports) -to import remote-only dependencies in the global scope. - -```python -image = modal.Image.debian_slim().uv_pip_install("pandas") - -with image.imports(): - import pandas as pd - - -@app.function(enable_memory_snapshot=True, image=image) -def my_func(): - print(f"pandas v{pd.__version__}") -``` - -## Container lifecycle hooks and Memory Snapshots - -Modal's [container lifecycle hooks](/docs/guide/lifecycle-functions) -provide additional control over what parts of container initialization work -are included in Memory Snapshots. Put initialization code that you want to run -before snapshotting inside methods decorated with `@modal.enter(snap=True)`. - -```python -@app.cls(enable_memory_snapshot=True) -class MyCls: - @modal.enter(snap=True) - def load(self): - ... # will be snapshot - - @modal.enter() - def load_more(self): - ... # will not be snapshot -``` - -## GPU Memory Snapshots - - - -GPU Memory Snapshots build on CPU Memory Snapshots and additionally capture GPU state. - -In addition to `enable_memory_snapshot=True`, -pass `experimental_options={"enable_gpu_snapshot": True}` to your Function or Cls -to enable GPU Memory Snapshots. - -```python -@app.function( - gpu="a10", - enable_memory_snapshot=True, - experimental_options={"enable_gpu_snapshot": True} - ) -def my_gpu_func(): - ... -``` - -You'll generally want to include any expensive initialization work that -requires the GPU in the Memory Snapshot. -Use a Modal [Cls](/docs/guide/lifecycle-functions) -and put that work inside a `@modal.enter` method, -like so: - -```python -image = modal.Image.debian_slim().uv_pip_install("transformers[torch]") - -with image.imports(): - import torch - from transformers import pipeline - - -@app.cls( - gpu="h100", - enable_memory_snapshot=True, - experimental_options={"enable_gpu_snapshot": True}, - image=image, -) -class Llm: - @modal.enter(snap=True) - def init(self): - self.pipeline = pipeline(model="Qwen/Qwen3-1.7B", device_map="cuda") - self.pipeline.model = torch.compile(self.pipeline.model, mode="reduce-overhead") - context = [{"role": "user", "content": DEFAULT_PROMPT}] - self.pipeline(context) -``` - -You can find a complete code sample [here](/docs/examples/gpu_snapshot). - -We recommend warming up your model by running a few forward passes on sample data -in the `@modal.enter(snap=True)` method to move more initialization work into the snapshotting phase. -Without warmup, this work is generally done on the first few requests after container start -(regardless of whether Memory Snapshots are used), -which shows up as tail latency. - -### Limitations of GPU Memory Snapshots - -[We've seen](/blog/gpu-mem-snapshots) that GPU Memory Snapshots can massively reduce cold start time, -but they are subject to certain limitations. -The underlying checkpoint/restore technology in the device drivers -is still quite new. We expect these limitations to be resolved as the drivers update. -We recommend reviewing the material below -before adding GPU Memory Snapshots to your Modal Functions. - -#### You may need to rewrite code for compatibility or to improve performance - -While most GPU-accelerated Modal Functions can take advantage of GPU Memory Snapshots, -apart from the limitations described below, -most Functions will need some of their code rewritten to ensure compatibility with GPU Memory Snapshots -or to deliver performance improvements. - -This is particularly true for more complex inference engines, -like those used to maximize [LLM inference performance](/docs/guide/high-performance-llm-inference). -For instance, it is often better to discard the initial, unfilled KV cache before the snapshot is taken, -then recreate it on restore, rather than writing and then reading the KV cache's meaningless pages in a snapshot. -See [this example with vLLM](/docs/examples/vllm_snapshot) -and [this example with SGLang](/docs/examples/sglang_snapshot) -for sample code, patterns, and other guidance. - -#### GPU Memory Snapshots are generally incompatible with multi-GPU code - -Though a few simple programs interacting with multiple GPUs can be successfully snapshot, -there are known issues with most practical uses of multiple GPUs, -stemming from multi-process and multi-GPU resource management concerns. -We anticipate improvements here in future drivers. - -#### GPU Memory Snapshots are generally incompatible with non-CUDA GPU code - -For instance, use of graphics capabilities prior to snapshotting will generally cause failures. - -#### GPU Memory Snapshots do not speed up model loading from storage - -Memory Snapshots use the same high-performance distributed filesystem -that delivers Modal [Images](/docs/guide/images) -and Modal [Volumes](/docs/guide/volumes) -to our worldwide fleet of containers at minimum latency and maximum throughput. - -That means that if the majority of your initialization latency is spent loading weights, -GPU Memory Snapshots will generally not improve your cold start times -- -and may even worsen them, by adding overhead. -Instead, Memory Snapshots should primarily be used to "skip past" work -that is not bottlenecked by storage bandwidth, like library initialization (imports) -and JIT compilation (Torch, DeepGEMM, Triton, etc.). - -#### GPU Memory Snapshots can interact poorly with `torch.compile` - -In certain cases, running the Torch Compiler can cause Memory Snapshot creation to fail. - -Some of these failures can be fixed by setting the environment variable `TORCHINDUCTOR_COMPILE_THREADS` to `1` before compiling. - -## Memory Snapshots FAQs - -### How do I know whether Memory Snapshots are being created or used? - -You can see Memory Snapshots in action in your Function's "Containers" tab. Containers that created a memory snapshot are marked with a icon in the Startup column. Containers that restored from a snapshot are marked with a icon. In the below screenshot, the container startup times when restoring from a memory snapshot are significantly faster. - -![snapshot icons](https://modal-cdn.com/cdnbot/memory-snapshot-iconss6tm168n_cb303ec9.webp) - -You can also search your Modal App's logs for the line `Snapshot created. Restoring Function from memory snapshot.` - -### When are Memory Snapshots updated? - -Redeploying your Function with new configuration (e.g. a [new GPU type](/docs/guide/gpu)) -or new code will cause previous Memory Snapshots to become obsolete. -Subsequent invocations to the new Function version will automatically create new Memory Snapshots with the new configuration and code. - -Changes to [Modal Volumes](/docs/guide/volumes) do not cause Memory Snapshots to update. -Deleting files in a Volume used during restore will cause restore failures. - -### I haven't changed my Function. Why do I still see Memory Snapshots being created sometimes? - -Modal recaptures Memory Snapshots to keep up with the platform's latest runtime and security changes. - -Additionally, you may observe your Function being snapshot multiple times during its first few invocations. -This happens because Memory Snapshots are specific to the underlying worker type that created them -(e.g. CPU flags), and Modal Functions run across a handful of worker types. - -Snapshot creation may add some latency to Function initialization. - -CPU-only Functions need around 6 snapshots for full coverage, and Functions targeting a specific -GPU (e.g. A100) need 2-3. - -### How do Memory Snapshots handle randomness? - -If your application depends on uniqueness of state, you must evaluate your -Function code and verify that it is resilient to snapshotting operations. For -example, if a variable is randomly initialized and that value included in a Memory Snapshot, -that variable will be identical after every restore, possibly breaking uniqueness expectations -of later code. - -## Advanced usage of Memory Snapshots - -### Using GPUs without using GPU Memory Snapshots - -CPU Memory Snapshots on their own block GPU access, -but GPU Functions can still benefit from Memory Snapshots. -This involves refactoring your initialization code to run across two separate `@modal.enter` functions: -one that runs before creating the snapshot (`snap=True`), -and one that runs after restoring from the snapshot (`snap=False`). - -For instance, you might load model weights into CPU memory in the `snap=True` method, -then move the weights onto GPU memory in the `snap=False` method. - -Even without GPU snapshotting, this technique reduces the startup time for `Embedder.run` -in the below example by about 3x, from ~6 seconds down to just ~2 seconds. - -```python -import modal - -image = modal.Image.debian_slim().uv_pip_install("sentence-transformers") -app = modal.App("sentence-transformers", image=image) - -with image.imports(): - from sentence_transformers import SentenceTransformer - -model_vol = modal.Volume.from_name("sentence-transformers-models", create_if_missing=True) - - -@app.cls(gpu="a10", volumes={"/models": model_vol}, enable_memory_snapshot=True) -class Embedder: - model_id = "BAAI/bge-small-en-v1.5" - - @modal.enter(snap=True) - def load(self): - # Create a memory snapshot with the model loaded in CPU memory. - self.model = SentenceTransformer(f"/models/{self.model_id}", device="cpu") - - @modal.enter(snap=False) - def setup(self): - self.model.to("cuda") # Move the model to the GPU! - - @modal.method() - def run(self, sentences:list[str]): - embeddings = self.model.encode(sentences, normalize_embeddings=True) - print(embeddings) - - -@app.local_entrypoint() -def main(): - Embedder().run.remote(sentences=["what is the meaning of life?"]) - - -if __name__ == "__main__": - cls = modal.Cls.from_name("sentence-transformers", "Embedder") - cls().run.remote(sentences=["what is the meaning of life?"]) -``` - -#### GPUs are not available in CPU-only Memory Snapshots - -If you are using the GPU Memory Snapshot feature (`enable_gpu_snapshot`), then -GPUs are available within `@modal.enter(snap=True)`. - -If you are using memory snapshots *without* `enable_gpu_snapshot`, then it's important -to note that GPUs will not be available within the `@modal.enter(snap=True)` method. - -```python -image = modal.Image.debian_slim().uv_pip_install("torch", "numpy") - - -@app.cls(enable_memory_snapshot=True, gpu="a10", image=image) -class GPUAvailability: - @modal.enter(snap=True) - def no_gpus_available_during_snapshots(self): - import torch - print(f"GPUs available: {torch.cuda.is_available()}") # False - - @modal.enter(snap=False) - def gpus_available_following_restore(self): - import torch - print(f"GPUs available: {torch.cuda.is_available()}") # True - - @modal.method() - def demo(self): - print(f"GPUs available: {torch.cuda.is_available()}") # True -``` - -#### Watch out for accidental GPU initialization during CPU-only Memory Snapshots - -The `torch.cuda` module has multiple functions which, if called during -snapshotting, will initialize CUDA as having zero GPU devices. Such functions -include `torch.cuda.is_available` and `torch.cuda.get_device_capability`. -If you're using a framework that calls these methods during its import phase, -it may not be compatible with memory snapshots. The problem can manifest as -confusing "cuda not available" or "no CUDA-capable device is detected" errors. - -We have found that importing PyTorch twice solves the problem in some cases: - -```python - -@app.cls(enable_memory_snapshot=True, gpu="A10") -class GPUAvailability: - @modal.enter(snap=True) - def pre_snap(self): - import torch - ... - @modal.enter(snap=False) - def post_snap(self): - import torch # re-import to re-init GPU availability state - ... -``` - -In particular, `xformers` is known to call `torch.cuda.get_device_capability` on -import, so if it is imported during snapshotting it can unhelpfully initialize -CUDA with zero GPUs. The -[workaround](https://github.com/facebookresearch/xformers/issues/1030) for this -is to set the `XFORMERS_ENABLE_TRITON` environment variable to `1` in your `modal.Image`. - -```python -image = modal.Image.debian_slim().pip_install("xformers>=0.28") # for instance -image = image.env({"XFORMERS_ENABLE_TRITON": "1"}) -``` diff --git a/.agents/skills/modal/references/guide/modal-1-0-migration.md b/.agents/skills/modal/references/guide/modal-1-0-migration.md deleted file mode 100644 index efa8b2b07748e4ef85bf95a2cfa9d423dcc334fc..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/modal-1-0-migration.md +++ /dev/null @@ -1,404 +0,0 @@ -# Modal 1.0 migration guide - -We released version 1.0 of the Modal Python SDK in May 2025. -This release signifies an increased commitment to API stability and implies -some changes to our development workflow. - -Preceding the 1.0 release, we introduced a number of deprecations and changes -based on feedback that we received from early users. These changes were intended -to address pain points and reduce confusion about some aspects of the Modal API. -While adapting to them requires some changes to existing code, we believe that -they’ll make it easier to use Modal going forward. - -This page highlights the major changes for 1.0 and provides some advice for how -to migrate your code to the new stable APIs. Most deprecations introduced prior -to the release of v1.0 will not be enforced (actually cause breaking changes) -until a subsequent minor (v1.x) release, but we recommend updating your code so -that you can take advantage of new features and avoid any future issues. - -## Deprecating `Image.copy_*` methods - -*Introduced in: v0.72.11* - -We recently introduced new `Image` methods — `Image.add_local_dir` and -`Image.add_local_file` — to replace the existing `Image.copy_local_dir` and -`Image.copy_local_file`. - -The new methods subsume the functionality of the old ones, but their default -behavior is different and more performant. By default, files will be mounted to -the container at runtime rather than copied into a new `Image` layer. This can -speed up development substantially when iterating on the contents of the files. - -Building a new `Image` layer should be necessary only when subsequent build -steps will use the added files. In that case, you can pass `copy=True` in -`Image.add_local_file` or `Image.add_local_dir`. - -The `Image.add_local_dir` method also has an `ignore=` parameter, which you can -use to pass file-matching patterns (using dockerignore rules) or predicate -functions to exclude files. - -## Deprecating `Mount` as part of the public API - -*Introduced in: v0.72.4* | *Enforced in: v1.0.0* - -Currently, local files can be mounted to the container filesystem either by -including them in the `Image` definition or by passing a `modal.Mount` object -directly to the `App.function` or `App.cls` decorators. As part of the 1.0 -release, we are simplifying the container filesystem configuration to be defined -only by the `Image` used for each Function. This implies deprecation of the -following: - -* The `mount=` parameter of `App.function` and `App.cls` -* The `context_mount=` parameter of several `modal.Image` methods -* The `Image.copy_mount` method -* The `Mount` object - -Code that uses the `mount=` parameter of `App.function` and `App.cls` should be -migrated to pass those files / directories to the `Image` used by that Function -or Cls, i.e. using the `Image.add_local_file`, `Image.add_local_dir`, or -`Image.add_local_python_source` methods: - -```python notest -# Mounting local files - -# Old way (deprecated) -mount = modal.Mount.from_local_dir("data").add_local_file("config.yaml") -@app.function(image=image, mount=mount) -def f(): - ... - -# New way -image = image.add_local_dir("data", "/root/data").add_local_file("config.yaml", "/root/config.yaml") -@app.function(image=image) -def f(): - ... - -## Mounting local Python source code - -# Old way (deprecated) -mount = modal.Mount.from_local_python_packages("my-lib")) -@app.function(image=image, mount=mount) -def f() - ... - -# New way -image = image.add_local_python_source("my-lib") -@app.function(image=image) -def f(...): - ... - -## Using Image.copy_mount - -# Old way (deprecated) -mount = modal.Mount.from_local_dir("data").add_local_file("config.yaml") -image.copy_mount(mount) - -# New way -image.add_local_dir("data", "root/data").add_local_file("config.yaml", "/root/config.yaml") -``` - -Code that uses the `context_mount=` parameter of `Image.from_dockerfile` and -`Image.dockerfile_commands` methods can delete that parameter; we now -automatically infer the files that need to be included in the context. - -## Deprecating the `@modal.build` decorator - -*Introduced in: v0.72.17* - -As part of consolidating the filesystem configuration API, we are also -deprecating the `modal.build` decorator. - -For use cases where `modal.build` would previously have been the suggested -approach (e.g., downloading model weights or other large assets to the -container filesystem), we now recommend using a `modal.Volume` instead. The -main advantage of storing weights in a `Volume` instead of an `Image` is that -the weights do not need to be re-downloaded every time you change something else -about the `Image` definition. - -Many frameworks, such as Hugging Face, automatically cache downloaded model -weights. When using these frameworks, you just need to ensure that you mount a -`modal.Volume` to the expected location of the framework’s cache: - -```python notest -cache_vol = modal.Volume.from_name("hf-hub-cache") -@app.cls( - image=image.env({"HF_HUB_CACHE": "/cache"}), - volumes={"/cache": cache_vol}, - ... -) -class Model: - @modal.enter() - def load_model(self): - self.model = ModelClass.from_pretrained(...) -``` - -For frameworks that don’t support automatic caching, you could write a separate -function to download the weights and write them directly to the Volume, then -`modal run` against this function before you deploy. - -In some cases (e.g., if the step runs very quickly), you may wish for the logic -currently decorated with `@modal.build` to continue modifying the Image -filesystem. In that case, you can extract the method as a standalone function -and pass it to `Image.run_function`: - -```python notest -def download_weights(): - ... - -image = image.run_function(download_weights) -``` - -## Requiring explicit inclusion of local Python dependencies - -*Introduced in: 0.73.11* | *Enforced in: 1.0.0* - -Prior to 1.0, Modal will inspect the modules that are imported when running -your App code and automatically include any "local" modules in the remote -container environment. This behavior is referred to as "automounting". - -While convenient, this approach has a number of edge cases and surprising -behaviors, such as ignoring modules with imports that are deferred using -`Image.imports`. Additionally, it is difficult to configure the automounting -behavior to, e.g., ignore large data files that are stored within your local -Python project directories. - -Going forward, it will be necessary to explicitly include the local dependencies -of your Modal App. The easiest way to do this is with -[`Image.add_local_python_source`](/docs/reference/modal.Image#add_local_python_source): - -```python notest -import modal -import helpers - -image = modal.Image.debian_slim().add_local_python_source("helpers") -``` - -In the period leading up to the change in default behavior, the Modal client -will issue deprecation warnings when automounted modules are not included -in the Image. Updating the Image definition will remove these warnings. - -Note that Modal will continue to automatically include the source module or -package defining the App itself. We're introducing a new App or Function-level -parameter, `include_source`, which can be set to `False` in cases where this is -not desired (i.e., because your Image definition already includes the App -source). - -## Renaming autoscaler parameters - -*Introduced in: v0.73.76* - -We're renaming several parameters that configure autoscaling behavior: - -* `keep_warm` is now `min_containers` -* `concurrency_limit` is now `max_containers` -* `container_idle_timeout` is now `scaledown_window` - -The renaming is intended to address some persistent confusion about -the meaning of these parameters. The migration path is a simple -find-and-replace operation. - -Additionally, we're promoting a fourth parameter, `buffer_containers`, -from experimental status (previously `_experimental_buffer_containers`). -Like `min_containers`, `buffer_containers` can help mitigate cold-start -penalties by overprovisioning containers while the Function is active. - -## Renaming `modal.web_endpoint` to `modal.fastapi_endpoint` - -*Introduced in: v0.73.89* - -We're renaming the `modal.web_endpoint` decorator to `modal.fastapi_endpoint` -so that the implicit dependency on FastAPI is more clear. This can be a -simple name substitution in your code as the semantics are otherwise identical. - -We may reintroduce a lightweight `modal.web_endpoint` without external -dependencies in the future. - -## Replacing `allow_concurrent_inputs` with `@modal.concurrent` - -*Introduced in: v0.73.148* - -The `allow_concurrent_inputs` parameter is being replaced with a new decorator, -`@modal.concurrent`. The decorator can be applied either to a Function or a Cls. -We're moving the input concurrency feature out of "Beta" status as part of this -change. - -The new decorator exposes two distinct parameters: `max_inputs` (the limit -on the number of inputs the Function will concurrently accept) and -`target_inputs` (the level of concurrency targeted by the Modal autoscaler). -The simplest migration path is to replace `allow_concurrent_inputs=N` with -`@modal.concurrent(max_inputs=N)`: - -```python notest -# Old way, with a function (deprecated) -@app.function(allow_concurrent_inputs=1000) -def f(...): - ... - -# New way, with a function -@app.function() -@modal.concurrent(max_inputs=1000) -def f(...): - ... - -# Old way, with a class (deprecated) -@app.cls(allow_concurrent_inputs=1000) -class MyCls: - ... - -# New way, with a class -@app.cls() -@modal.concurrent(max_inputs=1000) -class MyCls: - ... -``` - -Setting `target_inputs` along with `max_inputs` may benefit performance by -reducing latency during periods where the container pool is scaling up. See the -[input concurrency guide](/docs/guide/concurrent-inputs) for more information. - -## Deprecating the `.lookup` method on Modal objects - -*Introduced in: v0.72.56* - -Most Modal objects can be instantiated through two distinct methods: -`.from_name` and `.lookup`. The redundancy between these methods is a persistent -source of confusion. - -The `.from_name` method is lazy: it operates entirely locally and instantiates -only a shell for the object. The local object won’t be associated with its -identity on the Modal server until you interact with it. In contrast, the -`.lookup` method is eager: it triggers a remote call to the Modal server, and it -returns a fully-hydrated object. - -Because Modal objects can now be hydrated on-demand, when they are first -used, there is rarely any need to eagerly hydrate. Therefore, we’re deprecating -`.lookup` so that there’s only one obvious way to instantiate objects. - -In most cases, the migration is a simple find-and-replace of `.lookup` → -`.from_name`. - -One exception is when your code needs to access object metadata, such as its ID, -or a Web Function URL. In that case, you can explicitly force hydration of the -object by calling its `.hydrate()` method. There may be other subtle consequences, -such as errors being raised at a different location if no object exists with the -given name. - -## Removing support for custom Cls constructors - -*Introduced in: v0.74.0* - -Classes decorated with `App.cls` are no longer allowed to have a custom constructor -(`__init__` method). Instead, class parameterization should be exposed using -dataclass-style [`modal.parameter`](/docs/reference/modal.parameter) annotations: - -```python notest -# Old way (deprecated) -@app.cls() -class MyCls: - def __init__(self, name: str = "Bert"): - self.name = name - -# New way -@app.cls() -class MyCls: - name: str = modal.parameter(default="Bert") -``` - -Modal will provide a synthetic constructor for classes that use `modal.parameter`. -Arguments to the synthetic constructor must be passed using keywords, so you may -need to update your calling code as well: - -```python notest -obj = MyCls(name="Bert") # name= is now required -``` - -We're making this change to address some persistent confusion about when -constructors execute for remote calls and what operations are allowed to run in -them. If your custom constructor performs any setup logic beyond storing the -parameter values, you should move it to a method decorated with -`@modal.enter()`. - -Additionally, we're reducing the types that we support as class parameters to -a small number of primitives (`str`, `int`, `bool`, and `bytes`). - -Limiting class parameterization to primitive types will also allow us to provide -better observability over parameterized class instances in the web dashboard, -CLI, and other contexts where it is not possible to represent arbitrary Python -objects. - -If you need to parameterize classes across more complex types, you can implement -your own serialization logic, e.g. using strings as the wire format: - -```python notest -@app.cls() -class MyCls: - param_str: str = modal.parameter() - - @modal.enter() - def deserialize_parameters(self): - self.param_obj = SomeComplexType.from_str(self.param_str) -``` - -We recommend adopting interpretable constructor arguments (i.e., prefer -meaningful strings over pickled bytes) so that you will be able to get the most -benefit from future improvements to parameterized class observability. - -## Simplifying Cls lookup patterns - -*Introduced in: v0.73.26* - -Modal previously supported several different patterns for looking up a `modal.Cls` -and remotely invoking one of its methods: - -```python notest -# Documented pattern -MyCls = modal.Cls.from_name("my-app", "MyCls") -obj = MyCls() -obj.some_method.remote(...) - -# Alternate pattern: skipping the object instantiation -MyCls = modal.Cls.from_name("my-app", "MyCls") -MyCls.some_method.remote(...) - -# Alternate pattern: looking up the method as a Function -f = modal.Function.lookup("my-app", "MyCls.some_method") -f.remote(...) -``` - -While each pattern could successfully trigger a remote function call, there were -a number of subtle differences in behavior between them. - -Going forward, we will only support the first pattern. Making remote calls to a -method on a deployed Cls will require you to (a) look up the object using -`modal.Cls` and (b) instantiate the object before calling its methods. - -## Deprecating `modal.gpu` objects - -*Introduced in: v0.73.31* - -The `modal.gpu` objects are being deprecated; going forward, all GPU resource -configuration should be accomplished using strings. - -This should be an easy code substitution, e.g. `gpu=modal.gpu.H100()` can be -replaced with `gpu="H100"`. When using the `count=` parameter of the GPU class, -simply append it to the name with a colon (e.g. `gpu="H100:8"`). In the case of -the `modal.gpu.A100(size="80GB")` variant, the name of the corresponding gpu is -`"A100-80GB"`. - -Note that string arguments are case-insensitive, so `"H100"` and `"h100"` are -both accepted. - -The main rationale for this change is that it will allow us to introduce new -GPU models in the future without requring users to upgrade their SDK. - -## Requiring explicit invocation for module mode - -*Introduced in: 0.73.58* - -The Modal CLI allows you to reference the source code for your App as either -a file path (e.g. `src/my_app.py`) or as a module name (e.g. `src.my_app`). - -As in Python, the choice has some implications for how relative imports are -resolved. To make this more salient, Modal will mirror Python going forwared -and require that you explicitly invoke module mode by passing `-m` on your -command line (e.g., `modal deploy -m src.my_app`). diff --git a/.agents/skills/modal/references/guide/modal-user-account-setup.md b/.agents/skills/modal/references/guide/modal-user-account-setup.md deleted file mode 100644 index 413349c629704eca649d21e939a892af50856ef4..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/modal-user-account-setup.md +++ /dev/null @@ -1,24 +0,0 @@ -# Modal user account setup - -To run and deploy applications on Modal you'll need to sign up and create a user -account. - -You can visit the [signup](/signup) page to begin the process or execute -[`modal setup`](/docs/reference/cli/setup#modal-setup) on the command line. - -Users can also be provisioned through [Okta SSO](/docs/guide/okta-sso), which is -an enterprise feature that you can request. For the typical user you'll sign-up -using an existing GitHub account. If you're interested in authenticating with -other identity providers let us know at . - -## What GitHub permissions does signing up require? - -* `user:email` — gives us the emails associated with the GitHub account. -* `read:org` (invites only) — needed for Modal workspace invites. Note: this - only allows us to see what organization memberships you have - ([GitHub docs](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/scopes-for-oauth-apps)). - We won't be able to access any code repositories or other details. - -## How can I change my email? - -You can change your email on the [settings](/settings) page. diff --git a/.agents/skills/modal/references/guide/model-weights.md b/.agents/skills/modal/references/guide/model-weights.md deleted file mode 100644 index 2ce8e4330c07b15fee7fbc82373dedf72307556c..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/model-weights.md +++ /dev/null @@ -1,206 +0,0 @@ -# Storing model weights on Modal - -Efficiently managing the weights of large models is crucial for optimizing the -build times and startup latency of many ML and AI applications. - -Our recommended method for working with model weights is to store them in a Modal [Volume](/docs/guide/volumes), -which acts as a distributed file system, a "shared disk" all of your Modal Functions can access. - -## Storing weights in a Modal Volume - -To store your model weights in a Volume, you need to either -make the Volume available to a Modal Function that saves the model weights -or upload the model weights into the Volume from a client. - -### Saving model weights into a Modal Volume from a Modal Function - -If you're already generating the weights on Modal, you just need to -attach the Volume to your Modal Function, making it available for reading and writing: - -```python -from pathlib import Path - -volume = modal.Volume.from_name("model-weights-vol", create_if_missing=True) -MODEL_DIR = Path("/models") - -@app.function(gpu="any", volumes={MODEL_DIR: volume}) # attach the Volume -def train_model(data, config): - import run_training - - model = run_training(config, data) - model.save(config, MODEL_DIR) -``` - -Volumes are attached by including them in a dictionary that maps -a path on the remote machine to a `modal.Volume` object. -They look just like a normal file system, so model weights can be saved to them -without adding any special code. - -If the model weights are generated outside of Modal and made available -over the Internet, for example by an open-weights model provider -or your own training job on a dedicated cluster, -you can also download them into a Volume from a Modal Function: - -```python continuation -@app.function(volumes={MODEL_DIR: volume}) -def download_model(model_id): - import model_hub - - model_hub.download(model_id, local_dir=MODEL_DIR / model_id) -``` - -Add [Modal Secrets](/docs/guide/secrets) to access weights that require authentication. - -See [below](#storing-weights-from-the-hugging-face-hub-on-modal) for -more on downloading from the popular Hugging Face Hub. - -### Uploading model weights into a Modal Volume - -Instead of pulling weights into a Modal Volume from inside a Modal Function, -you might wish to push weights into Modal from a client, -like your laptop or a dedicated training cluster. - -For that, you can use the `batch_upload` method of -[`modal.Volume`](/docs/reference/modal.Volume)s -via the Modal Python client library: - -```python continuation -volume = modal.Volume.from_name("model-weights-vol", create_if_missing=True) - -@app.local_entrypoint() -def main(local_path: str, remote_path: str): - with volume.batch_upload() as upload: - upload.put_directory(local_path, remote_path) -``` - -Alternatively, you can upload model weights using the -[`modal volume`](/docs/reference/cli/volume) CLI command: - -```bash -modal volume put model-weights-vol path/to/model path/on/volume -``` - -### Mounting cloud buckets as Modal Volumes - -If your model weights are already in cloud storage, -for example in an S3 bucket, you can connect them -to Modal Functions with a `CloudBucketMount`. - -See [the guide](/docs/guide/cloud-bucket-mounts) for details. - -## Reading model weights from a Modal Volume - -You can read weights from a Volume as you would normally read them -from disk, so long as you attach the Volume to your Function. - -```python continuation -@app.function(gpu="any", volumes={MODEL_DIR: volume}) -def inference(prompt, model_id): - import load_model - - model = load_model(MODEL_DIR / model_id) - model.run(prompt) -``` - -## Storing weights in the Modal Image - -It is also possible to store weights in your Function's Modal [Image](/docs/guide/images), -the private file system state that a Function sees when it starts up. -The weights might be downloaded via shell commands with [`Image.run_commands`](/docs/guide/images) -or downloaded using a Python function with [`Image.run_function`](/docs/guide/images). - -We recommend storing model weights in a Modal [Volume](/docs/guide/volumes), -as described [above](#storing-weights-in-a-modal-volume). Performance is similar -for the two methods. Volumes are more flexible. -Images are rebuilt when their definition changes, starting from the changed layer, -which increases reproducibility for some builds but leads to unnecessary extra downloads -in most cases. - -## Optimizing model weight reads with `@modal.enter` - -In the above code samples, weights are loaded from disk into memory each time -the `inference` function is run. This isn't so bad if inference is much -slower than model loading (e.g. it is run on very large datasets) -or if the model loading logic is smart enough to skip reloading. - -To guarantee a particular model's weights are only loaded once, you can use the `@modal.enter` -[container lifecycle hook](/docs/guide/lifecycle-functions) -to load the weights only when a new container starts. - -```python continuation -MODEL_ID = "some-model-id" - -@app.cls(gpu="any", volumes={MODEL_DIR: volume}) -class Model: - @modal.enter() - def setup(self, model_id=MODEL_ID): - import load_model - - self.model = load_model(MODEL_DIR, model_id) - - @modal.method() - def inference(self, prompt): - return self.model.run(prompt) -``` - -Note that methods decorated with `@modal.enter` can't be passed dynamic arguments. - -If you need to load a single but possibly different model on each container start, you can -[parametrize](/docs/guide/parametrized-functions) your Modal Cls. -Below, we use the `modal.parameter` syntax. - -```python continuation -@app.cls(gpu="any", volumes={MODEL_DIR: volume}) -class ParametrizedModel: - model_id: str = modal.parameter() - - @modal.enter() - def setup(self): - import load_model - - self.model = load_model(MODEL_DIR, self.model_id) - - @modal.method() - def inference(self, prompt): - return self.model.run(prompt) -``` - -## Storing weights from the Hugging Face Hub on Modal - -The [Hugging Face Hub](https://huggingface.co/models) has over 1,000,000 models -with weights available for download. - -The snippet below shows some additional tricks for downloading models -from the Hugging Face Hub on Modal. - -```python -from typing import Optional -from pathlib import Path - -import modal - -# create a Volume, or retrieve it if it exists -volume = modal.Volume.from_name("model-weights-vol", create_if_missing=True) -MODEL_DIR = Path("/models") - -# define dependencies for downloading model -download_image = ( - modal.Image.debian_slim() - .pip_install("huggingface_hub") - .env({"HF_XET_HIGH_PERFORMANCE": "1"}) # enable fast data transfer -) -app = modal.App() - -@app.function( - volumes={MODEL_DIR.as_posix(): volume}, # "mount" the Volume, sharing it with your function - image=download_image, # only download dependencies needed here -) -def download_model( - repo_id: str = "hf-internal-testing/tiny-random-GPTNeoXForCausalLM", - revision: Optional[str] = None, # include a revision to prevent surprises! -): - from huggingface_hub import snapshot_download - - snapshot_download(repo_id=repo_id, local_dir=MODEL_DIR / repo_id, revision=revision) - print(f"Model downloaded to {MODEL_DIR / repo_id}") -``` diff --git a/.agents/skills/modal/references/guide/multi-node-training.md b/.agents/skills/modal/references/guide/multi-node-training.md deleted file mode 100644 index dcc190563bfd8db8887dfc8a33ed8c61a59eb426..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/multi-node-training.md +++ /dev/null @@ -1,166 +0,0 @@ -# Multi-node clusters - - - -Modal supports running a training job across several coordinated containers. Each container can saturate the available GPU devices on its host (aka node) and communicate with peer containers which do the same. By scaling a training job from a single GPU to 16 GPUs you can achieve nearly 16x improvements in training time. - -### Cluster compute capability - -Modal clusters provide: - -* A 50 Gbps [IPv6 private network](https://modal.com/docs/guide/private-networking) for orchestration, dataset downloading, etc. -* A 3,200 Gbps RDMA scale-out network ([RoCE](https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet)). -* Up to 64 devices. -* At least 1 TB of RAM and 4 TB of local NVMe SSD per node. -* Deep burn-in testing. -* Interoperability with all Modal platform functionality ([Volumes](/docs/guide/volumes), [Dicts](/docs/guide/dicts), [Tunnels](/docs/guide/tunnels), etc.). - -The guide will walk you through how the Modal client library enables multi-node training and integrates with `torchrun`. - -### `@clustered` - -Unlike standard Modal Function containers, containers in a multi-node training job must be able to: - -1. Perform fast, direct network communication between each other. -2. Be scheduled together, all or nothing, at the same time. - -The `@clustered` decorator enables this behavior. - -```python -import modal.experimental - -@app.function( - gpu="H100:8", - timeout=60 * 60 * 24, - retries=modal.Retries(initial_delay=0.0, max_retries=10), -) -@modal.experimental.clustered(size=4) -def train_model(): - cluster_info = modal.experimental.get_cluster_info() - - container_rank = cluster_info.rank - world_size = len(cluster_info.container_ips) - main_addr = cluster_info.container_ips[0] - is_main = "(main)" if container_rank == 0 else "" - - print(f"{container_rank=} {is_main} {world_size=} {main_addr=}") - ... -``` - -Applying this decorator under `@app.function` modifies the Function so that remote calls to it are serviced by a multi-node container group. The above configuration creates a group of four containers each having 8 H100 GPU devices, for a total of 32 devices. - - - -Starting May 31st, 2026, clustered functions must use the full number of GPU devices per node (e.g. `H100:4` is invalid but `H100:8` is valid). Clustered functions require GPUs, and CPU-only functions are not supported. If you have a special case that does not fall under above and would like to use clustered functions, please contact . - - - -## Scheduling - -A `modal.experimental.clustered` Function runs on multiple nodes in our cloud, but executes like a normal function call. For example, all nodes are scheduled together ([gang scheduling](https://en.wikipedia.org/wiki/Gang_scheduling)) so that your code runs on all of the requested hardware or not at all. - -Traditionally this kind of cluster and scheduling management would be handled by SLURM, Kubernetes, or manually. But with Modal it's all provided serverlessly with just a Python decorator! - -### Rank & input broadcast - -![diagram](https://modal-cdn.com/cdnbot/multinodepmgnla70_4b57a155.webp) - -You may notice above that a single `.remote` Function call created three input executions but returned only one output. This is how input-output is structured for multi-node training jobs on Modal. The Function call’s arguments are replicated to each container, but only the rank zero container’s is returned to the caller. - -A container’s rank is a key concept in multi-node jobs. Rank zero is the 'leader' rank and typically coordinates the job. Rank zero is also known as the "main" container. Rank zero's output will always be the output of a multi-node training run. - -## Networking - -Function containers cannot normally make direct network connections to other Function containers, but this is a requirement for multi-node training communication. So, along with gang scheduling, the `@clustered` decorator enables Modal’s workspace-private inter-container networking called [i6pn](https://www.notion.so/Multi-node-docs-1281e7f16949806f966adedfe8b2cb74?pvs=21). - -The [cluster networking guide](/docs/guide/private-networking) goes into more detail on i6pn, but the upshot is that each container in the cluster is made aware of the network address of all the other containers in the cluster, enabling them to communicate with each other quickly via [TCP](https://pytorch.org/docs/stable/elastic/rendezvous.html). - -### RDMA (Infiniband) - -Clusters are equipped with Infiniband providing up to 3,200 Gbps scale-out bandwidth for inter-node communication. -RDMA scale-out networking is enabled with the `rdma` parameter of `modal.experimental.clustered`. - -```python notest -@modal.experimental.clustered(size=2, rdma=True) -def train(): - ... -``` - -To run a simple Infiniband RDMA performance test see the [this sample code](https://github.com/modal-labs/multinode-training-guide/tree/main/benchmark). - -## Cluster Info - -`modal.experimental.get_cluster_info()` exposes the following information about the cluster: - -* `rank: int` is the current container's order within the cluster, starting from `0`, the leader. -* `cluster_id: str` is the unique identifier for the cluster. -* `container_ips: list[str]` contains the IPv6 addresses of each container in the cluster, sorted by rank. -* `container_ipv4_ips: list[str]` contains the IPv4 addresses of each container in the cluster, sorted by rank. - -## Fault Tolerance - -For a clustered Function, failures in inputs and containers are handled differently. - -If an input fails on any container, this failure **is not propagated** to other containers in the cluster. Containers are responsible for detecting and responding to input failures on other containers. - -Only rank 0's output matters: if an input fails on the leader container (rank 0), the input is marked as failed, even if the input succeeds on another container. Similarly, if an input succeeds on the leader container but fails on another container, the input will still be marked as successful. - -If a container in the cluster is preempted, Modal will terminate all remaining containers in the cluster, and retry the input. - -### Input Synchronization - -***Important:*** synchronization is not relevant for single training runs, and applies mostly to inference use-cases. - -Modal does not synchronize input execution across containers. Containers are responsible for ensuring that they do not process inputs faster than other containers in their cluster. - -In particular, it is important that the leader container (rank 0) only starts processing the next input after all other containers have finished processing the current input. - -## Examples - -To get hands-on with multi-node training you can jump into the [`multinode-training-guide` repository](https://github.com/modal-labs/multinode-training-guide) or [`modal-examples` repository](https://github.com/modal-labs/modal-examples/tree/main/14_clusters) and `modal run` something! - -* [Simple ‘hello world’ 4 x 1 H100 torch cluster example](https://github.com/modal-labs/modal-examples/blob/main/14_clusters/simple_torch_cluster.py) -* [Infiniband RDMA performance test](https://github.com/modal-labs/multinode-training-guide/tree/main/benchmark) -* [Use 2 x 8 H100s to train a ResNet50 model on the ImageNet dataset](https://github.com/modal-labs/multinode-training-guide/tree/main/resnet50) -* [Speedrun GPT-2 training with modded-nanogpt](https://github.com/modal-labs/multinode-training-guide/tree/main/nanoGPT) - - - -### Torchrun Example - -```python -import modal -import modal.experimental - -image = ( - modal.Image.debian_slim(python_version="3.12") - .pip_install("torch~=2.5.1", "numpy~=2.2.1") - .add_local_dir( - "training", remote_path="/root/training" - ) -) -app = modal.App("example-simple-torch-cluster", image=image) - -n_nodes = 4 - -@app.function(gpu=f"H100:8", timeout=60 * 60 * 24) -@modal.experimental.clustered(size=n_nodes, rdma=True) -def launch_torchrun(): - # import the 'torchrun' interface directly. - from torch.distributed.run import parse_args, run - - cluster_info = modal.experimental.get_cluster_info() - - run( - parse_args( - [ - f"--nnodes={n_nodes}", - f"--node-rank={cluster_info.rank}", - f"--master-addr={cluster_info.container_ips[0]}", - "--nproc-per-node=8", - "--master-port=1234", - "training/train.py", - ] - ) - ) -``` diff --git a/.agents/skills/modal/references/guide/named-images.md b/.agents/skills/modal/references/guide/named-images.md deleted file mode 100644 index 453212923637ee61270941e21457daf861cddcd5..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/named-images.md +++ /dev/null @@ -1,172 +0,0 @@ -# Named images - -Named Images let you publish a Modal Image under a name that you can reference -later to use the Image, akin to a container registry. - -This can be useful for stricter Image change management and for avoiding -unintended Image invalidation and rebuilds on latency-sensitive code paths. - -Unlike inline Image definitions, referencing an image by name will never -implicitly rebuild an Image. The Image reference for a name is mutable, -and because the reference is typically updated only after a successful -publish, callers keep using the previous working Image while the new build is running. - -A typical workflow using named images would be: - -1. Define, build, and publish the Image in an independently run Image build script -2. Reference the published Image by name in Sandbox or Function code, getting the latest image built in 1 at the time - -## Publishing an Image from a script - -Use [`Image.build`](/docs/reference/modal.Image#build) to build the -Image, then call `.publish()` on the resulting Image: - - - {#snippet python()} - -```python notest -# build_image.py -app = modal.App.lookup("image-builds", create_if_missing=True) - -image = ( - modal.Image.debian_slim(python_version="3.12") - .apt_install("git") - .uv_pip_install("numpy", "pandas", "scikit-learn") - .run_commands("python -c 'import sklearn; print(sklearn.__version__)'") -) - -with modal.enable_output(): - image.build(app).publish("analytics-runtime") -``` - -{/snippet} - -{#snippet javascript()} - -```javascript -// build_image.ts -const app = await modal.apps.fromName("image-builds", { - createIfMissing: true, -}); - -const image = modal.images - .fromRegistry("python:3.12-slim") - .dockerfileCommands([ - "RUN apt-get update && apt-get install -y git", - "RUN pip install numpy pandas scikit-learn", - "RUN python -c 'import sklearn; print(sklearn.__version__)'", - ]); - -const builtImage = await image.build(app); -await builtImage.publish("analytics-runtime"); -``` - -{/snippet} - -{#snippet go()} - -```go -// build_image.go -app, err := mc.Apps.FromName(ctx, "image-builds", &modal.AppFromNameParams{ - CreateIfMissing: true, -}) - -image := mc.Images.FromRegistry("python:3.12-slim", nil). - DockerfileCommands([]string{ - "RUN apt-get update && apt-get install -y git", - "RUN pip install numpy pandas scikit-learn", - "RUN python -c 'import sklearn; print(sklearn.__version__)'", - }, nil) - -builtImage, err := image.Build(ctx, app, nil) -err = builtImage.Publish(ctx, "analytics-runtime", nil) -``` - -{/snippet} - -## Starting Sandboxes using named Images - -Named Images are especially useful for Sandboxes because Sandbox creation often happens -on a latency-sensitive path and you typically never want to block Sandbox creation on -rebuilding an Image. - -Use [`Image.from_name`](/docs/reference/modal.Image#from_name) when referencing -a named Image that you have previously built, and start the Sandbox using that: - - - {#snippet python()} - -```python notest -# sandbox_launcher.py -sb = modal.Sandbox.create( - "python", - "-c", - "import pandas, sklearn; print('ready')", - image=modal.Image.from_name("analytics-runtime"), - app=app, -) -print(sb.stdout.read()) -``` - -{/snippet} - -{#snippet javascript()} - -```javascript -// sandbox_launcher.ts -const image = await modal.images.fromName("analytics-runtime"); -const sb = await modal.sandboxes.create(app, image); - -const p = await sb.exec([ - "python", - "-c", - "import pandas, sklearn; print('ready')", -]); -console.log(await p.stdout.readText()); -sb.detach(); -``` - -{/snippet} - -{#snippet go()} - -```go -// sandbox_launcher.go -image, err := mc.Images.FromName(ctx, "analytics-runtime", nil) -sb, err := mc.Sandboxes.Create(ctx, app, image, nil) -defer sb.Detach() - -p, err := sb.Exec(ctx, []string{ - "python", - "-c", - "import pandas, sklearn; print('ready')", -}, nil) -stdout, err := io.ReadAll(p.Stdout) -fmt.Println(string(stdout)) -``` - -{/snippet} - -## Running Functions using named Images - -Named Images can also be used when defining Modal Functions when you want more control over when -a Function starts using a new Image. To use a named Image, point the Function image attribute -to a [`Image.from_name`](/docs/reference/modal.Image#from_name) reference: - -```python notest -# app.py -@app.function(image=modal.Image.from_name("analytics-runtime")) -def train(): - import pandas as pd - from sklearn.linear_model import LinearRegression - ... -``` - -Note that publishing a new version of this named Image would not automatically -update your deployed Functions to use the updated Image. You still need to redeploy -the App that references that name for the change to propagate. - -## Tags - -Every named Image is represented using a `{name}:{tag}` name - if you do not specify the tag part, the `:latest` tag is automatically used. -You can publish the same Image using multiple names or tag which can be useful to do things like versioning of images. diff --git a/.agents/skills/modal/references/guide/oidc-integration.md b/.agents/skills/modal/references/guide/oidc-integration.md deleted file mode 100644 index a5b393f95d41f45924af376af0dec685675f858b..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/oidc-integration.md +++ /dev/null @@ -1,300 +0,0 @@ -# Using OIDC to authenticate with external services - -Your Functions in Modal may need to access external resources like S3 buckets. -Traditionally, you would need to store long-lived credentials in Modal Secrets -and reference those Secrets in your function code. With the Modal OIDC -integration, you can instead use automatically-generated identity -tokens to authenticate to external services. - -## How it works - -[OIDC](https://auth0.com/docs/authenticate/protocols/openid-connect-protocol) is -a standard protocol for authenticating users between systems. In Modal, we use -OIDC to generate short-lived tokens that external services can use to verify -that your function is authenticated. - -The OIDC integration has two components: the discovery document and the generated -tokens. - -The [OIDC discovery document](https://swagger.io/docs/specification/v3_0/authentication/openid-connect-discovery/) -describes how our OIDC server is configured. It primarily includes the supported -[claims](https://developer.okta.com/blog/2017/07/25/oidc-primer-part-1) and the [keys](https://auth0.com/docs/secure/tokens/json-web-tokens/json-web-key-sets) -we use to sign tokens. Discovery documents are always hosted at `/.well-known/openid-configuration`, and -you can view ours at . - -The generated tokens are [JWTs](https://jwt.io/) signed by Modal using the keys described in the -discovery document. These tokens contain the full identity of the Function -in the `sub` claim, and they use custom claims to make this information more -easily accessible. See our [discovery document](https://oidc.modal.com/.well-known/openid-configuration) -for a full list of claims. - -Generated tokens are injected into your Function's containers via the `MODAL_IDENTITY_TOKEN` -environment variable. Below is an example of what claims might be included in a token: - -```json -{ - "sub": "modal:workspace_id:ac-12345abcd:environment_name:modal-examples:app_name:oidc-token-test:function_name:jwt_return_func:container_id:ta-12345abcd", - "aud": "oidc.modal.com", - "exp": 1732137751, - "iat": 1731964951, - "iss": "https://oidc.modal.com", - "jti": "31f92dca-e847-4bc9-8d15-9f234567a123", - "workspace_id": "ac-12345abcd", - "environment_id": "en-12345abcd", - "environment_name": "modal-examples", - "app_id": "ap-12345abcd", - "app_name": "oidc-token-test", - "function_id": "fu-12345abcd", - "function_name": "jwt_return_func", - "container_id": "ta-12345abcd" -} -``` - -### App name format - -By default, Modal Apps can be created with arbitrary names. However, when using -OIDC, the App name has a stricter character set. Specifically, it must be 64 -characters or less and can only include alphanumeric characters, dashes, periods, -and underscores. If these constraints are violated, the OIDC token will not be -injected into the container. - -Note that these are the same constraints that are applied to [Deployed Apps](/docs/guide/managing-deployments). -This means that if an App is deployable, it will also be compatible with OIDC. - -## Demo usage with AWS S3 - -To see how OIDC tokens can be used, we'll demo a simple Function that lists -objects in an S3 bucket. - -### Step 0: Understand your OIDC claims - -Before we can configure OIDC policies, we need to know what claims we can match -against. We can run a Function and inspect its claims to find out. - -```python notest -app = modal.App("oidc-token-test") - -jwt_image = modal.Image.debian_slim().pip_install("pyjwt") - -@app.function(image=jwt_image) -def jwt_return_func(): - import jwt - - token = os.environ["MODAL_IDENTITY_TOKEN"] - claims = jwt.decode(token, options={"verify_signature": False}) - print(json.dumps(claims, indent=2)) - -@app.local_entrypoint() -def main(): - jwt_return_func.remote() -``` - -Run the function locally to see its claims: - -```bash -$ modal run oidc-token-test.py -{ - "sub": "modal:workspace_id:ac-12345abcd:environment_name:modal-examples:app_name:oidc-token-test:function_name:jwt_return_func:container_id:ta-12345abcd", - "aud": "oidc.modal.com", - "exp": 1732137751, - "iat": 1731964951, - "iss": "https://oidc.modal.com", - "jti": "31f92dca-e847-4bc9-8d15-9f234567a123", - "workspace_id": "ac-12345abcd", - "environment_id": "en-12345abcd", - "environment_name": "modal-examples", - "app_id": "ap-12345abcd", - "app_name": "oidc-token-test", - "function_id": "fu-12345abcd", - "function_name": "jwt_return_func", - "container_id": "ta-12345abcd" -} -``` - -Now we can match off these claims to configure our OIDC policies. - -### Step 1: Configure AWS to trust Modal's OIDC provider - -We need to make AWS accept Modal identity tokens. To do this, we need to add -Modal's OIDC provider as a trusted entity in our AWS account. - -```bash -aws iam create-open-id-connect-provider \ - --url https://oidc.modal.com \ - --client-id-list oidc.modal.com -``` - -This will trigger AWS to pull down our [JSON Web Key Set (JWKS)](https://auth0.com/docs/secure/tokens/json-web-tokens/json-web-key-sets) -and use it to verify the signatures of any tokens signed by Modal. - -### Step 2: Create an IAM policy that can be used by Modal Functions - -Let's create a simple IAM policy that allows listing objects in an S3 bucket. -Take the policy below and replace the bucket name with your own. - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": ["s3:PutObject", "s3:GetObject", "s3:ListBucket"], - "Resource": ["arn:aws:s3:::fun-bucket", "arn:aws:s3:::fun-bucket/*"] - } - ] -} -``` - -### Step 3: Create an IAM role that can be assumed by Modal Functions - -Now, we can create an IAM role that uses this policy. Visit the IAM console -to create this role. If you add this policy using the CLI, update the -OIDC provider ARN to match the one created in [Step 1](#step-1-configure-aws-to-trust-modals-oidc-provider). -Be sure to replace the Workspace ID placeholder with your own. You can find your Workspace ID -at https://modal.com/settings/workspaces or through the `modal token info` CLI. - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Federated": "arn:aws:iam::123456789abcd:oidc-provider/oidc.modal.com" - }, - "Action": "sts:AssumeRoleWithWebIdentity", - "Condition": { - "StringEquals": { - "oidc.modal.com:aud": "oidc.modal.com" - }, - "StringLike": { - "oidc.modal.com:sub": "modal:workspace_id:ac-12345abcd:*" - } - } - } - ] -} -``` - -Note how we use `workspace_id` to limit the scope of the role. This means that -the IAM role can only be assumed by Functions in your Workspace. You can further -limit this by specifying an Environment, App, or Function name. - -Ideally, we would use the custom claims for role limiting. Unfortunately, AWS -does not support [matching on custom claims](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_iam-condition-keys.html#condition-keys-wif), -so we use the `sub` claim instead. - -### Step 4: Use the OIDC token in your Function - -The AWS SDKs have built-in support for OIDC tokens, so you can use them as -follows: - -```python notest -import boto3 - -app = modal.App("oidc-token-test") - -boto3_image = modal.Image.debian_slim().pip_install("boto3") - -# Trade a Modal OIDC token for AWS credentials -def get_s3_client(role_arn): - sts_client = boto3.client("sts") - - # Assume role with Web Identity - credential_response = sts_client.assume_role_with_web_identity( - RoleArn=role_arn, RoleSessionName="OIDCSession", WebIdentityToken=os.environ["MODAL_IDENTITY_TOKEN"] - ) - - # Extract credentials - credentials = credential_response["Credentials"] - return boto3.client( - "s3", - aws_access_key_id=credentials["AccessKeyId"], - aws_secret_access_key=credentials["SecretAccessKey"], - aws_session_token=credentials["SessionToken"], - ) - -# List the contents of an S3 bucket -@app.function(image=boto3_image) -def list_bucket_contents(bucket_name, role_arn): - s3_client = get_s3_client(role_arn) - response = s3_client.list_objects_v2(Bucket=bucket_name) - for obj in response["Contents"]: - print(f"- {obj['Key']} (Size: {obj['Size']} bytes)") - -@app.local_entrypoint() -def main(): - # Replace with the role ARN and bucket name from step 2 - list_bucket_contents.remote("fun-bucket", "arn:aws:iam::123456789abcd:role/oidc_test_role") -``` - -Run the function locally to see the contents of the bucket: - -```bash -$ modal run oidc-token-test.py -- test-file.txt (Size: 10 bytes) -``` - -## Demo usage with AWS Elastic Container Registry (ECR) - -You can also use OIDC to authenticate [Private Registries](/docs/guide/existing-images) on AWS. - -### Prerequisites - -1. Configure AWS to trust Modal's OIDC provider ([Step 1 above](#step-1-configure-aws-to-trust-modals-oidc-provider)) - -2. [Create an AWS Policy with read-only ECR access](/docs/guide/existing-images#elastic-container-registry-ecr) - -3. Create an IAM role that uses this policy ([Step 3 above](#step-3-create-an-iam-role-that-can-be-assumed-by-modal-functions)) - -### Test with a sample image - -Create sample Dockerfile: - -```dockerfile -FROM python:3.11-slim -WORKDIR /app -CMD ["python3"] -``` - -Build and push the image to ECR: - -```bash -# Login with the AWS CLI -aws ecr get-login-password --region [ECR_REGION] | docker login --username AWS --password-stdin [ECR_REPO_ARN] - -# Build the Docker Image -docker build -t modal-oidc-test-image . - -# Push the image to ECR -docker tag modal-oidc-test-image:latest [ECR_REPO_ARN]:latest -docker push [ECR_REPO_ARN]:latest -``` - -Test pulling the image from ECR: - -```python -import modal - -app = modal.App("image-from-ecr-test") -sample_image = modal.Image.from_aws_ecr( - "[ECR_IMAGE_URI]", #eg. "12345678.dkr.ecr.us-east-1.amazonaws.com/repository:latest" - secret=modal.Secret.from_dict( - { - "AWS_ROLE_ARN": "[IAM_ROLE_ARN]", # eg. "arn:aws:iam::123456789abcd:role/oidc_test_role" - "AWS_REGION": "[ECR_REGION]", # eg. "us-east-1" - } - ), -) - -@app.function(image=sample_image) -def hello(): - print("Hello, World!") -``` - -## Next steps - -The OIDC integration can be used for much more than just AWS. With this same pattern, -you can configure automatic access to [Vault](https://developer.hashicorp.com/vault/docs/auth/jwt), -[GCP](https://cloud.google.com/identity-platform/docs/web/oidc), [Azure](https://learn.microsoft.com/en-us/entra/identity-platform/v2-protocols-oidc), and more. -At this time, OIDC-authenticated container image pulling is only support with AWS ECR. diff --git a/.agents/skills/modal/references/guide/okta-sso.md b/.agents/skills/modal/references/guide/okta-sso.md deleted file mode 100644 index 3a1fe2910e2dd5faac8922db4aca2b485d48f369..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/okta-sso.md +++ /dev/null @@ -1,85 +0,0 @@ -# Okta SSO - - -Okta SSO is available on the Enterprise plan. Contact sales@modal.com for more information. - - -## Prerequisites - -* A Workspace that's on an [Enterprise](/pricing) plan -* Admin access to the Workspace you want to configure with Okta Single-Sign-On (SSO) -* Admin privileges for your Okta Organization - -## Supported features - -* Identity Provider (IdP) initiated SSO -* Service Provider (SP) initiated SSO -* Just-In-Time account provisioning - -For more information on the listed features, visit the -[Okta Glossary](https://help.okta.com/okta_help.htm?type=oie\&id=ext_glossary). - -## Configuration - -### Read this before you enable "Require SSO" - -Enabling "Require SSO" will force all users to sign in via Okta. Ensure that you -have admin access to your Modal Workspace through an Okta account before -enabling. - -### Configuration steps - -#### Step 1: Add Modal app to Okta Applications - -1. Sign in to your Okta admin dashboard - -2. Navigate to the Applications tab and click "Browse App Catalog". - ![Okta browse application](../../assets/docs/okta-browse-applications.png) - -3. Select "Modal" and click "Done". - -4. Select the "Sign On" tab and click "Edit". - ![Okta sign on edit](../../assets/docs/okta-sign-on-edit.png) - -5. Fill out Workspace field to configure for your specific Modal workspace. See - [Step 2](/docs/guide/okta-sso#step-2-link-your-workspace-to-okta-modal-application) - if you're unsure what this is. - ![Okta add workspace](../../assets/docs/okta-add-workspace-username.png) - -#### Step 2: Link your Workspace to Okta Modal application - -1. Navigate to your application on the Okta Admin page. - -2. Copy the Metadata URL from the Okta Admin Console (It's under the "Sign On" - tab). ![Okta metadata url](../../assets/docs/okta-metadata-url.png) - -3. Sign in to https://modal.com and visit your [Workspace Management](/settings/workspace-management/identity-and-provisioning) page's `Identity and Provisioning` tab. - -4. Paste the Metadata URL in the input and click "Save Changes" - -#### Step 3: Assign users / groups and test the integration - -1. Navigate back to your Okta application on the Okta Admin dashboard. -2. Click on the "Assignments" tab and add the appropriate people or groups. - -![Okta Assign Users](../../assets/docs/okta-assign-people.png) - -3. To test the integration, sign in as one of the users you assigned in the previous step. -4. Click on the Modal application on the Okta Dashboard to initiate Single Sign-On. - -#### Notes - -The following SAML attributes are used by the integration: - -| Name | Value | -| --------- | -------------- | -| email | user.email | -| firstName | user.firstName | -| lastName | user.lastName | - -## SP-initiated SSO - -The sign-in process is initiated from https://modal.com/login/sso - -1. Enter your workspace name in the input -2. Click "continue with SSO" to authenticate with Okta diff --git a/.agents/skills/modal/references/guide/otel-integration.md b/.agents/skills/modal/references/guide/otel-integration.md deleted file mode 100644 index e4cc59964cb7d5e9db264ce3b4a3dbbf3d917c6d..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/otel-integration.md +++ /dev/null @@ -1,98 +0,0 @@ -# Connecting Modal to your OpenTelemetry Provider - -You can export Modal logs to your [OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) -provider using the Modal OpenTelemetry integration. This integration is compatible with -any observability provider that supports the OpenTelemetry HTTP APIs. - -## What this integration does - -This integration allows you to: - -1. Export Modal audit logs to your provider -2. Export Modal function logs to your provider -3. Export container metrics to your provider - -## Metrics - -The Modal OpenTelemetry Integration will forward the following metrics to your provider: - -* `modal.cpu.utilization` -* `modal.memory.usage` -* `modal.gpu.memory.usage` -* `modal.gpu.compute.utilization` -* `modal.container.running` -* `modal.input_events.elapsed_time_us` -* `modal.input_events.input_queue_time_us` -* `modal.input_events.coldstart_time_us` -* `modal.input_events.successes` -* `modal.input_events.total_inputs` -* `modal.function.pending_inputs` -* `modal.function.running_inputs` - -Deprecated metrics: - -* `modal.memory.utilization` (use `modal.memory.usage`) -* `modal.gpu.memory.utilization` (use `modal.gpu.memory.usage`) - -`modal.input_events.successes` and `modal.input_events.total_inputs` can be used to measure the success rate of a certain function or app. - -These metrics are tagged with `container_id`, `environment_name`, `app_name`, -`app_id`, `function_name`, `function_id`, `workspace_name`, and `workspace_id`. - -## Custom metrics - - - -Contact us to enable custom metrics for your workspace. - - - -The Modal OpenTelemetry Integration allows you to send custom metrics and spans to your provider. You will -then need to export our collector environment variables. These configure the OpenTelemetry SDK -to send messages to our collector in HTTP format. You don't need to do this to get the -out-of-the-box metrics above, only for your own custom metrics. - -```python -@app.function( - secrets=[modal.Secret.from_dict({ - "OTEL_EXPORTER_OTLP_ENDPOINT": "otlp-collector.modal.local:4317", - "OTEL_EXPORTER_OTLP_INSECURE": "true", - "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf", - })], -) -def custom_metrics(): - ... -``` - -All OpenTelemetry SDKs should pick this configuration up, and your custom metrics and spans will be -sent to your configured provider. - -## Installing the integration - -1. Find out the endpoint URL for your OpenTelemetry provider. This is the URL that - the Modal integration will send logs to. Note that this should be the base URL - of the OpenTelemetry provider, and not a specific endpoint. For example, for the - [US New Relic instance](https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-otlp/#configure-endpoint-port-protocol), - the endpoint URL is `https://otlp.nr-data.net`, not `https://otlp.nr-data.net/v1/logs`. -2. Find out the API key or other authentication method required to send logs to your - OpenTelemetry provider. This is the key that the Modal integration will use to authenticate - with your provider. Modal can provide any key/value HTTP header pairs. For example, for - [New Relic](https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-otlp/#api-key), - the header is `api-key`. -3. Create a new OpenTelemetry Secret in Modal with one key per header. These keys should be - prefixed with `OTEL_HEADER_`, followed by the name of the header. The value of this - key should be the value of the header. For example, for New Relic, an example Secret - might look like `OTEL_HEADER_api-key: YOUR_API_KEY`. If you use the OpenTelemetry Secret - template, this will be pre-filled for you. -4. Navigate to the [Modal metrics settings page](http://modal.com/settings/metrics) and configure - the OpenTelemetry push URL from step 1 and the Secret from step 3. -5. Save your changes and use the test button to confirm that logs are being sent to your provider. - If it's all working, you should see a `Hello from Modal! 🚀` log from the `modal.test_logs` service. - -## Uninstalling the integration - -Once the integration is uninstalled, all logs will stop being sent to -your provider. - -1. Navigate to the [Modal metrics settings page](http://modal.com/settings/metrics) - and disable the OpenTelemetry integration. diff --git a/.agents/skills/modal/references/guide/parametrized-functions.md b/.agents/skills/modal/references/guide/parametrized-functions.md deleted file mode 100644 index fa1d2f87877a8fb1c544343cd79f984c0ac7cc6e..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/parametrized-functions.md +++ /dev/null @@ -1,121 +0,0 @@ -# Parametrized functions - -A single Modal Function can be parametrized by a set of arguments, so that each unique combination of arguments will behave like an individual -Modal Function with its own auto-scaling and lifecycle logic. - -For example, you might want to have a separate pool of containers for each unique user that invokes your Function. In this scenario, you would -parametrize your Function by a user ID. - -To parametrize a Modal Function, you need to use Modal's [class syntax](/docs/guide/lifecycle-functions) and the -[`@app.cls`](/docs/reference/modal.App#cls) decorator. Specifically, you'll need to: - -1. Convert your function to a method by making it a member of a class. -2. Decorate the class with `@app.cls(...)` with the same arguments you previously - had for `@app.function(...)` or your [Web Function decorator](/docs/guide/webhooks). -3. If you previously used the `@app.function()` decorator on your function, replace it with `@modal.method()`. -4. Define dataclass-style, type-annotated instance attributes with `modal.parameter()` and optionally set default values: - -```python -import modal - -app = modal.App() - -@app.cls() -class MyClass: - - foo: str = modal.parameter() - bar: int = modal.parameter(default=10) - - @modal.method() - def baz(self, qux: str = "default") -> str: - return f"This code is running in container pool ({self.foo}, {self.bar}), with input qux={qux}" -``` - -The parameters create a keyword-only constructor for your class, and the methods can be called as follows: - -```python -@app.local_entrypoint() -def main(): - m1 = MyClass(foo="hedgehog", bar=7) - m1.baz.remote() - - m2 = MyClass(foo="fox") - m2.baz.remote(qux="override") -``` - -Function calls for each unique combination of values for `foo` and `bar` will run in their own separate container pools. -If you re-constructed a `MyClass` with the same arguments in a different context, the calls to `baz` would be routed to the same set of containers as before. - -Some things to note: - -* The total size of the arguments is limited to 16 KiB. -* Modal classes can still annotate types of regular class attributes, which are independent of parametrization, by either omitting `= modal.parameter()` or using `= modal.parameter(init=False)` to satisfy type checkers. -* The support types are these primitives: `str`, `int`, `bool`, and `bytes`. -* The legacy `__init__` constructor method is being removed, see [the 1.0 migration for details.](/docs/guide/modal-1-0-migration#removing-support-for-custom-cls-constructors) - -## Looking up a parametrized function - -If you want to call your parametrized function from a Python script running -anywhere, you can use `Cls.lookup`: - -```python notest -import modal - -MyClass = modal.Cls.from_name("parametrized-function-app", "MyClass") # returns a class-like object -m = MyClass(foo="snake", bar=12) -m.baz.remote() -``` - -## Parametrized Web Functions - -Modal [Web Functions](/docs/guide/webhooks) can also be parametrized: - -```python -app = modal.App("parametrized-endpoint") - -@app.cls() -class MyClass(): - - foo: str = modal.parameter() - bar: int = modal.parameter(default=10) - - @modal.fastapi_endpoint() - def baz(self, qux: str = "default") -> str: - ... -``` - -Parameters are specified in the URL as query parameter values. - -```bash -curl "https://parametrized-endpoint.modal.run?foo=hedgehog&bar=7&qux=override" -curl "https://parametrized-endpoint.modal.run?foo=hedgehog&qux=override" -curl "https://parametrized-endpoint.modal.run?foo=hedgehog&bar=7" -curl "https://parametrized-endpoint.modal.run?foo=hedgehog" -``` - -## Using parametrized functions with lifecycle functions - -Parametrized functions can be used with [lifecycle functions](/docs/guide/lifecycle-functions). - -For example, here is how you might parametrize the [`@modal.enter`](/docs/guide/lifecycle-functions#enter) lifecycle function to load a specific model: - -```python -@app.cls() -class Model: - - name: str = modal.parameter() - size: int = modal.parameter(default=100) - - @modal.enter() - def load_model(self): - print(f"Loading model {self.name} with size {self.size}") - self.model = load_model_util(self.name, self.size) - - @modal.method() - def generate(self, prompt: str) -> str: - return self.model.generate(prompt) -``` - -## Performance - -Currently, parametrized Function creation is rate limited to 1 per second, with the ability to burst to 1000. Please [get in touch](mailto:support@modal.com) if you need higher rate limits. diff --git a/.agents/skills/modal/references/guide/preemption.md b/.agents/skills/modal/references/guide/preemption.md deleted file mode 100644 index 225e2e107d9635639f95115df20faaef043de1d2..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/preemption.md +++ /dev/null @@ -1,38 +0,0 @@ -# Preemption - -All Modal Functions are subject to preemption by default. -If a preemption event interrupts a running Function, Modal will gracefully terminate -the Function and restart it on the same input. - -Preemptions are rare, but it is always possible that your Function is -interrupted. Long-running Functions such as model training Functions should take -particular care to tolerate interruptions, as likelihood of interruption increases -with Function run duration. - -## Preparing for interruptions - -Design your applications to be fault and preemption tolerant. Modal will send an -interrupt signal to your container when preemption occurs. This will cause the -Function's [exit handler](/docs/guide/lifecycle-functions#exit) to run, which -can perform any cleanup within its grace period. - -Other best practices for handling preemptions include: - -* Divide long-running operations into small tasks or use checkpoints so that you - can save your work frequently. See our [long training example](/docs/examples/long-training) - for a practical demonstration of checkpointing. -* Ensure preemptible operations are safely retryable (ie. idempotent). - -## Non-preemptible Functions - -If you require Functions that are guaranteed not to be preempted, you may set the `nonpreemptible` -parameter (available starting in client version v1.2.3) to `True` in the `@app.function()` or `@app.cls()` decorator. -Note that a 3x multiplier will be applied to the [list price](https://modal.com/pricing) for CPU and Memory usage when -`nonpreemptible` is set to `True`. - -**Note:** The `nonpreemptible` parameter is not supported for GPU Functions. - -## Non-preemptible Sandboxes - -Modal Sandboxes are not subject to preemption, except in the case where a `gpu` -requirement is specified. This is because of availability and scheduling latency constraints. diff --git a/.agents/skills/modal/references/guide/private-networking.md b/.agents/skills/modal/references/guide/private-networking.md deleted file mode 100644 index 575652ad4ce52246dfef340700e8ae5abd9a26f1..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/private-networking.md +++ /dev/null @@ -1,44 +0,0 @@ -# Cluster networking - -i6pn (IPv6 private networking) is Modal's private container-to-container networking solution. It allows users to create clusters of Modal containers which can send network traffic to each other with low latency and high bandwidth (≥ 50Gbps). - -Normally, `modal.Function` containers can initiate outbound network connections to the internet but they are not directly addressable by other containers. i6pn-enabled containers, on the other hand, can be directly connected to by other i6pn-enabled containers and this is a key enabler of Modal's Beta `@modal.experimental.clustered` functionality. - -You can enable i6pn on any `modal.Function`: - -```python -@app.function(i6pn=True) -def hello_private_network(): - import socket - - i6pn_addr = socket.getaddrinfo("i6pn.modal.local", None, socket.AF_INET6)[0][4][0] - print(i6pn_addr) # fdaa:5137:3ebf:a70:1b9d:3a11:71f2:5f0f -``` - -In this snippet we see that the i6pn-enabled container is able to retrieve its own IPv6 address by -resolving `i6pn.modal.local`. For this Function container to discover the addresses of *other* containers, -address sharing must be implemented using an auxiliary data structure, such as a shared `modal.Dict` or `modal.Queue`. - -## Private networking - -All i6pn network traffic is *Workspace private*. - -![i6pn-diagram](https://modal-cdn.com/cdnbot/i6pn-1eksk4vuy_c4c4a0df.webp) - -In the image above, Workspace A has subnet `fdaa:1::/48`, while Workspace B has subnet `fdaa:2::/48`. - -You'll notice they share the first 16 bits. This is because the `fdaa::/16` prefix contains all of our private network IPv6 addresses, while each workspace is assigned a random 32-bit identifier when it is created. Together, these form the 48-bit subnet. - -The upshot of this is that only containers in the same workspace can see each other and send each other network packets. i6pn networking is secure by default. - -## Region boundaries - -Modal operates a [global fleet](/docs/guide/region-selection) and allows containers to run on multiple cloud providers and in many regions. i6pn networking is however region-scoped functionality, meaning that only i6pn-enabled containers in the same region can perform network communication. - -Modal's i6pn-enabled primitives such as `@modal.experimental.clustered` automatically restrict container geographic placement and cloud placement to ensure inter-container connectivity. - -## Public network access to cluster networking - -For cluster networked containers that need to be publicly accessible, you need to expose ports with [modal.Tunnel](/docs/guide/tunnels) because i6pn addresses are not publicly exposed. - -Consider having a container setup a Tunnel and act as the gateway to the private cluster networking. diff --git a/.agents/skills/modal/references/guide/project-structure.md b/.agents/skills/modal/references/guide/project-structure.md deleted file mode 100644 index e69eb7038975c3f660ef505d0f238240b3d81511..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/project-structure.md +++ /dev/null @@ -1,106 +0,0 @@ -# Project structure - -## Apps spanning multiple files - -When your project spans multiple files, more care is required to package the -full structure for running or deploying on Modal. - -There are two main considerations: (1) ensuring that all of your Functions get -registered to the App, and (2) ensuring that any local dependencies get included -in the Modal container. - -Say that you have a simple project that's distributed across three files: - -``` -src/ -├── app.py # Defines the `modal.App` as a variable named `app` -├── llm.py # Imports `app` and decorates some functions -└── web.py # Imports `app` and decorates other functions -``` - -With this structure, if you deploy using `modal deploy src/app.py`, Modal won't -discover the Functions defined in the other two modules, because they never get -imported. - -If you instead run `modal deploy src/llm.py`, Modal will deploy the App with -just the Functions defined in that module. - -One option would be to ensure that one module in the project transitively -imports all of the other modules and to point the `modal deploy` CLI at it, but -this approach can lead to an awkward project structure. - -### Defining your project as a Python package - -A better approach would be to define your project as a Python *package* and to -use the Modal CLI's "module mode" invocation pattern. - -In Python, a package is a directory containing an `__init__.py` file (and -usually some other Python modules). If you have a `src/__init__.py` that -imports all of the member modules, it will ensure that any decorated Functions -contained within them get registered to the App: - -```python notest -# Contents of __init__.py -import .app -import .llm -import .web -``` - -*Important: use *relative* imports (`import .app`) between member modules.* - -Unfortunately, it's not enough just to set this up and make your deploy command -`modal deploy src/app.py`. Instead, you need to invoke Modal in *module mode*: -`modal deploy -m src.app`. Note the use of the `-m` flag and the module path -(`src.app` instead of `src/app.py`). Akin to `python -m ...`, this incantation -treats the target as a package rather than just a single script. - -### App composition - -As your project grows in scope, it may become helpful to organize it into -multiple component Apps, rather than having the project defined as one large -monolith. That way, as you iterate during development, you can target a specific -component, which will build faster and avoid any conflicts with concurrent work -on other parts of the project. - -Projects set up this way can still be deployed as one unit by using `App.include`. -Say our project from above defines separate Apps in `llm.py` and `web.py` and then -adds a new `deploy.py` file: - -```python notest -# Contents of deploy.py -import modal - -from .llm import llm_app -from .web import web_app - -app = modal.App("full-app").include(llm_app).include(web_app) -``` - -This lets you run `modal deploy -m src.deploy` to package everything in one -step. - -**Note:** Since the multi-file app still has a single namespace for all -functions, it's important to name your Modal functions uniquely across the -project even when splitting it up across files: otherwise you risk some -functions "shadowing" others with the same name. - -## Including local dependencies - -Another factor to consider is whether Modal will package all of the local -dependencies that your App requires. - -Even if your Modal App itself can be contained to a single file, any local -modules that file imports (like, say, a `helpers.py`) also need to be available -in the Modal container. - -By default, Modal will automatically include the module or package where a -Function is defined in all containers that run that Function. So if the project -is set up as a package and the helper modules are part of that package, you -should be all set. If you're not using a package setup, or if the local -dependencies are external to your project's package, you'll need to explicitly -include them in the Image, i.e. with `modal.Image.add_local_python_source`. - -**Note:** This behavior changed in Modal 1.0. Previously, Modal would -"automount" any local dependencies that were imported by your App source into a -container. This was changed to be more selective to avoid unnecessary inclusion -of large local packages. diff --git a/.agents/skills/modal/references/guide/proxy-ips.md b/.agents/skills/modal/references/guide/proxy-ips.md deleted file mode 100644 index 867addbd3001a7a166ab83722ec8da5098dfbb32..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/proxy-ips.md +++ /dev/null @@ -1,98 +0,0 @@ -# Proxies - - - -You can securely connect with resources in your private network -using a Modal Proxy. Proxies are a secure tunnel between -Apps and exit nodes with static IPs. You can allow-list those static IPs -in your network firewall, making sure that only traffic originating from these -IP addresses is allowed into your network. - -Proxies are unique and not shared between workspaces. All traffic -between your Apps and the Proxy server is encrypted using -[WireGuard](https://www.wireguard.com/). - -Modal Proxies are built on top of [vprox](https://github.com/modal-labs/vprox), -a Modal open-source project used to create highly available proxy servers -using WireGuard. - -## Creating a Proxy - - - -Proxies are available on the Team and Enterprise plans. Visit workspace settings to upgrade. - - - -You can create Proxies in your workspace [Settings](/settings) page. -Team Plan users can create one Proxy and Enterprise users three Proxies. Each Proxy -can have a maximum of five static IP addresses. - -Please reach out to if you need greater limits. - -## Using a Proxy - -After a Proxy is online, add it to a Modal Function with the argument -`proxy=Proxy.from_name("")`. For example: - -```python -import modal -import subprocess - -app = modal.App(image=modal.Image.debian_slim().apt_install("curl")) - -@app.function(proxy=modal.Proxy.from_name("")) -def my_ip(): - subprocess.run(["curl", "-s", "ifconfig.me"]) - -@app.local_entrypoint() -def main(): - my_ip.remote() -``` - -All network traffic from your Function will now use the Proxy as a tunnel. - -The program above will always print the same IP address independent -of where it runs in Modal's infrastructure. If that same program -were to run without a Proxy, it would print a different IP -address depending on where it runs. - -## Proxy performance - -All traffic that goes through a Proxy is encrypted by WireGuard. This adds -latency to your Function's networking. If you are experiencing networking issues -with Proxies related to performance, first add more IP addresses to your -Proxy (see [Adding more IP addresses to a Proxy](#adding-more-ip-addresses-to-a-proxy)). - -## Adding more IP addresses to a Proxy - -Proxies support up to five static IP addresses. Adding IP addresses improves -throughput linearly. - -You can add an IP address to your workspace in [Settings](/settings) > Proxies. -Select the desired Proxy and add a new IP. - -If a Proxy has multiple IPs, Modal will randomly pick one when running your Function. - -## Proxies and Sandboxes - -Proxies can also be used with [Sandboxes](/docs/guide/sandboxes). For example: - -```python notest -import modal - -app = modal.App.lookup("sandbox-proxy", create_if_missing=True) -sb = modal.Sandbox.create( - app=app, - image=modal.Image.debian_slim().apt_install("curl"), - proxy=modal.Proxy.from_name("")) - -process = sb.exec("curl", "-s", "https://ifconfig.me") -stdout = process.stdout.read() -print(stdout) - -sb.terminate() -``` - -Similarly to our Function implementation, this Sandbox program will -always print the same IP address. diff --git a/.agents/skills/modal/references/guide/queues.md b/.agents/skills/modal/references/guide/queues.md deleted file mode 100644 index 3dc887a639761b4f884ac778ffc05bc34bf3d230..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/queues.md +++ /dev/null @@ -1,127 +0,0 @@ -# Queues - -Modal Queues provide distributed FIFO queues to your Modal Apps. - -```python runner:ModalRunner retry:2 -import modal - -app = modal.App() -queue = modal.Queue.from_name("simple-queue", create_if_missing=True) - - -def producer(x): - queue.put(x) # adding a value - - -@app.function() -def consumer(): - return queue.get() # retrieving a value - - -@app.local_entrypoint() -def main(x="some object"): - # produce and consume tasks from local or remote code - producer(x) - print(consumer.remote()) -``` - -This page is a high-level guide to using Modal Queues. -For reference documentation on the `modal.Queue` object, see -[this page](/docs/reference/modal.Queue). -For reference documentation on the `modal queue` CLI command, see -[this page](/docs/reference/cli/queue). - -## Modal Queues are Python queues in the cloud - -Like [Python `Queue`s](https://docs.python.org/3/library/queue.html), -Modal Queues are multi-producer, multi-consumer first-in-first-out (FIFO) queues. - -Queues are particularly useful when you want to handle tasks or process -data asynchronously, or when you need to pass messages between different -components of your distributed system. - -Queues are cleared 24 hours after the last `put` operation and are backed by -a replicated in-memory database, so persistence is likely, but not guaranteed. -As such, `Queue`s are best used for communication between active functions and -not relied on for persistent storage. - -[Please get in touch](mailto:support@modal.com) if you need durability for Queue objects. - -## Queues are partitioned by key - -Queues are split into separate FIFO partitions via a string key. By default, one -partition (corresponding to an empty key) is used. - -A single `Queue` can contain up to 100,000 partitions, each with up to 5,000 -items. Each item can be up to 1 MiB. These limits also apply to the default -partition. - -Each partition has an independent TTL, by default 24 hours. -Lower TTLs can be specified by the `partition_ttl` argument in the `put` or -`put_many` methods. - -```python -with modal.Queue.ephemeral() as q: - q.put("some value") # first in - q.put(123) - - assert q.get() == "some value" # first out - assert q.get() == 123 - - q.put(0) - q.put(1, partition="foo") - q.put(2, partition="bar") - - # Default and "foo" partition are ignored by the get operation. - assert q.get(partition="bar") == 2 - - # Set custom 10s expiration time on "foo" partition. - q.put(3, partition="foo", partition_ttl=10) - - # Iterate through items in place (read immutably) - q.put(1) - assert [v for v in q.iterate()] == [0, 1] -``` - -## You can access Modal Queues synchronously or asynchronously, blocking or non-blocking - -Queues are synchronous and blocking by default. Consumers will block and wait -on an empty Queue and producers will block and wait on a full Queue, -both with an `Optional`, configurable `timeout`. If the `timeout` is `None`, -they will wait indefinitely. If a `timeout` is provided, `get` methods will raise -[`queue.Empty`](https://docs.python.org/3/library/queue.html#queue.Empty) -exceptions and `put` methods will raise -[`queue.Full`](https://docs.python.org/3/library/queue.html#queue.Full) -exceptions, both from the Python standard library. - -The `get` and `put` methods can be made non-blocking by setting the `block` argument to `False`. -They raise `queue` exceptions without waiting on the `timeout`. - -Queues are stored in the cloud, so all interactions require communication over the network. -This adds some extra latency to calls, apart from the `timeout`, on the order of tens of milliseconds. -To avoid this latency impacting application latency, you can asynchronously interact with Queues -by adding the `.aio` function suffix to access methods. - -```python notest -@app.local_entrypoint() -async def main(value=None): - await my_queue.put.aio(value or 200) - assert await my_queue.get.aio() == value -``` - -See the guide to [asynchronous functions](/docs/guide/async) for more -information. - -## Modal Queues are not *exactly* Python Queues - -Python Queues can have values of any type. - -Modal Queues can store Python objects of any serializable type. - -Objects are serialized using [`cloudpickle`](https://github.com/cloudpipe/cloudpickle), -so precise support is inherited from that library. `cloudpickle` can serialize a surprising variety of objects, -like `lambda` functions or even Python modules, but it can't serialize a few things that don't -really make sense to serialize, like live system resources (sockets, writable file descriptors). - -Note that you will need to have the library defining the type installed in the environment -where you retrieve the object so that it can be deserialized. diff --git a/.agents/skills/modal/references/guide/rbac.md b/.agents/skills/modal/references/guide/rbac.md deleted file mode 100644 index 149183b2c40cfa4d91c8e1fe05651edbc50a9c1c..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/rbac.md +++ /dev/null @@ -1,184 +0,0 @@ -# Role-Based Access Control (RBAC) - - -RBAC is available on the Team and Enterprise plans. Visit workspace settings to upgrade. - - -Role-Based Access Control (RBAC) gives Workspace administrators more granular control over who can access and modify resources. - -This is especially useful for protecting production while allowing broader access to development and staging. - -Modal's RBAC system operates at two levels: - -* **Workspace Roles** control overall Workspace permissions. -* **Environment Roles** control access to specific RBAC restricted Environments. - -## Workspace Roles - -Modal [Workspaces](/docs/guide/workspaces) organize Modal Apps and other resources for a group of users. These roles control access at the level of the entire Workspace. - -All Workspace Members have one of three Roles that determine their overall permissions: - -* **Owner** — Full read-write access to everything in the Workspace, including billing, Workspace management, and all Environments. Can assign any Role to other members. -* **Manager** — Same as Owner, but cannot modify the Owner Role. -* **Member** — Can deploy and manage Apps, but cannot access billing, Workspace management, or other Workspace settings. - -## Environment Roles - -Modal [Environments](/docs/guide/environments) isolate Modal Apps and other resources from one another within a Workspace. - -Environments can be set as **restricted** to enable RBAC at the Environment level. Restricted Environments introduce two specific Roles for more granular control: - -* **Viewer** — Read-only access to resources in the Environment, including dashboards, logs, metrics, app and function configuration. -* **Contributor** — Full read and write access to the Environment. Workspace Owners and Managers automatically have Contributor access to all restricted Environments. - -## Setting up restricted Environments - -To enable RBAC for specific Environments: - -1. **Enable RBAC**: This requires a Team or Enterprise plan. See our [pricing page](/pricing) for more information. - -2. **Create or restrict an Environment**: To create a new restricted Environment, use - [`modal environment create --restricted NAME`](/docs/reference/cli/environment#modal-environment-create). To restrict an existing unrestricted Environment, navigate to your Workspace's - Environment Management page in [Settings](/settings), select the Environment, - and click **Make Restricted**. - -3. **Configure Environment Roles**: Once an Environment is restricted: - * All Workspace Members automatically get **Viewer** access. - * Workspace Owners and Managers automatically get **Contributor** access. - * You can assign specific users or service users **Contributor** access. - -4. **Manage access**: Use the **Manage** button next to any restricted Environment to: - * Add users or service users with specific Roles - * Change existing Roles - * Remove access - -### Default access by actor - -| Workspace Role | Unrestricted Environment Default | Restricted Environment Default | -| ---------------------------- | -------------------------------- | -------------------------------------------------------- | -| Workspace Owner | Contributor | Contributor | -| Workspace Manager | Contributor | Contributor | -| Workspace Member | Contributor | Viewer by default, or Contributor if explicitly assigned | -| Service user / service token | Contributor | Viewer by default, or Contributor if explicitly assigned | - -In restricted Environments, you can grant specific users or service users **Contributor** access for deployment and management workflows. - -## Service users and service tokens - -[Service users](/docs/guide/service-users) are programmatic identities authenticated with API tokens. They are useful for CI/CD pipelines, deployment bots, and other machine-to-machine communication needs. - -Unlike human users, service users do not have a Workspace-level role. Their access is controlled through Environment Roles. - -| Use case | Recommended identity | How access works | -| -------------------------------------------- | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | -| Interactive development or manual management | Human user | Access is based on the user's Workspace Role, plus any Environment Role for restricted Environments | -| Automation in CI/CD or deployment workflows | Service user authenticated with a service token | Access is based only on the service user's Environment Role | -| Deploying to an unrestricted Environment | Human user or service user | Both have Contributor access | -| Deploying to a restricted Environment | Human user or service user with explicit Contributor access | Viewer by default; must be granted Contributor to deploy or manage resources | - -This makes service users the recommended way to let automation deploy to a restricted Environment without granting broad Workspace permissions. - -## Proxy auth tokens for Web Functions - -[Web Functions](/docs/guide/webhooks) can be protected with proxy auth tokens, which authenticate inbound HTTP requests before they reach your function. - -On workspaces with RBAC enabled, proxy auth tokens are **scoped** — each token is explicitly associated with one or more Environments, and will only be accepted for Web Functions deployed to those Environments. This prevents a token intended for a staging endpoint from being used to call a production one. - -### Creating a scoped proxy auth token - -1. Navigate to **Settings → Proxy Auth Tokens** and click **New Token**. -2. Copy the token ID and secret — the secret is only shown once. -3. You will be prompted to select the Environments this token should be valid for. -4. Use the **Manage Environments** button on any existing scoped token to update its Environment associations. Changes take effect immediately, so removing an Environment will instantly revoke access for any clients using that token to call endpoints in that Environment. - -### Scoped vs. workspace-wide tokens - -| Token type | Who gets it | Valid for | -| -------------- | ---------------------------- | ---------------------------------------------------------- | -| Scoped | Workspaces with RBAC enabled | Only the Environments explicitly associated with the token | -| Workspace-wide | Workspaces without RBAC | Any Web Function in the workspace | - -Existing workspace-wide tokens continue to work as-is. New tokens created on workspaces with RBAC enabled are scoped by default. - -If RBAC is disabled on a workspace, scoped tokens fall back to workspace-wide access. - -## Cross-Environment access - -Restricted Environments prevent app and task identities in other Environments from accessing resources inside the restricted Environment. For more detail, see [Cross-Environment Lookups](/docs/guide/environments#cross-environment-lookups). - -In practice, this means a task can access objects in its own Environment and other unrestricted Environments, but code running in another Environment cannot use APIs such as `modal.App.lookup()`, `Secret.from_name()`, or `Volume.lookup()` to reach into a restricted Environment. - -This prevents privilege escalation from a less trusted Environment into a more sensitive one. - -### Cross-Environment behavior for app and task identities - -Access checks are evaluated against the **target** Environment. That means workloads running inside a restricted Environment can still access objects in an **unrestricted** Environment, but workloads running outside a restricted Environment cannot reach into it. - -| Source Environment | Target Environment | Cross-Environment access | -| ------------------ | ------------------ | ------------------------ | -| Unrestricted | Unrestricted | Allowed | -| Unrestricted | Restricted | Denied | -| Restricted | Unrestricted | Allowed | -| Restricted | Restricted | Denied | - -Same-Environment access is unaffected by these cross-Environment rules. - -### Example: inbound vs. outbound access - -Suppose you have two Environments: - -* `prod` — restricted -* `test` — unrestricted - -A task running in `test` cannot look up secrets, volumes, or Apps in `prod`. - -A task running in `prod` can still access objects in `test`, because `test` is not restricted. - -If both `prod` and `test` are restricted, then tasks in one cannot access objects in the other. - -## Protecting production secrets with restricted Environments - -A common RBAC setup is to place production secrets in a restricted production Environment and grant **Contributor** access only to the human users and service users that should be allowed to deploy or manage production. - -| Scenario | Result | -| ------------------------------------------------------------------------------ | ------- | -| Developer in `dev` tries to edit a secret in restricted `prod` | Denied | -| CI service user with Contributor access to restricted `prod` deploys to `prod` | Allowed | -| Task running in `prod` reads a secret in `prod` | Allowed | -| Task running in `prod` accesses objects in unrestricted `test` | Allowed | - -This setup lets you keep development and testing more open while protecting production resources, including secrets, from accidental or unauthorized access. - -## Common access patterns - -| Pattern | Allowed? | Notes | -| ------------------------------------------------------------------------------ | -------------------------------------- | ------------------------------------------------------------------------------------------- | -| Workspace Member views logs in a restricted Environment | Yes | Workspace Members have Viewer access by default in restricted Environments | -| Workspace Member deploys to a restricted Environment | No | Contributor access is required to deploy or modify resources | -| Workspace Owner or Manager deploys to a restricted Environment | Yes | Owners and Managers automatically have Contributor access | -| Service user deploys to a restricted Environment | Yes, if explicitly granted Contributor | Service users have Viewer access by default in restricted Environments | -| Task running in `dev` reads a secret in restricted `prod` | No | Cross-Environment access into a restricted Environment is denied | -| Task running in restricted `prod` accesses objects in unrestricted `test` | Yes | Cross-Environment access is allowed when the target Environment is unrestricted | -| User views dashboards or app details in a restricted Environment | Yes | Viewer access includes read-only views such as dashboards, logs, metrics, and configuration | -| Task accesses resources in its own Environment | Yes | Same-Environment access is unaffected by cross-Environment restrictions | -| Scoped proxy auth token used on a Web Function in an associated Environment | Yes | Token must be explicitly associated with the target Environment | -| Scoped proxy auth token used on a Web Function in a non-associated Environment | No | Token is not valid for Environments it has not been associated with | - -## FAQ - -**Can I make Environments completely private?** - -No. Workspace Members will always have at least **Viewer** access to restricted Environments. Fully private Environments are planned for the future. - -**How do service tokens work with restricted Environments?** - -Service tokens authenticate service users. In restricted Environments, service users have **Viewer** access by default unless you explicitly assign them **Contributor** access. This allows automated systems and CI/CD pipelines to deploy and manage production without granting broad Workspace permissions. - -**Can I use `modal.App.lookup()` across different restricted Environments?** - -No. Apps cannot look up, read from, or write to objects in a different restricted Environment. - -**Can code running in a restricted Environment access other Environments?** - -Yes, but only when the target Environment is not restricted. A restricted Environment blocks access **into** it from other Environments. diff --git a/.agents/skills/modal/references/guide/region-selection.md b/.agents/skills/modal/references/guide/region-selection.md deleted file mode 100644 index 0fac7a82fb76121148c716d4051724694149cd89..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/region-selection.md +++ /dev/null @@ -1,133 +0,0 @@ -# Region selection - -Modal runs containers globally across multiple different clouds. By default, all inputs to Modal Functions are routed through our servers in Virginia, USA (`us-east`) before being sent to a container for execution. - -You can observe the location identifier of a container [via an environment variable](/docs/guide/environment_variables). Logging this environment variable alongside latency information can reveal when geography is impacting your application performance. - -## Specifying a container region - -To run your Modal Function containers in a specific region, pass a `region=` argument to the `function` decorator: - -```python -@app.function(region=["us-west"]) -def f(): - ... -``` - -Sandboxes accept the same `region=` argument on `Sandbox.create`: - -```python notest -sb = modal.Sandbox.create(region=["us-west"], app=app) -``` - -This can be particularly useful when running a latency-sensitive app that needs to run near an external DB. - -### Pricing - -A multiplier on top of our [base usage pricing](/pricing) will be applied to any Function or Sandbox that has a container region defined. - -| **Region type** | **Multiplier** | -| ----------------------- | -------------- | -| Broad (e.g. `us`) | 1.5x | -| Narrow (e.g. `us-west`) | 1.75x | - -Here's an example: let's say you have a Function or Sandbox container that uses 1 T4, 1 CPU core, and 1GB memory. You've specified that it should run in `us-west`. The cost to run it for 1 hour would be `((T4 hourly cost) + (CPU hourly cost for one core) + (Memory hourly cost for one GB)) * 1.75`. - -If you specify multiple container regions and they span the two categories above, we will apply the smaller of the two multipliers. - -### Container region options - -Modal offers different levels of granularity for container regions. Use broader regions when possible, as this increases the pool of available resources your Function or Sandbox containers can be assigned to, which improves cold-start time and availability. - - - -``` - Broad Narrow Notes - =========================================================== - "us" United States - "us-east" - "us-central" - "us-south" - "us-west" ------------------------------------------------------------- - "eu" European Economic Area - "eu-west" - "eu-north" - "eu-south" ------------------------------------------------------------- - "ap" Asia-Pacific - "ap-northeast" - "ap-southeast" - "ap-south" - "ap-melbourne" - "jp" Japan - "au" Australia ------------------------------------------------------------- - "uk" United Kingdom ------------------------------------------------------------- - "ca" Canada ------------------------------------------------------------- - "me" Middle East ------------------------------------------------------------- - "sa" South America ------------------------------------------------------------- - "af" Africa ------------------------------------------------------------- - "mx" Mexico -``` - -Need access to more granular region definitions? Contact . - -## Regional routing - - - -In addition to letting you specify the region a Function's containers run in, Modal also allows you to specify which region your inputs and outputs will be routed through to reduce network overhead. By default, this is `us-east` (Virginia, USA). - -This doesn't apply to Sandboxes, as most operations go directly to the container (with some minor exceptions that are routed through `us-east`). - -### Specifying a routing region - -To have your Modal Function's traffic route through a specific region, pass a `routing_region=` argument to the `function` decorator. - -```python -@app.function(routing_region="us-west") -def f(): - ... -``` - -The valid options for `routing_region=` are: - -* `us-east` (Virginia, USA) -* `us-west` (Oregon, USA) -* `eu-west` (Dublin, Ireland) -* `ap-south` (Mumbai, India) - -### Current restrictions - -`routing_region=` can only be set during the initial deployment of a Function and cannot be changed in a subsequent redeployment. To change the routing region, a new Function should be created. Functions specifying a routing region outside of `us-east` can only be invoked with `.remote()` or `.map()` or via HTTP for [Web Functions](/docs/guide/webhooks). - -[Inputs and outputs larger than 2 MiB](/docs/guide/security#function-inputs-and-outputs) are still uploaded to object storage in `us-east`. - -## Optimizing latency - -Modal has a variety of tools to optimize network latency--even down to ~10ms in extreme cases like real-time robotics. Using container region selection in conjunction with a nearby routing region can eliminate significant network overhead. - -[Cloudping.co](https://www.cloudping.co) provides good estimates of the latency between regions. For example, the round-trip latency between AWS `us-east` (Virginia, USA) and `us-west` (Oregon, USA) is around 60ms. - -Splitting out regional deployments with separate Functions can be done like so: - -```python -def f(): - ... - -@app.function(region=["us-central", "us-west"], routing_region="us-west") -def f_us_west(): - return f() - -@app.function(region="ap", routing_region="ap-south") -def f_ap_south(): - return f() -``` - -To optimize latency further, please contact us on [Slack](https://modal.com/slack) or at . diff --git a/.agents/skills/modal/references/guide/resources.md b/.agents/skills/modal/references/guide/resources.md deleted file mode 100644 index 9812d49f6991e5089882742c70772b4ba5a85871..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/resources.md +++ /dev/null @@ -1,112 +0,0 @@ -# Configuring CPU, memory, and disk - -Each Modal Function or Sandbox container has a default request of 0.125 CPU cores and 128 MiB of memory. -Containers can exceed this minimum if the worker has available CPU or memory. -You can also guarantee access to more resources by requesting larger values, [similarly to Kubernetes](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/). - -This guide covers resource configuration for both [Functions](/docs/guide/apps#apps-functions-and-entrypoints) -and [Sandboxes](/docs/guide/sandboxes). For Sandbox-specific guidance on pricing and -cost optimization, see [Sandbox pricing and resources](/docs/guide/sandbox-resources). - -## CPU cores - -If you have code that must run on a larger number of cores, you can -request that using the `cpu` argument. This allows you to specify a -floating-point number of CPU cores: - -```python -import modal - -app = modal.App() - -@app.function(cpu=8.0) -def my_function(): - # code here will have access to at least 8.0 cores - ... -``` - -Note that this value corresponds to physical cores, not vCPUs. - -Modal also will set several environment variables that control multi-threading -behavior in linear algebra libraries (e.g., `OPENBLAS_NUM_THREADS`, -`OMP_NUM_THREADS`, `MKL_NUM_THREADS`) based on your CPU request. - -## Memory - -If you have code that needs more guaranteed memory, you can request it using the -`memory` argument. This expects an integer number of megabytes: - -```python -import modal - -app = modal.App() - -@app.function(memory=32768) -def my_function(): - # code here will have access to at least 32 GiB of RAM - ... -``` - -## How much can I request? - -For both CPU and memory, a maximum is enforced at Function or Sandbox creation time to -ensure your containers can be scheduled for execution. Requests exceeding the -maximum will be rejected with an -[`InvalidError`](/docs/reference/modal.exception#modalexceptioninvaliderror). - -## Billing - -For CPU and memory, you'll be charged based on whichever is higher: your request or actual usage. - -Disk requests are billed by increasing the memory request at a 20:1 ratio. For example, requesting 500 GiB of disk will increase the memory request to 25 GiB, if it is not already set higher. - -## Resource limits - -### CPU limits - -Modal containers have a default soft CPU limit that is set at 16 physical cores above the CPU request. -Given that the default CPU request is 0.125 cores, the default soft CPU limit is 16.125 cores. -Above this limit, the host will begin to throttle the CPU usage of the container. - -You can alternatively set the CPU limit explicitly: - -```python -cpu_request = 1.0 -cpu_limit = 4.0 -@app.function(cpu=(cpu_request, cpu_limit)) -def f(): - ... -``` - -### Memory limits - -Modal containers can have a hard memory limit which will 'Out of Memory' (OOM) kill -containers which attempt to exceed the limit. This functionality is useful when a process -has a serious memory leak. You can set the limit and have the container killed to avoid paying -for the leaked GBs of memory. - -Specify this limit using the `memory` parameter on [`@app.function()`](/docs/reference/modal.App#function) or [`Sandbox.create()`](/docs/reference/modal.Sandbox#create): - -```python -mem_request = 1024 -mem_limit = 2048 -@app.function( - memory=(mem_request, mem_limit), -) -def f(): - ... -``` - -### Disk limits - -Running Modal containers have access to many GBs of SSD disk, but the amount -of writes is limited by: - -1. The size of the underlying worker's SSD disk capacity -2. A per-container disk quota that defaults to 512 GiB. - -Hitting either limit will cause the container's disk writes to be rejected, which -typically manifests as an `OSError`. - -Increased disk sizes can be requested with the `ephemeral_disk` parameter on [`@app.function()`](/docs/reference/modal.App#function). The maximum -disk size is 3.0 TiB (3,145,728 MiB). Larger disks are intended to be used for [dataset processing](/docs/guide/dataset-ingestion). diff --git a/.agents/skills/modal/references/guide/restricted-access.md b/.agents/skills/modal/references/guide/restricted-access.md deleted file mode 100644 index 3de595ea1274074a3152d8e96aad496ac63322de..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/restricted-access.md +++ /dev/null @@ -1,171 +0,0 @@ -# Run untrusted code with Restricted Functions - -This guide page documents Restricted Functions, which can be used to safely execute untrusted code in Modal Functions. - -## Create a Restricted Function - -To create a Restricted Function, set `restrict_modal_access=True` in the Function definition: - -```python -@app.function(restrict_modal_access=True) -def run_untrusted_code(code_input: str): - # This code cannot access Modal resources - return eval(code_input) -``` - -When `restrict_modal_access` is enabled, the Function cannot - -* access Modal resources (Queues, Dicts, etc.) -* call other Functions -* access Modal's internal APIs - -## Sandboxes offer an alternative interface for untrusted code - -Modal provides two primitives for running untrusted code: Restricted Functions and [Sandboxes](/docs/guide/sandboxes). -While both can be used for running untrusted code, they provide different interfaces: -Sandboxes provide a process interface, -while Restricted Functions provide a function-calling interface. -Process interfaces are especially useful for stateful, multi-stage communication, -while function-calling interfaces are especially useful for stateless, input/output communication. - -These differences are summarized in the table below. - -| Feature | Restricted Function | Sandbox | -| --------- | ------------------------------ | ---------------------------------------------- | -| State | Stateless | Stateful | -| Interface | Function-like | Container-like | -| Setup | Simple decorator | Requires explicit creation/termination | -| Use case | Quick, isolated code execution | Interactive development, long-running sessions | - -## Best Practices - -When running untrusted code, consider these additional security measures: - -1. Use `single_use_containers=True` to ensure each container only handles one request. Containers that get reused could cause information leakage between users. - -```python -@app.function(restrict_modal_access=True, single_use_containers=True) -def isolated_function(input_data): - # Each input gets a fresh container - return process(input_data) -``` - -Note: Prior to v1.3.0, single-use containers were configured by setting `max_inputs=1`. - -2. Set appropriate timeouts to prevent long-running operations: - -```python -@app.function( - restrict_modal_access=True, - timeout=30, # 30 second timeout - single_use_containers=True -) -def time_limited_function(input_data): - return process(input_data) -``` - -3. Consider using `block_network=True` to prevent the container from making outbound network requests: - -```python -@app.function( - restrict_modal_access=True, - block_network=True, - single_use_containers=True -) -def network_isolated_function(input_data): - return process(input_data) -``` - -4. Minimize the App source that's included in the container - -A restricted Modal Function will have read access to its source files in the -container, so you'll want to avoid including anything that would be harmful -if exfiltrated by the untrusted process. - -If deploying an App from within a [larger package](/docs/guide/project-structure), -the entire package source may be automatically included by default. A best -practice would be to make the untrusted Function part of a standalone App that -includes the minimum necessary files to run: - -```python -restricted_app = modal.App("restricted-app", include_source=False) - -image = ( - modal.Image.debian_slim() - .add_local_file("restricted_executor.py", "/root/restricted_executor.py") -) - -@restricted_app.function( - restrict_modal_access=True, - block_network=True, - single_use_containers=True -) -def isolated_function(input_data): - return process(input_data) -``` - -## Example: Running LLM-generated Code - -Below is a complete example of running code generated by a language model: - -```python -import modal - -app = modal.App("restricted-access-example") - - -@app.function(restrict_modal_access=True, single_use_containers=True, timeout=30, block_network=True) -def run_llm_code(generated_code: str): - try: - # Create a restricted environment - execution_scope = {} - - # Execute the generated code - exec(generated_code, execution_scope) - - # Return the result if it exists - return execution_scope.get("result", None) - except Exception as e: - return f"Error executing code: {str(e)}" - - -@app.local_entrypoint() -def main(): - # Example LLM-generated code - code = """ -def calculate_fibonacci(n): - if n <= 1: - return n - return calculate_fibonacci(n-1) + calculate_fibonacci(n-2) - -result = calculate_fibonacci(10) - """ - - result = run_llm_code.remote(code) - print(f"Result: {result}") - -``` - -This example locks down the container to ensure that the code is safe to execute by: - -* Restricting Modal access -* Using a fresh container for each execution -* Setting a timeout -* Blocking network access -* Catching and handling potential errors - -## Error Handling - -When a Restricted Function attempts to access Modal resources, it will raise an `AuthError`: - -```python -@app.function(restrict_modal_access=True) -def restricted_function(q: modal.Queue): - try: - # This will fail because the Function is restricted - return q.get() - except modal.exception.AuthError as e: - return f"Access denied: {e}" -``` - -The error message will indicate that the operation is not permitted due to restricted Modal access. diff --git a/.agents/skills/modal/references/guide/retries.md b/.agents/skills/modal/references/guide/retries.md deleted file mode 100644 index c909df774b7f2314d6ae2f426b0abd89f16121b5..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/retries.md +++ /dev/null @@ -1,41 +0,0 @@ -# Failures and retries - -Failure is part of life. Sometimes you just have to retry. This guide page documents how to do this on Modal. - -For reference documentation on the `modal.Retries` object, see [this page](/docs/reference/modal.Retries). - -## Automatically recover from flakes with `retries` - -You can configure Modal to automatically retry Function failures if you set the -`retries` option when declaring your Function: - -```python -@app.function(retries=3) -def my_flaky_function(): - pass -``` - -The basic configuration shown provides a fixed 1s delay between retry attempts. -For fine-grained control over retry delays, including exponential backoff -configuration, use [`modal.Retries`](/docs/reference/modal.Retries). - -## Handle failures in `Function.map` - -By default, failures are propagated back to the caller. -To treat exceptions like successful results and aggregate them in the results list instead, -pass in [`return_exceptions=True`](/docs/guide/scale#exceptions). - -When used with [`Function.map()`](/docs/guide/scale#parallel-execution-of-inputs), -each input is retried independently. - -## Container crashes - -If a `modal.Function` container crashes (either on start-up, e.g. while handling imports in global scope, or during execution, e.g. an out-of-memory error), -Modal will reschedule the container and any work it was currently assigned. - -For [ephemeral apps](/docs/guide/apps#ephemeral-apps), container crashes will be retried until a failure rate is exceeded, -after which all pending inputs will be failed and the exception will be propagated to the caller. - -For [deployed apps](/docs/guide/apps#deployed-apps), container crashes will be retried indefinitely, so as to not disrupt service. -Modal will instead apply a crash-loop backoff and the rate of new container creation for the Function will be slowed down. -Crash-looping containers are displayed in the [App dashboard](/apps). diff --git a/.agents/skills/modal/references/guide/s3-gateway-endpoints.md b/.agents/skills/modal/references/guide/s3-gateway-endpoints.md deleted file mode 100644 index 17999e2de7fdb6309a06806aad9b1234ccb182f2..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/s3-gateway-endpoints.md +++ /dev/null @@ -1,27 +0,0 @@ -# S3 Gateway endpoints - -When running workloads in AWS, our system automatically uses a corresponding -[S3 Gateway endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html) -to ensure low costs, optimal performance, and network reliability between Modal and S3. - -Workloads running on Modal should not incur egress or ingress fees associated -with S3 operations. No configuration is needed in order for your app to use S3 Gateway endpoints. -S3 Gateway endpoints are automatically used when your app runs on AWS. - -## Endpoint configuration - -Only use the region-specific endpoint (`s3..amazonaws.com`) or the -global AWS endpoint (`s3.amazonaws.com`). Using an S3 endpoint from one region -in another **will not use the S3 Gateway Endpoint incurring networking costs**. - -Avoid specifying regional endpoints manually, as this can lead to unexpected cost -or performance degradation. - -## Inter-region costs - -S3 Gateway endpoints guarantee no costs for network traffic within the same AWS region. -However, if your Modal Function runs in one region but your bucket resides in a -different region you will be billed for inter-region traffic. - -You can prevent this by scheduling your Modal App in the same region of your -S3 bucket with [Region selection](https://modal.com/docs/guide/region-selection#region-selection). diff --git a/.agents/skills/modal/references/guide/saml-sso.md b/.agents/skills/modal/references/guide/saml-sso.md deleted file mode 100644 index 62866fbcdf1e70d700ac3064ff6f300472c9a99b..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/saml-sso.md +++ /dev/null @@ -1,84 +0,0 @@ -# Custom SAML SSO - - -Custom SAML SSO is available on the Enterprise plan. Contact sales@modal.com for more information. - - -If you use an identity provider (IdP) other than Okta, you can configure custom SAML SSO for your Modal workspace. - -For Okta-specific setup, see our [Okta SSO documentation](/docs/guide/okta-sso). - -## Prerequisites - -* A Workspace that's on an [Enterprise](/pricing) plan -* Admin access to the Workspace you want to configure with SSO -* Admin privileges for your identity provider - -## Supported features - -* Identity Provider (IdP) initiated SSO -* Service Provider (SP) initiated SSO -* Just-In-Time account provisioning - -## Configuration - -### Modal SAML settings - -Configure your IdP with the following settings: - -| Setting | Value | -| --------- | ------------------------------------------------- | -| Entity ID | `https://www.modal.com` | -| ACS URL | `https://modal.com/api/okta/saml/sso/` | - -Replace `` with your Modal Workspace name. - -### Required SAML attributes - -Your IdP must send the following SAML attributes: - -| Attribute | Description | -| --------- | -------------------- | -| email | User's email address | -| firstName | User's first name | -| lastName | User's last name | - -### Configuration steps - -#### Step 1: Configure your IdP - -1. Create a new SAML application in your identity provider -2. Set the Entity ID to `https://www.modal.com` -3. Set the ACS URL to `https://modal.com/api/okta/saml/sso/` (replace `` with your Workspace name) -4. Configure the required SAML attributes (email, firstName, lastName) -5. Ensure your IdP signs SAML assertions - -#### Step 2: Link your Workspace to your IdP - -1. Obtain the SAML Metadata URL from your IdP -2. Sign in to https://modal.com and visit your [Workspace Management](/settings/workspace-management/identity-and-provisioning) page's `Identity and Provisioning` tab -3. Paste the Metadata URL in the input and click "Save Changes" - -#### Step 3: Test the integration - -1. Assign users in your IdP -2. Test IdP-initiated SSO by clicking the Modal application in your IdP dashboard -3. Test SP-initiated SSO by visiting the login URL below - -#### Step 4: Read this before you enable "Require SSO" - -Enabling "Require SSO" will force all users to sign in via SSO. Ensure that you -have admin access to your Modal Workspace through your identity provider before -enabling. - -## Login URL - -This URL can be used so that users can sign-in to the correct workspace from your IdP. - -`https://modal.com/login/sso?workspace=` (replace `` with your workspace name) - -## Troubleshooting - -### Microsoft Entra SAML - -Make sure the SAML attributes are mapped correctly. For example, `email` should be lowercase and the SAML attribute should not have a namespace. Read more about Microsoft Entra SAML attributes [here](https://learn.microsoft.com/en-us/entra/identity-platform/saml-claims-customization). diff --git a/.agents/skills/modal/references/guide/sandbox-files.md b/.agents/skills/modal/references/guide/sandbox-files.md deleted file mode 100644 index 40603183a9fa7e7771d378b30da86f6d539855fa..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/sandbox-files.md +++ /dev/null @@ -1,497 +0,0 @@ -# Filesystem Access - -There are multiple options for uploading files to a Sandbox and accessing them -from outside the Sandbox. - -## Filesystem API - - - -This API brings significant reliability improvements compared to the previous Sandbox filesystem API, which was available in releases prior to v1.4.0 and is now deprecated. - - - -The most convenient way to pass data in and out of the Sandbox during -execution is to use our filesystem API: - - - {#snippet python()} - -```python -import modal - -app = modal.App.lookup("sandbox-fs-demo", create_if_missing=True) - -sb = modal.Sandbox.create(app=app) - -# Write text to a file in the Sandbox. -sb.filesystem.write_text("Hello World!\n", "/tmp/test.txt") - -# Read the file back from the Sandbox into a string. -contents = sb.filesystem.read_text("/tmp/test.txt") -print(contents) - -sb.terminate() -sb.detach() -``` - -{/snippet} -{#snippet javascript()} - -```javascript notest -import { ModalClient } from "modal"; - -const modal = new ModalClient(); -const app = await modal.apps.fromName("sandbox-fs-demo", { - createIfMissing: true, -}); -const image = modal.images.fromRegistry("python:3.13-slim"); - -const sb = await modal.sandboxes.create(app, image); - -// Write text to a file in the Sandbox. -await sb.filesystem.writeText("Hello World!\n", "/tmp/test.txt"); - -// Read the file back from the Sandbox into a string. -const contents = await sb.filesystem.readText("/tmp/test.txt"); -console.log(contents); - -await sb.terminate(); -``` - -{/snippet} -{#snippet go()} - -```go notest -package main - -import ( - "context" - "fmt" - - modal "github.com/modal-labs/modal-client/go" -) - -func main() { - ctx := context.Background() - mc, _ := modal.NewClient() - - app, _ := mc.Apps.FromName(ctx, "sandbox-fs-demo", &modal.AppFromNameParams{ - CreateIfMissing: true, - }) - image := mc.Images.FromRegistry("python:3.13-slim", nil) - - sb, _ := mc.Sandboxes.Create(ctx, app, image, nil) - defer sb.Terminate(ctx, nil) - - fs := sb.Filesystem() - - // Write text to a file in the Sandbox. - fs.WriteText(ctx, "Hello World!\n", "/tmp/test.txt", nil) - - // Read the file back from the Sandbox into a string. - contents, _ := fs.ReadText(ctx, "/tmp/test.txt", nil) - fmt.Println(contents) -} -``` - -{/snippet} - -It has convenience APIs for streaming file copies in both directions: - - - {#snippet python()} - -```python -from pathlib import Path -import modal - -# Write a local file. -with open("local-file.txt", "w") as f: - f.write("Hello World!\n") - -app = modal.App.lookup("sandbox-fs-demo", create_if_missing=True) - -sb = modal.Sandbox.create(app=app) - -# Copy the local file into the Sandbox. -sb.filesystem.copy_from_local("local-file.txt", "/tmp/file-in-sandbox.txt") - -# Copy it back to the local filesystem. -sb.filesystem.copy_to_local("/tmp/file-in-sandbox.txt", "local-file-copy.txt") - -print(Path("local-file-copy.txt").read_text()) - -sb.terminate() -sb.detach() -``` - -{/snippet} -{#snippet javascript()} - -```javascript notest -import { readFile, writeFile } from "node:fs/promises"; - -const sb = await modal.sandboxes.create(app, image); - -// Write a local file. -await writeFile("local-file.txt", "Hello World!\n", "utf-8"); - -// Copy the local file into the Sandbox. -await sb.filesystem.copyFromLocal("local-file.txt", "/tmp/file-in-sandbox.txt"); - -// Copy it back to the local filesystem. -await sb.filesystem.copyToLocal( - "/tmp/file-in-sandbox.txt", - "local-file-copy.txt", -); - -console.log(await readFile("local-file-copy.txt", "utf-8")); - -await sb.terminate(); -``` - -{/snippet} -{#snippet go()} - -```go notest -sb, _ := mc.Sandboxes.Create(ctx, app, image, nil) -defer sb.Terminate(ctx, nil) - -fs := sb.Filesystem() - -// Write a local file. -os.WriteFile("local-file.txt", []byte("Hello World!\n"), 0o644) - -// Copy the local file into the Sandbox. -fs.CopyFromLocal(ctx, "local-file.txt", "/tmp/file-in-sandbox.txt", nil) - -// Copy it back to the local filesystem. -fs.CopyToLocal(ctx, "/tmp/file-in-sandbox.txt", "local-file-copy.txt", nil) - -data, _ := os.ReadFile("local-file-copy.txt") -fmt.Println(string(data)) -``` - -{/snippet} - -It also offers APIs for inspecting and managing files: - - - {#snippet python()} - -```python -import modal - -app = modal.App.lookup("sandbox-fs-demo", create_if_missing=True) - -sb = modal.Sandbox.create(app=app) - -# Set up a structured project. -sb.filesystem.make_directory("/tmp/project/results") - -# Let the Sandbox do some work and write outputs to files. -sb.filesystem.write_text("42\n", "/tmp/project/results/answer.txt") -sb.filesystem.write_text("debug info\n", "/tmp/project/results/debug.log") - -# Inspect what was produced. -for entry in sb.filesystem.list_files("/tmp/project/results"): - print(entry.name, entry.type.value, entry.size) - -# Check that the result file has content before downloading it. -info = sb.filesystem.stat("/tmp/project/results/answer.txt") -if info.size > 0: - answer = sb.filesystem.read_text("/tmp/project/results/answer.txt") - print(answer) - -# Clean up the whole project. -sb.filesystem.remove("/tmp/project", recursive=True) - -sb.terminate() -sb.detach() -``` - -{/snippet} -{#snippet javascript()} - -```javascript notest -const sb = await modal.sandboxes.create(app, image); - -// Set up a structured project. -await sb.filesystem.makeDirectory("/tmp/project/results"); - -// Let the Sandbox do some work and write outputs to files. -await sb.filesystem.writeText("42\n", "/tmp/project/results/answer.txt"); -await sb.filesystem.writeText("debug info\n", "/tmp/project/results/debug.log"); - -// Inspect what was produced. -const entries = await sb.filesystem.listFiles("/tmp/project/results"); -for (const entry of entries) { - console.log(entry.name, entry.type, entry.size); -} - -// Check that the result file has content before downloading it. -const info = await sb.filesystem.stat("/tmp/project/results/answer.txt"); -if (info.size > 0) { - const answer = await sb.filesystem.readText( - "/tmp/project/results/answer.txt", - ); - console.log(answer); -} - -// Clean up the whole project. -await sb.filesystem.remove("/tmp/project", { recursive: true }); - -await sb.terminate(); -``` - -{/snippet} -{#snippet go()} - -```go notest -sb, _ := mc.Sandboxes.Create(ctx, app, image, nil) -defer sb.Terminate(ctx, nil) - -fs := sb.Filesystem() - -// Set up a structured project. -fs.MakeDirectory(ctx, "/tmp/project/results", nil) - -// Let the Sandbox do some work and write outputs to files. -fs.WriteText(ctx, "42\n", "/tmp/project/results/answer.txt", nil) -fs.WriteText(ctx, "debug info\n", "/tmp/project/results/debug.log", nil) - -// Inspect what was produced. -entries, _ := fs.ListFiles(ctx, "/tmp/project/results", nil) -for _, entry := range entries { - fmt.Println(entry.Name, entry.Type, entry.Size) -} - -// Check that the result file has content before downloading it. -info, _ := fs.Stat(ctx, "/tmp/project/results/answer.txt", nil) -if info.Size > 0 { - answer, _ := fs.ReadText(ctx, "/tmp/project/results/answer.txt", nil) - fmt.Println(answer) -} - -// Clean up the whole project. -fs.Remove(ctx, "/tmp/project", &modal.SandboxFilesystemRemoveParams{Recursive: true}) -``` - -{/snippet} - -These APIs may be used to read files of up to 5GB and write files of any size. - -However, if you have a large dataset that you want to use repeatedly from many sandboxes, -consider [using Volumes](#using-volumes). - -## Using Volumes - -It's possible to use Modal [Volume](/docs/reference/modal.Volume)s or -[CloudBucketMount](/docs/guide/cloud-bucket-mounts)s with Sandboxes. - -Volumes and CloudBucketMounts allow you to upload data once and access that -data efficiently from many sandboxes. - -To access a Volume from a Sandbox, you can use the `volumes` parameter of `Sandbox.create`: - -```python notest -# Find or create a Volume with the name "my-volume". -vol = modal.Volume.from_name("my-volume", create_if_missing=True) -sb = modal.Sandbox.create( - volumes={"/cache": vol}, - app=my_app, -) -# Read a file in the Volume. -p = sb.exec("bash", "-c", "cat /cache/some-file.txt") -print(p.stdout.read()) -p.wait() - -# Write a file to the Volume. -p = sb.exec("bash", "-c", "echo foo > /cache/a.txt") -p.wait() -sb.terminate(wait=True) -sb.detach() - -# Access the Volume file from outside the Sandbox. -for data in vol.read_file("a.txt"): - print(data) -``` - -File syncing behavior differs between Volumes and CloudBucketMounts. For -Volumes, files are only synced back to the Volume when the Sandbox terminates. -For CloudBucketMounts, files are synced automatically. - -### Mounting a subdirectory - -You can mount a subdirectory of a Volume instead of the entire Volume using -[`with_mount_options`](/docs/guide/volumes#mount-options). This is especially -useful when many Sandboxes share a single Volume but each Sandbox should only -access its own data: - - - {#snippet python()} - -```python notest -sb_app = modal.App.lookup("my-app", create_if_missing=True) - -vol = modal.Volume.from_name("shared-volume", create_if_missing=True) - -# Each Sandbox only sees its own subdirectory of the Volume. -sb = modal.Sandbox.create( - volumes={"/data": vol.with_mount_options(sub_path="/users/user_123")}, - app=sb_app, -) -# /data inside the Sandbox maps to /users/user_123 in the Volume. -# The Sandbox cannot see or modify files belonging to other users. -p = sb.exec("bash", "-c", "echo hello > /data/output.txt") -p.wait() -sb.terminate(wait=True) -sb.detach() -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const app = await modal.apps.fromName("my-app", { - createIfMissing: true, -}); -const vol = await modal.volumes.fromName("shared-volume", { - createIfMissing: true, -}); -const image = modal.images.fromRegistry("python:3.13-slim"); - -// Each Sandbox only sees its own subdirectory of the Volume. -const sb = await modal.sandboxes.create(app, image, { - volumes: { "/data": vol.withMountOptions({ subPath: "/users/user_123" }) }, -}); -// /data inside the Sandbox maps to /users/user_123 in the Volume. -// The Sandbox cannot see or modify files belonging to other users. -const p = await sb.exec(["bash", "-c", "echo hello > /data/output.txt"]); -await p.wait(); -await sb.terminate({ wait: true }); -``` - -{/snippet} - -{#snippet go()} - -```go notest -app, _ := mc.Apps.FromName(ctx, "volume-subdir-test", &modal.AppFromNameParams{CreateIfMissing: true}) - -vol, _ := mc.Volumes.FromName(ctx, "shared-volume", &modal.VolumeFromNameParams{ - CreateIfMissing: true, -}) -image := mc.Images.FromRegistry("python:3.13-slim", nil) - -// Each Sandbox only sees its own subdirectory of the Volume. -subPath := "/users/user_123" -sb, _ := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Volumes: map[string]*modal.Volume{ - "/data": vol.WithMountOptions(&modal.VolumeMountOptions{SubPath: &subPath}), - }, -}) -defer sb.Terminate(ctx, nil) - -// /data inside the Sandbox maps to /users/user_123 in the Volume. -// The Sandbox cannot see or modify files belonging to other users. -p, _ := sb.Exec(ctx, []string{"bash", "-c", "echo hello > /data/output.txt"}, nil) -p.Wait(ctx) -``` - -{/snippet} - -For more details on Volume mount options, see the -[Volumes guide](/docs/guide/volumes#mount-options). - -### Committing Volume changes with `sync` (v2 only) - -For [Volumes v2](/docs/guide/volumes#volumes-v2-overview), you can explicitly -commit changes at any point during Sandbox execution by running the `sync` -command on the mountpoint. This persists all data and metadata changes to the -Volume's storage without waiting for the Sandbox to terminate: - -```python notest -sb = modal.Sandbox.create( - volumes={"/data": modal.Volume.from_name("my-v2-volume")}, - app=my_app, -) - -# Write files to the volume -sb.exec("bash", "-c", "echo 'hello' > /data/output.txt").wait() - -# Commit changes immediately -p = sb.exec("sync", "/data") -p.wait() -if p.returncode != 0: - raise Exception(f"sync failed with exit code {p.returncode}") - -# Changes are now persisted and visible to other containers -sb.terminate() -sb.detach() -``` - -This is particularly useful for long-running Sandboxes where you want to -persist intermediate results, or when you need changes to be visible to other -containers before the Sandbox terminates. - -## Adding files to an Image - -In some cases, you may want to [add a file to an Image itself](/docs/guide/images#add-local-files-with-add_local_dir-and-add_local_file). -This is useful if the file will be used by many Sandboxes, or if you -want to access that file from the Sandbox's entrypoint command. - -This can be done using the -[`add_local_file`](/docs/reference/modal.Image#add_local_file) and -[`add_local_dir`](/docs/reference/modal.Image#add_local_dir) methods on the -[`Image`](/docs/reference/modal.Image) class: - -```python notest -# Eagerly build the image - otherwise the Image will lazily build when the -# Sandbox is created. -image = ( - modal.Image.debian_slim() - .add_local_dir( - local_path="/home/user/my_dir", - remote_path="/app", - ) - .build(my_app) -) - -sb = modal.Sandbox.create(app=my_app, image=image) -p = sb.exec("ls", "/app") -print(p.stdout.read()) -p.wait() -sb.detach() -``` - - - - diff --git a/.agents/skills/modal/references/guide/sandbox-networking.md b/.agents/skills/modal/references/guide/sandbox-networking.md deleted file mode 100644 index 61f9d0dba28ca7f0dd0fbb28a2aa6f927cb051e0..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/sandbox-networking.md +++ /dev/null @@ -1,376 +0,0 @@ -# Networking and security - -Sandboxes are built to be secure-by-default, meaning that a default Sandbox has -no ability to accept incoming network connections or access your Modal resources. - -## Outbound access control - -By default, Sandboxes can make outbound connections to any public IP address. -Modal provides three levels of outbound network restriction: - -| Level | Parameter | What it controls | -| ----------------------------- | --------------------------- | -------------------------------------------------------------- | -| **Full block** | `block_network=True` | Drops all outbound traffic. | -| **IP-range allowlist** | `outbound_cidr_allowlist` | Only allows traffic to the listed CIDR ranges (any protocol). | -| **Domain allowlist** *(Beta)* | `outbound_domain_allowlist` | Only allows TLS traffic (port 443) to the listed domain names. | - -`outbound_cidr_allowlist` and `outbound_domain_allowlist` can be combined additively - traffic that meets either criteria will be let through. - -### Blocking all network access - -Set `block_network=True` to prevent the Sandbox from making any outbound -connections: - - - {#snippet python()} - -```python notest -sb = modal.Sandbox.create( - "python", "my_script.py", - block_network=True, - app=app, -) -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const sb = await modal.sandboxes.create(app, image, { - command: ["python", "my_script.py"], - blockNetwork: true, -}); -``` - -{/snippet} - -{#snippet go()} - -```go notest -sb, err := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Command: []string{"python", "my_script.py"}, - BlockNetwork: true, -}) -``` - -{/snippet} - -When `block_network` is enabled, `outbound_cidr_allowlist`, -`outbound_domain_allowlist`, and `inbound_cidr_allowlist` cannot be used. - -### Restricting by IP range (CIDR allowlist) - -Use `outbound_cidr_allowlist` to restrict outbound traffic to a set of IP -ranges. All traffic to IPs outside these ranges (except traffic allowed by `outbound_domain_allowlist`) is blocked. - - - {#snippet python()} - -```python notest -sb = modal.Sandbox.create( - "sleep", "infinity", - outbound_cidr_allowlist=["52.0.0.0/8", "10.0.1.0/24"], - app=app, -) -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const sb = await modal.sandboxes.create(app, image, { - command: ["sleep", "infinity"], - outboundCidrAllowlist: ["52.0.0.0/8", "10.0.1.0/24"], -}); -``` - -{/snippet} - -{#snippet go()} - -```go notest -sb, err := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Command: []string{"sleep", "infinity"}, - OutboundCIDRAllowlist: []string{"52.0.0.0/8", "10.0.1.0/24"}, -}) -``` - -{/snippet} - -### Restricting by domain name (domain allowlist) - - - -Use `outbound_domain_allowlist` to restrict outbound TLS traffic to a set of -domain names: - - - {#snippet python()} - -```python notest -sb = modal.Sandbox.create( - "sleep", "infinity", - outbound_domain_allowlist=["api.openai.com", "*.github.com"], - app=app, -) -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const sb = await modal.sandboxes.create(app, image, { - command: ["sleep", "infinity"], - outboundDomainAllowlist: ["api.openai.com", "*.github.com"], -}); -``` - -{/snippet} - -{#snippet go()} - -```go notest -sb, err := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Command: []string{"sleep", "infinity"}, - OutboundDomainAllowlist: []string{"api.openai.com", "*.github.com"}, -}) -``` - -{/snippet} - -When a domain allowlist is set: - -* **TLS (port 443)** connections are allowed only to the listed domains. - Connections to non-allowlisted domains are securely blocked and logged to - the Sandbox's system output stream. -* **Non-TLS traffic** (HTTP, raw TCP, UDP) to IPs that are not on a CIDR - allowlist is **blocked**. - -Entries prefixed with `*.` match the parent domain and any subdomain: - -| Allowlist entry | Matches | Does not match | -| --------------- | ------------------------------------------------- | ----------------- | -| `example.com` | `example.com` | `sub.example.com` | -| `*.example.com` | `example.com`, `a.example.com`, `a.b.example.com` | `evilexample.com` | - -## Inbound access control - -Use `inbound_cidr_allowlist` to restrict which IP addresses can connect -**inbound** to the Sandbox through tunnels and Sandbox Connect Tokens: - - - {#snippet python()} - -```python notest -sb = modal.Sandbox.create( - "python", "-m", "http.server", "8080", - encrypted_ports=[8080], - inbound_cidr_allowlist=["203.0.113.0/24"], - app=app, -) -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const sb = await modal.sandboxes.create(app, image, { - command: ["python", "-m", "http.server", "8080"], - encryptedPorts: [8080], - inboundCidrAllowlist: ["203.0.113.0/24"], -}); -``` - -{/snippet} - -{#snippet go()} - -```go notest -sb, err := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Command: []string{"python", "-m", "http.server", "8080"}, - EncryptedPorts: []int{8080}, - InboundCIDRAllowlist: []string{"203.0.113.0/24"}, -}) -``` - -{/snippet} - -## Connecting to Sandboxes with HTTP and WebSockets - -You can make authenticated HTTP and WebSocket requests to a Sandbox by generating -Sandbox Connect Tokens. They work like this: - -```python notest -# Start a Sandbox with a server running on port 8080. -sb = modal.Sandbox.create( - "bash", "-c", "python3 -m http.server 8080", - app=my_app, -) - -# Create a connect token, optionally including arbitrary user metadata. -creds = sb.create_connect_token(user_metadata={"user_id": "foo"}) - -# Make an HTTP request, passing the token in the Authorization header. -requests.get(creds.url, headers={"Authorization": f"Bearer {creds.token}"}) - -# You can also put the token in a `_modal_connect_token` query param. -url = f"{creds.url}/?_modal_connect_token={creds.token}" -ws_url = url.replace("https://", "wss://") -with websockets.connect(ws_url) as socket: - socket.send("Hello world!") - -sb.detach() -``` - -The server running on port 8080 in the container will receive an authenticated -request with an unspoofable `X-Verified-User-Data` header whose value is the -JSON-serialized Python dict that was passed as `user_metadata` to the -`create_connect_token()` function. This can be used by the application to -determine access control, for example. - -There are a few things to remember with Sandbox Connect Tokens: - -1. The server inside the container must be listening on port 8080. -2. The token may be sent in an `Authorization` header, in a `_modal_connect_token` - query param, or in a `_modal_connect_token` cookie. -3. If `_modal_connect_token` is set as a query param, the resulting response will - include a `Set-Cookie` header that sets it as a cookie. -4. The `user_metadata` must be JSON-serializable and must be less than 512 - characters after serialization. - -### Forwarding ports - -While it is recommended to use [Sandbox Connect Tokens](#connecting-to-sandboxes-with-http-and-websockets) -for HTTP requests and WebSocket connections to the container, you can also expose -raw TCP ports to the internet. This is useful if, for example, you want to run a -server inside the Sandbox that expects a raw TCP connection and handles -authentication itself. - -Use the `encrypted_ports` and `unencrypted_ports` parameters of `Sandbox.create` -to specify which ports to forward. You can then access the public URL of a tunnel -using the [`Sandbox.tunnels`](/docs/reference/modal.Sandbox#tunnels) method: - -```python notest -import requests -import time - -sb = modal.Sandbox.create( - "python", - "-m", - "http.server", - "12345", - encrypted_ports=[12345], - app=my_app, -) - -tunnel = sb.tunnels()[12345] - -time.sleep(1) # Wait for server to start. - -print(f"Connecting to {tunnel.url}...") -print(requests.get(tunnel.url, timeout=5).text) - -sb.detach() -``` - -It is also possible to create an encrypted port that uses `HTTP/2` rather than `HTTP/1.1` with the `h2_ports` option. This will return -a URL that you can make H2 (HTTP/2 + TLS) requests to. If you want to run an `HTTP/2` server inside a sandbox, this feature may be useful. -Here is an example: - -```python notest -import time - -port = 4359 -sb = modal.Sandbox.create( - app=my_app, - image=my_image, - h2_ports=[port], -) -p = sb.exec("python", "my_http2_server.py") - -tunnel = sb.tunnels()[port] -time.sleep(1) -print(f"Tunnel URL: {tunnel.url}") - -sb.detach() -``` - -For more details on how tunnels work, see the [tunnels guide](/docs/guide/tunnels). - -### Custom domains - - - -Custom domains for Sandbox tunnels are available on the Team and Enterprise plans. Visit workspace settings to upgrade. - - - - - -The infrastructure is production-grade, but onboarding requires a manual setup step. - - - -By default, Sandbox tunnels are served from subdomains of `w.modal.host`. -In some cases, it's necessary to have a tunnel served through a custom domain -for security reasons. This is possible with manual setup. - -Note that tunnel custom domains are distinct from other custom domains in Modal. -Other custom domains use `CNAME` forwarding. For tunnels, we need to use an -`NS` record to delegate the domain to Modal's nameservers. - -**1. Delegate a (sub)domain to Modal's nameservers.** - -Add `NS` records to your DNS zone pointing to Modal's nameservers. For example, -to use `sandbox.example.com`, add the following records in your DNS provider's -control panel: - -| Name | Type | Value | -| --------------------- | ---- | -------------------- | -| `sandbox.example.com` | NS | `w-ns-a.modal.host.` | -| `sandbox.example.com` | NS | `w-ns-b.modal.host.` | -| `sandbox.example.com` | NS | `w-ns-c.modal.host.` | -| `sandbox.example.com` | NS | `w-ns-d.modal.host.` | - -You can delegate any subdomain depth you like (e.g. `tunnels.a.b.c.example.com`). - -**2. Ask Modal to set up the domain.** - -Reach out to us on Slack and provide the domain name. We'll enable it for your -workspace. - -**3. Pass `custom_domain` to `Sandbox.create`.** - -```python notest -import modal - -app = modal.App.lookup("my-app", create_if_missing=True) -sb = modal.Sandbox.create( - "python", "-m", "http.server", "8080", - encrypted_ports=[8080], - custom_domain="sandbox.example.com", - app=app, -) - -tunnel = sb.tunnels()[8080] -print(tunnel.url) # https://[...].sandbox.example.com -``` - -Modal will provision a TLS certificate automatically. Sandbox Connect Tokens generated -for this sandbox will also use the custom domain. - -## Security model - -Sandboxes are built on top of [gVisor](https://gvisor.dev/), a container runtime -by Google that provides strong isolation properties. gVisor has custom logic to -prevent Sandboxes from making malicious system calls, giving you stronger isolation -than most other container runtimes. - -Additionally, Sandboxes are not authorized to access other resources in your Modal -workspace the way that Modal Functions are [by default](/docs/guide/restricted-access). -As a result, the blast radius of any malicious code will be limited to the Sandbox -container itself. diff --git a/.agents/skills/modal/references/guide/sandbox-resources.md b/.agents/skills/modal/references/guide/sandbox-resources.md deleted file mode 100644 index 5e8c23f592878f82afd128366ab8283ceb83e76a..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/sandbox-resources.md +++ /dev/null @@ -1,168 +0,0 @@ -# Sandbox resources and pricing - -This page covers resource configuration and pricing for Modal Sandboxes. -For general documentation on CPU and memory options, see -[Reserving CPU and memory](/docs/guide/resources). - -## Pay for what you use - -Modal Sandboxes are billed by the second based on whichever is higher: -your resource request or your actual usage. - -Sandboxes can burst beyond their CPU and memory requests when additional -resources are available on the underlying host. Your request guarantees -a minimum level of resources, but when spare capacity exists, your Sandbox -can use more. You pay for `max(request, actual)`. - -See [Billing](/docs/guide/resources#billing) in the resource guide for more details. - -## Configuring resources - -Set CPU and memory requests using the `cpu` and `memory` parameters when creating your Sandbox. -The `cpu` parameter specifies physical CPU cores (1 core = 2 vCPUs), -and `memory` specifies MiB: - - - {#snippet python()} - -```python notest -import modal - -app = modal.App.lookup("my-app", create_if_missing=True) - -sb = modal.Sandbox.create( - cpu=0.5, - memory=512, - app=app, -) -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -import { ModalClient } from "modal"; - -const modal = new ModalClient(); -const app = await modal.apps.fromName("my-app", { createIfMissing: true }); -const image = modal.images.fromRegistry("python:3.13-slim"); - -const sb = await modal.sandboxes.create(app, image, { - cpu: 0.5, - memoryMiB: 512, -}); -``` - -{/snippet} - -{#snippet go()} - -```go notest -package main - -import ( - "context" - - modal "github.com/modal-labs/modal-client/go" -) - -func main() { - ctx := context.Background() - mc, _ := modal.NewClient() - - app, _ := mc.Apps.FromName(ctx, "my-app", &modal.AppFromNameParams{ - CreateIfMissing: true, - }) - image := mc.Images.FromRegistry("python:3.13-slim", nil) - - sb, _ := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - CPU: 0.5, - MemoryMiB: 512, - }) -} -``` - -{/snippet} - -For details on default values and maximum limits, see -[Reserving CPU and memory](/docs/guide/resources). - -### Resource limits - -You can set upper limits to cap how much a Sandbox can burst. -This is particularly useful when an AI agent controls what runs inside the Sandbox, -as it prevents misbehaving or adversarial workloads from consuming unbounded resources: - - - {#snippet python()} - -```python notest -sb = modal.Sandbox.create( - cpu=(0.5, 4.0), # Request 0.5 cores, limit to 4 cores - memory=(512, 2048), # Request 512 MiB, limit to 2048 MiB - app=app, -) -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const sb = await modal.sandboxes.create(app, image, { - cpu: 0.5, - cpuLimit: 4.0, - memoryMiB: 512, - memoryLimitMiB: 2048, -}); -``` - -{/snippet} - -{#snippet go()} - -```go notest -sb, _ := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - CPU: 0.5, // Request 0.5 cores - CPULimit: 4.0, // Limit to 4 cores - MemoryMiB: 512, // Request 512 MiB - MemoryLimitMiB: 2048, // Limit to 2048 MiB -}) -``` - -{/snippet} - -See [Resource limits](/docs/guide/resources#resource-limits) for details on -how CPU and memory limits behave. - -## Tuning your requests - -For maximum cost-efficiency, base your requests on observed usage percentiles rather than peaks: -around p50–75 for CPU and p90–95 for memory. - -If your observed usage is consistently higher than your request, you may -run into resource contention on the host. This can manifest as OOM errors or CPU throttling. - -The goal should be to set requests that correspond to your base load while letting bursting handle occasional spikes. - -1. Start with default values for CPU and memory. - -2. Run your typical workload and observe actual resource usage - in the [Modal dashboard](/apps) to understand your baseline needs. - -3. Set your request to match the resource level your workload consistently needs. - -## GPU Sandboxes - -You can also run Sandboxes with GPUs. See [GPU acceleration](/docs/guide/gpu) for available -GPU types and configuration. - -Unlike CPU Sandboxes, GPU Sandboxes are subject to [preemption](/docs/guide/preemption). -Design your GPU workloads to handle interruptions gracefully. - -## Additional resources - -* [Sandbox pricing](/pricing#sandboxes): current pricing for Sandboxes -* [Reserving CPU and memory](/docs/guide/resources): CPU, memory, and disk configuration -* [Billing](/docs/guide/billing): billing cycles, budgets, and cost attribution diff --git a/.agents/skills/modal/references/guide/sandbox-snapshots.md b/.agents/skills/modal/references/guide/sandbox-snapshots.md deleted file mode 100644 index e374d394c04e24b8ad6db090276a48da23f210be..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/sandbox-snapshots.md +++ /dev/null @@ -1,456 +0,0 @@ -# Snapshots - -Sandboxes support snapshotting, allowing you to save your Sandbox's state -and restore it later. This is useful for: - -* Reducing startup latency -* Creating custom environments for your Sandboxes to run in -* Backing up your Sandbox's state for debugging -* Running large-scale experiments with the same initial state -* Branching your Sandbox's state to test different code changes independently - -Modal currently supports three different kinds of Sandbox snapshots: - -1. [Filesystem Snapshots](#filesystem-snapshots) -2. [Directory Snapshots](#directory-snapshots) -3. [Memory Snapshots](#memory-snapshots) - -## Snapshot Retention - -Different snapshot types have different retention policies: - -| Snapshot Type | Default Retention Period | -| ------------------- | ------------------------ | -| Filesystem Snapshot | 30 days after creation | -| Directory Snapshot | 30 days after creation | -| Memory Snapshot | 7 days after creation | - - - -**Breaking change in v1.5 (Python) / v0.8.0 (Go/JS):** Filesystem Snapshots now default to a 30-day TTL. Previously, Filesystem Snapshots persisted indefinitely and Directory Snapshots already defaulted to 30 days. Both `snapshot_filesystem()` and `snapshot_directory()` now accept an explicit TTL parameter that you can use to override the default, including opting out of expiry entirely. - - - -Filesystem Snapshots and Directory Snapshots are [Images](/docs/reference/modal.Image) and are automatically garbage collected after their TTL expires (30 days by default). You can configure a custom TTL when creating a snapshot, or opt out of expiry entirely to retain snapshots indefinitely. Memory Snapshots expire 7 days after creation and cannot currently be extended. - -Here is how to configure custom TTLs for each snapshot type: - - - {#snippet python()} - -```python notest -# Filesystem snapshot with custom TTL of 7 days -image = sb.snapshot_filesystem(ttl=7 * 24 * 3600) - -# Filesystem snapshot with no expiry (retain indefinitely, like the pre-v1.5 default) -image = sb.snapshot_filesystem(ttl=None) - -# Directory snapshot with custom TTL of 7 days -snapshot = sb.snapshot_directory("/project", ttl=7 * 24 * 3600) - -# Directory snapshot with no expiry -snapshot = sb.snapshot_directory("/project", ttl=None) -``` - -{/snippet} -{#snippet javascript()} - -```javascript notest -// Filesystem snapshot with custom TTL of 7 days -let image = await sb.snapshotFilesystem({ ttlMs: 7 * 24 * 3600 * 1000 }); - -// Filesystem snapshot with no expiry (retain indefinitely, like the pre-v0.8.0 default) -image = await sb.snapshotFilesystem({ ttlMs: null }); - -// Directory snapshot with custom TTL of 7 days -let snapshot = await sb.snapshotDirectory("/project", { - ttlMs: 7 * 24 * 3600 * 1000, -}); - -// Directory snapshot with no expiry -snapshot = await sb.snapshotDirectory("/project", { ttlMs: null }); -``` - -{/snippet} -{#snippet go()} - -```go notest -// Filesystem snapshot with custom TTL of 7 days -image, _ := sb.SnapshotFilesystem(ctx, &modal.SandboxSnapshotFilesystemParams{ - TTL: 7 * 24 * time.Hour, -}) - -// Filesystem snapshot with no expiry (retain indefinitely, like the pre-v0.8.0 default) -image, _ = sb.SnapshotFilesystem(ctx, &modal.SandboxSnapshotFilesystemParams{ - TTL: modal.NoExpiryTTL, -}) - -// Directory snapshot with custom TTL of 7 days -snapshot, _ := sb.SnapshotDirectory(ctx, "/project", &modal.SandboxSnapshotDirectoryParams{ - TTL: 7 * 24 * time.Hour, -}) - -// Directory snapshot with no expiry -snapshot, _ = sb.SnapshotDirectory(ctx, "/project", &modal.SandboxSnapshotDirectoryParams{ - TTL: modal.NoExpiryTTL, -}) -``` - -{/snippet} - -If you try to use an expired snapshot, Modal will raise a `NotFoundError` — immediately when mounting the Image into a running sandbox, or upon first interaction (e.g. `exec` or `wait`) when starting a new sandbox from the expired image. Note that `Image.from_id()` is itself lazy and will not raise an error on construction even if the provided image id has been deleted. - -To manage storage for long-lived snapshots, you can delete them programmatically when no longer needed. See [Deleting Snapshots](#deleting-snapshots) for details. - -## Filesystem Snapshots - -Filesystem Snapshots are copies of the Sandbox's filesystem at a given point in time. -These Snapshots are [Images](/docs/reference/modal.Image) and can be used to create -new Sandboxes. - -To create a Filesystem Snapshot, you can use the -[`Sandbox.snapshot_filesystem()`](/docs/reference/modal.Sandbox#snapshot_filesystem) method: - -```python notest -import modal - -app = modal.App.lookup("sandbox-fs-snapshot-test", create_if_missing=True) - -sb = modal.Sandbox.create(app=app) -p = sb.exec("bash", "-c", "echo 'test' > /test") -p.wait() -assert p.returncode == 0, "failed to write to file" -image = sb.snapshot_filesystem() -sb.terminate() - -sb2 = modal.Sandbox.create(image=image, app=app) -p2 = sb2.exec("bash", "-c", "cat /test") -assert p2.stdout.read().strip() == "test" -``` - -Filesystem Snapshots are optimized for performance: they are calculated as the difference -from your base image, so only modified files are stored. Restoring a Filesystem Snapshot -utilizes the same infrastructure we use to get fast cold starts for your Sandboxes. - -See [Snapshot Retention](#snapshot-retention) for TTL configuration options and [Deleting Snapshots](#deleting-snapshots) to learn how to manage snapshot storage. - -## Directory Snapshots - - - -Directory Snapshots allow you to snapshot a specific directory within a running Sandbox. The resulting snapshot is an Image that can then be mounted into another already-running Sandbox (typically at a later time), which can be useful for: - -* **Updating system dependencies separately from application code**: Base dependencies can be updated by starting a new Sandbox from an updated base Image, and then mounting in previously snapshotted application code. -* **Using warm pools in combination with snapshots**: For use cases that benefit from a [warm pool](/docs/examples/sandbox_pool) of Sandboxes to reduce start-up latency, the first initialization can now happen in the warm pool without losing the ability to restore application-specific code at a later point in time. -* **Speeding up resumptions of previous sessions**: Files in mounted Images are prioritized when containers load files, so mounting a directory can speed up Sandbox resumptions vs. starting from a full file system image. - -### Usage - -Use `snapshot_directory` to snapshot a directory, -`mount_image` to mount a previous directory snapshot at a directory path, -and `unmount_image` to remove that mounted Image later. -To protect directory snapshots with customer-held key material, see -[Customer Supplied Encryption Keys](/docs/guide/customer-supplied-encryption-keys#directory-snapshots). - - - {#snippet python()} - -```python notest -sb = modal.Sandbox.create(app=app) -# Write some dummy data -sb.exec("bash", "-c", "mkdir /project && echo 'data' > /project/file.txt").wait() - -# Snapshot the directory -snapshot = sb.snapshot_directory("/project") - -# Ok to throw away the old sandbox at this point -sb.terminate() - -# Mount the snapshot in a new Sandbox -sb2 = modal.Sandbox.create(app=app) -try: - sb2.mount_image("/project", snapshot) -except modal.exception.NotFoundError: - # Handle a potential ttl expiry of the old snapshot here - ... - -# The sandbox now has access to the previous project state -assert sb2.exec("cat", "/project/file.txt").stdout.read().strip() == "data" - -``` - -{/snippet} -{#snippet javascript()} - -```javascript notest -import { NotFoundError } from "modal"; - -const sb = await modal.sandboxes.create(app, image); -// Write some dummy data -const p = await sb.exec([ - "bash", - "-c", - "mkdir /project && echo 'data' > /project/file.txt", -]); -await p.wait(); - -// Snapshot the directory -const snapshot = await sb.snapshotDirectory("/project"); - -// Ok to throw away the old sandbox at this point -await sb.terminate(); -sb.detach(); - -// Mount the snapshot in a new Sandbox -const sb2 = await modal.sandboxes.create(app, image); -try { - await sb2.mountImage("/project", snapshot); -} catch (e) { - if (e instanceof NotFoundError) { - // Handle a potential ttl expiry of the old snapshot here - } -} - -// The sandbox now has access to the previous project state -const p2 = await sb2.exec(["cat", "/project/file.txt"]); -console.assert((await p2.stdout.readText()).trim() === "data"); -sb2.detach(); -``` - -{/snippet} -{#snippet go()} - -```go notest -sb, _ := mc.Sandboxes.Create(ctx, app, image, nil) -defer sb.Detach() - -// Write some dummy data -p, _ := sb.Exec(ctx, []string{"bash", "-c", "mkdir /project && echo 'data' > /project/file.txt"}, nil) -p.Wait(ctx, nil) - -// Snapshot the directory -snapshot, _ := sb.SnapshotDirectory(ctx, "/project", nil) - -// Ok to throw away the old sandbox at this point -sb.Terminate(ctx, nil) - -// Mount the snapshot in a new Sandbox -sb2, _ := mc.Sandboxes.Create(ctx, app, image, nil) -defer sb2.Detach() - -if err := sb2.MountImage(ctx, "/project", snapshot, nil); err != nil { - var notFound modal.NotFoundError - if errors.As(err, ¬Found) { - // Handle a potential ttl expiry of the old snapshot here - } -} - -// The sandbox now has access to the previous project state -p2, _ := sb2.Exec(ctx, []string{"cat", "/project/file.txt"}, nil) -stdout, _ := io.ReadAll(p2.Stdout) -fmt.Println(strings.TrimSpace(string(stdout))) // "data" -``` - -{/snippet} - -### Unmounting a mounted Image - -To unmount a previously mounted Image, -call `unmount_image` on the exact path you passed to `mount_image`. -After unmounting, the underlying Sandbox filesystem at that path becomes visible again. - - - {#snippet python()} - -```python notest -sb2.unmount_image("/project") -``` - -{/snippet} -{#snippet javascript()} - -```javascript notest -await sb2.unmountImage("/project"); -``` - -{/snippet} -{#snippet go()} - -```go notest -_ = sb2.UnmountImage(ctx, "/project", nil) -``` - -{/snippet} - -## Memory Snapshots - - - -A number of known [limitations](#limitations) currently apply. - - - -Sandbox memory snapshots are copies of a Sandbox’s entire state, both in memory and on the filesystem. These Snapshots can be restored later to create a new Sandbox, which is an exact clone of the original Sandbox. - -To snapshot a Sandbox, create it with `_experimental_enable_snapshot` set to `True`, and use the `_experimental_snapshot` method, which returns a `SandboxSnapshot` object: - -```python notest -image = modal.Image.debian_slim().apt_install("curl", "procps") -app = modal.App.lookup("sandbox-snapshot", create_if_missing=True) - -with modal.enable_output(): - sb = modal.Sandbox.create( - "python3", "-m", "http.server", "8000", - app=app, image=image, _experimental_enable_snapshot=True - ) - -print(f"Performing snapshot of {sb.object_id} ...") -snapshot = sb._experimental_snapshot() -``` - -Create a new Sandbox from the returned SandboxSnapshot with `Sandbox._experimental_from_snapshot`: - -```python notest -print(f"Restoring from snapshot {sb.object_id} ...") -sb2 = modal.Sandbox._experimental_from_snapshot(snapshot) - -print("Let's see that the http.server is still running...") -p = sb2.exec("ps", "aux") -print(p.stdout.read()) - -# Talk to snapshotted sandbox http.server -p = sb2.exec("curl", "http://localhost:8000/") -reply = p.stdout.read() -print(reply) # - -Deletion is irreversible. Deleted snapshots cannot be recovered, and any Sandboxes configured to use a deleted snapshot will fail to start. - - - - - {#snippet python()} - -```python notest -import modal.experimental - -# Get the image ID from a filesystem or directory snapshot -image = sb.snapshot_filesystem() -# or: image = sb.snapshot_directory("/project") -image_id = image.object_id # e.g., "im-abc123" - -# Later, delete the snapshot when no longer needed -modal.experimental.image_delete(image_id) -``` - -{/snippet} -{#snippet javascript()} - -```javascript notest -// Get the image ID from a filesystem or directory snapshot -const image = await sb.snapshotFilesystem(); -// or: const image = await sb.snapshotDirectory("/project"); -const imageId = image.imageId; // e.g., "im-abc123" - -// Later, delete the snapshot when no longer needed -await modal.images.delete(imageId); -``` - -{/snippet} -{#snippet go()} - -```go notest -// Get the image ID from a filesystem or directory snapshot -image, _ := sb.SnapshotFilesystem(ctx, nil) -// or: image, _ := sb.SnapshotDirectory(ctx, "/project", nil) -imageId := image.ImageID // e.g., "im-abc123" - -// Later, delete the snapshot when no longer needed -mc.Images.Delete(ctx, imageId, nil) -``` - -{/snippet} - -To delete snapshots, you need to track the image IDs yourself (e.g., in a database or [Modal Dict](/docs/guide/dicts)), since there is currently no API to list all snapshots you have created. diff --git a/.agents/skills/modal/references/guide/sandbox-spawn.md b/.agents/skills/modal/references/guide/sandbox-spawn.md deleted file mode 100644 index e6a388554b284af60a23a5d75346821f87de1029..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/sandbox-spawn.md +++ /dev/null @@ -1,156 +0,0 @@ -# Running commands in Sandboxes - -Once you have created a Sandbox, you can run commands inside it using the -[`Sandbox.exec`](/docs/reference/modal.Sandbox#exec) method. - -```python notest -sb = modal.Sandbox.create(app=my_app) - -process = sb.exec("echo", "hello", timeout=3) -print(process.stdout.read()) - -process = sb.exec("python", "-c", "print(1 + 1)", timeout=3) -print(process.stdout.read()) - -process = sb.exec( - "bash", - "-c", - "for i in $(seq 1 10); do echo foo $i; sleep 0.1; done", - timeout=5, -) -for line in process.stdout: - print(line, end="") - -sb.terminate() -sb.detach() -``` - -`Sandbox.exec` returns a [`ContainerProcess`](/docs/reference/modal.container_process#modalcontainer_processcontainerprocess) -object, which allows access to the process's `stdout`, `stderr`, and `stdin`. -The `timeout` parameter ensures that the `exec` command will run for at most -`timeout` seconds. - -## Input - -The Sandbox and ContainerProcess `stdin` handles are [`StreamWriter`](/docs/reference/modal.io_streams#modalio_streamsstreamwriter) -objects. This object supports flushing writes with both synchronous and asynchronous APIs: - -```python notest -import asyncio - -sb = modal.Sandbox.create(app=my_app) - -p = sb.exec("bash", "-c", "while read line; do echo $line; done") -p.stdin.write(b"foo bar\n") -p.stdin.write_eof() -p.stdin.drain() -p.wait() -sb.terminate() -sb.detach() - -async def run_async(): - sb = await modal.Sandbox.create.aio(app=my_app) - p = await sb.exec.aio("bash", "-c", "while read line; do echo $line; done") - p.stdin.write(b"foo bar\n") - p.stdin.write_eof() - await p.stdin.drain.aio() - await p.wait.aio() - await sb.terminate.aio() - await sb.detach.aio() - -asyncio.run(run_async()) -``` - -## Output - -The Sandbox and ContainerProcess `stdout` and `stderr` handles are [`StreamReader`](/docs/reference/modal.io_streams#modalio_streamsstreamreader) -objects. These objects support reading from the stream in both synchronous and asynchronous manners. -These handles also respect the timeout given to `Sandbox.exec`. - -To read from a stream after the underlying process has finished, you can use the `read` -method, which blocks until the process finishes and returns the entire output stream. - -```python notest -sb = modal.Sandbox.create(app=my_app) -p = sb.exec("echo", "hello") -print(p.stdout.read()) -sb.terminate() -sb.detach() -``` - -To stream output, take advantage of the fact that `stdout` and `stderr` are -iterable: - -```python notest -import asyncio - -sb = modal.Sandbox.create(app=my_app) - -p = sb.exec("bash", "-c", "for i in $(seq 1 10); do echo foo $i; sleep 0.1; done") - -for line in p.stdout: - # Lines preserve the trailing newline character, so use end="" to avoid double newlines. - print(line, end="") -p.wait() -sb.terminate() -sb.detach() - -async def run_async(): - sb = await modal.Sandbox.create.aio(app=my_app) - p = await sb.exec.aio("bash", "-c", "for i in $(seq 1 10); do echo foo $i; sleep 0.1; done") - async for line in p.stdout: - # Avoid double newlines by using end="". - print(line, end="") - await p.wait.aio() - await sb.terminate.aio() - await sb.detach.aio() - -asyncio.run(run_async()) -``` - -### Stream types - -By default, all streams are buffered in memory, waiting to be consumed by the -client. You can control this behavior with the `stdout` and `stderr` parameters. -These parameters are conceptually similar to the `stdout` and `stderr` -parameters of the [`subprocess`](https://docs.python.org/3/library/subprocess.html#subprocess.DEVNULL) module. - -```python notest -from modal.stream_type import StreamType - -sb = modal.Sandbox.create(app=my_app) - -# Default behavior: buffered in memory. -p = sb.exec( - "bash", - "-c", - "echo foo; echo bar >&2", - stdout=StreamType.PIPE, - stderr=StreamType.PIPE, -) -print(p.stdout.read()) -print(p.stderr.read()) - -# Print the stream to STDOUT as it comes in. -p = sb.exec( - "bash", - "-c", - "echo foo; echo bar >&2", - stdout=StreamType.STDOUT, - stderr=StreamType.STDOUT, -) -p.wait() - -# Discard all output. -p = sb.exec( - "bash", - "-c", - "echo foo; echo bar >&2", - stdout=StreamType.DEVNULL, - stderr=StreamType.DEVNULL, -) -p.wait() - -sb.terminate() -sb.detach() -``` diff --git a/.agents/skills/modal/references/guide/sandboxes.md b/.agents/skills/modal/references/guide/sandboxes.md deleted file mode 100644 index 887feb635b829f58ea250f9452227d1cae8198d9..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/sandboxes.md +++ /dev/null @@ -1,1417 +0,0 @@ -# Sandboxes - -This page is a high-level guide to Sandboxes, -secure containers for executing untrusted user or agent code on Modal. - -For reference documentation on the `modal.Sandbox` interface, -see [this page](/docs/reference/modal.Sandbox). - -## What are Sandboxes and why should I use them? - -In addition to the Function interface, Modal has a direct -interface for defining containers *at runtime* and securely running arbitrary code -inside them. - -This can be useful if, for example, you want to: - -* Execute code generated by a language model. -* Create isolated environments for running untrusted code. -* Check out a git repository and run a command against it, like a test suite, or - `npm lint`. -* Run containers with arbitrary dependencies and setup scripts. - -Each individual job is called a **Sandbox** and can be created using the -[`Sandbox.create`](/docs/reference/modal.Sandbox#create) constructor: - - - {#snippet python()} - -```python -sb_app = modal.App.lookup("my-app", create_if_missing=True) -sb = modal.Sandbox.create(app=sb_app) - -p = sb.exec("python", "-c", "print('hello')", timeout=3) -print(p.stdout.read()) - -p = sb.exec("bash", "-c", "for i in {1..10}; do date +%T; sleep 0.5; done", timeout=5) -for line in p.stdout: - # Avoid double newlines by using end="". - print(line, end="") - -sb.terminate() -sb.detach() -``` - -{/snippet} - -{#snippet python\_async()} - -```python -sb_app = await modal.App.lookup.aio("my-app", create_if_missing=True) -sb = await modal.Sandbox.create.aio(app=sb_app) - -p = await sb.exec.aio("python", "-c", "print('hello')", timeout=3) -print(await p.stdout.read.aio()) - -p = await sb.exec.aio("bash", "-c", "for i in {1..10}; do date +%T; sleep 0.5; done", timeout=5) -async for line in p.stdout: - # Avoid double newlines by using end="". - print(line, end="") - -await sb.terminate.aio() -await sb.detach.aio() -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -import { ModalClient } from "modal"; - -const modal = new ModalClient(); -const app = await modal.apps.fromName("my-app", { - createIfMissing: true, -}); -const image = modal.images.fromRegistry("python:3.13-slim"); - -const sb = await modal.sandboxes.create(app, image); - -const p = await sb.exec(["python", "-c", "print('hello')"], { - timeoutMs: 3 * 1000, -}); -console.log(await p.stdout.readText()); - -const p2 = await sb.exec( - ["bash", "-c", "for i in {1..10}; do date +%T; sleep 0.5; done"], - { timeoutMs: 5 * 1000 }, -); -for await (const line of p2.stdout) { - process.stdout.write(line); -} - -await sb.terminate(); -``` - -{/snippet} - -{#snippet go()} - -```go notest -package main - -import ( - "context" - "fmt" - "io" - "os" - "time" - - modal "github.com/modal-labs/modal-client/go" -) - -func main() { - ctx := context.Background() - mc, _ := modal.NewClient() - - app, _ := mc.Apps.FromName(ctx, "my-app", &modal.AppFromNameParams{ - CreateIfMissing: true, - }) - image := mc.Images.FromRegistry("python:3.13-slim", nil) - - sb, _ := mc.Sandboxes.Create(ctx, app, image, nil) - defer sb.Terminate(ctx, nil) - - p, _ := sb.Exec(ctx, []string{"python", "-c", "print('hello')"}, &modal.SandboxExecParams{ - Timeout: 3 * time.Second, - }) - stdout, _ := io.ReadAll(p.Stdout) - fmt.Println(string(stdout)) - - p2, _ := sb.Exec(ctx, []string{"bash", "-c", "for i in {1..10}; do date +%T; sleep 0.5; done"}, &modal.SandboxExecParams{ - Timeout: 5 * time.Second, - }) - io.Copy(os.Stdout, p2.Stdout) -} -``` - -{/snippet} - -**Note:** you can run the above example as a script directly with `python my_script.py`. `modal run` is not needed here since there is no [entrypoint](/docs/guide/apps#entrypoints-for-ephemeral-apps). - -Sandboxes require an [`App`](/docs/guide/apps) to be passed when spawned from outside -of a Modal container. You may pass in a regular `App` object or look one up by name with -[`App.lookup`](/docs/reference/modal.App#lookup). The `create_if_missing` flag on `App.lookup` -will create an `App` with the given name if it doesn't exist. - -## Lifecycle - -### Events - -Every Sandbox moves through a series of lifecycle events as it progresses from -creation to completion. Understanding these events is useful for monitoring, -debugging, and building automations that react to Sandbox state changes. - -The lifecycle events, in order, are: - -1. **Created** — The Sandbox has been requested and registered with Modal. At this - point the Sandbox object exists and has an ID, but no compute resources have been - allocated yet. This is the initial state immediately after calling `Sandbox.create`. - -2. **Scheduled** — The Sandbox has been scheduled to a specific worker. The - worker is now provisioning the resources the Sandbox needs (CPU, memory, GPU, - volumes, etc.) and preparing the container environment. The Sandbox will - transition to **Started** once the container is fully initialized. - -3. **Started** — The Sandbox's container has been launched on a worker and the - entrypoint process (if any) is running. At this point you can begin executing - commands inside the Sandbox with `sandbox.exec(...)`. Network tunnels and volume - mounts are active. - -4. **Ready** — If [readiness probes](/docs/guide/sandboxes#readiness-probes) are - enabled for the Sandbox, this event fires once the probe succeeds, indicating that - the service inside the Sandbox is fully initialized and ready to accept traffic. - This is especially useful for Sandboxes running web servers or other services - that need warm-up time before they can handle requests. If readiness probes are - not configured, this event is skipped. - -5. **Finished** — The Sandbox has stopped running. This can happen for several - reasons: the entrypoint process exited on its own, the Sandbox was explicitly - terminated (via the dashboard or `sandbox.terminate()`), the timeout or idle timeout - was reached, or an out-of-memory condition occurred. Once finished, no further - commands can be executed inside the Sandbox. You can learn more about why a Sandbox - stopped running in the dashboard or by examining the exit code returned from - `sandbox.poll()`. - -### Timeouts - -Sandboxes have a default maximum lifetime of 5 minutes. You can change this by passing -a `timeout` of up to 24 hours to the `Sandbox.create(...)` function. - - - {#snippet python()} - -```python fixture:sb_app -sb = modal.Sandbox.create(app=sb_app, timeout=10*60) # 10 minutes -sb.detach() -``` - -{/snippet} - -{#snippet python\_async()} - -```python fixture:sb_app -sb = await modal.Sandbox.create.aio(app=sb_app, timeout=10*60) # 10 minutes -await sb.detach.aio() -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const sb = await modal.sandboxes.create(app, image, { - timeoutMs: 10 * 60 * 1000, // 10 minutes -}); -sb.detach(); -``` - -{/snippet} - -{#snippet go()} - -```go notest -sb, err := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Timeout: 10 * time.Minute, -}) -defer sb.Detach() -``` - -{/snippet} - -If you need a Sandbox to run for more than 24 hours, we recommend using -[Filesystem Snapshots](/docs/guide/sandbox-snapshots) to preserve its state, -and then restore from that snapshot with a subsequent Sandbox. - -### Idle Timeouts - -Sandboxes can also be automatically terminated after a period of inactivity - you can do this by setting the `idle_timeout` parameter. A Sandbox is considered active if any of the following are true: - -1. It has an active [command](/docs/guide/sandbox-spawn) running (via [`sb.exec(...)`](/docs/reference/modal.Sandbox#exec)) -2. Its stdin is being written to (via [`sb.stdin.write()`](/docs/reference/modal.Sandbox#stdin)) -3. It has an open TCP connection over one of its [Tunnels](/docs/guide/tunnels) - -### Readiness Probes - - - -After a Sandbox starts, you often need to run custom initialization logic before it's -ready for use — pulling code with `git pull`, installing dependencies, starting a server, -writing config files, or other setup that isn't baked into the image. Readiness probes -give you a way to track when that initialization is complete, so you don't have to build -the polling or signaling yourself. Modal also uses probe results to give you -observability into how long this startup phase typically takes. - -A readiness probe is a check that Modal runs automatically inside the Sandbox at a -configurable interval. You can then call `wait_until_ready()` to block until the probe -succeeds. - -There are two types of readiness probes: - -* **TCP probe** — Checks whether a TCP port inside the Sandbox is accepting connections. - This is the most common choice when your startup logic includes launching a server. -* **Exec probe** — Runs an arbitrary command inside the Sandbox and succeeds when the - command exits with status code 0. Use this for any other readiness condition: checking - that a file exists, verifying a setup script has completed, confirming dependencies are - installed, etc. - -Both probe types accept an `interval_ms` parameter (default: 100ms) that controls how -frequently the check is retried until it succeeds. - -#### TCP readiness probe - -Use a TCP probe when your Sandbox starts a server and you want to wait until -it's listening on a port: - - - {#snippet python()} - -```python fixture:sb_app -sb = modal.Sandbox.create( - "python3", "-m", "http.server", "8080", - readiness_probe=modal.Probe.with_tcp(8080), - app=sb_app, -) - -# Blocks until port 8080 is accepting connections -sb.wait_until_ready() - -# The server is now ready — interact with it via tunnels, exec, etc. -sb.terminate() -sb.detach() -``` - -{/snippet} - -{#snippet python\_async()} - -```python fixture:sb_app -sb = await modal.Sandbox.create.aio( - "python3", "-m", "http.server", "8080", - readiness_probe=modal.Probe.with_tcp(8080), - app=sb_app, -) - -# Blocks until port 8080 is accepting connections -await sb.wait_until_ready.aio() - -# The server is now ready — interact with it via tunnels, exec, etc. -await sb.terminate.aio() -await sb.detach.aio() -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -import { ModalClient, Probe } from "modal"; - -const modal = new ModalClient(); -const app = await modal.apps.fromName("my-app", { createIfMissing: true }); -const image = modal.images.fromRegistry("python:3.13-slim"); - -const sb = await modal.sandboxes.create(app, image, { - command: ["python3", "-m", "http.server", "8080"], - readinessProbe: Probe.withTcp(8080), -}); - -// Blocks until port 8080 is accepting connections -await sb.waitUntilReady(); - -// The server is now ready — interact with it via tunnels, exec, etc. -await sb.terminate(); -``` - -{/snippet} - -{#snippet go()} - -```go notest -package main - -import ( - "context" - "time" - - modal "github.com/modal-labs/modal-client/go" -) - -func main() { - ctx := context.Background() - mc, _ := modal.NewClient() - - app, _ := mc.Apps.FromName(ctx, "my-app", &modal.AppFromNameParams{ - CreateIfMissing: true, - }) - image := mc.Images.FromRegistry("python:3.13-slim", nil) - probe, _ := modal.NewTCPProbe(8080, nil) - - sb, _ := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Command: []string{"python3", "-m", "http.server", "8080"}, - ReadinessProbe: probe, - }) - defer sb.Detach() - - // Blocks until port 8080 is accepting connections - sb.WaitUntilReady(ctx, 5*time.Minute) - - // The server is now ready — interact with it via tunnels, exec, etc. - sb.Terminate(ctx, nil) -} -``` - -{/snippet} - -#### Exec readiness probe - -Use an exec probe when readiness depends on something other than a TCP port — for -example, waiting for a file to be created or a setup script to complete: - - - {#snippet python()} - -```python fixture:sb_app -sb = modal.Sandbox.create( - "bash", "-c", "sleep 5 && touch /tmp/ready && sleep 3600", - readiness_probe=modal.Probe.with_exec( - "sh", "-c", "test -f /tmp/ready", - interval_ms=250, - ), - app=sb_app, -) - -# Blocks until "test -f /tmp/ready" exits with code 0 -sb.wait_until_ready() - -# The sandbox is now ready -p = sb.exec("cat", "/tmp/ready") -sb.terminate() -sb.detach() -``` - -{/snippet} - -{#snippet python\_async()} - -```python fixture:sb_app -sb = await modal.Sandbox.create.aio( - "bash", "-c", "sleep 5 && touch /tmp/ready && sleep 3600", - readiness_probe=modal.Probe.with_exec( - "sh", "-c", "test -f /tmp/ready", - interval_ms=250, - ), - app=sb_app, -) - -# Blocks until "test -f /tmp/ready" exits with code 0 -await sb.wait_until_ready.aio() - -# The sandbox is now ready -p = await sb.exec.aio("cat", "/tmp/ready") -await sb.terminate.aio() -await sb.detach.aio() -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -import { ModalClient, Probe } from "modal"; - -const modal = new ModalClient(); -const app = await modal.apps.fromName("my-app", { createIfMissing: true }); -const image = modal.images.fromRegistry("python:3.13-slim"); - -const sb = await modal.sandboxes.create(app, image, { - command: ["bash", "-c", "sleep 5 && touch /tmp/ready && sleep 3600"], - readinessProbe: Probe.withExec(["sh", "-c", "test -f /tmp/ready"], { - intervalMs: 250, - }), -}); - -// Blocks until "test -f /tmp/ready" exits with code 0 -await sb.waitUntilReady(); - -// The sandbox is now ready -await sb.terminate(); -``` - -{/snippet} - -{#snippet go()} - -```go notest -package main - -import ( - "context" - "time" - - modal "github.com/modal-labs/modal-client/go" -) - -func main() { - ctx := context.Background() - mc, _ := modal.NewClient() - - app, _ := mc.Apps.FromName(ctx, "my-app", &modal.AppFromNameParams{ - CreateIfMissing: true, - }) - image := mc.Images.FromRegistry("python:3.13-slim", nil) - probe, _ := modal.NewExecProbe( - []string{"sh", "-c", "test -f /tmp/ready"}, - &modal.ExecProbeParams{Interval: 250 * time.Millisecond}, - ) - - sb, _ := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Command: []string{"bash", "-c", "sleep 5 && touch /tmp/ready && sleep 3600"}, - ReadinessProbe: probe, - }) - defer sb.Detach() - - // Blocks until "test -f /tmp/ready" exits with code 0 - sb.WaitUntilReady(ctx, 5*time.Minute) - - // The sandbox is now ready - sb.Terminate(ctx, nil) -} -``` - -{/snippet} - -**Note:** Readiness probes will run for a maximum of 5 minutes. If the probe does not -succeed within that window, `wait_until_ready()` will raise a `TimeoutError`. The probe -timeout does **not** automatically terminate the Sandbox — you may want to catch the -`TimeoutError` and explicitly terminate the Sandbox if readiness is never achieved: - - - {#snippet python()} - -```python notest -try: - sb.wait_until_ready() -except TimeoutError: - print("Sandbox failed to become ready") - sb.terminate() - sb.detach() -``` - -{/snippet} - -{#snippet python\_async()} - -```python notest -try: - await sb.wait_until_ready.aio() -except TimeoutError: - print("Sandbox failed to become ready") - await sb.terminate.aio() - await sb.detach.aio() -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -try { - await sb.waitUntilReady(); -} catch (err) { - console.log("Sandbox failed to become ready"); - await sb.terminate(); -} -``` - -{/snippet} - -{#snippet go()} - -```go notest -if err := sb.WaitUntilReady(ctx, 5*time.Minute); err != nil { - fmt.Println("Sandbox failed to become ready") - sb.Terminate(ctx, nil) -} -``` - -{/snippet} - -If you call `wait_until_ready()` on a Sandbox that was not configured with a readiness -probe, an error will be raised. Similarly, calling it after the Sandbox has been -terminated will raise an error. However, calling `wait_until_ready()` after the Sandbox -has already become ready returns immediately. - -## Return Codes - -[Unix-style exit codes](https://tldp.org/LDP/abs/html/exitcodes.html) are provided to help diagnose conditions such as success, manual termination, or out-of-memory. - -They are available on both: - -* Processes in the sandbox (via [`ContainerProcess.returncode`](/docs/reference/modal.container_process#returncode) / [`ContainerProcess.poll()`](/docs/reference/modal.container_process#poll)) -* The Sandbox itself (via [`Sandbox.returncode`](/docs/reference/modal.Sandbox#returncode) / [`Sandbox.poll()`](/docs/reference/modal.Sandbox#poll)) - - - {#snippet python()} - -```python fixture:sb_app -sb = modal.Sandbox.create(app=sb_app) - -# Read returncode of individual process -p = sb.exec("sh", "-c", "exit 42") -p.wait() -print(p.returncode) # 42 - -# Read returncode of finished sandbox -# Terminate sends a SIGKILL, code 137 -sb.terminate(wait=True) -print(sb.returncode) # 137 -``` - -{/snippet} - -{#snippet python\_async()} - -```python fixture:sb_app -sb = await modal.Sandbox.create.aio(app=sb_app) - -# Read returncode of individual process -p = await sb.exec.aio("sh", "-c", "exit 42") -await p.wait.aio() -print(p.returncode) # 42 - -# Read returncode of finished sandbox -# Terminate sends a SIGKILL, code 137 -await sb.terminate.aio(wait=True) -print(sb.returncode) # 137 -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const sb = await modal.sandboxes.create(app, image); - -// Read returncode of individual process -const p = await sb.exec(["sh", "-c", "exit 42"]); -const returnCode = await p.wait(); -console.log(returnCode); // 42 - -// Read returncode of finished sandbox -// Terminate sends a SIGKILL, code 137 -const returnCodeSb = await sb.terminate({ wait: true }); -console.log(returnCodeSb); // 137 -``` - -{/snippet} - -{#snippet go()} - -```go notest -sb, _ := mc.Sandboxes.Create(ctx, app, image, nil) - -// Read returncode of individual process -p, _ := sb.Exec(ctx, []string{"sh", "-c", "exit 42"}, nil) -returnCode, _ := p.Wait(ctx) -fmt.Println(returnCode) // 42 - -// Read returncode of finished sandbox -// Terminate sends a SIGKILL, code 137 -returnCodeSb, _ := sb.Terminate(ctx, &modal.SandboxTerminateParams{Wait: true}) -fmt.Println(returnCodeSb) // 137 -``` - -{/snippet} - -## Configuration - -Sandboxes support nearly all configuration options found in regular `modal.Function`s. -Refer to [`Sandbox.create`](/docs/reference/modal.Sandbox#create) for further documentation -on Sandbox configs. - -For example, Images and Volumes can be used just as with functions: - - - {#snippet python()} - -```python fixture:sb_app -sb = modal.Sandbox.create( - image=modal.Image.debian_slim().pip_install("pandas"), - volumes={"/data": modal.Volume.from_name("data-volume", create_if_missing=True)}, - app=sb_app, -) -sb.detach() -``` - -{/snippet} - -{#snippet python\_async()} - -```python fixture:sb_app -sb = await modal.Sandbox.create.aio( - image=modal.Image.debian_slim().pip_install("pandas"), - volumes={"/data": modal.Volume.from_name("data-volume", create_if_missing=True)}, - app=sb_app, -) -await sb.detach.aio() -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const image = modal.images.fromRegistry("python:3.13-slim"); -const volume = modal.volumes.fromName("my-volume"); -const sb = await modal.sandboxes.create(app, image, { - volumes: { "/data": volume }, - workdir: "/repo", -}); -sb.detach(); -``` - -{/snippet} - -{#snippet go()} - -```go notest -image := mc.Images.FromRegistry("python:3.13-slim", nil) -volume := mc.Volumes.FromName("my-volume", nil) -sb, err := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Volumes: map[string]*modal.Volume{"/data": volume}, - Workdir: "/repo", -}) -defer sb.Detach() -``` - -{/snippet} - -## Environments - -### Environment variables - -You can set environment variables using inline secrets: - - - {#snippet python()} - -```python fixture:sb_app -secret = modal.Secret.from_dict({"MY_SECRET": "hello"}) - -sb = modal.Sandbox.create( - secrets=[secret], - app=sb_app, -) -p = sb.exec("bash", "-c", "echo $MY_SECRET") -print(p.stdout.read()) -sb.detach() -``` - -{/snippet} - -{#snippet python\_async()} - -```python fixture:sb_app -secret = modal.Secret.from_dict({"MY_SECRET": "hello"}) - -sb = await modal.Sandbox.create.aio( - secrets=[secret], - app=sb_app, -) -p = await sb.exec.aio("bash", "-c", "echo $MY_SECRET") -print(await p.stdout.read.aio()) -await sb.detach.aio() -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const secret = modal.secrets.fromObject({ MY_SECRET: "hello" }); -const image = modal.images.fromRegistry("python:3.13-slim"); - -const sb = await modal.sandboxes.create(app, image, { - secrets: [secret], -}); -const p = await sb.exec(["bash", "-c", "echo $MY_SECRET"]); -console.log(await p.stdout.readText()); -sb.detach(); -``` - -{/snippet} - -{#snippet go()} - -```go notest -secret, err := mc.Secrets.FromMap(ctx, map[string]string{"MY_SECRET": "hello"}, nil) -image := mc.Images.FromRegistry("python:3.13-slim", nil) - -sb, err := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Secrets: []*modal.Secret{secret}, -}) -defer sb.Detach() -p, err := sb.Exec(ctx, []string{"bash", "-c", "echo $MY_SECRET"}, nil) -stdout, err := io.ReadAll(p.Stdout) -fmt.Println(string(stdout)) -``` - -{/snippet} - -### Custom Images - -Sandboxes support [custom images](/docs/guide/images) just as Functions do. These can be defined using [method chaining](/docs/guide/images) or by referencing an [existing Image in an external container registry](/docs/guide/existing-images). - -#### Separating Image builds from Sandbox creation - -To avoid blocking creation of new Sandboxes on rebuilding an invalidated Image, -it's recommended to use Modal's named Images with sandboxes, rather than using -inline Image definitions. - -Use [`Image.build`](/docs/reference/modal.Image#build) to trigger -Image builds as part of a deployment flow or at a regular interval (e.g., in a -[scheduled job](/docs/guide/cron) or CI pipeline), then publish the result as a -[named image](/docs/guide/named-images) - -Sandboxes can then use [`Image.from_name`](/docs/reference/modal.Image#from_name) -to reference the Image in a way that's guaranteed to not block on rebuilds. - - - {#snippet python()} - -```python notest -# build_sandbox_image.py -app = modal.App.lookup("sandbox-app", create_if_missing=True) - -# Method-chained image -image = modal.Image.debian_slim().pip_install("pandas") - -# Or, for an external registry image with a fixed tag: -# image = modal.Image.from_registry("ubuntu:24.04") - -with modal.enable_output(): - image.build(app=app).publish("sandbox-runtime") - -# sandbox_app.py -app = modal.App.lookup("sandbox-app", create_if_missing=True) - -image = modal.Image.from_name("sandbox-runtime") -modal.Sandbox.create(app=app, image=image) -``` - -{/snippet} - -{#snippet python\_async()} - -```python notest -# build_sandbox_image.py -app = await modal.App.lookup.aio("sandbox-app", create_if_missing=True) - -# Method-chained image -image = modal.Image.debian_slim().pip_install("pandas") - -# Or, for an external registry image with a fixed tag: -# image = modal.Image.from_registry("ubuntu:24.04") - -with modal.enable_output(): - await image.build.aio(app) -await image.publish.aio("sandbox-runtime") - -# app.py -app = await modal.App.lookup.aio("sandbox-app", create_if_missing=True) - -image = modal.Image.from_name("sandbox-runtime") -await modal.Sandbox.create.aio(app=app, image=image) -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -// build_sandbox_image.ts -const app = await modal.apps.fromName("sandbox-app", { - createIfMissing: true, -}); - -const image = modal.images - .fromRegistry("python:3.13-slim") - .dockerfileCommands(["RUN pip install pandas"]); - -await image.build(app); -await image.publish("sandbox-runtime"); - -// app.ts -const app = await modal.apps.fromName("sandbox-app", { - createIfMissing: true, -}); - -const image = await modal.images.fromName("sandbox-runtime"); -await modal.sandboxes.create(app, image); -``` - -{/snippet} - -{#snippet go()} - -```go notest -// build_sandbox_image.go -app, err := mc.Apps.FromName(ctx, "sandbox-app", &modal.AppFromNameParams{ - CreateIfMissing: true, -}) - -image := mc.Images.FromRegistry("python:3.13-slim", nil). - DockerfileCommands([]string{"RUN pip install pandas"}, nil) - -builtImage, err := image.Build(ctx, app, nil) -err = builtImage.Publish(ctx, "sandbox-runtime", nil) - -// app.go -app, err = mc.Apps.FromName(ctx, "sandbox-app", &modal.AppFromNameParams{ - CreateIfMissing: true, -}) - -image, err = mc.Images.FromName(ctx, "sandbox-runtime", nil) -sb, err := mc.Sandboxes.Create(ctx, app, image, nil) -defer sb.Detach() -``` - -{/snippet} - - - -* **Modal treats external Image tags as immutable once pulled.** For [external registry](/docs/guide/existing-images) images, `Image.build` always returns the cached version — Modal does not detect upstream changes to mutable tags like `:latest`. -* To pick up a new version of an external registry image, update the tag in your deploy script (for example, `ubuntu:24.04` → `ubuntu:24.04-20240523`). - - - -#### Image build logs - -You may need to manually enable output streaming to see your image build logs: - - - {#snippet python()} - -```python fixture:sb_app -image = modal.Image.debian_slim().pip_install("pandas", "numpy") - -with modal.enable_output(): - sb = modal.Sandbox.create(image=image, app=sb_app) -sb.detach() -``` - -{/snippet} - -{#snippet python\_async()} - -```python fixture:sb_app -image = modal.Image.debian_slim().pip_install("pandas", "numpy") - -with modal.enable_output(): - sb = await modal.Sandbox.create.aio(image=image, app=sb_app) -await sb.detach.aio() -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const image = modal.images - .fromRegistry("python:3.13-slim") - .dockerfileCommands(["RUN pip install pandas numpy"]); - -const sb = await modal.sandboxes.create(app, image); -sb.detach(); -``` - -{/snippet} - -{#snippet go()} - -```go notest -image := mc.Images.FromRegistry("python:3.13-slim", nil). - DockerfileCommands([]string{"RUN pip install pandas numpy"}, nil) - -// Note: Image build logs are automatically streamed in Go -sb, err := mc.Sandboxes.Create(ctx, app, image, nil) -defer sb.Detach() -``` - -{/snippet} - -## Running a Sandbox with an entrypoint - -In most cases, Sandboxes are treated as a generic container that can run arbitrary -commands. However, in some cases, you may want to run a single command or script -as the entrypoint of the Sandbox. You can do this by passing command arguments to the -Sandbox constructor: - - - {#snippet python()} - -```python fixture:sb_app -sb = modal.Sandbox.create("python", "-m", "http.server", "8080", app=sb_app, timeout=10) -for line in sb.stdout: - print(line, end="") -sb.detach() -``` - -{/snippet} - -{#snippet python\_async()} - -```python fixture:sb_app -sb = await modal.Sandbox.create.aio("python", "-m", "http.server", "8080", app=sb_app, timeout=10) -async for line in sb.stdout: - print(line, end="") -await sb.detach.aio() -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const sb = await modal.sandboxes.create(app, image, { - command: ["python", "-m", "http.server", "8080"], - timeoutMs: 10 * 1000, -}); -sb.detach(); -``` - -{/snippet} - -{#snippet go()} - -```go notest -sb, err := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Command: []string{"python", "-m", "http.server", "8080"}, - Timeout: 10 * time.Second, -}) -sb.Detach() -``` - -{/snippet} - -This functionality is most useful for running long-lived services that you want -to keep running in the background. See our [Jupyter notebook example](/docs/examples/jupyter_sandbox) -for a more concrete example of this. - -## Referencing Sandboxes from other code - -If you have a running Sandbox, you can retrieve it using the `from_id` method. - - - {#snippet python()} - -```python fixture:sb_app -sb = modal.Sandbox.create(app=sb_app) -sb_id = sb.object_id -sb.detach() - -# ... later in the program ... - -sb2 = modal.Sandbox.from_id(sb_id) -p = sb2.exec("echo", "hello") -print(p.stdout.read()) -sb2.terminate() -sb2.detach() -``` - -{/snippet} - -{#snippet python\_async()} - -```python fixture:sb_app -sb = await modal.Sandbox.create.aio(app=sb_app) -sb_id = sb.object_id -await sb.detach.aio() - -# ... later in the program ... - -sb2 = await modal.Sandbox.from_id.aio(sb_id) -p = await sb2.exec.aio("echo", "hello") -print(await p.stdout.read.aio()) -await sb2.terminate.aio() -await sb2.detach.aio() -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const sb = await modal.sandboxes.create(app, image); -const sbId = sb.sandboxId; -await sb.detach(); - -// ... later in the program ... - -const sb2 = await modal.sandboxes.fromId(sbId); -const p = await sb2.exec(["echo", "hello"]); -console.log(await p.stdout.readText()); -await sb2.terminate(); -``` - -{/snippet} - -{#snippet go()} - -```go notest -sb, err := mc.Sandboxes.Create(ctx, app, image, nil) -defer sb.Detach() -sbId := sb.SandboxID - -// ... later in the program ... - -sb2, err := mc.Sandboxes.FromID(ctx, sbId) -defer sb2.Terminate(ctx, nil) -p, err := sb2.Exec(ctx, []string{"echo", "hello"}, nil) -stdout, err := io.ReadAll(p.Stdout) -fmt.Println(string(stdout)) -``` - -{/snippet} - -A common use case for this is keeping a pool of Sandboxes available for executing tasks -as they come in. You can keep a list of `object_id`s of Sandboxes that are "open" and -reuse them, closing over the `object_id` in whatever function is using them. - -## Named Sandboxes - -You can assign a name to a Sandbox when creating it. Each name must be unique within an app - -only one *running* Sandbox can use a given name at a time. Note that the associated app must be -a deployed app. Once a Sandbox completely stops running, its name becomes available for reuse. -Some applications find Sandbox Names to be useful for ensuring that no more than one Sandbox is -running per resource or project. If a Sandbox with the given name is already running, `create()` -will raise an error. - - - {#snippet python()} - -```python notest -sb1 = modal.Sandbox.create(app=sb_app, name="my-name") -# This will raise a modal.exception.AlreadyExistsError. -sb2 = modal.Sandbox.create(app=sb_app, name="my-name") -``` - -{/snippet} - -{#snippet python\_async()} - -```python notest -sb1 = await modal.Sandbox.create.aio(app=sb_app, name="my-name") -# This will raise a modal.exception.AlreadyExistsError. -sb2 = await modal.Sandbox.create.aio(app=sb_app, name="my-name") -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const sb1 = await modal.sandboxes.create(app, image, { name: "my-name" }); -// this will raise an AlreadyExistsError -const sb2 = await modal.sandboxes.create(app, image, { name: "my-name" }); -``` - -{/snippet} - -{#snippet go()} - -```go notest -sb1, err := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Name: "my-name", -}) -// this will return an error -sb2, err := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Name: "my-name", -}) -``` - -{/snippet} - -A named Sandbox may be fetched from a deployed app using `from_name()` *but only -if the Sandbox is currently running*. If no running Sandbox is found, `from_name()` will raise -an error. - - - {#snippet python()} - -```python notest -sb_app = modal.App.lookup("my-app", create_if_missing=True) -sb1 = modal.Sandbox.create(app=sb_app, name="my-name") -# Returns the currently running Sandbox with the name "my-name" from the -# deployed app named "my-app". -sb2 = modal.Sandbox.from_name("my-app", "my-name") -assert sb1.object_id == sb2.object_id # sb1 and sb2 refer to the same Sandbox -sb1.detach() -sb2.detach() -``` - -{/snippet} - -{#snippet python\_async()} - -```python notest -sb_app = await modal.App.lookup.aio("my-app", create_if_missing=True) -sb1 = await modal.Sandbox.create.aio(app=sb_app, name="my-name") -# Returns the currently running Sandbox with the name "my-name" from the -# deployed app named "my-app". -sb2 = await modal.Sandbox.from_name.aio("my-app", "my-name") -assert sb1.object_id == sb2.object_id # sb1 and sb2 refer to the same Sandbox -await sb1.detach.aio() -await sb2.detach.aio() -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const app = await modal.apps.fromName("my-app", { createIfMissing: true }); -const sb1 = await modal.sandboxes.create(app, image, { name: "my-name" }); -// returns the currently running Sandbox with the name "my-name" from the -// deployed app named "my-app". -const sb2 = await modal.sandboxes.fromName("my-app", "my-name"); -console.assert(sb1.sandboxId === sb2.sandboxId); // sb1 and sb2 refer to the same Sandbox -sb1.detach(); -sb2.detach(); -``` - -{/snippet} - -{#snippet go()} - -```go notest -app, err := mc.Apps.FromName(ctx, "my-app", &modal.AppFromNameParams{ - CreateIfMissing: true, -}) -sb1, err := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Name: "my-name", -}) -// returns the currently running Sandbox with the name "my-name" from the -// deployed app named "my-app". -sb2, err := mc.Sandboxes.FromName(ctx, "my-app", "my-name", nil) -// sb1 and sb2 refer to the same Sandbox -fmt.Println(sb1.SandboxID == sb2.SandboxID) -defer sb1.Detach() -defer sb2.Detach() -``` - -{/snippet} - -Sandbox Names may contain only alphanumeric characters, dashes, periods, and underscores, and must -be shorter than 64 characters. - -## Tagging - -Sandboxes can also be tagged with arbitrary key-value pairs. These tags can be used -to filter results in `Sandbox.list`. - - - {#snippet python()} - -```python fixture:sb_app -sandbox_v1_1 = modal.Sandbox.create("sleep", "10", app=sb_app) -sandbox_v1_2 = modal.Sandbox.create("sleep", "20", app=sb_app) - -sandbox_v1_1.set_tags({"major_version": "1", "minor_version": "1"}) -sandbox_v1_2.set_tags({"major_version": "1", "minor_version": "2"}) - -for sandbox in modal.Sandbox.list(app_id=sb_app.app_id): # All sandboxes. - print(sandbox.object_id) - -for sandbox in modal.Sandbox.list( - app_id=sb_app.app_id, - tags={"major_version": "1"}, -): # Also all sandboxes. - print(sandbox.object_id) - -for sandbox in modal.Sandbox.list( - app_id=sb_app.app_id, - tags={"major_version": "1", "minor_version": "2"}, -): # Just the latest sandbox. - print(sandbox.object_id) - -sandbox_v1_1.detach() -sandbox_v1_2.detach() -``` - -{/snippet} - -{#snippet python\_async()} - -```python fixture:sb_app -sandbox_v1_1 = await modal.Sandbox.create.aio("sleep", "10", app=sb_app) -sandbox_v1_2 = await modal.Sandbox.create.aio("sleep", "20", app=sb_app) - -await sandbox_v1_1.set_tags.aio({"major_version": "1", "minor_version": "1"}) -await sandbox_v1_2.set_tags.aio({"major_version": "1", "minor_version": "2"}) - -async for sandbox in modal.Sandbox.list.aio(app_id=sb_app.app_id): # All sandboxes. - print(sandbox.object_id) - -async for sandbox in modal.Sandbox.list.aio( - app_id=sb_app.app_id, - tags={"major_version": "1"}, -): # Also all sandboxes. - print(sandbox.object_id) - -async for sandbox in modal.Sandbox.list.aio( - app_id=sb_app.app_id, - tags={"major_version": "1", "minor_version": "2"}, -): # Just the latest sandbox. - print(sandbox.object_id) - -await sandbox_v1_1.detach.aio() -await sandbox_v1_2.detach.aio() -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const sandboxV1_1 = await modal.sandboxes.create(app, image, { - command: ["sleep", "10"], -}); -const sandboxV1_2 = await modal.sandboxes.create(app, image, { - command: ["sleep", "20"], -}); - -await sandboxV1_1.setTags({ major_version: "1", minor_version: "1" }); -await sandboxV1_2.setTags({ major_version: "1", minor_version: "2" }); - -// All sandboxes. -for await (const sandbox of modal.sandboxes.list({ appId: app.appId })) { - console.log(sandbox.sandboxId); -} - -// Also all sandboxes. -for await (const sandbox of modal.sandboxes.list({ - appId: app.appId, - tags: { major_version: "1" }, -})) { - console.log(sandbox.sandboxId); -} - -// Just the latest sandbox. -for await (const sandbox of modal.sandboxes.list({ - appId: app.appId, - tags: { major_version: "1", minor_version: "2" }, -})) { - console.log(sandbox.sandboxId); -} -sandboxV1_1.detach(); -sandboxV1_2.detach(); -``` - -{/snippet} - -{#snippet go()} - -```go notest -sandboxV1_1, err := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Command: []string{"sleep", "10"}, -}) -sandboxV1_2, err := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Command: []string{"sleep", "20"}, -}) -defer sandboxV1_1.Detach() -defer sandboxV1_2.Detach() - -sandboxV1_1.SetTags(ctx, map[string]string{"major_version": "1", "minor_version": "1"}) -sandboxV1_2.SetTags(ctx, map[string]string{"major_version": "1", "minor_version": "2"}) - -// All sandboxes. -it, _ := mc.Sandboxes.List(ctx, &modal.SandboxListParams{ - AppID: app.AppID, -}) -for sandbox := range it { - fmt.Println(sandbox.SandboxID) -} - -// Also all sandboxes. -it, _ = mc.Sandboxes.List(ctx, &modal.SandboxListParams{ - AppID: app.AppID, - Tags: map[string]string{"major_version": "1"}, -}) -for sandbox := range it { - fmt.Println(sandbox.SandboxID) -} - -// Just the latest sandbox. -it, _ = mc.Sandboxes.List(ctx, &modal.SandboxListParams{ - AppID: app.AppID, - Tags: map[string]string{"major_version": "1", "minor_version": "2"}, -}) -for sandbox := range it { - fmt.Println(sandbox.SandboxID) -} -``` - -{/snippet} - -## Cleaning up Client-side Connections - -Unlike other Modal objects, the local Sandbox will hold a direct connection to -its compute substrate. While this connection should be automatically closed -during garbage collection, we recommend explicitly cleaning up the resources -once you are finished interacting with the Sandbox by calling its `detach()` method: - - - {#snippet python()} - -```python fixture:sb_app -sb = modal.Sandbox.create(app=sb_app) -sb.detach() -``` - -{/snippet} - -{#snippet python\_async()} - -```python fixture:sb_app -sb = await modal.Sandbox.create.aio(app=sb_app) -await sb.detach.aio() -``` - -{/snippet} - -{#snippet javascript()} - -```javascript notest -const sb = await modal.sandboxes.create(app, image); -sb.detach(); -``` - -{/snippet} - -{#snippet go()} - -```go notest -sb, err := mc.Sandboxes.Create(ctx, app, image, nil) -defer sb.Detach() -``` - -{/snippet} - -After calling `detach`, any operation using the Sandbox object is not guaranteed to -work. If you want to continue interacting with a running sandbox, use `Sandbox.from_id` -to get a new Sandbox object that references the original sandbox. In the Python SDK, -`terminate` leaves your sandbox attached, so we recommend calling `detach` after you -are done with your terminated sandbox. In the Go/JS SDK, `Terminate` will also detach -your sandbox. diff --git a/.agents/skills/modal/references/guide/scale.md b/.agents/skills/modal/references/guide/scale.md deleted file mode 100644 index 602875133bc70555e3613e6bac3766e4a90db617..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/scale.md +++ /dev/null @@ -1,213 +0,0 @@ -# Scaling out - -Modal makes it trivially easy to scale compute across thousands of containers. -You won't have to worry about your App crashing if it goes viral or need to wait -a long time for your batch jobs to complete. - -For the the most part, scaling out will happen automatically, and you won't need -to think about it. But it can be helpful to understand how Modal's autoscaler -works and how you can control its behavior when you need finer control. - -## How does autoscaling work on Modal? - -Every Modal Function corresponds to an autoscaling pool of containers. The size -of the pool is managed by Modal's autoscaler. The autoscaler will spin up new -containers when there is no capacity available for new inputs, and it will spin -down containers when resources are idling. By default, Modal Functions will -scale to zero when there are no inputs to process. - -Autoscaling decisions are made quickly and frequently so that your batch jobs -can ramp up fast and your deployed Apps can respond to any sudden changes in -traffic. - -## Configuring autoscaling behavior - -Modal exposes a few settings that allow you to configure the autoscaler's -behavior. These settings can be passed to the `@app.function` or `@app.cls` -decorators: - -* `max_containers`: The upper limit on containers for the specific Function. -* `min_containers`: The minimum number of containers that should be kept warm, - even when the Function is inactive. -* `buffer_containers`: The size of the buffer to maintain while the Function is - active, so that additional inputs will not need to queue for a new container. -* `scaledown_window`: The maximum duration (in seconds) that individual - containers can remain idle when scaling down. - -In general, these settings allow you to trade off cost and latency. Maintaining -a larger warm pool or idle buffer will increase costs but reduce the chance that -inputs will need to wait for a new container to start. - -Similarly, a longer scaledown window will let containers idle for longer, which -might help avoid unnecessary churn for Apps that receive regular but infrequent -inputs. Note that containers may not wait for the entire scaledown window before -shutting down if the App is substantially overprovisioned. - -## Dynamic autoscaler updates - -It's also possible to update the autoscaler settings dynamically (i.e., without redeploying -the App) using the [`Function.update_autoscaler()`](/docs/reference/modal.Function#update_autoscaler) -method: - -```python notest -f = modal.Function.from_name("my-app", "f") -f.update_autoscaler(max_containers=100) -``` - -The autoscaler settings will revert to the configuration in the function -decorator the next time you deploy the App. Or they can be overridden by -further dynamic updates: - -```python notest -f.update_autoscaler(min_containers=2, max_containers=10) -f.update_autoscaler(min_containers=4) # max_containers=10 will still be in effect -``` - -A common pattern is to run this method in a [scheduled function](/docs/guide/cron) -that adjusts the size of the warm pool (or container buffer) based on the time of day: - -```python -@app.function() -def inference_server(): - ... - -@app.function(schedule=modal.Cron("0 6 * * *", timezone="America/New_York")) -def increase_warm_pool(): - inference_server.update_autoscaler(min_containers=4) - -@app.function(schedule=modal.Cron("0 22 * * *", timezone="America/New_York")) -def decrease_warm_pool(): - inference_server.update_autoscaler(min_containers=0) -``` - -When you have a [`modal.Cls`](/docs/reference/modal.Cls), `update_autoscaler` -is a method on an *instance* and will control the autoscaling behavior of -containers serving the Function with that specific set of parameters: - -```python notest -MyClass = modal.Cls.from_name("my-app", "MyClass") -obj = MyClass(model_version="3.5") -obj.update_autoscaler(buffer_containers=2) # type: ignore -``` - -Note that it's necessary to disable type checking on this line, because the -object will appear as an instance of the class that you defined rather than the -Modal wrapper type. - -## Parallel execution of inputs - -If your code is running the same function repeatedly with different independent -inputs (e.g., a grid search), the easiest way to increase performance is to run -those function calls in parallel using Modal's -[`Function.map()`](/docs/reference/modal.Function#map) method. - -Here is an example if we had a function `evaluate_model` that takes a single -argument: - -```python -import modal - -app = modal.App() - - -@app.function() -def evaluate_model(x): - ... - - -@app.local_entrypoint() -def main(): - inputs = list(range(100)) - for result in evaluate_model.map(inputs): # runs many inputs in parallel - ... -``` - -In this example, `evaluate_model` will be called with each of the 100 inputs -(the numbers 0 - 99 in this case) roughly in parallel and the results are -returned as an iterable with the results ordered in the same way as the inputs. - -### Exceptions - -By default, if any of the function calls raises an exception, the exception will -be propagated. To treat exceptions as successful results and aggregate them in -the results list, pass in -[`return_exceptions=True`](/docs/reference/modal.Function#map). - -```python -@app.function() -def my_func(a): - if a == 2: - raise Exception("ohno") - return a ** 2 - -@app.local_entrypoint() -def main(): - print(list(my_func.map(range(3), return_exceptions=True))) - # [0, 1, Exception('ohno'))] -``` - -### Starmap - -If your function takes multiple variable arguments, you can either use -[`Function.map()`](/docs/reference/modal.Function#map) with one input iterator -per argument, or [`Function.starmap()`](/docs/reference/modal.Function#starmap) -with a single input iterator containing sequences (like tuples) that can be -spread over the arguments. This works similarly to Python's built in `map` and -`itertools.starmap`. - -```python -@app.function() -def my_func(a, b): - return a + b - -@app.local_entrypoint() -def main(): - assert list(my_func.starmap([(1, 2), (3, 4)])) == [3, 7] -``` - -### Gotchas - -Note that `.map()` is a method on the modal function object itself, so you don't -explicitly *call* the function. - -Incorrect usage: - -```python notest -results = evaluate_model(inputs).map() -``` - -Modal's map is also not the same as using Python's builtin `map()`. While the -following will technically work, it will execute all inputs in sequence rather -than in parallel. - -Incorrect usage: - -```python notest -results = map(evaluate_model, inputs) -``` - -## Asynchronous usage - -All Modal APIs are available in both blocking and asynchronous variants. If you -are comfortable with asynchronous programming, you can use it to create -arbitrary parallel execution patterns, with the added benefit that any Modal -functions will be executed remotely. See the [async guide](/docs/guide/async) or -the examples for more information about asynchronous usage. - -## GPU acceleration - -Sometimes you can speed up your applications by utilizing GPU acceleration. See -the [GPU section](/docs/guide/gpu) for more information. - -## Scaling Limits - -Modal enforces the following limits for every function: - -* 2,000 pending inputs (inputs that haven't been assigned to a container yet) -* 25,000 total inputs (which include both running and pending inputs) - -For inputs created with `.spawn()` for async jobs, Modal allows up to 1 million pending inputs instead of 2,000. - -If you try to create more inputs and exceed these limits, you'll receive a `Resource Exhausted` error, and you should retry your request later. If you need higher limits, please reach out! - -Additionally, each `.map()` invocation can process at most 1000 inputs concurrently. diff --git a/.agents/skills/modal/references/guide/sdk-javascript-go.md b/.agents/skills/modal/references/guide/sdk-javascript-go.md deleted file mode 100644 index 1daefad0db44ed1311b617c4a3979c5252e5dec4..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/sdk-javascript-go.md +++ /dev/null @@ -1,100 +0,0 @@ -# Modal SDKs for JavaScript and Go - - - -Modal also provides SDKs that enable using Modal Functions and Sandboxes from JavaScript/TypeScript and Go projects. - -While Python is the primary language for building Modal applications and implementing Modal Functions, these SDKs enable use cases like: - -* Using Sandboxes in JS/Go projects, to safely execute arbitrary commands, run untrusted user code, or as a safe environment for AI agents. -* Directly calling Modal Functions without having to define it as a public Web Function and address it via HTTP requests -* Interacting with Modal resources like Volumes, Secrets, Queues, etc. directly from JS/Go. - -We're working towards feature parity with the main Modal Python SDK, although defining Modal Functions will likely remain exclusive to Python. - -## Installation - -For installation instructions, see the READMEs for [JavaScript](https://github.com/modal-labs/modal-client/tree/main/js) and [Go](https://github.com/modal-labs/modal-client/tree/main/go) on GitHub. - -## JavaScript/TypeScript - -The `modal` package is [distributed via npm](https://www.npmjs.org/package/modal). See the [JS API reference documentation](https://modal-labs.github.io/libmodal/) for details. - -### Simple JavaScript Example - -```ts -import { ModalClient } from "modal"; - -const modal = new ModalClient(); - -const app = await modal.apps.fromName("libmodal-example", { - createIfMissing: true, -}); - -// Create a Sandbox with the specified Image, and mount a Volume -const volume = await modal.volumes.fromName("libmodal-example-volume", { - createIfMissing: true, -}); -const image = modal.images.fromRegistry("alpine:3.21"); -const sb = await modal.sandboxes.create(app, image, { - volumes: { "/mnt/volume": volume }, -}); -const p = await sb.exec(["cat", "/mnt/volume/message.txt"]); -console.log(`Message: ${await p.stdout.readText()}`); -await sb.terminate(); - -// Call a previously deployed Modal Function -const echo = await modal.functions.fromName("libmodal-example", "echo"); -console.log(await echo.remote(["Hello world!"])); -``` - -There are [many more examples available on GitHub](https://github.com/modal-labs/modal-client/blob/main/js/README.md#documentation). - -## Go - -The `modal-go` package is [installed via go get](https://pkg.go.dev/github.com/modal-labs/modal-client/go). See the [Go API reference documentation](https://pkg.go.dev/github.com/modal-labs/modal-client/go#section-documentation) for details. - -### Simple Go Example - -```go -package main - -import ( - "context" - "fmt" - "io" - - modal "github.com/modal-labs/modal-client/go" -) - -func main() { - // Skipping err handling throughout for brevity - ctx := context.Background() - - mc, _ := modal.NewClient() - - app, _ := mc.Apps.FromName(ctx, "libmodal-example", &modal.AppFromNameParams{CreateIfMissing: true}) - - // Create a Sandbox with the specified Image, and mount a Volume - volume, _ := mc.Volumes.FromName(ctx, "libmodal-example-volume", &modal.VolumeFromNameParams{CreateIfMissing: true}) - image := mc.Images.FromRegistry("alpine:3.21", nil) - sb, _ := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{ - Volumes: map[string]*modal.Volume{"/mnt/volume": volume}, - }) - defer sb.Terminate(context.Background(), nil) - p, _ := sb.Exec(ctx, []string{"cat", "/mnt/volume/message.txt"}, nil) - stdout, _ := io.ReadAll(p.Stdout) - fmt.Printf("Message: %s\n", stdout) - - // Call a previously deployed Modal Function - echo, _ := mc.Functions.FromName(ctx, "libmodal-example", "echo", nil) - result, _ := echo.Remote(ctx, []any{"Hello world!"}, nil) - fmt.Println(result) -} -``` - -There are [many more examples available on GitHub](https://github.com/modal-labs/modal-client/blob/main/go/README.md#documentation). - -## Support - -The JS and Go Modal SDKs are in active development, and we love to hear your feedback. If you have questions or suggestions, please reach out on the [Modal Community Slack](https://modal.com/slack). diff --git a/.agents/skills/modal/references/guide/secrets.md b/.agents/skills/modal/references/guide/secrets.md deleted file mode 100644 index 716aec35a8dbfd9602d2402da5236779ca439c06..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/secrets.md +++ /dev/null @@ -1,125 +0,0 @@ -# Secrets - -Securely provide credentials and other sensitive information to your Modal Functions with Secrets. - -You can create and edit Secrets via -the [dashboard](/secrets), -the command line interface ([`modal secret`](/docs/reference/cli/secret)), and -programmatically from Python code ([`modal.Secret`](/docs/reference/modal.Secret)). - -To inject Secrets into the container running your Function, add the -`secrets=[...]` argument to your `app.function` or `app.cls` decoration. - -## Limits - -Each key-value pair in a Secret is subject to the following limits: - -* **Key names** can be at most **16,384 characters** long. They must contain only letters, digits, and underscores, and cannot start with a digit. -* **Values** can be at most **32,768 characters** long. - -If you need to provide larger values to your containers, consider writing the data to a [Volume](/docs/guide/volumes) and reading it at runtime. - -## Deploy Secrets from the Modal Dashboard - -The most common way to create a Modal Secret is to use the -[Secrets panel of the Modal dashboard](/secrets), -which also shows any existing Secrets. - -When you create a new Secret, you'll be prompted with a number of templates to help you get started. -These templates demonstrate standard formats for credentials for everything from Postgres and MongoDB -to Weights & Biases and Hugging Face. - -## Use Secrets in your Modal Apps - -You can then use your Secret by constructing it `from_name` when defining a Modal App -and then accessing its contents as environment variables. -For example, if you have a Secret called `secret-keys` containing the key -`MY_PASSWORD`: - -```python -@app.function(secrets=[modal.Secret.from_name("secret-keys")]) -def some_function(): - import os - - secret_key = os.environ["MY_PASSWORD"] - ... -``` - -Each Secret can contain multiple keys and values but you can also inject -multiple Secrets, allowing you to separate Secrets into smaller reusable units: - -```python -@app.function(secrets=[ - modal.Secret.from_name("my-secret-name"), - modal.Secret.from_name("other-secret"), -]) -def other_function(): - ... -``` - -The Secrets are applied in order, so key-values from later `modal.Secret` -objects in the list will overwrite earlier key-values in the case of a clash. -For example, if both `modal.Secret` objects above contained the key `FOO`, then -the value from `"other-secret"` would always be present in `os.environ["FOO"]`. - -## Create Secrets programmatically - -In addition to defining Secrets on the web dashboard, you can -programmatically create a Secret directly in your script and send it along to -your Function using `Secret.from_dict(...)`. This can be useful if you want to -send Secrets from your local development machine to the remote Modal App. - -```python -import os - -if modal.is_local(): - local_secret = modal.Secret.from_dict({"FOO": os.environ["LOCAL_FOO"]}) -else: - local_secret = modal.Secret.from_dict({}) - - -@app.function(secrets=[local_secret]) -def some_function(): - import os - - print(os.environ["FOO"]) -``` - -If you have [`python-dotenv`](https://pypi.org/project/python-dotenv/) installed, -you can also use `Secret.from_dotenv()` to create a Secret from the variables in a `.env` -file - -```python -@app.function(secrets=[modal.Secret.from_dotenv()]) -def some_other_function(): - print(os.environ["USERNAME"]) -``` - -## Interact with Secrets from the command line - -You can create, list, and delete your Modal Secrets with the `modal secret` command line interface. - -View your Secrets and their timestamps with - -```bash -modal secret list -``` - -Create a new Secret by passing `{KEY}={VALUE}` pairs to `modal secret create`: - -```bash -modal secret create database-secret PGHOST=uri PGPORT=5432 PGUSER=admin PGPASSWORD=hunter2 -``` - -or using environment variables (assuming below that the `PGPASSWORD` environment variable is set -e.g. by your CI system): - -```bash -modal secret create database-secret PGHOST=uri PGPORT=5432 PGUSER=admin PGPASSWORD="$PGPASSWORD" -``` - -Remove Secrets by passing their name to `modal secret delete`: - -```bash -modal secret delete database-secret -``` diff --git a/.agents/skills/modal/references/guide/security.md b/.agents/skills/modal/references/guide/security.md deleted file mode 100644 index 97070122927cb21f3a4af275b2a6381b586a2036..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/security.md +++ /dev/null @@ -1,169 +0,0 @@ -# Security and privacy at Modal - -The document outlines Modal's security and privacy commitments. - -## Application security (AppSec) - -AppSec is the practice of building software that is secure by design, secured -during development, secured with testing and review, and deployed securely. - -* We build our software using memory-safe programming languages, including Rust - (for our worker runtime and storage infrastructure) and Python (for our API - servers and Modal client). -* Software dependencies are audited by Github's Dependabot. -* We make decisions that minimize our attack surface. Most interactions with - Modal are well-described in a gRPC API, and occur through - [`modal`](https://pypi.org/project/modal), our open-source command-line tool - and Python client library. -* We have automated synthetic monitoring test applications that continuously - check for network and application isolation within our runtime. -* We use HTTPS for secure connections. Modal forces HTTPS for all services using - TLS (SSL), including our public website and the Dashboard to ensure secure - connections. Modal's [client library](https://pypi.org/project/modal) connects - to Modal's servers over TLS and verify TLS certificates on each connection. -* All user data is encrypted in transit and at rest. -* All public Modal APIs use - [TLS 1.3](https://datatracker.ietf.org/doc/html/rfc8446), the latest and - safest version of the TLS protocol. -* Internal code reviews are performed using a modern, PR-based development - workflow (Github), and engage external penetration testing firms to assess our - software security. - -## Corporate security (CorpSec) - -CorpSec is the practice of making sure Modal employees have secure access to -Modal company infrastructure, and also that exposed channels to Modal are -secured. CorpSec controls are the primary concern of standards such as SOC2. - -* Access to our services and applications is gated on a SSO Identity Provider - (IdP). -* We mandate phishing-resistant multi-factor authentication (MFA) in all - enrolled IdP accounts. -* We regularly audit access to internal systems. -* Employee laptops are protected by full disk encryption using FileVault2, and - managed by Secureframe MDM. - -## Network and infrastructure security (InfraSec) - -InfraSec is the practice of ensuring a hardened, minimal attack surface for -components we deploy on our network. - -* Modal uses logging and metrics observability providers, including Datadog and - Sentry.io. -* Compute jobs at Modal are containerized and virtualized using - [gVisor](https://github.com/google/gvisor), the sandboxing technology - developed at Google and used in their *Google Cloud Run* and *Google - Kubernetes Engine* cloud services. -* We conduct annual business continuity and security incident exercises. - - - -## Vulnerability remediation - -Security vulnerabilities directly affecting Modal's systems and services will be -patched or otherwise remediated within a timeframe appropriate for the severity -of the vulnerability, subject to the public availability of a patch or other -remediation mechanisms. - -If there is a CVSS severity rating accompanying a vulnerability disclosure, we -rely on that as a starting point, but may upgrade or downgrade the severity -using our best judgement. - -### Severity timeframes - -* **Critical:** 24 hours -* **High:** 1 week -* **Medium:** 1 month -* **Low:** 3 months -* **Informational:** 3 months or longer - -## Shared responsibility model - -Modal prioritizes the integrity, security, and availability of customer data. Under our shared responsibility model, customers also have certain responsibilities regarding data backup, recovery, and availability. - -1. **Data backup**: Customers are responsible for maintaining backups of their data. Performing daily backups is recommended. Customers must routinely verify the integrity of their backups. -2. **Data recovery**: Customers should maintain a comprehensive data recovery plan that includes detailed procedures for data restoration in the event of data loss, corruption, or system failure. Customers must routinely test their recovery process. -3. **Availability**: While Modal is committed to high service availability, customers must implement contingency measures to maintain business continuity during service interruptions. Customers are also responsible for the reliability of their own IT infrastructure. -4. **Security measures**: Customers must implement appropriate security measures, such as encryption and access controls, to protect their data throughout the backup, storage, and recovery processes. These processes must comply with all relevant laws and regulations. - -## SOC 2 - -We have successfully completed a [System and Organization Controls (SOC) 2 Type 2 -audit](/blog/soc2type2). Go to our [Security Portal](https://trust.modal.com) to request access to the report. - -## HIPAA - -HIPAA, which stands for the Health Insurance Portability and Accountability Act, establishes a set of standards that protect health information, including individuals’ medical records and other individually identifiable health information. HIPAA guidelines apply to both covered entities and business associates—of which Modal is the latter if you are processing PHI on Modal. - -Modal's services can be used in a HIPAA compliant manner. It is important to note that unlike other security standards, there is no officially recognized certification process for HIPAA compliance. Instead, we demonstrate our compliance with regulations such as HIPAA via the practices outlined in this doc, our technical and operational security measures, and through official audits for standards compliance such as SOC 2 certification. - -To use Modal services for HIPAA-compliant workloads, a Business Associate Agreement (BAA) should be established with us prior to submission of any PHI. This is available on our Enterprise plan. Contact us at security@modal.com to get started. At the moment, [Volumes v1](https://modal.com/docs/guide/volumes), [Images](https://modal.com/docs/guide/images) (excluding [Filesystem and Directory Snapshots](/docs/guide/sandbox-snapshots)), [Memory Snapshots](https://modal.com/docs/guide/memory-snapshots), and user code are out of scope of the commitments within our BAA, so PHI should not be used in those areas of the product. - -[Volumes v2](https://modal.com/docs/guide/volumes#volumes-v2) are HIPAA compliant. - -## PCI - -*Payment Card Industry Data Security Standard* (PCI) is a standard that defines -the security and privacy requirements for payment card processing. - -Modal uses [Stripe](https://stripe.com) to securely process transactions and -trusts their commitment to best-in-class security. We do not store personal -credit card information for any of our customers. Stripe is certified as "PCI -Service Provider Level 1", which is the highest level of certification in the -payments industry. - -## Bug bounty program - -Keeping user data secure is a top priority at Modal. We welcome contributions -from the security community to identify vulnerabilities in our product and -disclose them to us in a responsible manner. We currently run a private bug -bounty program through HackerOne. If you have found a vulnerability and -wish to participate, please send an email to security@modal.com with your -HackerOne username or email and we will invite you to the program. - -## Data privacy - -Modal will never access or use: - -* your source code. -* the inputs (function arguments) or outputs (function return values) to your Modal Functions. -* any data you store in Modal, such as in Images or Volumes. - -App logs and metadata are stored on Modal. Modal will not access this data -unless permission is granted by the user to help with troubleshooting. - -### Data retention - -Different Modal products have different retention policies for the data they -handle. The table below summarizes how long each type of data is retained. - -| Data | Product | Retention | -| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | -| Inputs and outputs | [Functions](/docs/guide) (`.remote`, `.spawn`, `.map`, [Web Functions](/docs/guide/webhooks), Scheduled Functions) | Up to 7 days, then deleted | -| Request and response payloads | Modal Inference endpoints | Not stored — proxied directly to your container | -| App and container logs | [Functions](/docs/guide), [Sandboxes](/docs/guide/sandboxes) | Plan-dependent: 1 day on Starter, 30 days on Team, configurable on Enterprise (see [pricing](/pricing)) | -| Audit logs | Workspace (Enterprise) | Per Enterprise contract (see [Audit logs](/docs/guide/audit-logs)) | -| Files | [Volumes](/docs/guide/volumes), [Images](/docs/guide/images) | Persistent until you delete them | -| Memory snapshots | [Function memory snapshots](/docs/guide/memory-snapshots), [Sandbox memory snapshots](/docs/guide/sandbox-snapshots) | 7 days after creation | -| Filesystem snapshots | [Sandbox filesystem snapshots](/docs/guide/sandbox-snapshots) | 30 days after creation (configurable; stored as Images) | -| Directory snapshots | [Sandbox directory snapshots](/docs/guide/sandbox-snapshots) | 30 days after creation (configurable) | -| Entries | [Dicts](/docs/guide/dicts) | 7 days after last read or write | -| Partitions | [Queues](/docs/guide/queues) | Configurable per-partition TTL (default 24 hours) | -| App, Function, and container metadata | All products | Stored for the lifetime of your account | - -#### Function inputs and outputs - -Function inputs and outputs are stored encrypted at rest. Small payloads -(≤ 2 MiB) are stored inline in our metadata store; larger payloads are stored -in object storage. Both are deleted within a maximum TTL of 7 days. - -#### Modal Inference endpoints - -Modal Inference endpoints are zero data retention: request and response payloads -are never written to disk and pass through Modal's infrastructure only as -in-flight network traffic. Inference endpoints terminate TLS at Modal's edge -proxy and forward requests directly to your containers over an internal tunnel. - -## Questions? - -[Email us!](mailto:security@modal.com) diff --git a/.agents/skills/modal/references/guide/service-users.md b/.agents/skills/modal/references/guide/service-users.md deleted file mode 100644 index 7d73238529f6f764a48de11af4b137d830c9dd7a..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/service-users.md +++ /dev/null @@ -1,56 +0,0 @@ -# Service Users - -Service users are programmatic accounts that allow automated systems to interact with Modal. They're ideal for CI/CD pipelines, automated deployments, and other workflows that need to authenticate. - -## Create a Service User - -Service users are only available for shared workspaces. You will need workspace owner or manager privileges to create service users. - -To create a service user: - -1. Go to your workspace [tokens settings page](/settings/tokens/service-users) -2. Click **New Service User** -3. Enter a name for your service user (must be lowercase alphanumeric, can contain hyphens or underscores) -4. Click **Create** - -After creation, you'll see the `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`. **This is the only time you can view the token secret** for security reasons. - -## Use Service User Tokens - -Set the service user credentials as environment variables in your automated environment: - -```bash -export MODAL_TOKEN_ID=your-token-id -export MODAL_TOKEN_SECRET=your-token-secret -``` - -Once configured, you can use Modal's CLI and Python SDK as usual: - -```bash -modal deploy your_app.py -``` - -## Delete a Service User - -To remove a service user: - -1. Go to the [tokens settings page](/settings/tokens/service-users) -2. Find the service user in the table -3. Click **Delete** when you hover over the row - -## Permissions - -Service users have the same permissions as workspace members. They cannot do actions that are only permitted for a workspace owner or manager. To learn more about members, managers, and owners, see this [workspace](/docs/guide/workspaces#administrating-workspace-members) section. - -## Securing Service Users - -Because service user tokens are long-lived and used in automated environments, it's important to limit their access to only what's necessary: - -* **Store tokens securely.** Use a secrets manager or your CI/CD platform's built-in secrets storage rather than hardcoding tokens in source code or configuration files. -* **Use restricted Environments.** Assign service users the **Contributor** role only on the specific Environments they need access to, keeping production isolated from development and staging. - - - -Modal supports [Role-Based Access Control (RBAC)](/docs/guide/rbac) to restrict access at both the Workspace and Environment level. Use RBAC to ensure your service users can only deploy to the Environments they're authorized for. - - diff --git a/.agents/skills/modal/references/guide/slack-notifications.md b/.agents/skills/modal/references/guide/slack-notifications.md deleted file mode 100644 index dfeddf92ccd24f66668616259c14cccf97f1c748..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/slack-notifications.md +++ /dev/null @@ -1,52 +0,0 @@ -# Slack notifications - - - -You can integrate your Modal Workspace with Slack to receive timely essential notifications. - -## Prerequisites - -* You are a [Workspace Manager](/docs/guide/workspaces#administrating-workspace-members) in the Modal Workspace you're installing the Slack integration in. -* You have permissions to install apps in your Slack workspace. - -## Supported notifications - -* Alerts for failed scheduled function runs. -* Alerts for crash-looping containers in a function. -* Alerts when any of your apps have client versions that are out of date. -* Alerts when you hit your GPU resource limits. - -## Slack Permissions - -The Modal Slack app requests the following permissions to integrate with Slack: - -* Start direct messages with people -* Send messages as @modal -* Add shortcuts and/or slash commands that people can use -* View basic information about public channels in a workspace -* View basic information about private channels that Modal has been added to -* View basic information about direct messages that Modal has been added to -* View basic information about group direct messages that Modal has been added to -* View people in a workspace - -## Configuration - -### Step 1: Install the Slack integration - -Visit the *Slack Notifications* section on your [settings](/settings/slack-notifications) page in your Modal Workspace and click the **Add to Slack** button. - -### Step 2: Invite the Modal app to your Slack channel - -Navigate to the Slack channel and `/invite` the Modal app so that the app can post messages to the channel. - -![Adding an app to Slack channel](https://modal-cdn.com/cdnbot/slack-invite-app_vpxfskj_f0dc9524.webp) - -### Step 3: Add the Modal app to your Slack channel - -Navigate to the Slack channel you want to add the Modal app to and click on the channel header. On the integrations tab you can add the Modal app. - -![Add Modal app to Slack channel](../../assets/docs/slack-add-modal-app.jpg) - -### Step 4: Use `/modal link` to link the Slack channel to your Modal Workspace - -You'll be prompted to select the Workspace you want to link to the Slack channel. You can always unlink the Slack channel by visiting the *Slack Notifications* section on your [settings](/settings/slack-notifications) page in your Modal Workspace. diff --git a/.agents/skills/modal/references/guide/streaming-endpoints.md b/.agents/skills/modal/references/guide/streaming-endpoints.md deleted file mode 100644 index 5895f2d8b200388f054183000c46e8078a33e4fc..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/streaming-endpoints.md +++ /dev/null @@ -1,126 +0,0 @@ -# Streaming endpoints - -Modal `fastapi_endpoint`s support streaming responses using FastAPI's -[`StreamingResponse`](https://fastapi.tiangolo.com/advanced/custom-response/#streamingresponse) -class. This class accepts asynchronous generators, synchronous generators, or -any Python object that implements the -[*iterator protocol*](https://docs.python.org/3/library/stdtypes.html#typeiter), -and can be used with Modal Functions! - -## Simple example - -This simple example combines Modal's `@modal.fastapi_endpoint` decorator with a -`StreamingResponse` object to produce a real-time SSE response. - -```python -import time - -def fake_event_streamer(): - for i in range(10): - yield f"data: some data {i}\n\n".encode() - time.sleep(0.5) - - -@app.function(image=modal.Image.debian_slim().pip_install("fastapi[standard]")) -@modal.fastapi_endpoint() -def stream_me(): - from fastapi.responses import StreamingResponse - return StreamingResponse( - fake_event_streamer(), media_type="text/event-stream" - ) -``` - -If you serve this Web Function and hit it with `curl`, you will see the ten SSE -events progressively appear in your terminal over a ~5 second period. - -```shell -curl --no-buffer https://modal-labs--example-streaming-stream-me.modal.run -``` - -The MIME type of `text/event-stream` is important in this example, as it tells -the downstream web server to return responses immediately, rather than buffering -them in byte chunks (which is more efficient for compression). - -You can still return other content types like large files in streams, but they -are not guaranteed to arrive as real-time events. - -## Streaming responses with `.remote` - -A Modal Function wrapping a generator function body can have its response passed -directly into a `StreamingResponse`. This is particularly useful if you want to -do some GPU processing in one Modal Function that is called by a CPU-based web -endpoint Modal Function. - -```python -@app.function(gpu="any") -def fake_video_render(): - for i in range(10): - yield f"data: finished processing some data from GPU {i}\n\n".encode() - time.sleep(1) - - -@app.function(image=modal.Image.debian_slim().pip_install("fastapi[standard]")) -@modal.fastapi_endpoint() -def hook(): - from fastapi.responses import StreamingResponse - return StreamingResponse( - fake_video_render.remote_gen(), media_type="text/event-stream" - ) -``` - -## Streaming responses with `.map` and `.starmap` - -You can also combine Modal Function parallelization with streaming responses, -enabling applications to service a request by farming out to dozens of -containers and iteratively returning result chunks to the client. - -```python -@app.function() -def map_me(i): - return f"segment {i}\n" - - -@app.function(image=modal.Image.debian_slim().pip_install("fastapi[standard]")) -@modal.fastapi_endpoint() -def mapped(): - from fastapi.responses import StreamingResponse - return StreamingResponse(map_me.map(range(10)), media_type="text/plain") -``` - -This snippet will spread the ten `map_me(i)` executions across containers, and -return each string response part as it completes. By default the results will be -ordered, but if this isn't necessary pass `order_outputs=False` as keyword -argument to the `.map` call. - -### Asynchronous streaming - -The example above uses a synchronous generator, which automatically runs on its -own thread, but in asynchronous applications, a loop over a `.map` or `.starmap` -call can block the event loop. This will stop the `StreamingResponse` from -returning response parts iteratively to the client. - -To avoid this, you can use the `.aio()` method to convert a synchronous `.map` -into its async version. Also, other blocking calls should be offloaded to a -separate thread with `asyncio.to_thread()`. For example: - -```python -@app.function(gpu="any", image=modal.Image.debian_slim().pip_install("fastapi[standard]")) -@modal.fastapi_endpoint() -async def transcribe_video(request): - from fastapi.responses import StreamingResponse - - segments = await asyncio.to_thread(split_video, request) - return StreamingResponse(wrapper(segments), media_type="text/event-stream") - - -# Notice that this is an async generator. -async def wrapper(segments): - async for partial_result in transcribe_video.map.aio(segments): - yield "data: " + partial_result + "\n\n" -``` - -## Further examples - -* Complete code for the simple examples given above is available - [in our modal-examples Github repository](https://github.com/modal-labs/modal-examples/blob/main/07_web_endpoints/streaming.py). -* [An end-to-end example of streaming Youtube video transcriptions with OpenAI's whisper model.](https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/openai_whisper/streaming/main.py) diff --git a/.agents/skills/modal/references/guide/timeouts.md b/.agents/skills/modal/references/guide/timeouts.md deleted file mode 100644 index e0079d4896f52ba8026bccef80e4f5fd09a518cf..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/timeouts.md +++ /dev/null @@ -1,80 +0,0 @@ -# Timeouts - -All Modal [Function](/docs/reference/modal.Function) executions have a default -execution timeout of 300 seconds (5 minutes), but users may specify timeout -durations between 1 second and 24 hours. - -```python -import time - - -@app.function() -def f(): - time.sleep(599) # Timeout! - - -@app.function(timeout=600) -def g(): - time.sleep(599) - print("*Just* made it!") -``` - -The timeout duration is a measure of a Function's *execution* time. It does not -include scheduling time or any other period besides the time your code is -executing in Modal. This duration is also per execution attempt, meaning -Functions configured with [`modal.Retries`](/docs/reference/modal.Retries) will -start new execution timeouts on each retry. For example, an infinite-looping -Function with a 100 second timeout and 3 allowed retries will run for least 400 -seconds within Modal. - -### Container startup timeout - -A Function's `startup_timeout` configures the container's *startup* time. Your container -may be taking a long time to startup because it is loading large data, initializing a -large model or importing many packages. In these cases, you can extend the -`startup_timeout` of your Function. - -```python -@app.cls(startup_timeout=30, timeout=10) -class MyFunction: - @modal.enter() - def startup(self): - time.sleep(20) - - @modal.method() - def f(self): - time.sleep(1) -``` - -`startup_timeout` was added in v1.1.4. Prior to v1.1.4, `timeout` configures the -*execution* time and *startup* time. If `startup_timeout` is not set, `timeout` will -still configure both times. - -## Handling timeouts - -After exhausting any specified retries, a timeout in a Function will produce a -`modal.exception.FunctionTimeoutError` which you may catch in your code. - -```python -import modal.exception - - -@app.function(timeout=100) -def f(): - time.sleep(200) # Timeout! - - -@app.local_entrypoint() -def main(): - try: - f.remote() - except modal.exception.FunctionTimeoutError: - ... # Handle the timeout. -``` - -## Timeout accuracy - -Functions will run for *at least* as long as their timeout allows, but they may -run a handful of seconds longer. If you require accurate and precise timeout -durations on your Function executions, it is recommended that you implement -timeout logic in your user code. diff --git a/.agents/skills/modal/references/guide/trigger-deployed-functions.md b/.agents/skills/modal/references/guide/trigger-deployed-functions.md deleted file mode 100644 index fbc0d06eb46ad243b39b5ca6f617b19d4c972f2c..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/trigger-deployed-functions.md +++ /dev/null @@ -1,205 +0,0 @@ -# Invoking deployed functions - -Modal lets you take a function created by a -[deployment](/docs/guide/managing-deployments) and call it from other contexts. - -There are two ways of invoking deployed functions. If the invoking client is -running Python, then the same -[Modal client library](https://pypi.org/project/modal/) used to write Modal code -can be used. HTTPS is used if the invoking client is not running Python and -therefore cannot import the Modal client library. - -## Invoking with Python - -Some use cases for Python invocation include: - -* An existing Python web server (eg. Django, Flask) wants to invoke Modal - functions. -* You have split your product or system into multiple Modal applications that - deploy independently and call each other. - -### Function lookup and invocation basics - -Let's say you have a script `my_shared_app.py` and this script defines a Modal -app with a function that computes the square of a number: - -```python -import modal - -app = modal.App("my-shared-app") - - -@app.function() -def square(x: int): - return x ** 2 -``` - -You can deploy this app to create a persistent deployment: - -``` -% modal deploy shared_app.py -✓ Initialized. -✓ Created objects. -├── 🔨 Created square. -├── 🔨 Mounted /Users/erikbern/modal/shared_app.py. -✓ App deployed! 🎉 - -View Deployment: https://modal.com/apps/erikbern/my-shared-app -``` - -Let's try to run this function from a different context. For instance, let's -fire up the Python interactive interpreter: - -```bash -% python -Python 3.9.5 (default, May 4 2021, 03:29:30) -[Clang 12.0.0 (clang-1200.0.32.27)] on darwin -Type "help", "copyright", "credits" or "license" for more information. ->>> import modal ->>> f = modal.Function.from_name("my-shared-app", "square") ->>> f.remote(42) -1764 ->>> -``` - -This works exactly the same as a regular modal `Function` object. For example, -you can `.map()` over functions invoked this way too: - -```bash ->>> f = modal.Function.from_name("my-shared-app", "square") ->>> f.map([1, 2, 3, 4, 5]) -[1, 4, 9, 16, 25] -``` - -#### Authentication - -The Modal Python SDK will read the token from `~/.modal.toml` which typically is -created using `modal token new`. - -Another method of providing the credentials is to set the environment variables -`MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`. If you want to call a Modal function -from a context such as a web server, you can expose these environment variables -to the process. - -#### Lookup of lifecycle functions - -[Lifecycle functions](/docs/guide/lifecycle-functions) are defined on classes, -which you can look up in a different way. Consider this code: - -```python -import modal - -app = modal.App("my-shared-app") - - -@app.cls() -class MyLifecycleClass: - @modal.enter() - def enter(self): - self.var = "hello world" - - @modal.method() - def foo(self): - return self.var -``` - -Let's say you deploy this app. You can then call the function by doing this: - -```bash ->>> cls = modal.Cls.from_name("my-shared-app", "MyLifecycleClass") ->>> obj = cls() # You can pass any constructor arguments here ->>> obj.foo.remote() -'hello world' -``` - -### Asynchronous invocation - -In certain contexts, a Modal client will need to trigger Modal functions without -waiting on the result. This is done by spawning functions and receiving a -[`FunctionCall`](/docs/reference/modal.FunctionCall) as a -handle to the triggered execution. - -The following is an example of a Flask web server (running outside Modal) which -accepts model training jobs to be executed within Modal. Instead of the HTTP -POST request waiting on a training job to complete, which would be infeasible, -the relevant Modal function is spawned and the -[`FunctionCall`](/docs/reference/modal.FunctionCall) -object is stored for later polling of execution status. - -```python -from uuid import uuid4 -from flask import Flask, jsonify, request - -app = Flask(__name__) -pending_jobs = {} - -... - -@app.route("/jobs", methods = ["POST"]) -def create_job(): - predict_fn = modal.Function.from_name("example", "train_model") - job_id = str(uuid4()) - function_call = predict_fn.spawn( - job_id=job_id, - params=request.json, - ) - pending_jobs[job_id] = function_call - return { - "job_id": job_id, - "status": "pending", - } -``` - -### Importing a Modal function between Modal apps - -You can also import one function defined in an app from another app: - -```python -import modal - -app = modal.App("another-app") - -square = modal.Function.from_name("my-shared-app", "square") - - -@app.function() -def cube(x): - return x * square.remote(x) - - -@app.local_entrypoint() -def main(): - assert cube.remote(42) == 74088 -``` - -### Comparison with HTTPS - -Compared with HTTPS invocation, Python invocation has the following benefits: - -* Uses the Modal client library's built-in authentication. - * Web Functions are public to the entire internet, whereas Function lookup - is private to your workspace and authenticated via your Modal token. -* You can work with shared Modal Functions as if they are normal Python - functions, which might be more convenient. - -## Invoking with HTTPS - -Any application that can make HTTPS requests can interact with deployed Modal -applications via [Web Functions](/docs/guide/webhooks). Note that -all deployed Web Functions have [a stable HTTPS -URL](/docs/guide/webhook-urls). - -Some use cases for HTTPS invocation include: - -* Calling Modal functions from a web browser client running JavaScript -* Calling Modal functions from backend services in languages we don't yet have - official SDKs for (Java, Ruby, etc.) -* Calling Modal functions using UNIX tools (`curl`, `wget`) - -However, if the client of your Modal deployment is running Python, JavaScript, -or Go, it's better to use the [Modal Python -SDK](https://pypi.org/project/modal/) or [Modal SDKs for JavaScript and -Go](/docs/guide/sdk-javascript-go) to invoke your Modal code. - -For more detail on setting up functions for invocation over HTTP see the -[Web Functions guide](/docs/guide/webhooks). diff --git a/.agents/skills/modal/references/guide/troubleshooting.md b/.agents/skills/modal/references/guide/troubleshooting.md deleted file mode 100644 index 891d4476913479b36436522a7c1d8879bcccfa53..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/troubleshooting.md +++ /dev/null @@ -1,138 +0,0 @@ -# Troubleshooting - -This guide page documents solutions for common Modal issues. - -For tips on troubleshooting your own code running on Modal, -see [this guide page](/docs/guide/developing-debugging). - -## "Command not found" errors - -If you installed Modal but you're seeing an error like -`modal: command not found` when trying to run the CLI, this means that the -installation location of Python package executables ("binaries") are not present -on your system path. This is a common problem; you need to reconfigure your -system's environment variables to fix it. - -One workaround is to use `python -m modal` instead of `modal`. However, this -is just a patch. There's no single solution for the problem, because Python -installs dependencies on different locations depending on your environment. See -this [popular StackOverflow question](https://stackoverflow.com/q/35898734) for -pointers on how to resolve your system path issue. - -## Function side effects - -The same container *can* be reused for multiple invocations of the same Function -within an App. This means that if your Function has side effects like modifying -files on disk, they may or may not be present for subsequent calls to that -Function. You should not rely on the side effects to be present, but you might -have to be careful so they don't cause problems. - -For example, if you create a disk-backed database using sqlite3: - -```python -import modal -import sqlite3 - -app = modal.App() - -@app.function() -def db_op(): - db = sqlite3("db_file.sqlite3") - db.execute("CREATE TABLE example (col_1 TEXT)") - ... -``` - -This Function *can* (but will not necessarily) fail on the second invocation -with an `OperationalError: table foo already exists` error. - -To get around this, take care to either clean up your side effects (e.g. -deleting the db file at the end your function call above) or make your Functions -take them into consideration (e.g. adding an -`if os.path.exists("db_file.sqlite")` condition or randomize the filename -above). Alternatively, you can set `single_use_containers=True` so that every -Function call will spin up a new container; however, note that this will result -in higher cost and worse latency as every invocation will require a cold start. - -## Heartbeat timeout - -The Modal client in `modal.Function` containers runs a heartbeat loop that the host uses to healthcheck the container's main process. -If the container stops heartbeating for a long period (minutes), the container will be terminated due to a `heartbeat timeout`, which is displayed in logs. - -Container heartbeat timeouts are rare, and they are typically caused by one of two application-level sources: - -* [Global Interpreter Lock](https://wiki.python.org/moin/GlobalInterpreterLock) is held for a long time, stopping the heartbeat thread from making progress. [py-spy](https://github.com/benfred/py-spy?tab=readme-ov-file#how-does-gil-detection-work) can detect GIL holding. We include `py-spy` [automatically in `modal shell`](/docs/guide/developing-debugging#debug-shells) for convenience. A quick fix for GIL holding is to run the code which holds the GIL [in a subprocess](https://docs.python.org/3/library/multiprocessing.html#the-process-class). -* Container process initiates shutdown, intentionally stopping the heartbeats, but it does not complete shutdown. - -In both cases [turning on debug logging](/docs/guide/developing-debugging#debug-logs) will help diagnose the issue. - -## `413 Content Too Large` errors - -If you receive a `413 Content Too Large` error, this might be because you are -hitting our gRPC payload size limits. - -The size limit is currently 100MB. - -## Outdated kernel version (4.4.0) - -Our secure runtime [reports a misleadingly old](https://github.com/google/gvisor/issues/11117) kernel version, 4.4.0. -Certain software libraries will detect this and report a warning. These warnings can be ignored because the runtime -actually implements Linux kernel features from versions 5.15+. - -If the outdated kernel version reporting creates errors in your application please contact us [in our Slack](https://modal.com/slack). - -## CUDA driver initialization failed on L4 GPU type - -Certain L4 instance types within Modal's fleet have a flaky issue in the NVIDIA driver which causes -the following CUDA context initialization error: - -``` -RuntimeError: CUDA driver initialization failed, you might not have a CUDA gpu. -``` - -A workaround to ensure reliable container startup is given below: - -```python -@modal.enter() -def warmup_cuda(self): - import ctypes - import time - import modal - cu = ctypes.CDLL("libcuda.so.1") - max_retries = 10 - retry_delay_secs = 0.5 - for attempt in range(max_retries): - rc = cu.cuInit(0) - if rc == 0: - break - else: - if attempt < max_retries - 1: - print(f"cuInit failed on attempt {attempt + 1}/{max_retries} with code {rc}, retrying...") - time.sleep(retry_delay_secs) - else: - print(f"CUDA initialization failed after {max_retries} attempts; stopping container") - modal.experimental.stop_fetching_inputs() -``` - -We are investigating a root cause fix for this problem. -Multi-cloud GPU reliability at the scale of many thousands of GPUs is a tough technical challenge! -Read more about our solution [here](/blog/gpu-health). - -## Connection issues in forked processes - -When a process is forked, the child may inherit stale network state from -the parent. If you're using Modal from a forked process (e.g. Celery prefork -workers, `multiprocessing`), create a fresh client after the fork and pass it -explicitly: - -```python -import multiprocessing -import modal - -def child(): - client = modal.Client.from_credentials(token_id, token_secret) - fc = modal.FunctionCall.from_id(call_id, client=client) - result = fc.get(timeout=0) - -p = multiprocessing.Process(target=child) -p.start() -``` diff --git a/.agents/skills/modal/references/guide/tunnels.md b/.agents/skills/modal/references/guide/tunnels.md deleted file mode 100644 index 96d530e46d5098bd883214a38a57ac3ada1ca05e..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/tunnels.md +++ /dev/null @@ -1,180 +0,0 @@ -# Tunnels - -Modal allows you to expose live TCP ports on a Modal container. This is done by -creating a *tunnel* that forwards the port to the public Internet. - -```python -import modal - -app = modal.App() - - -@app.function() -def start_app(): - # Inside this `with` block, port 8000 on the container can be accessed by - # the address at `tunnel.url`, which is randomly assigned. - with modal.forward(8000) as tunnel: - print(f"tunnel.url = {tunnel.url}") - print(f"tunnel.tls_socket = {tunnel.tls_socket}") - # ... start some web server at port 8000, using any framework -``` - -Tunnels are direct connections and terminate TLS automatically. Within a few -milliseconds of container startup, this function prints a message such as: - -``` -tunnel.url = https://wtqcahqwhd4tu0.r5.modal.host -tunnel.tls_socket = ('wtqcahqwhd4tu0.r5.modal.host', 443) -``` - -You can also create tunnels on a [Sandbox](/docs/guide/sandbox-networking#forwarding-ports) -to directly expose the container's ports. - -## Build with tunnels - -Tunnels are the fastest way to get a low-latency, direct connection to a running -container. You can use them to run live browser applications with **interactive -terminals**, **Jupyter notebooks**, **VS Code servers**, and more. - -As a quick example, here is how you would expose a Jupyter notebook: - -```python -import os -import secrets -import subprocess - -import modal - - -image = modal.Image.debian_slim().pip_install("jupyterlab") -app = modal.App(image=image) - - -@app.function() -def run_jupyter(): - token = secrets.token_urlsafe(13) - with modal.forward(8888) as tunnel: - url = tunnel.url + "/?token=" + token - print(f"Starting Jupyter at {url}") - subprocess.run( - [ - "jupyter", - "lab", - "--no-browser", - "--allow-root", - "--ip=0.0.0.0", - "--port=8888", - "--LabApp.allow_origin='*'", - "--LabApp.allow_remote_access=1", - ], - env={**os.environ, "JUPYTER_TOKEN": token, "SHELL": "/bin/bash"}, - stderr=subprocess.DEVNULL, - ) -``` - -When you run the function, it starts Jupyter and gives you the public URL. It's -as simple as that. - -All Modal features are supported. If you -[need GPUs](https://modal.com/docs/guide/gpu), pass `gpu=` to the -`@app.function()` decorator. If you -[need more CPUs, RAM](https://modal.com/docs/guide/resources), or to attach -[volumes](https://modal.com/docs/guide/volumes), those -also just work. - -### Programmable startup - -The tunnel API is completely on-demand, so you can start them as the result of a -web request. - -For example, you could make something like Jupyter Hub without leaving Modal, -giving your users their own Jupyter notebooks when they visit a URL: - -```python -import modal - - -image = modal.Image.debian_slim().pip_install("fastapi[standard]") -app = modal.App(image=image) - - -@app.function(timeout=900) # 15 minutes -def run_jupyter(q): - ... # as before, but return the URL on app.q - - -@app.function() -@modal.fastapi_endpoint(method="POST") -def jupyter_hub(): - from fastapi import HTTPException - from fastapi.responses import RedirectResponse - - ... # do some validation on the secret or bearer token - - if is_valid: - with modal.Queue.ephemeral() as q: - run_jupyter.spawn(q) - url = q.get() - return RedirectResponse(url, status_code=303) - - else: - raise HTTPException(401, "Not authenticated") -``` - -This gives every user who sends a POST request to the Web Function their own -Jupyter notebook server, on a fully isolated Modal container. - -You could do the same with VS Code and get some basic version of an instant, -serverless IDE! - -### Advanced: Unencrypted TCP tunnels - -By default, tunnels are only exposed to the Internet at a secure random URL, and -connections have automatic TLS (the "S" in HTTPS). However, sometimes you might -need to expose a protocol like an SSH server that goes directly over TCP. In -this case, we have support for *unencrypted* tunnels: - -```python notest -with modal.forward(8000, unencrypted=True) as tunnel: - print(f"tunnel.tcp_socket = {tunnel.tcp_socket}") -``` - -Might produce an output like: - -``` -tunnel.tcp_socket = ('r3.modal.host', 23447) -``` - -You can then connect over TCP, for example with `nc r3.modal.host 23447`. Unlike -encrypted TLS sockets, these cannot be given a non-guessable, cryptographically -random URL due to how the TCP protocol works, so they are assigned a random port -number instead. - -## Pricing - -Modal only charges for containers based on -[the resources you use](https://modal.com/pricing). There is no additional -charge for having an active tunnel. - -For example, if you start a Jupyter notebook on port 8888 and access it via -tunnel, you can use it for an hour for development (with 0.01 CPUs) and then -actually run an intensive job with 16 CPUs for one minute. The amount you would -be billed for in that hour is 0.01 + 16 \* (1/60) = **0.28 CPUs**, even though -you had access to 16 CPUs without needing to restart your notebook. - -## Security - -Tunnels are run on Modal's private global network of Internet relays. On -startup, your container will connect to the nearest tunnel so you get the -minimum latency, very similar in performance to a direct connection with the -machine. - -This makes them ideal for live debugging sessions, using web-based terminals -like [ttyd](https://github.com/tsl0922/ttyd). - -The generated URLs are cryptographically random, but they are also public on the -Internet, so anyone can access your application if they are given the URL. - -We do not currently do any detection of requests above L4, so if you are running -a web server, we will not add special proxy HTTP headers or translate HTTP/2. -You're just getting the TLS-encrypted TCP stream directly! diff --git a/.agents/skills/modal/references/guide/vm-sandboxes.md b/.agents/skills/modal/references/guide/vm-sandboxes.md deleted file mode 100644 index 74f02a518a2525a9fa0f02fe93795c53ab533c39..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/vm-sandboxes.md +++ /dev/null @@ -1,98 +0,0 @@ -# VM Sandboxes - - - -Access to this feature is restricted to allowlisted workspaces only. Compatibility with [Sandbox Snapshots](/docs/guide/sandbox-snapshots) is partial: [Filesystem Snapshots](/docs/guide/sandbox-snapshots#filesystem-snapshots) are supported, but [Memory Snapshots](/docs/guide/sandbox-snapshots#memory-snapshots) are not yet supported. - - - -Sandboxes can be run on top of a full virtual machine rather than on top of gVisor. This gives -each Sandbox a real Linux kernel, which makes certain workloads (e.g. Docker systems) behave -the way they would on a normal Linux host. - -You can use the VM runtime for your Sandbox by passing `experimental_options={"vm_runtime": True}` -to `Sandbox.create()`. - -```python fixture:sb_app -with modal.enable_output(): - sb = modal.Sandbox.create( - app=sb_app, - cpu=2, # physical cores - memory=4096, # MiB - experimental_options={"vm_runtime": True}, - ) - -# add a script that uses VM Sandbox features -sb.filesystem.write_text( - """ - # Format an ext4 filesystem onto a regular file. - truncate -s 100M /tmp/disk.img - mkfs.ext4 -F /tmp/disk.img - - # Mount it. This works in a VM, but isn't supported in gVisor. - mkdir -p /mnt/loop - mount -o loop /tmp/disk.img /mnt/loop -""", - "/tmp/mount_loopback_filesystem.sh", -) - -p = sb.exec("bash", "/tmp/mount_loopback_filesystem.sh") -p.wait() - -print(p.stdout.read()) - -print(p.stderr.read()) - -assert p.returncode == 0 # error if the program in the Sandbox fails - -sb.terminate() -``` - -Additionally, quickly provision a VM runtime sandbox with a PTY shell via the CLI using: - -``` -modal shell --experimental-option vm_runtime=1 -``` - -## Improvements over gVisor sandboxes - -Docker workloads behave more like they do in a non-container environment. In particular: - -* Docker state (e.g. `/var/lib/docker`) is included in [Filesystem Snapshots](/docs/guide/sandbox-snapshots#filesystem-snapshots) -* Docker features that previously needed special treatment on gVisor (e.g. inter-container networking) will also work normally - -Features that only make sense in a bona fide Linux environment are now available: - -* Custom [init systems](https://arxiv.org/pdf/0706.2748) (such as [`systemd`](https://man7.org/linux/man-pages/man1/systemd.1.html)) are supported -* [eBPF](https://ebpf.io/) is supported -* Resource isolation within the Sandbox via [cgroups](https://man7.org/linux/man-pages/man7/cgroups.7.html) is supported - -Finally, for most workloads, the root filesystem will perform better on a VM Sandbox than in a gVisor Sandbox. The benefit is less pronounced for metadata-heavy workloads. - -## Resource model - -Unlike [resource provisioning](/docs/guide/resources) in other runtimes, -memory provisioning is **static** for VM Sandboxes: you get exactly as much -RAM as you request via `memory` argument to `Sandbox.create`. By default, VM -sandboxes get 1GiB of RAM. - -However, CPU provisioning is elastic. You can burst above your requested amount. - -Costs for both resources are calculated based on the requested amount, used amount, -the duration of Sandbox execution, and [our rates for `cpu` and `memory`](/pricing). - -## Limitations - -The following limitations are known and we're tracking them: - -* **GPUs are not supported.** VM Sandboxes currently only support CPU workloads. -* **`setuid` bits are not faithfully preserved in Modal [Images](/docs/guide/images).** - * Executables that rely on the `setuid` bit (e.g. `sudo`, `ping`, `mount`) may not work properly. - * This can be worked around if it matters for your workload: re-apply the `setuid` bit at runtime as the `root` user (e.g. by running `chmod u+s /path/to/binary`). -* **The [Sandbox filesystem API](/docs/guide/sandbox-files#filesystem-api-beta) is only available in new SDK versions**. For the Python SDK, it requires version ≥ 1.4.0 and for the JS/TS/Go SDKs, it requires versions ≥ 0.7.6. -* **[`Sandbox.reload_volumes()`](/docs/reference/modal.Sandbox#reload_volumes) is not supported.** VM Sandboxes do not currently support reloading volumes at runtime. -* **[Memory Snapshots](/docs/guide/sandbox-snapshots#memory-snapshots) are not yet supported.** Only - [Filesystem Snapshots](/docs/guide/sandbox-snapshots#filesystem-snapshots) work on VM Sandboxes today. -* **Root images ≥ 512 GiB are not supported.** The VM root filesystem is currently limited to 512 GiB. Sandboxes created from container images exceeding this size will fail to start. - -If you hit a rough edge that isn't listed here, please reach out via [Slack](/slack) or email us at . diff --git a/.agents/skills/modal/references/guide/volumes.md b/.agents/skills/modal/references/guide/volumes.md deleted file mode 100644 index 8dbaeb604ad738e7f710b7e5e857a9c3f0c4761c..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/volumes.md +++ /dev/null @@ -1,598 +0,0 @@ -# Volumes - -Volumes are a high-performance distributed file system for Modal applications. They are optimized for write-once, read-many I/O workloads, like creating machine learning model weights and distributing them for inference. - -Key benefits: - -* Volumes are distributed by default, so you can use them alongside Modal’s global compute pool without needing to manage replicas across regions. -* Volumes have caching and chunking optimizations built-in to maximize throughput. -* Volumes come with a fully-featured filesystem interface for easy integration into your favorite ML tools and frameworks. -* Volumes are backed by multiple underlying cloud providers to guarantee high availability. - -This page is a high-level guide to using Modal Volumes. -For reference documentation on the `modal.Volume` object, see -[this page](/docs/reference/modal.Volume). -For reference documentation on the `modal volume` CLI command, see -[this page](/docs/reference/cli/volume). - -## Pricing - -Please refer to our [pricing page](/pricing) for up-to-date prices. We calculate usage by snapshotting your total storage once a day. When you delete data, you may still be billed for that storage for up to four days, to reflect our underlying processing costs. - -## Volumes v2 - - - -Instructions specific to v2 Volumes will be annotated with 🌱 below. - - - -Read more about [Volumes v2](#volumes-v2-overview) below. - -## Creating a Volume - -The easiest way to create a Volume and use it as a part of your App is to use -the [`modal volume create`](/docs/reference/cli/volume#modal-volume-create) CLI command. This will create the Volume and output -some sample code: - -```bash -% modal volume create my-volume -Created volume 'my-volume' in environment 'main'. -``` - -> 🌱 To create a v2 Volume, pass `--version=2` in the command above. - -## Using a Volume on Modal - -To attach an existing Volume to a Modal Function, use [`Volume.from_name`](/docs/reference/modal.Volume#from_name): - -```python -vol = modal.Volume.from_name("my-volume") - - -@app.function(volumes={"/data": vol}) -def run(): - with open("/data/xyz.txt", "w") as f: - f.write("hello") - vol.commit() # Needed to make sure all changes are persisted before exit -``` - -You can also browse and manipulate Volumes from an ad hoc Modal Shell: - -```bash -% modal shell --volume my-volume --volume another-volume -``` - -Volumes will be mounted under `/mnt`. - -Volumes are designed to provide up to 2.5 GB/s of bandwidth. -Actual throughput is not guaranteed and may be lower depending on network conditions. - -## Mount options - -When attaching a Volume to a Function or Sandbox, you can configure mount options using -[`Volume.with_mount_options`](/docs/reference/modal.Volume#with_mount_options). -These options are not stored on the Volume itself — they apply per container mount, -so the same Volume can be mounted differently for distinct containers. - -### Read-only mounts - -To prevent a container from writing to a Volume, mount it in read-only mode: - -```python notest -import modal - -volume = modal.Volume.from_name("my-volume") - -sb = modal.Sandbox.create( - volumes={"/data": volume.with_mount_options(read_only=True)}, - app=app, -) -sb.exec("cat", "/data/config.json").wait() # ok! -sb.exec("touch", "/data/new-file").wait() # error! -``` - -### Mounting a sub-path - -You can mount a subdirectory of a Volume instead of the entire Volume using the `sub_path` mount option. -If the subdirectory doesn't exist yet, it will be created when the container starts. - -```python notest -import modal - -volume = modal.Volume.from_name("my-volume") - -sb = modal.Sandbox.create( - volumes={"/user_data": volume.with_mount_options(sub_path="/users/user_123")}, - app=app, -) -# /user_data inside of the contianer is now referencing /users/user_123 in the Volume -sb.exec("ls", "/user_data").wait() -``` - -Sub-path mounting is especially useful when you want a single Volume to serve -multiple end user sessions, but don't want a session to access or even see -files for other sessions in the Volume. - -**Note:** Sub-path is currently restricted to directories - you can not mount an individual file. - -## Downloading a file from a Volume - -While there’s no file size limit for individual files in a volume, the frontend only supports downloading files up to 16 MB. For larger files, please use the CLI: - -```bash -% modal volume get my-volume xyz.txt xyz-local.txt -``` - -### Creating Volumes lazily from code - -You can also create Volumes lazily from code using: - -```python -vol = modal.Volume.from_name("my-volume", create_if_missing=True) -``` - -> 🌱 To create a v2 Volume, pass `version=2` to the call to `from_name()` in the code above. - -This will create the Volume if it doesn't exist. - -## Using a Volume from outside of Modal - -Volumes can also be used outside Modal via the [Python SDK](/docs/reference/modal.Volume#modalvolume) or our [CLI](/docs/reference/cli/volume). - -### Using a Volume from local code - -You can interact with Volumes from anywhere you like using the `modal` Python client library. - -```python notest -vol = modal.Volume.from_name("my-volume") - -with vol.batch_upload() as batch: - batch.put_file("local-path.txt", "/remote-path.txt") - batch.put_directory("/local/directory/", "/remote/directory") - batch.put_file(io.BytesIO(b"some data"), "/foobar") -``` - -For more details, see the [reference documentation](/docs/reference/modal.Volume). - -### Using a Volume via the command line - -You can also interact with Volumes using the command line interface. You can run -`modal volume` to get a full list of its subcommands: - -```bash -% modal volume -Usage: modal volume [OPTIONS] COMMAND [ARGS]... - - Read and edit modal.Volume volumes. - Note: users of modal.NetworkFileSystem should use the modal nfs command instead. - -╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ -│ --help Show this message and exit. │ -╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -╭─ File operations ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ -│ cp Copy within a modal.Volume. Copy source file to destination file or multiple source files to destination directory. │ -│ get Download files from a modal.Volume object. │ -│ ls List files and directories in a modal.Volume volume. │ -│ put Upload a file or directory to a modal.Volume. │ -│ rm Delete a file or directory from a modal.Volume. │ -╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -╭─ Management ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ -│ create Create a named, persistent modal.Volume. │ -│ delete Delete a named, persistent modal.Volume. │ -│ list List the details of all modal.Volume volumes in an Environment. │ -╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ -``` - -For more details, see the [reference documentation](/docs/reference/cli/volume). - -## Volume commits and reloads - -Unlike a normal filesystem, you need to explicitly reload the Volume to see -changes made since it was first mounted. This reload is handled by invoking the -[`.reload()`](/docs/reference/modal.Volume#reload) method on a Volume object. -Similarly, any Volume changes made within a container need to be committed for -those the changes to become visible outside the current container. This is handled -periodically by [background commits](#background-commits) and directly by invoking -the [`.commit()`](/docs/reference/modal.Volume#commit) -method on a `modal.Volume` object. - -At container creation time the latest state of an attached Volume is mounted. If -the Volume is then subsequently modified by a commit operation in another -running container, that Volume modification won't become available until the -original container does a [`.reload()`](/docs/reference/modal.Volume#reload). - -Consider this example which demonstrates the effect of a reload: - -```python -import pathlib -import modal - -app = modal.App() - -volume = modal.Volume.from_name("my-volume") - -p = pathlib.Path("/root/foo/bar.txt") - - -@app.function(volumes={"/root/foo": volume}) -def f(): - p.write_text("hello") - print(f"Created {p=}") - volume.commit() # Persist changes - print(f"Committed {p=}") - - -@app.function(volumes={"/root/foo": volume}) -def g(reload: bool = False): - if reload: - volume.reload() # Fetch latest changes - if p.exists(): - print(f"{p=} contains '{p.read_text()}'") - else: - print(f"{p=} does not exist!") - - -@app.local_entrypoint() -def main(): - g.remote() # 1. container for `g` starts - f.remote() # 2. container for `f` starts, commits file - g.remote(reload=False) # 3. reuses container for `g`, no reload - g.remote(reload=True) # 4. reuses container, but reloads to see file. -``` - -The output for this example is this: - -``` -p=PosixPath('/root/foo/bar.txt') does not exist! -Created p=PosixPath('/root/foo/bar.txt') -Committed p=PosixPath('/root/foo/bar.txt') -p=PosixPath('/root/foo/bar.txt') does not exist! -p=PosixPath('/root/foo/bar.txt') contains hello -``` - -This code runs two containers, one for `f` and one for `g`. Only the last -function invocation reads the file created and committed by `f` because it was -configured to reload. - -### Background commits - -Modal Volumes run background commits: -every few seconds while your Function executes, -the contents of attached Volumes will be committed -without your application code calling `.commit`. -A final snapshot and commit is also automatically performed on container shutdown. - -Being able to persist changes to Volumes without changing your application code -is especially useful when [training or fine-tuning models using frameworks](#model-checkpointing). - -## Model serving - -A single ML model can be served by simply baking it into a `modal.Image` at -build time using [`run_function`](/docs/reference/modal.Image#run_function). But -if you have dozens of models to serve, or otherwise need to decouple image -builds from model storage and serving, use a `modal.Volume`. - -Volumes can be used to save a large number of ML models and later serve any one -of them at runtime with great performance. This snippet below shows the -basic structure of the solution. - -```python -import modal - -app = modal.App() -volume = modal.Volume.from_name("model-store") -model_store_path = "/vol/models" - - -@app.function(volumes={model_store_path: volume}, gpu="any") -def run_training(): - model = train(...) - save(model_store_path, model) - volume.commit() # Persist changes - - -@app.function(volumes={model_store_path: volume}) -def inference(model_id: str, request): - try: - model = load_model(model_store_path, model_id) - except NotFound: - volume.reload() # Fetch latest changes - model = load_model(model_store_path, model_id) - return model.run(request) -``` - -For more details, see our [guide to storing model weights on Modal](/docs/guide/model-weights). - -## Model checkpointing - -Checkpoints are snapshots of an ML model and can be configured by the callback -functions of ML frameworks. You can use saved checkpoints to restart a training -job from the last saved checkpoint. This is particularly helpful in managing -[preemption](/docs/guide/preemption). - -For more, see our [example code for long-running training](/docs/examples/long-training). - -### Hugging Face `transformers` - -To periodically checkpoint into a `modal.Volume`, just set the `Trainer`'s -[`output_dir`](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.output_dir) -to a directory in the Volume. - -```python -import pathlib - -volume = modal.Volume.from_name("my-volume") -VOL_MOUNT_PATH = pathlib.Path("/vol") - -@app.function( - gpu="A10G", - timeout=2 * 60 * 60, # run for at most two hours - volumes={VOL_MOUNT_PATH: volume}, -) -def finetune(): - from transformers import Seq2SeqTrainer - ... - - training_args = Seq2SeqTrainingArguments( - output_dir=str(VOL_MOUNT_PATH / "model"), - # ... more args here - ) - - trainer = Seq2SeqTrainer( - model=model, - args=training_args, - train_dataset=tokenized_xsum_train, - eval_dataset=tokenized_xsum_test, - ) -``` - -## Volume performance - -Volumes work best when they contain less than 50,000 files and directories. The -latency to attach or modify a Volume scales linearly with the number of files in -the Volume, and past a few tens of thousands of files the linear component -starts to dominate the fixed overhead. - -There is currently a hard limit of 500,000 inodes (files, directories and -symbolic links) per Volume. If you reach this limit, any further attempts to -create new files or directories will error with -[`ENOSPC` (No space left on device)](https://pubs.opengroup.org/onlinepubs/9799919799/). - -If you need to work with a large number of files, consider using Volumes v2! -It is currently in Beta. See below for more info. - -## Filesystem consistency - -### Concurrent modification - -Concurrent modification from multiple containers is supported, but concurrent -modifications of the same files should be avoided. Last write wins in case of -concurrent modification of the same file — any data the last writer didn't have -when committing changes will be lost! - -The number of commits you can run concurrently is limited. If you run too many -concurrent commits each commit will take longer due to contention. If you are -committing small changes, avoid doing more than 5 concurrent commits (the number -of concurrent commits you can make is proportional to the size of the changes -being committed). - -As a result, Volumes are typically not a good fit for use cases where you need -to make concurrent modifications to the same file (nor is distributed file -locking supported). - -While a reload is in progress the Volume will appear empty to the container that -initiated the reload. That means you cannot read from or write to a Volume in a -container where a reload is ongoing (note that this only applies to the -container where the reload was issued, other containers remain unaffected). - -### Busy Volume errors - -You can only reload a Volume when there no open files on the Volume. If you have -open files on the Volume the [`.reload()`](/docs/reference/modal.Volume#reload) -operation will fail with "volume busy". The following is a simple example of how -a "volume busy" error can occur: - -```python -volume = modal.Volume.from_name("my-volume") - - -@app.function(volumes={"/vol": volume}) -def reload_with_open_files(): - f = open("/vol/data.txt", "r") - volume.reload() # Cannot reload when files in the Volume are open. -``` - -### Can't find file on Volume errors - -When accessing files in your Volume, don't forget to pre-pend where your Volume -is mounted in the container. - -In the example below, where the Volume has been mounted at `/data`, "hello" is -being written to `/data/xyz.txt`. - -```python -import modal - -app = modal.App() -vol = modal.Volume.from_name("my-volume") - - -@app.function(volumes={"/data": vol}) -def run(): - with open("/data/xyz.txt", "w") as f: - f.write("hello") - vol.commit() -``` - -If you instead write to `/xyz.txt`, the file will be saved to the local disk of the Modal Function. -When you dump the contents of the Volume, you will not see the `xyz.txt` file. - -## Disk usage reporting - -Modal Volumes are not block devices and do not have a fixed capacity. -Additionally, used space is not currently reported at the filesystem level. -As a result, tools that query disk usage via the `statfs` syscall -(e.g. `shutil.disk_usage()`, `os.statvfs()`, `df`) will return placeholder -values. - -If you need to check the size of a Volume, refer to the size shown in the Modal -dashboard, or use `du` on the mounted Volume within a container. - -## Volumes v2 overview - -Volumes v2 generally behave just like Volumes v1, and most of the existing APIs -and CLI commands that you are used to will work the same between versions. -Because the file system implementation is completely different, there will be -some significant performance characteristics that can differ from version 1 -Volumes. Below is an outline of the key differences you should be aware of. - -### Volumes v2 are still in Beta - - - -We cannot yet guarantee that no data will be lost, so we don't recommend using Volumes v2 for mission-critical data at this time. - - - -You can still reap the benefits of v2 for -data that isn't precious, or that is easy to rebuild, such as log files, -regularly updated training data and model weights, caches, and more. - -### Volumes v2 are HIPAA compliant - -If you delete the volume, the data is guaranteed to be lost according to HIPAA requirements. - -### Volumes v2 is more scaleable - -Volumes v2 support more files, higher throughput, and more irregular access -patterns. Commits and reloads are also faster. - -Additionally, Volumes v2 supports hard-linking of files, where multiple paths -can point to the same inode. - -### In v2, you can store as many files as you want - -There is no limit on the number of files in Volumes v2. - -By contrast, in Volumes v1, there is a limit on the number of files of 500,000, -and we recommend keeping the count to 50,000 or less. - -### In v2, you can write concurrently from hundreds of containers - -The file system should not experience any performance degradation as more -containers write to distinct files simultaneously. - -By contrast, in Volumes v1, we recommend no more than five writers access the -Volume at once. - -Note, however, that concurrent access to a particular *file* in a Volume still -has last-write-wins semantics in many circumstances. These semantics are -unacceptable for most applications, so any particular file should only be -written to by a single container at a time. - -### In v2, random accesses have improved performance - -In v1, writes to locations inside a file would sometimes incur substantial -overhead, like a rewrite of the entire file. - -In v2, this overhead is removed, and only changes are written. - -### In v2, you can commit using `sync` - -For Volumes v2, you can trigger a commit from within a Sandbox or modal shell -by running the `sync` command on the mountpoint: - -```bash -sync /path/to/mountpoint -``` - -This is useful when you don't have access to the Python SDK's -[`.commit()`](/docs/reference/modal.Volume#commit) method, such as when running -shell commands in a Sandbox or during an interactive `modal shell` session. - -Running `sync` on the mountpoint will flush any pending writes to the kernel -and then persist all data and metadata changes to the Volume's persistent -storage. - -For example, to commit changes in a modal shell session: - -```bash -% modal shell --volume my-v2-volume -root / → echo "hello" > /mnt/my-v2-volume/test.txt -root / → sync /mnt/my-v2-volume # Persist changes before exiting -``` - -Or to commit from within a Sandbox: - -```python notest -sb = modal.Sandbox.create( - volumes={"/data": modal.Volume.from_name("my-v2-volume")}, - app=my_app, -) -sb.exec("bash", "-c", "echo 'hello' > /data/test.txt").wait() - -# Persist changes and check for errors -p = sb.exec("sync", "/data") -p.wait() -if p.returncode != 0: - raise Exception(f"sync failed with exit code {p.returncode}") -``` - -> ⚠️ This feature is only available for Volumes v2. - -### Volumes v2 has a few limits in place - -While we work out performance trade-offs and listen to user feedback, we have -put some artificial limits in place. - -* Files must be less than 1 TiB. -* At most 262,144 files can be stored in a single directory. - Directory depth is unbounded, so the total file count is unbounded. -* Traversing the filesystem can be slower in v2 than in v1, due to demand - loading of the filesystem tree. - -### Upgrading v1 Volumes - -Currently, there is no automated tool for upgrading v1 Volumes to v2. We are -planning to implement an automated migration path but for now v1 Volumes need -to be manually migrated by creating a new v2 Volume and either copying files -over from the v1 Volume or writing new files. - -To reuse the name of an existing v1 Volume for a new v2 Volume, first stop all -apps that are utilizing the v1 Volume before deleting it. If this is not -feasible, e.g. due to wanting to avoid downtime, use a new name for the v2 -Volume. - -**Warning:** When deleting an existing Volume, any deployed apps or running -functions utilizing that Volume will cease to function, even if a new Volume is -created with the same name. This is because Volumes are identified with opaque -unique IDs that are resolved at application deployment or start time. A newly -created Volume with the same name as a deleted Volume will have a new Volume ID -and any deployed or running apps will still be referring to the old ID until -these apps are re-deployed or restarted. - -In order to create a new volume and copy data over from the old volume, you can -use a tool like `cp` if you intend to copy all the data in one go, or `rsync` -if you want to incrementally copy the data across a longer time span: - -```shell -$ modal volume create --version=2 2files2furious -$ modal shell --volume files-and-furious --volume 2files2furious -Welcome to Modal's debug shell! -We've provided a number of utilities for you, like `curl` and `ps`. -# Option 1: use `cp` -root / → cp -rp /mnt/files-and-furious/. /mnt/2files2furious/. -root / → sync /mnt/2files2furious # Ensure changes are persisted before exiting - -# Option 2: use `rsync` -root / → apt install -y rsync -root / → rsync -a /mnt/files-and-furious/. /mnt/2files2furious/. -root / → sync /mnt/2files2furious # Ensure changes are persisted before exiting -``` - -## Further examples - -* [Character LoRA fine-tuning](/docs/examples/diffusers_lora_finetune) with model storage on a Volume -* [Protein folding](/docs/examples/chai1) with model weights and output files stored on Volumes -* [Dataset visualization with Datasette](/docs/examples/cron_datasette) using a SQLite database on a Volume diff --git a/.agents/skills/modal/references/guide/webhook-proxy-auth.md b/.agents/skills/modal/references/guide/webhook-proxy-auth.md deleted file mode 100644 index ad1bee2a0f5fc3531d23ae954f8ab2d4fe23449c..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/webhook-proxy-auth.md +++ /dev/null @@ -1,61 +0,0 @@ -# Proxy Auth Tokens - -Use Proxy Auth Tokens to prevent unauthorized clients from triggering your Web Functions. - -```python -import modal - -image = modal.Image.debian_slim().pip_install("fastapi") -app = modal.App("proxy-auth-public", image=image) - - -@app.function() -@modal.fastapi_endpoint() -def public(): - return "hello world" - - -@app.function() -@modal.fastapi_endpoint(requires_proxy_auth=True) -def private(): - return "hello friend" -``` - -The `public` endpoint can be hit by any client over the Internet. - -```bash -curl https://public-url--goes-here.modal.run -``` - -The `private` endpoint cannot. - -```bash -curl --fail-with-body https://private-url--goes-here.modal.run -# modal-http: missing credentials for proxy authorization -# curl: (22) The requested URL returned error: 401 -# https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401 -``` - -Authorization is demonstrated via a Proxy Auth Token. You can create a Proxy Auth Token for your workspace [here](/settings/proxy-auth-tokens). -In requests to the Web Function, clients supply the Token ID and Token Secret in the `Modal-Key` and `Modal-Secret` HTTP headers. - -```bash -export TOKEN_ID=wk-1234abcd -export TOKEN_SECRET=ws-1234abcd -curl -H "Modal-Key: $TOKEN_ID" \ - -H "Modal-Secret: $TOKEN_SECRET" \ - https://private-url--goes-here.modal.run -``` - -Proxy authorization can be added to [Web Functions](/docs/guide/webhooks) created by the -[`fastapi_endpoint`](/docs/reference/modal.fastapi_endpoint), -[`asgi_app`](/docs/reference/modal.asgi_app), -[`wsgi_app`](/docs/reference/modal.wsgi_app), or -[`web_server`](/docs/reference/modal.web_server) decorators, -which are otherwise publicly available. - -Everyone within the workspace of the Web Function can manage its Proxy Auth Tokens. - -## Restricting tokens to specific Environments - -On workspaces with RBAC enabled, tokens can be scoped to specific Environments, restricting which Web Functions they are valid for. See [Proxy auth tokens for Web Functions](/docs/guide/rbac#proxy-auth-tokens-for-web-functions) in the RBAC guide for more. diff --git a/.agents/skills/modal/references/guide/webhook-timeouts.md b/.agents/skills/modal/references/guide/webhook-timeouts.md deleted file mode 100644 index 7928a90f10ec003f8c2e47704f1ef5d06acf7991..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/webhook-timeouts.md +++ /dev/null @@ -1,80 +0,0 @@ -# Request timeouts - -Web Function requests should complete quickly, ideally within a -few seconds. All Web Function types -([`modal.fastapi_endpoint`](/docs/reference/modal.fastapi_endpoint), -[`modal.asgi_app`](/docs/reference/modal.asgi_app), -[`modal.wsgi_app`](/docs/reference/modal.wsgi_app), -and [`modal.web_server`](/docs/reference/modal.web_server)) -have a maximum HTTP request timeout of 150 seconds enforced. However, the -underlying Modal Function can have a longer [timeout](/docs/guide/timeouts). - -In case the Function takes more than 150 seconds to complete, an HTTP status 303 -redirect response is returned pointing at the original URL with a special query -parameter linking it that request. This is the *result URL* for your function. -Most web browsers allow for up to 20 such redirects, effectively allowing up to -50 minutes (20 \* 150 s) for Web Functions before the request times out. - -(**Note:** This does not work with requests that require -[CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS), since the -response will not have been returned from your code in time for the server to -populate CORS headers.) - -Some libraries and tools might require you to add a flag or option in order to -follow redirects automatically, e.g. `curl -L ...` or `http --follow ...`. - -The *result URL* can be reloaded without triggering a new request. It will block -until the request completes. - -(**Note:** As of March 2025, the Python standard library's `urllib` module has the -maximum number of redirects to any single URL set to 4 by default ([source](https://github.com/python/cpython/blob/main/Lib/urllib/request.py)), which would limit the total timeout to 12.5 minutes (5 \* 150 s = 750 s) unless this setting is overridden.) - -## Polling solutions - -Sometimes it can be useful to be able to poll for results rather than wait for a -long running HTTP request. The easiest way to do this is to have your web -endpoint spawn a `modal.Function` call and return the function call id that -another endpoint can use to poll the submitted function's status. Here is an -example: - -```python -import fastapi - -import modal - - -image = modal.Image.debian_slim().pip_install("fastapi[standard]") -app = modal.App(image=image) - -web_app = fastapi.FastAPI() - - -@app.function() -@modal.asgi_app() -def fastapi_app(): - return web_app - - -@app.function() -def slow_operation(): - ... - - -@web_app.post("/accept") -async def accept_job(request: fastapi.Request): - call = slow_operation.spawn() - return {"call_id": call.object_id} - - -@web_app.get("/result/{call_id}") -async def poll_results(call_id: str): - function_call = modal.FunctionCall.from_id(call_id) - try: - return function_call.get(timeout=0) - except TimeoutError: - http_accepted_code = 202 - return fastapi.responses.JSONResponse({}, status_code=http_accepted_code) -``` - -[*Document OCR Web App*](/docs/examples/doc_ocr_webapp) is an example that uses -this pattern. diff --git a/.agents/skills/modal/references/guide/webhook-urls.md b/.agents/skills/modal/references/guide/webhook-urls.md deleted file mode 100644 index 0533d7a5cabfa84d8ea2046cf005c9dc7bdfc07d..0000000000000000000000000000000000000000 --- a/.agents/skills/modal/references/guide/webhook-urls.md +++ /dev/null @@ -1,228 +0,0 @@ -# Web Function URLs - -This guide documents the behavior of URLs for [Web Function](/docs/guide/webhooks) -on Modal: automatic generation, configuration, programmatic retrieval, and more. - -## Determine the Web Function URL from code - -Modal Functions with the -[`fastapi_endpoint`](/docs/reference/modal.fastapi_endpoint), -[`asgi_app`](/docs/reference/modal.asgi_app), -[`wsgi_app`](/docs/reference/modal.wsgi_app), -or [`web_server`](/docs/reference/modal.web_server) decorator -are made available over the Internet when they are -[`serve`d](/docs/reference/cli/serve) or [`deploy`ed](/docs/reference/cli/deploy) -and so they have a URL. - -This URL is displayed in the `modal` CLI output -and is available in the Modal [dashboard](/apps) for the Function. - -To determine a Function's URL programmatically, -check its [`get_web_url()`](/docs/reference/modal.Function#get_web_url) -property: - -```python -@app.function(image=modal.Image.debian_slim().pip_install("fastapi[standard]")) -@modal.fastapi_endpoint(docs=True) -def show_url() -> str: - return show_url.get_web_url() -``` - -For deployed Functions, this also works from other Python code! -You just need to do a [`from_name`](/docs/reference/modal.Function#from_name) -based on the name of the Function and its [App](/docs/guide/apps): - -```python notest -import requests - -remote_function = modal.Function.from_name("app", "show_url") -remote_function.get_web_url() == requests.get(handle.get_web_url()).json() -``` - -## Auto-generated URLs - -By default, Modal Functions -will be served from the `modal.run` domain. -The full URL will be constructed from a number of pieces of information -to uniquely identify the endpoint. - -At a high-level, Web Function URLs for deployed Apps have the -following structure: `https://--