cosmos-framework/docs/prompt_upsampling.md

Prompt Upsampling

Skill: .agents/skills/cosmos3-inference/SKILL.md

---

Table of Contents

• Overview
• Setup
• Inputs
• Outputs
• Quick Start
  • Text-to-Image
  • Text-to-Video
  • Image-to-Video
• Reasoner Templates
• Posttrained I2V
• Template Files

---

Overview

The prompt upsampler converts one prompt per line into structured Cosmos3 JSON prompts. It supports:

• text2image
• text2video
• image2video
• posttrain_image2video

Run it as:

python -m cosmos_framework.inference.prompt_upsampling --help

Setup

Run commands from the repository root. Configure the OpenAI-compatible endpoint and API token:

export PROMPT_UPSAMPLER_ENDPOINT_URL="[required endpoint url]"
export PROMPT_UPSAMPLER_API_TOKEN="[api key]"
export PROMPT_UPSAMPLER_MODEL_NAME="[model name]"

Arguments:

• --input: Text file with one prompt per non-empty line.
• --output: Output directory. The CLI writes one prompt_<index>.json file per input prompt.
• --mode: One of text2image, text2video, image2video, or posttrain_image2video.
• --endpoint-url: OpenAI-compatible endpoint URL. Defaults to PROMPT_UPSAMPLER_ENDPOINT_URL.
• --model: Model name. Defaults to PROMPT_UPSAMPLER_MODEL.
• --api-token: API token. Defaults to PROMPT_UPSAMPLER_API_TOKEN.
• --prompt-template: Optional prompt template override for the selected mode.
• --json-template: Optional JSON schema template override for the selected mode.

Inputs

Prompt files contain one non-empty prompt per line:

A humanoid robot carefully assembles a gearbox on a workbench.
A red sports car drives through rain at night.

For image2video and posttrain_image2video, provide either one shared image:

--image-url inputs/prompt_upsampler/image_inputs/car_driving.jpg

Or provide one image per prompt:

--image-list inputs/prompt_upsampler/images.txt

Image-list files must have the same number of non-empty lines as the prompt file:

inputs/prompt_upsampler/image_inputs/car_driving.jpg
inputs/prompt_upsampler/image_inputs/humanoid_robot.jpg

Local image paths, HTTP(S) URLs, and data: URLs are accepted.

Outputs

The CLI creates the output directory and writes one JSON file per input prompt:

outputs/prompt_upsampler/upsampled_t2i_prompts_opus/
+-- prompt_0.json
`-- prompt_1.json

Every mode writes one record per prompt with the same shape: the upsampled JSON is a compact string under prompt. A record also includes a negative_prompt string when the template or model produces one.

{
  "prompt": "{\"subjects\": [...], \"resolution\": {\"H\": 480, \"W\": 832}, ...}"
}

Quick Start

The built-in external API templates are used by default. They live in cosmos_framework/inference/prompting_templates/external_api.

For example, to use Opus to generate upsampled prompts, set

export PROMPT_UPSAMPLER_ENDPOINT_URL="https://api.anthropic.com/v1/"
export PROMPT_UPSAMPLER_MODEL_NAME="claude-opus-4-6"

Text-to-Image

python -m cosmos_framework.inference.prompt_upsampling \
    --input inputs/prompt_upsampler/prompts_t2i.txt \
    --output outputs/prompt_upsampler/upsampled_t2i_prompts_opus \
    --mode text2image \
    --endpoint-url "${PROMPT_UPSAMPLER_ENDPOINT_URL}" \
    --model "${PROMPT_UPSAMPLER_MODEL_NAME}" \
    --api-token "${PROMPT_UPSAMPLER_API_TOKEN}" \
    --resolution 720 \
    --aspect-ratio "1,1"

Text-to-Video

python -m cosmos_framework.inference.prompt_upsampling \
    --input inputs/prompt_upsampler/prompts_t2v.txt \
    --output outputs/prompt_upsampler/upsampled_t2v_prompts_opus \
    --mode text2video \
    --endpoint-url "${PROMPT_UPSAMPLER_ENDPOINT_URL}" \
    --model "${PROMPT_UPSAMPLER_MODEL_NAME}" \
    --api-token "${PROMPT_UPSAMPLER_API_TOKEN}" \
    --resolution 720 \
    --aspect-ratio "16,9" \
    --duration "5s" \
    --fps 24

Image-to-Video

python -m cosmos_framework.inference.prompt_upsampling \
    --input inputs/prompt_upsampler/prompts_i2v.txt \
    --image-list inputs/prompt_upsampler/images.txt \
    --output outputs/prompt_upsampler/upsampled_i2v_prompts_opus \
    --mode image2video \
    --endpoint-url "${PROMPT_UPSAMPLER_ENDPOINT_URL}" \
    --model "${PROMPT_UPSAMPLER_MODEL_NAME}" \
    --api-token "${PROMPT_UPSAMPLER_API_TOKEN}" \
    --resolution 720 \
    --aspect-ratio "16,9" \
    --duration "5s" \
    --fps 24

Reasoner Templates

Reasoner-style prompts use the templates in cosmos_framework/inference/prompting_templates/reasoner. Pass them explicitly with --prompt-template and --json-template.

For using reasoner prompt upsampler in this script, launch a VLLM server. Set the model name, endpoint url and API token served by your endpoint:

export PROMPT_UPSAMPLER_REASONER_MODEL="cosmos3-reasoner"

Text-to-image:

python -m cosmos_framework.inference.prompt_upsampling \
    --input inputs/prompt_upsampler/prompts_t2i.txt \
    --output outputs/prompt_upsampler/upsampled_t2i_prompts_reason \
    --mode text2image \
    --endpoint-url "${PROMPT_UPSAMPLER_ENDPOINT_URL}" \
    --model "${PROMPT_UPSAMPLER_REASONER_MODEL}" \
    --api-token "${PROMPT_UPSAMPLER_API_TOKEN}" \
    --prompt-template cosmos_framework/inference/prompting_templates/reasoner/reasoner_t2i_prompt.txt \
    --json-template cosmos_framework/inference/prompting_templates/reasoner/reasoner_t2i_json_schema.json \
    --resolution 720 \
    --aspect-ratio "1,1"

Text-to-video:

python -m cosmos_framework.inference.prompt_upsampling \
    --input inputs/prompt_upsampler/prompts_t2v.txt \
    --output outputs/prompt_upsampler/upsampled_t2v_prompts_reason \
    --mode text2video \
    --endpoint-url "${PROMPT_UPSAMPLER_ENDPOINT_URL}" \
    --model "${PROMPT_UPSAMPLER_REASONER_MODEL}" \
    --api-token "${PROMPT_UPSAMPLER_API_TOKEN}" \
    --prompt-template cosmos_framework/inference/prompting_templates/reasoner/reasoner_t2v_prompt.txt \
    --json-template cosmos_framework/inference/prompting_templates/reasoner/reasoner_t2v_json_schema.json \
    --resolution 720 \
    --aspect-ratio "16,9" \
    --duration "5s" \
    --fps 24

Image-to-video:

python -m cosmos_framework.inference.prompt_upsampling \
    --input inputs/prompt_upsampler/prompts_i2v.txt \
    --image-list inputs/prompt_upsampler/images.txt \
    --output outputs/prompt_upsampler/upsampled_i2v_prompts_reason \
    --mode image2video \
    --endpoint-url "${PROMPT_UPSAMPLER_ENDPOINT_URL}" \
    --model "${PROMPT_UPSAMPLER_REASONER_MODEL}" \
    --api-token "${PROMPT_UPSAMPLER_API_TOKEN}" \
    --prompt-template cosmos_framework/inference/prompting_templates/reasoner/reasoner_i2v_prompt.txt \
    --json-template cosmos_framework/inference/prompting_templates/reasoner/reasoner_i2v_json_schema.json \
    --resolution 720 \
    --aspect-ratio "16,9" \
    --duration "5s" \
    --fps 24

Posttrained I2V

The posttrain_image2video mode targets the post-trained Cosmos3-Super-Image2Video model. It produces both an upsampled JSON prompt and a per-sample contextual negative prompt. We use "claude-opus-4-7" for our Cosmos3-Super-Image2Video model.

python -m cosmos_framework.inference.prompt_upsampling \
    --input inputs/prompt_upsampler/prompts_i2v.txt \
    --image-list inputs/prompt_upsampler/images.txt \
    --output outputs/prompt_upsampler/upsampled_i2v_pos_neg \
    --mode posttrain_image2video \
    --endpoint-url "${PROMPT_UPSAMPLER_ENDPOINT_URL}" \
    --model "${PROMPT_UPSAMPLER_MODEL_NAME}" \
    --api-token "${PROMPT_UPSAMPLER_API_TOKEN}" \
    --prompt-template cosmos_framework/inference/prompting_templates/external_api/posttrained_i2v_prompt.txt \
    --json-template cosmos_framework/inference/prompting_templates/external_api/posttrained_i2v_json_schema.json \
    --resolution 480 \
    --aspect-ratio "16,9" \
    --duration "8s" \
    --fps 24

The posttrained I2V templates emit a <negative_prompt> block, so each output record also carries a negative_prompt field.

Template Files

Default external API templates:

• cosmos_framework/inference/prompting_templates/external_api/t2i_prompt.txt
• cosmos_framework/inference/prompting_templates/external_api/t2i_json_schema.json
• cosmos_framework/inference/prompting_templates/external_api/t2v_i2v_video_prompt.txt
• cosmos_framework/inference/prompting_templates/external_api/t2v_i2v_video_json_schema.json
• cosmos_framework/inference/prompting_templates/external_api/posttrained_i2v_prompt.txt
• cosmos_framework/inference/prompting_templates/external_api/posttrained_i2v_json_schema.json

Reasoner templates:

• cosmos_framework/inference/prompting_templates/reasoner/reasoner_t2i_prompt.txt
• cosmos_framework/inference/prompting_templates/reasoner/reasoner_t2i_json_schema.json
• cosmos_framework/inference/prompting_templates/reasoner/reasoner_t2v_prompt.txt
• cosmos_framework/inference/prompting_templates/reasoner/reasoner_t2v_json_schema.json
• cosmos_framework/inference/prompting_templates/reasoner/reasoner_i2v_prompt.txt
• cosmos_framework/inference/prompting_templates/reasoner/reasoner_i2v_json_schema.json