MickJ commited on
Commit
1ea4ae8
·
1 Parent(s): 4f702f1

Add SGLang serving instructions

Browse files

Add SGLang-Diffusion model card examples and tags for Cosmos3 serving.

Files changed (1) hide show
  1. README.md +61 -0
README.md CHANGED
@@ -992,6 +992,67 @@ Cosmos3 may produce imperfect outputs in challenging scenarios. Generation artif
992
 
993
  Cosmos3 outputs should not be treated as physically accurate simulation, reliable ground-truth reasoning, or safety-certified decision making. Applications involving robotics control, autonomous systems, scientific simulation, or safety-critical planning require additional validation, external constraints, system-level safety analysis, and domain-specific guardrails before deployment.
994
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
995
  ## Inference
996
 
997
  **Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM](https://github.com/vllm-project/vllm), [vLLM-Omni](https://github.com/vllm-project/vllm-omni), [Hugging Face Diffusers](https://github.com/huggingface/diffusers)
 
992
 
993
  Cosmos3 outputs should not be treated as physically accurate simulation, reliable ground-truth reasoning, or safety-certified decision making. Applications involving robotics control, autonomous systems, scientific simulation, or safety-critical planning require additional validation, external constraints, system-level safety analysis, and domain-specific guardrails before deployment.
994
 
995
+ ### SGLang
996
+
997
+ SGLang-Diffusion can serve `nvidia/Cosmos3-Super` through OpenAI-compatible image and video endpoints. Install SGLang from source with diffusion dependencies, then start the server:
998
+
999
+ ```bash
1000
+ git clone https://github.com/sgl-project/sglang.git
1001
+ cd sglang
1002
+ pip install -e "python[diffusion]"
1003
+ pip install "cosmos-guardrail==0.3.1"
1004
+
1005
+ sglang serve \
1006
+ --model-path nvidia/Cosmos3-Super \
1007
+ --num-gpus 4
1008
+ ```
1009
+
1010
+ For the video-specialized checkpoint:
1011
+
1012
+ ```bash
1013
+ sglang serve \
1014
+ --model-path nvidia/Cosmos3-Super-Image2Video \
1015
+ --num-gpus 4
1016
+ ```
1017
+
1018
+ Supported SGLang endpoints:
1019
+
1020
+ | Mode | Endpoint | Notes |
1021
+ | --- | --- | --- |
1022
+ | Text to image | `POST /v1/images/generations` | Returns base64 image data by default |
1023
+ | Text to video | `POST /v1/videos` | Creates an async job; poll `GET /v1/videos/{id}` and download `/content` |
1024
+ | Image to video | `POST /v1/videos` | Upload the conditioning image with `input_reference` |
1025
+
1026
+ Example text-to-video request:
1027
+
1028
+ ```bash
1029
+ job_id=$(curl -sS -X POST http://localhost:30000/v1/videos \
1030
+ --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
1031
+ --form-string "negative_prompt=blurry, distorted, low quality" \
1032
+ --form-string "size=1280x720" \
1033
+ --form-string "num_frames=81" \
1034
+ --form-string "fps=24" \
1035
+ --form-string "num_inference_steps=35" \
1036
+ --form-string "guidance_scale=4.0" \
1037
+ --form-string "flow_shift=10.0" \
1038
+ --form-string "seed=42" \
1039
+ --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
1040
+ | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')
1041
+
1042
+ while true; do
1043
+ status=$(curl -sS "http://localhost:30000/v1/videos/${job_id}" \
1044
+ | python -c 'import json, sys; print(json.load(sys.stdin)["status"])')
1045
+ [ "$status" = "completed" ] && break
1046
+ [ "$status" = "failed" ] && exit 1
1047
+ sleep 1
1048
+ done
1049
+
1050
+ curl -sS -L "http://localhost:30000/v1/videos/${job_id}/content" \
1051
+ -o cosmos3_super_t2v_output.mp4
1052
+ ```
1053
+
1054
+ Video-to-video, video-with-sound, and action generation are not supported by SGLang yet.
1055
+
1056
  ## Inference
1057
 
1058
  **Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM](https://github.com/vllm-project/vllm), [vLLM-Omni](https://github.com/vllm-project/vllm-omni), [Hugging Face Diffusers](https://github.com/huggingface/diffusers)