liang1225 commited on
Commit
0fae1e9
·
verified ·
1 Parent(s): 6f58f6b

Add image-to-video with negative prompt example

Browse files

Adds an 'Image to video with negative prompt' usage example that mirrors the existing image-to-video section, reusing the same input image and prompt but passing the curated assets/negative_prompt.json as the per-sample negative prompt (JSON-encoded string, matching the model's default negative caption format). Includes the generated example output (assets/edge_i2v_negprompt_output.mp4, 832x480, 121 frames @ 24fps).

.gitattributes CHANGED
@@ -51,3 +51,4 @@ assets/edge_i2v_output.mp4 filter=lfs diff=lfs merge=lfs -text
51
  assets/edge_action_fd_umi_2chunk_output.mp4 filter=lfs diff=lfs merge=lfs -text
52
  assets/edge_action_id_av_0_output.png filter=lfs diff=lfs merge=lfs -text
53
  assets/edge_action_id_av_1_output.png filter=lfs diff=lfs merge=lfs -text
 
 
51
  assets/edge_action_fd_umi_2chunk_output.mp4 filter=lfs diff=lfs merge=lfs -text
52
  assets/edge_action_id_av_0_output.png filter=lfs diff=lfs merge=lfs -text
53
  assets/edge_action_id_av_1_output.png filter=lfs diff=lfs merge=lfs -text
54
+ assets/edge_i2v_negprompt_output.mp4 filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -514,6 +514,49 @@ Example output:
514
 
515
  <video controls width="832" height="480" src="https://huggingface.co/nvidia/Cosmos3-Edge/resolve/main/assets/edge_i2v_output.mp4"></video>
516
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
517
  #### Action — forward dynamics
518
 
519
  Forward dynamics rolls out one action chunk at a time. This self-contained script fetches the UMI conditioning frame and action chunks, runs `cosmos_framework.scripts.inference` per chunk, feeds each chunk's final generated frame in as the next chunk's conditioning image, and stitches the 2-chunk rollout. Install `imageio[ffmpeg]` first.
 
514
 
515
  <video controls width="832" height="480" src="https://huggingface.co/nvidia/Cosmos3-Edge/resolve/main/assets/edge_i2v_output.mp4"></video>
516
 
517
+ #### Image to video with negative prompt
518
+
519
+ The same image-to-video setup as above, but with a structured **negative prompt** to steer the model away from common failure modes (temporal flicker, morphing geometry, broken physics, unstable camera push-in). We reuse the same input image and prompt, and pass the curated `negative_prompt.json` from `assets/` as the per-sample `negative_prompt` (JSON-encoded as a string, mirroring the model's default negative caption format).
520
+
521
+ ```shell
522
+ BASE="https://huggingface.co/nvidia/Cosmos3-Edge/resolve/main/assets"
523
+ python - "$BASE" <<'PY'
524
+ import json, sys, urllib.request
525
+ base = sys.argv[1]
526
+ prompt = urllib.request.urlopen(f"{base}/example_i2v_prompt.json").read().decode()
527
+ negative = urllib.request.urlopen(f"{base}/negative_prompt.json").read().decode()
528
+ json.dump(
529
+ {
530
+ "model_mode": "image2video",
531
+ "prompt": prompt,
532
+ "vision_path": f"{base}/example_i2v_input.jpg",
533
+ "negative_prompt": json.dumps(json.loads(negative)),
534
+ },
535
+ open("i2v_edge_neg.json", "w"), indent=2,
536
+ )
537
+ PY
538
+
539
+ python -m cosmos_framework.scripts.inference \
540
+ --parallelism-preset=latency \
541
+ -i i2v_edge_neg.json \
542
+ -o outputs/omni_edge_neg \
543
+ --checkpoint-path Cosmos3-Edge \
544
+ --resolution 480 \
545
+ --num-frames 121 \
546
+ --fps 24 \
547
+ --num-steps 50 \
548
+ --guidance 5.0 \
549
+ --shift 3.0 \
550
+ --sampler unipc \
551
+ --seed 0
552
+ ```
553
+
554
+ The generated video is written to `outputs/omni_edge_neg/<sample_name>/vision.mp4`.
555
+
556
+ Example output:
557
+
558
+ <video controls width="832" height="480" src="https://huggingface.co/nvidia/Cosmos3-Edge/resolve/main/assets/edge_i2v_negprompt_output.mp4"></video>
559
+
560
  #### Action — forward dynamics
561
 
562
  Forward dynamics rolls out one action chunk at a time. This self-contained script fetches the UMI conditioning frame and action chunks, runs `cosmos_framework.scripts.inference` per chunk, feeds each chunk's final generated frame in as the next chunk's conditioning image, and stitches the 2-chunk rollout. Install `imageio[ffmpeg]` first.
assets/edge_i2v_negprompt_output.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f2d41ae997adea1eef0f046b837404d6add2798e050974ea36696cee4b3efd00
3
+ size 23017533