PROMPTING.md

Viggle Remake: painted frames and the prompt

Part of Viggle Remake: what to pass as --ref, --pin and --prompt (the Space and the ComfyUI node take the same three).

Painted frames and pins

A painted frame is a frame of the source, repainted in an image editor to show the target: the new character, the object gone, the storm rolled in. We made the ones in this repository with OpenAI's gpt-image. They decide the identity and the look; the prompt and the source decide the rest.

• --ref, one to four times, takes the painted frames. The prompt calls them <Picture 1>, <Picture 2>, … in this order.
• --pin is either left out entirely or given once per --ref, in the same order. Each value is the source frame that ref was painted from.
  • A pinned ref is a keyframe: it is placed inside the output's timeline at that frame, and held there for the first four of the six steps (--keyframe-until 0.9).
  • It is also passed as a plain reference throughout.
  • Without --pin, the refs are references only: they set the look, and the model decides where it lands.
• Pin when the clip has cuts or the painted frame's composition matters. Pinning keeps shots on time; its cost is that the output copies the painted frame's composition at that frame.
• One painted frame per shot is usually enough. We add a second only where a character turns to show another side or a new one enters the shot.
• Paint at the canvas size or the source's aspect ratio. A pinned ref of another size is fitted to the canvas: the first one stretched, the rest cover-cropped.

Writing the prompt

Remake reads MiniMax-H3's six-section prompt, in this order:

1. subject_definitions
2. summary
3. retention_analysis
4. detailed_description
5. overall_soundscape
6. non_diegetic_music

The prompt uses three kinds of label:

• <Video 1> is the source.
• <Picture N> is your N-th painted frame.
• <Subject N> is each thing that changes or stays.

The detailed_description places [Shot N] events at MM:SS.mmm timestamps, so those timestamps must fit --frames. MiniMax's own writing guide is docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md in the base repository. The ten prompts in examples/prompts/ are the exact prompts behind the example clips of the same name in examples/media/.

Three habits matter most:

• Write it as a description of the target video, not as an instruction. Write "a pineapple mascot tosses the dough", not "replace the man with a pineapple".
• Say in retention_analysis what changes. That section is obeyed literally: if it says something is preserved, it is.
• Describe each <Subject N> well enough to exclude everyone similar in frame.

Every prompt we have evaluated Remake with uses this format. Free-form prompts are untested.