| # Viggle Remake: painted frames and the prompt |
|
|
| Part of [Viggle Remake](README.md): what to pass as `--ref`, `--pin` and `--prompt` (the Space and the ComfyUI node take the same three). |
|
|
| ## Painted frames and pins |
|
|
| A painted frame is a frame of the source, repainted in an image editor to show the target: the new |
| character, the object gone, the storm rolled in. We made the ones in this repository with OpenAI's |
| `gpt-image`. They decide the identity and the look; the prompt and the source decide the rest. |
|
|
| - **`--ref`**, one to four times, takes the painted frames. The prompt calls them `<Picture 1>`, `<Picture 2>`, … in this order. |
| - **`--pin`** is either left out entirely or given once per `--ref`, in the same order. Each value is the source frame that ref was painted from. |
| - A pinned ref is a **keyframe**: it is placed inside the output's timeline at that frame, and held there for the first four of the six steps (`--keyframe-until 0.9`). |
| - It is also passed as a plain reference throughout. |
| - Without `--pin`, the refs are references only: they set the look, and the model decides where it lands. |
| - **Pin when the clip has cuts or the painted frame's composition matters.** Pinning keeps shots on time; its cost is that the output copies the painted frame's composition at that frame. |
| - **One painted frame per shot is usually enough.** We add a second only where a character turns to show another side or a new one enters the shot. |
| - **Paint at the canvas size or the source's aspect ratio.** A pinned ref of another size is fitted to the canvas: the first one stretched, the rest cover-cropped. |
|
|
| ## Writing the prompt |
|
|
| Remake reads MiniMax-H3's **six-section prompt**, in this order: |
|
|
| 1. `subject_definitions` |
| 2. `summary` |
| 3. `retention_analysis` |
| 4. `detailed_description` |
| 5. `overall_soundscape` |
| 6. `non_diegetic_music` |
|
|
| The prompt uses three kinds of label: |
|
|
| - `<Video 1>` is the source. |
| - `<Picture N>` is your N-th painted frame. |
| - `<Subject N>` is each thing that changes or stays. |
|
|
| The `detailed_description` places `[Shot N]` events at `MM:SS.mmm` timestamps, so those timestamps |
| must fit `--frames`. MiniMax's own writing guide is |
| [`docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md`](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md) |
| in the base repository. The ten prompts in [`examples/prompts/`](examples/prompts) are the exact |
| prompts behind the example clips of the same name in `examples/media/`. |
|
|
| Three habits matter most: |
|
|
| - **Write it as a description of the target video, not as an instruction.** Write "a pineapple mascot tosses the dough", not "replace the man with a pineapple". |
| - **Say in `retention_analysis` what changes.** That section is obeyed literally: if it says something is preserved, it is. |
| - **Describe each `<Subject N>` well enough to exclude everyone similar in frame.** |
| |
| Every prompt we have evaluated Remake with uses this format. Free-form prompts are untested. |
| |
| |
