Image-to-Video
Wan2.2
English
remind
remind-5b
video-generation
world-model
video-world-model
causal-video-generation
dynamic-memory
out-of-sight-state-evolution
teaching-video-generators-to-remember
lora
Instructions to use AppliedIntuitionResearch/ReMind with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Wan2.2
How to use AppliedIntuitionResearch/ReMind with Wan2.2:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Link released Qwen prompt pipeline
Browse filesReplace the manual prompt-alignment outline with executable CLI instructions from the public release.
README.md
CHANGED
|
@@ -110,43 +110,40 @@ trajectories.
|
|
| 110 |
|
| 111 |
## Qwen3-VL prompt alignment for custom inputs
|
| 112 |
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
criteria, or a hand-written target caption. Use greedy decoding and ask for a
|
| 117 |
-
JSON object containing:
|
| 118 |
-
|
| 119 |
-
- `scene_caption`: an English, content-only description of the initial state,
|
| 120 |
-
physical progression, and requested final state;
|
| 121 |
-
- `control_requests`: any explicit `camera`, `occluder`, or `light` request,
|
| 122 |
-
with an exact supporting substring from the user text; and
|
| 123 |
-
- `clarification_questions`: unresolved intent or conflicting controls.
|
| 124 |
-
|
| 125 |
-
Preserve actions, negations, identities, directions, and stated end goals.
|
| 126 |
-
Keep camera motion and reversible visibility events out of `scene_caption`:
|
| 127 |
-
camera geometry belongs in a bundled trajectory, while occluder/light timing
|
| 128 |
-
belongs in the preset's `control` block. Never ask Qwen to invent camera
|
| 129 |
-
matrices or replace a bundled pair-fixed trajectory with an analytic path.
|
| 130 |
-
|
| 131 |
-
Copy the accepted `scene_caption` into the closest preset, configure at most
|
| 132 |
-
one supported control, and validate the resolved seven-chunk schedule before
|
| 133 |
-
loading weights:
|
| 134 |
|
| 135 |
```bash
|
| 136 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 137 |
|
| 138 |
python inference.py \
|
| 139 |
-
--preset /
|
| 140 |
--model-folder checkpoints/Wan2.2-TI2V-5B \
|
| 141 |
--base-checkpoint checkpoints/ReMind-5B/ReMind-5B.safetensors \
|
| 142 |
--ema-checkpoint checkpoints/ReMind-5B/ReMind-5b-dmd-ema.safetensors \
|
| 143 |
-
--output outputs/
|
| 144 |
```
|
| 145 |
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 150 |
|
| 151 |
## Checkpoint lineage and validation
|
| 152 |
|
|
|
|
| 110 |
|
| 111 |
## Qwen3-VL prompt alignment for custom inputs
|
| 112 |
|
| 113 |
+
The optional adapter turns one initial image and an ordinary user request into
|
| 114 |
+
an audited, executable ReMind preset using greedy
|
| 115 |
+
`Qwen/Qwen3-VL-8B-Instruct` inference:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
|
| 117 |
```bash
|
| 118 |
+
pip install -r requirements_prompt_engineering.txt
|
| 119 |
+
|
| 120 |
+
python tools/rewrite_remind_prompt.py \
|
| 121 |
+
--input-image /path/to/first_frame.png \
|
| 122 |
+
--user-prompt '继续把谷物倒进盘子里,让它逐渐堆积,但不要洒出来。' \
|
| 123 |
+
--name cereal_user \
|
| 124 |
+
--output-dir outputs/prompt_alignment/cereal_user
|
| 125 |
+
|
| 126 |
+
python inference.py \
|
| 127 |
+
--preset outputs/prompt_alignment/cereal_user/preset.yaml \
|
| 128 |
+
--dry-run
|
| 129 |
|
| 130 |
python inference.py \
|
| 131 |
+
--preset outputs/prompt_alignment/cereal_user/preset.yaml \
|
| 132 |
--model-folder checkpoints/Wan2.2-TI2V-5B \
|
| 133 |
--base-checkpoint checkpoints/ReMind-5B/ReMind-5B.safetensors \
|
| 134 |
--ema-checkpoint checkpoints/ReMind-5B/ReMind-5b-dmd-ema.safetensors \
|
| 135 |
+
--output outputs/cereal_user.mp4
|
| 136 |
```
|
| 137 |
|
| 138 |
+
The adapter keeps scene dynamics separate from camera/visibility controls,
|
| 139 |
+
compiles seven chunk-local prompts, and only exports a preset after schema and
|
| 140 |
+
heuristic VLM self-audit checks pass. It never invents camera geometry: camera
|
| 141 |
+
requests must select one of the bundled pair-fixed trajectories.
|
| 142 |
+
|
| 143 |
+
See [`docs/prompt_alignment.md`](docs/prompt_alignment.md) for camera,
|
| 144 |
+
occluder, light, local-model, and failure-handling options. The self-audit is
|
| 145 |
+
heuristic, so review unusual requests and keep fixed project-page captions
|
| 146 |
+
unchanged.
|
| 147 |
|
| 148 |
## Checkpoint lineage and validation
|
| 149 |
|