Txu647 commited on
Commit
302e162
·
verified ·
1 Parent(s): a134dae

Link released Qwen prompt pipeline

Browse files

Replace the manual prompt-alignment outline with executable CLI instructions from the public release.

Files changed (1) hide show
  1. README.md +25 -28
README.md CHANGED
@@ -110,43 +110,40 @@ trajectories.
110
 
111
  ## Qwen3-VL prompt alignment for custom inputs
112
 
113
- For a new first frame and an ordinary user request, we recommend using a local
114
- `Qwen/Qwen3-VL-8B-Instruct` call before ReMind inference. Give Qwen only the
115
- initial image and the user's text; do not provide future frames, evaluation
116
- criteria, or a hand-written target caption. Use greedy decoding and ask for a
117
- JSON object containing:
118
-
119
- - `scene_caption`: an English, content-only description of the initial state,
120
- physical progression, and requested final state;
121
- - `control_requests`: any explicit `camera`, `occluder`, or `light` request,
122
- with an exact supporting substring from the user text; and
123
- - `clarification_questions`: unresolved intent or conflicting controls.
124
-
125
- Preserve actions, negations, identities, directions, and stated end goals.
126
- Keep camera motion and reversible visibility events out of `scene_caption`:
127
- camera geometry belongs in a bundled trajectory, while occluder/light timing
128
- belongs in the preset's `control` block. Never ask Qwen to invent camera
129
- matrices or replace a bundled pair-fixed trajectory with an analytic path.
130
-
131
- Copy the accepted `scene_caption` into the closest preset, configure at most
132
- one supported control, and validate the resolved seven-chunk schedule before
133
- loading weights:
134
 
135
  ```bash
136
- python inference.py --preset /path/to/custom_case.yaml --dry-run
 
 
 
 
 
 
 
 
 
 
137
 
138
  python inference.py \
139
- --preset /path/to/custom_case.yaml \
140
  --model-folder checkpoints/Wan2.2-TI2V-5B \
141
  --base-checkpoint checkpoints/ReMind-5B/ReMind-5B.safetensors \
142
  --ema-checkpoint checkpoints/ReMind-5B/ReMind-5b-dmd-ema.safetensors \
143
- --output outputs/custom_case.mp4
144
  ```
145
 
146
- Treat Qwen's rewrite and self-review as heuristic. Reject or manually review
147
- outputs that drop a negation, invent an event, mix control language into the
148
- scene caption, request multiple controls, or leave the intent ambiguous.
149
- Fixed project-page captions should be used verbatim rather than rewritten.
 
 
 
 
 
150
 
151
  ## Checkpoint lineage and validation
152
 
 
110
 
111
  ## Qwen3-VL prompt alignment for custom inputs
112
 
113
+ The optional adapter turns one initial image and an ordinary user request into
114
+ an audited, executable ReMind preset using greedy
115
+ `Qwen/Qwen3-VL-8B-Instruct` inference:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
116
 
117
  ```bash
118
+ pip install -r requirements_prompt_engineering.txt
119
+
120
+ python tools/rewrite_remind_prompt.py \
121
+ --input-image /path/to/first_frame.png \
122
+ --user-prompt '继续把谷物倒进盘子里,让它逐渐堆积,但不要洒出来。' \
123
+ --name cereal_user \
124
+ --output-dir outputs/prompt_alignment/cereal_user
125
+
126
+ python inference.py \
127
+ --preset outputs/prompt_alignment/cereal_user/preset.yaml \
128
+ --dry-run
129
 
130
  python inference.py \
131
+ --preset outputs/prompt_alignment/cereal_user/preset.yaml \
132
  --model-folder checkpoints/Wan2.2-TI2V-5B \
133
  --base-checkpoint checkpoints/ReMind-5B/ReMind-5B.safetensors \
134
  --ema-checkpoint checkpoints/ReMind-5B/ReMind-5b-dmd-ema.safetensors \
135
+ --output outputs/cereal_user.mp4
136
  ```
137
 
138
+ The adapter keeps scene dynamics separate from camera/visibility controls,
139
+ compiles seven chunk-local prompts, and only exports a preset after schema and
140
+ heuristic VLM self-audit checks pass. It never invents camera geometry: camera
141
+ requests must select one of the bundled pair-fixed trajectories.
142
+
143
+ See [`docs/prompt_alignment.md`](docs/prompt_alignment.md) for camera,
144
+ occluder, light, local-model, and failure-handling options. The self-audit is
145
+ heuristic, so review unusual requests and keep fixed project-page captions
146
+ unchanged.
147
 
148
  ## Checkpoint lineage and validation
149