Txu647 commited on
Commit
a134dae
·
verified ·
1 Parent(s): 5bf2930

Document Qwen prompt alignment

Browse files

Add the first-frame Qwen3-VL prompt-alignment workflow and mark its documentation complete.

Files changed (1) hide show
  1. README.md +46 -0
README.md CHANGED
@@ -48,6 +48,12 @@ world state through camera motion, occlusion, and illumination changes.
48
  This repository currently releases only the validated ReMind 5B weights. It
49
  does not contain ReMind 1.3B weights.
50
 
 
 
 
 
 
 
51
  ## Files
52
 
53
  | File | Size | Purpose |
@@ -102,6 +108,46 @@ three-latent-frame chunks, and the shifted four-step schedule
102
  `[1000, 938, 833, 625]`. Camera examples use the bundled pair-fixed GT camera
103
  trajectories.
104
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
105
  ## Checkpoint lineage and validation
106
 
107
  - ReMind-5B generator: pair-fixed 480p checkpoint after an additional 2,000
 
48
  This repository currently releases only the validated ReMind 5B weights. It
49
  does not contain ReMind 1.3B weights.
50
 
51
+ Release checklist:
52
+
53
+ - [x] Release ReMind-5B checkpoints
54
+ - [x] Document Qwen3-VL prompt alignment for custom inputs
55
+ - [ ] Release ReMind-1.3B checkpoints
56
+
57
  ## Files
58
 
59
  | File | Size | Purpose |
 
108
  `[1000, 938, 833, 625]`. Camera examples use the bundled pair-fixed GT camera
109
  trajectories.
110
 
111
+ ## Qwen3-VL prompt alignment for custom inputs
112
+
113
+ For a new first frame and an ordinary user request, we recommend using a local
114
+ `Qwen/Qwen3-VL-8B-Instruct` call before ReMind inference. Give Qwen only the
115
+ initial image and the user's text; do not provide future frames, evaluation
116
+ criteria, or a hand-written target caption. Use greedy decoding and ask for a
117
+ JSON object containing:
118
+
119
+ - `scene_caption`: an English, content-only description of the initial state,
120
+ physical progression, and requested final state;
121
+ - `control_requests`: any explicit `camera`, `occluder`, or `light` request,
122
+ with an exact supporting substring from the user text; and
123
+ - `clarification_questions`: unresolved intent or conflicting controls.
124
+
125
+ Preserve actions, negations, identities, directions, and stated end goals.
126
+ Keep camera motion and reversible visibility events out of `scene_caption`:
127
+ camera geometry belongs in a bundled trajectory, while occluder/light timing
128
+ belongs in the preset's `control` block. Never ask Qwen to invent camera
129
+ matrices or replace a bundled pair-fixed trajectory with an analytic path.
130
+
131
+ Copy the accepted `scene_caption` into the closest preset, configure at most
132
+ one supported control, and validate the resolved seven-chunk schedule before
133
+ loading weights:
134
+
135
+ ```bash
136
+ python inference.py --preset /path/to/custom_case.yaml --dry-run
137
+
138
+ python inference.py \
139
+ --preset /path/to/custom_case.yaml \
140
+ --model-folder checkpoints/Wan2.2-TI2V-5B \
141
+ --base-checkpoint checkpoints/ReMind-5B/ReMind-5B.safetensors \
142
+ --ema-checkpoint checkpoints/ReMind-5B/ReMind-5b-dmd-ema.safetensors \
143
+ --output outputs/custom_case.mp4
144
+ ```
145
+
146
+ Treat Qwen's rewrite and self-review as heuristic. Reject or manually review
147
+ outputs that drop a negation, invent an event, mix control language into the
148
+ scene caption, request multiple controls, or leave the intent ambiguous.
149
+ Fixed project-page captions should be used verbatim rather than rewritten.
150
+
151
  ## Checkpoint lineage and validation
152
 
153
  - ReMind-5B generator: pair-fixed 480p checkpoint after an additional 2,000