mrjackspade commited on
Commit
c802bec
·
verified ·
1 Parent(s): b566e32

Rewrite model card for reader-facing clarity

Browse files
Files changed (1) hide show
  1. README.md +41 -48
README.md CHANGED
@@ -17,8 +17,8 @@ tags:
17
 
18
  This repository contains an experimental, stock-shaped replacement for the Qwen3-VL 8B text encoder
19
  used by Ideogram 4, plus the matching training LoRA. It is intended to make Ideogram 4 respond to
20
- ordinary natural-language prompts more like the stock model responds to structured Magic-Prompt JSON,
21
- without a runtime prompt rewriter, custom node, companion language model, or patched ComfyUI.
22
 
23
  This release is the first strength-0.2 milestone. The nominal 500-step milestone resolved to the first
24
  complete checkpoint in that checkpoint band, **optimizer step 510**.
@@ -33,8 +33,9 @@ complete checkpoint in that checkpoint band, **optimizer step 510**.
33
  | `manifests/merge_step_00000510.json` | Merge inputs, output hash, and per-projection quantization metrics | — | See `SHA256SUMS` |
34
  | `COMPATIBILITY.md` | Component boundary, provenance, and third-party compatibility notice | — | See `SHA256SUMS` |
35
 
36
- The LoRA uses this project's native tensor names (`layers.N.<projection>.lora_a` and `lora_b`) and is
37
- not advertised as a PEFT or generic ComfyUI LoRA. Most users should use the merged text encoder.
 
38
 
39
  ## Installation
40
 
@@ -45,8 +46,7 @@ ComfyUI/models/text_encoders/
45
  ```
46
 
47
  Select it in the same Ideogram 4 text-encoder loader used for the stock
48
- `qwen3vl_8b_fp8_scaled.safetensors`. No inference-time LoRA loader or source-code modification is
49
- required.
50
 
51
  ## What was trained
52
 
@@ -60,9 +60,8 @@ detailed natural language -> weak/broken stock conditioning
60
  structured Magic-Prompt JSON -> functional stock conditioning
61
  ```
62
 
63
- Each natural-language prompt therefore has a structured teacher target. The adapted text encoder is
64
- not trained to reproduce the teacher's token sequence or Qwen activations. Natural-language and JSON
65
- sequences differ in length, so the student is instead optimized at Ideogram's image-side response:
66
 
67
  ```text
68
  c_T = stock_text_encoder(teacher_json)
@@ -74,9 +73,8 @@ v_S = ideogram_conditional(z_t, t, c_S)
74
  loss = mean((v_S - v_T) ** 2)
75
  ```
76
 
77
- Ideogram parameters are frozen, but gradients remain enabled through the student conditional forward
78
- so the velocity loss can update the text-encoder adapters. There is no stock-natural-language retention
79
- term: preserving stock NL behavior would preserve the behavior this project is replacing.
80
 
81
  ### Stock text-encoder path
82
 
@@ -97,7 +95,7 @@ then projected to the 4,608-wide Ideogram DiT. The student preserves this archit
97
  - Teacher conditioning encoded through the exact stock Comfy tokenizer, Qwen taps, and Ideogram ingress
98
  projection, stored as float16.
99
 
100
- The source prompts, teacher JSON, and adult-content corpus are not distributed in this model repository.
101
 
102
  ### Eight-state trajectory supervision
103
 
@@ -125,7 +123,7 @@ authoritative Ideogram 4 debanner direction:
125
  - image tokens only;
126
  - subtract the nearest-neighbor-resized 8×8 spatial direction, then restore each image token's
127
  original L2 norm;
128
- - unconditional model unchanged.
129
 
130
  Only the first cached target velocity is directly corrected. The later seven target velocities are
131
  uncorrected but evaluated on the trajectory produced by the corrected first update. The student always
@@ -137,7 +135,7 @@ are rejected before model loading.
137
 
138
  ## Adapter geometry
139
 
140
- The geometry is fixed and was not reduced for this checkpoint:
141
 
142
  - rank: 64;
143
  - alpha: 64;
@@ -147,9 +145,8 @@ The geometry is fixed and was not reduced for this checkpoint:
147
  - trainable parameters: 174,587,904;
148
  - LoRA tensors: 504.
149
 
150
- Per block, the adapter contains 4,849,664 parameters. The all-block topology is intentional: the task is
151
- a distributed change in how the 13 tapped Qwen layers express conditioning, not memorization in a small
152
- upper-block adapter.
153
 
154
  ## Optimization and numerics
155
 
@@ -161,7 +158,7 @@ upper-block adapter.
161
  - Adaptive packing: batch 2 through 1,017 student tokens; longer examples use singleton microbatches.
162
  - Gradient clipping: global norm `1.0`.
163
  - Optimizer state is held on CPU between updates.
164
- - Unused multimodal Qwen components are pruned from the training graph; all 36 text blocks remain.
165
  - Initial learning rate: `3.2e-4`.
166
 
167
  Training is quantization-aware for the final deployment format. For each adapted projection, the forward
@@ -169,12 +166,12 @@ path forms the dequantized stock weight plus the LoRA delta, then requantizes us
169
  scale to `float8_e4m3fn`. A straight-through estimator supplies LoRA gradients without constructing a
170
  dense base-weight gradient.
171
 
172
- Learning rate is not tied to a fixed step horizon. A transactional controller evaluates fixed 50-step
173
- windows every 10 steps, estimates log-loss descent relative to robust residual noise, confirms a failed
174
- window for 20 additional steps, locates the elbow, restores the complete optimizer/sampler/RNG state at
175
- the rollback boundary, and continues at half the LR. A rollback is issued only when both deterministic
176
- held-out loss and interval-averaged training loss fail their descent tests. The LR floor is positive and
177
- does not terminate training.
178
 
179
  ## Step-510 checkpoint metrics
180
 
@@ -190,9 +187,8 @@ does not terminate training.
190
  | Peak reserved VRAM | 24.463 GiB |
191
  | Behavior microbatches | 8 batch-2, 0 singleton |
192
 
193
- Velocity MSE is a training/health proxy, not a complete image-quality metric. The release decision was
194
- made from fixed-seed stock/teacher/checkpoint image comparisons and owner visual review. No standardized
195
- perceptual benchmark, OCR benchmark, or broad safety evaluation is claimed for this checkpoint.
196
 
197
  ## Merge and deployment format
198
 
@@ -204,38 +200,35 @@ The standalone file was merged on CPU into the exact stock Comfy scaled-FP8 chec
204
  - output tensors: 1,254;
205
  - merged projections: 252;
206
  - stock per-tensor FP8 scales preserved;
207
- - no training-only adapter keys added to the merged file.
208
 
209
  The base file is byte-identical to the text encoder published in `Comfy-Org/Qwen3-VL` and
210
  `Comfy-Org/Ideogram-4` under the base SHA above.
211
 
212
  ## Limitations
213
 
214
- - This is an early step-510 research checkpoint, not the end of the longer training lineage.
215
- - The learned conditioning is specific to Ideogram 4's frozen conditional model and Comfy text-encoder
216
- ingress path. It is not a general-purpose Qwen3-VL chat model.
217
  - Training and validation use cached 512×512 trajectories from one fixed sampler contract. Behavior at
218
  other resolutions, schedules, samplers, and seeds requires independent evaluation.
219
  - Teacher targets can be imperfect or semantically inconsistent despite validation.
220
- - The corpus deliberately spans adult source-rating groups. The model has not been safety-aligned or
221
- certified for unrestricted deployment.
222
  - Static text conditioning may only approximate a first-step, latent-dependent spatial intervention.
223
- - FP8 requantization does not retain every low-magnitude LoRA delta exactly; per-projection retention
224
- metrics are included in the merge manifest.
225
 
226
  ## Component and licensing boundary
227
 
228
- The files in this repository contain only modified Qwen3-VL text-encoder parameters. They start from
229
- the independently published `Comfy-Org/Qwen3-VL` scaled-FP8 checkpoint and are released under its
230
- Apache-2.0 license. They contain no Ideogram DiT, VAE, sampler, tokenizer implementation, source code,
231
- or other Ideogram model parameters.
232
 
233
- During training, a separately obtained, frozen Ideogram 4 model was used as a differentiable downstream
234
- evaluator of the conditioning interface. Its weights were never trainable, copied into the adapter, or
235
- included in either release file. The released text encoder cannot generate images or substitute for
236
- Ideogram by itself.
237
 
238
- Ideogram 4 is third-party software/modeling infrastructure and is not distributed here. Anyone choosing
239
- to use these Qwen weights with Ideogram must obtain Ideogram separately and is independently responsible
240
- for reviewing and complying with Ideogram's license and usage policy. This repository is not affiliated
241
- with or endorsed by Ideogram, Inc.
 
17
 
18
  This repository contains an experimental, stock-shaped replacement for the Qwen3-VL 8B text encoder
19
  used by Ideogram 4, plus the matching training LoRA. It is intended to make Ideogram 4 respond to
20
+ ordinary natural-language prompts more like the stock model responds to structured Magic-Prompt JSON.
21
+ It operates as a direct checkpoint replacement through the stock ComfyUI loader.
22
 
23
  This release is the first strength-0.2 milestone. The nominal 500-step milestone resolved to the first
24
  complete checkpoint in that checkpoint band, **optimizer step 510**.
 
33
  | `manifests/merge_step_00000510.json` | Merge inputs, output hash, and per-projection quantization metrics | — | See `SHA256SUMS` |
34
  | `COMPATIBILITY.md` | Component boundary, provenance, and third-party compatibility notice | — | See `SHA256SUMS` |
35
 
36
+ The LoRA uses this project's native tensor names (`layers.N.<projection>.lora_a` and `lora_b`). The
37
+ merged text encoder is the recommended artifact for inference; the LoRA is provided for research,
38
+ inspection, and reproducibility.
39
 
40
  ## Installation
41
 
 
46
  ```
47
 
48
  Select it in the same Ideogram 4 text-encoder loader used for the stock
49
+ `qwen3vl_8b_fp8_scaled.safetensors`.
 
50
 
51
  ## What was trained
52
 
 
60
  structured Magic-Prompt JSON -> functional stock conditioning
61
  ```
62
 
63
+ Each natural-language prompt therefore has a structured teacher target. Because the natural-language
64
+ and JSON sequences differ in length, training matches their effect at Ideogram's image-side response:
 
65
 
66
  ```text
67
  c_T = stock_text_encoder(teacher_json)
 
73
  loss = mean((v_S - v_T) ** 2)
74
  ```
75
 
76
+ Ideogram parameters remain frozen while gradients pass through the student conditional forward to the
77
+ text-encoder adapters. The objective is entirely focused on matching the structured teacher response.
 
78
 
79
  ### Stock text-encoder path
80
 
 
95
  - Teacher conditioning encoded through the exact stock Comfy tokenizer, Qwen taps, and Ideogram ingress
96
  projection, stored as float16.
97
 
98
+ This release contains the trained weights, configuration, checkpoint metadata, and merge provenance.
99
 
100
  ### Eight-state trajectory supervision
101
 
 
123
  - image tokens only;
124
  - subtract the nearest-neighbor-resized 8×8 spatial direction, then restore each image token's
125
  original L2 norm;
126
+ - the correction is confined to the positive conditional pass.
127
 
128
  Only the first cached target velocity is directly corrected. The later seven target velocities are
129
  uncorrected but evaluated on the trajectory produced by the corrected first update. The student always
 
135
 
136
  ## Adapter geometry
137
 
138
+ This checkpoint uses the following adapter geometry:
139
 
140
  - rank: 64;
141
  - alpha: 64;
 
145
  - trainable parameters: 174,587,904;
146
  - LoRA tensors: 504.
147
 
148
+ Per block, the adapter contains 4,849,664 parameters. Adapting all 36 blocks gives every tapped depth a
149
+ trainable path for reshaping the conditioning features consumed by Ideogram.
 
150
 
151
  ## Optimization and numerics
152
 
 
158
  - Adaptive packing: batch 2 through 1,017 student tokens; longer examples use singleton microbatches.
159
  - Gradient clipping: global norm `1.0`.
160
  - Optimizer state is held on CPU between updates.
161
+ - The training graph contains the Qwen text transformer with all 36 text blocks.
162
  - Initial learning rate: `3.2e-4`.
163
 
164
  Training is quantization-aware for the final deployment format. For each adapted projection, the forward
 
166
  scale to `float8_e4m3fn`. A straight-through estimator supplies LoRA gradients without constructing a
167
  dense base-weight gradient.
168
 
169
+ Learning rate is managed by a transactional loss-curve controller. It evaluates fixed 50-step windows
170
+ every 10 steps, estimates log-loss descent relative to robust residual noise, confirms a failed window
171
+ for 20 additional steps, locates the elbow, restores the complete optimizer/sampler/RNG state at the
172
+ rollback boundary, and continues at half the LR. A rollback requires both deterministic held-out loss
173
+ and interval-averaged training loss to fail their descent tests. Reaching the positive LR floor allows
174
+ training to continue.
175
 
176
  ## Step-510 checkpoint metrics
177
 
 
187
  | Peak reserved VRAM | 24.463 GiB |
188
  | Behavior microbatches | 8 batch-2, 0 singleton |
189
 
190
+ Evaluation combines velocity MSE with fixed-seed stock/teacher/checkpoint image comparisons and owner
191
+ visual review. Standardized perceptual, OCR, and broad safety benchmarking remain future work.
 
192
 
193
  ## Merge and deployment format
194
 
 
200
  - output tensors: 1,254;
201
  - merged projections: 252;
202
  - stock per-tensor FP8 scales preserved;
203
+ - tensor inventory remains compatible with the stock text-encoder checkpoint layout.
204
 
205
  The base file is byte-identical to the text encoder published in `Comfy-Org/Qwen3-VL` and
206
  `Comfy-Org/Ideogram-4` under the base SHA above.
207
 
208
  ## Limitations
209
 
210
+ - Step 510 is the first public research milestone from a longer training lineage.
211
+ - The learned conditioning targets Ideogram 4's frozen conditional model and Comfy text-encoder ingress
212
+ path; its intended use is image-conditioning inference through that interface.
213
  - Training and validation use cached 512×512 trajectories from one fixed sampler contract. Behavior at
214
  other resolutions, schedules, samplers, and seeds requires independent evaluation.
215
  - Teacher targets can be imperfect or semantically inconsistent despite validation.
216
+ - The corpus deliberately spans adult source-rating groups. Deployment requires an application-level
217
+ safety policy appropriate to the intended use.
218
  - Static text conditioning may only approximate a first-step, latent-dependent spatial intervention.
219
+ - FP8 requantization rounds low-magnitude LoRA deltas to the available scaled-FP8 grid; per-projection
220
+ retention metrics are included in the merge manifest.
221
 
222
  ## Component and licensing boundary
223
 
224
+ The release artifacts consist exclusively of modified Qwen3-VL text-encoder parameters. They start
225
+ from the independently published `Comfy-Org/Qwen3-VL` scaled-FP8 checkpoint and are released under its
226
+ Apache-2.0 license.
 
227
 
228
+ During training, a separately obtained, frozen Ideogram 4 model served as a differentiable downstream
229
+ evaluator of the conditioning interface. Optimization updated the Qwen adapter parameters, and the
230
+ release files contain the resulting Qwen text-encoder weights.
 
231
 
232
+ Ideogram 4 is acquired separately under its own license and usage policy. Users combining the two
233
+ components are responsible for the terms applicable to each. This repository is independently produced
234
+ and has no affiliation with Ideogram, Inc.