Rewrite model card for reader-facing clarity
Browse files
README.md
CHANGED
|
@@ -17,8 +17,8 @@ tags:
|
|
| 17 |
|
| 18 |
This repository contains an experimental, stock-shaped replacement for the Qwen3-VL 8B text encoder
|
| 19 |
used by Ideogram 4, plus the matching training LoRA. It is intended to make Ideogram 4 respond to
|
| 20 |
-
ordinary natural-language prompts more like the stock model responds to structured Magic-Prompt JSON
|
| 21 |
-
|
| 22 |
|
| 23 |
This release is the first strength-0.2 milestone. The nominal 500-step milestone resolved to the first
|
| 24 |
complete checkpoint in that checkpoint band, **optimizer step 510**.
|
|
@@ -33,8 +33,9 @@ complete checkpoint in that checkpoint band, **optimizer step 510**.
|
|
| 33 |
| `manifests/merge_step_00000510.json` | Merge inputs, output hash, and per-projection quantization metrics | — | See `SHA256SUMS` |
|
| 34 |
| `COMPATIBILITY.md` | Component boundary, provenance, and third-party compatibility notice | — | See `SHA256SUMS` |
|
| 35 |
|
| 36 |
-
The LoRA uses this project's native tensor names (`layers.N.<projection>.lora_a` and `lora_b`)
|
| 37 |
-
|
|
|
|
| 38 |
|
| 39 |
## Installation
|
| 40 |
|
|
@@ -45,8 +46,7 @@ ComfyUI/models/text_encoders/
|
|
| 45 |
```
|
| 46 |
|
| 47 |
Select it in the same Ideogram 4 text-encoder loader used for the stock
|
| 48 |
-
`qwen3vl_8b_fp8_scaled.safetensors`.
|
| 49 |
-
required.
|
| 50 |
|
| 51 |
## What was trained
|
| 52 |
|
|
@@ -60,9 +60,8 @@ detailed natural language -> weak/broken stock conditioning
|
|
| 60 |
structured Magic-Prompt JSON -> functional stock conditioning
|
| 61 |
```
|
| 62 |
|
| 63 |
-
Each natural-language prompt therefore has a structured teacher target.
|
| 64 |
-
|
| 65 |
-
sequences differ in length, so the student is instead optimized at Ideogram's image-side response:
|
| 66 |
|
| 67 |
```text
|
| 68 |
c_T = stock_text_encoder(teacher_json)
|
|
@@ -74,9 +73,8 @@ v_S = ideogram_conditional(z_t, t, c_S)
|
|
| 74 |
loss = mean((v_S - v_T) ** 2)
|
| 75 |
```
|
| 76 |
|
| 77 |
-
Ideogram parameters
|
| 78 |
-
|
| 79 |
-
term: preserving stock NL behavior would preserve the behavior this project is replacing.
|
| 80 |
|
| 81 |
### Stock text-encoder path
|
| 82 |
|
|
@@ -97,7 +95,7 @@ then projected to the 4,608-wide Ideogram DiT. The student preserves this archit
|
|
| 97 |
- Teacher conditioning encoded through the exact stock Comfy tokenizer, Qwen taps, and Ideogram ingress
|
| 98 |
projection, stored as float16.
|
| 99 |
|
| 100 |
-
|
| 101 |
|
| 102 |
### Eight-state trajectory supervision
|
| 103 |
|
|
@@ -125,7 +123,7 @@ authoritative Ideogram 4 debanner direction:
|
|
| 125 |
- image tokens only;
|
| 126 |
- subtract the nearest-neighbor-resized 8×8 spatial direction, then restore each image token's
|
| 127 |
original L2 norm;
|
| 128 |
-
-
|
| 129 |
|
| 130 |
Only the first cached target velocity is directly corrected. The later seven target velocities are
|
| 131 |
uncorrected but evaluated on the trajectory produced by the corrected first update. The student always
|
|
@@ -137,7 +135,7 @@ are rejected before model loading.
|
|
| 137 |
|
| 138 |
## Adapter geometry
|
| 139 |
|
| 140 |
-
|
| 141 |
|
| 142 |
- rank: 64;
|
| 143 |
- alpha: 64;
|
|
@@ -147,9 +145,8 @@ The geometry is fixed and was not reduced for this checkpoint:
|
|
| 147 |
- trainable parameters: 174,587,904;
|
| 148 |
- LoRA tensors: 504.
|
| 149 |
|
| 150 |
-
Per block, the adapter contains 4,849,664 parameters.
|
| 151 |
-
|
| 152 |
-
upper-block adapter.
|
| 153 |
|
| 154 |
## Optimization and numerics
|
| 155 |
|
|
@@ -161,7 +158,7 @@ upper-block adapter.
|
|
| 161 |
- Adaptive packing: batch 2 through 1,017 student tokens; longer examples use singleton microbatches.
|
| 162 |
- Gradient clipping: global norm `1.0`.
|
| 163 |
- Optimizer state is held on CPU between updates.
|
| 164 |
-
-
|
| 165 |
- Initial learning rate: `3.2e-4`.
|
| 166 |
|
| 167 |
Training is quantization-aware for the final deployment format. For each adapted projection, the forward
|
|
@@ -169,12 +166,12 @@ path forms the dequantized stock weight plus the LoRA delta, then requantizes us
|
|
| 169 |
scale to `float8_e4m3fn`. A straight-through estimator supplies LoRA gradients without constructing a
|
| 170 |
dense base-weight gradient.
|
| 171 |
|
| 172 |
-
Learning rate is
|
| 173 |
-
|
| 174 |
-
|
| 175 |
-
|
| 176 |
-
|
| 177 |
-
|
| 178 |
|
| 179 |
## Step-510 checkpoint metrics
|
| 180 |
|
|
@@ -190,9 +187,8 @@ does not terminate training.
|
|
| 190 |
| Peak reserved VRAM | 24.463 GiB |
|
| 191 |
| Behavior microbatches | 8 batch-2, 0 singleton |
|
| 192 |
|
| 193 |
-
|
| 194 |
-
|
| 195 |
-
perceptual benchmark, OCR benchmark, or broad safety evaluation is claimed for this checkpoint.
|
| 196 |
|
| 197 |
## Merge and deployment format
|
| 198 |
|
|
@@ -204,38 +200,35 @@ The standalone file was merged on CPU into the exact stock Comfy scaled-FP8 chec
|
|
| 204 |
- output tensors: 1,254;
|
| 205 |
- merged projections: 252;
|
| 206 |
- stock per-tensor FP8 scales preserved;
|
| 207 |
-
-
|
| 208 |
|
| 209 |
The base file is byte-identical to the text encoder published in `Comfy-Org/Qwen3-VL` and
|
| 210 |
`Comfy-Org/Ideogram-4` under the base SHA above.
|
| 211 |
|
| 212 |
## Limitations
|
| 213 |
|
| 214 |
-
-
|
| 215 |
-
- The learned conditioning
|
| 216 |
-
|
| 217 |
- Training and validation use cached 512×512 trajectories from one fixed sampler contract. Behavior at
|
| 218 |
other resolutions, schedules, samplers, and seeds requires independent evaluation.
|
| 219 |
- Teacher targets can be imperfect or semantically inconsistent despite validation.
|
| 220 |
-
- The corpus deliberately spans adult source-rating groups.
|
| 221 |
-
|
| 222 |
- Static text conditioning may only approximate a first-step, latent-dependent spatial intervention.
|
| 223 |
-
- FP8 requantization
|
| 224 |
-
metrics are included in the merge manifest.
|
| 225 |
|
| 226 |
## Component and licensing boundary
|
| 227 |
|
| 228 |
-
The
|
| 229 |
-
the independently published `Comfy-Org/Qwen3-VL` scaled-FP8 checkpoint and are released under its
|
| 230 |
-
Apache-2.0 license.
|
| 231 |
-
or other Ideogram model parameters.
|
| 232 |
|
| 233 |
-
During training, a separately obtained, frozen Ideogram 4 model
|
| 234 |
-
evaluator of the conditioning interface.
|
| 235 |
-
|
| 236 |
-
Ideogram by itself.
|
| 237 |
|
| 238 |
-
Ideogram 4 is
|
| 239 |
-
|
| 240 |
-
|
| 241 |
-
with or endorsed by Ideogram, Inc.
|
|
|
|
| 17 |
|
| 18 |
This repository contains an experimental, stock-shaped replacement for the Qwen3-VL 8B text encoder
|
| 19 |
used by Ideogram 4, plus the matching training LoRA. It is intended to make Ideogram 4 respond to
|
| 20 |
+
ordinary natural-language prompts more like the stock model responds to structured Magic-Prompt JSON.
|
| 21 |
+
It operates as a direct checkpoint replacement through the stock ComfyUI loader.
|
| 22 |
|
| 23 |
This release is the first strength-0.2 milestone. The nominal 500-step milestone resolved to the first
|
| 24 |
complete checkpoint in that checkpoint band, **optimizer step 510**.
|
|
|
|
| 33 |
| `manifests/merge_step_00000510.json` | Merge inputs, output hash, and per-projection quantization metrics | — | See `SHA256SUMS` |
|
| 34 |
| `COMPATIBILITY.md` | Component boundary, provenance, and third-party compatibility notice | — | See `SHA256SUMS` |
|
| 35 |
|
| 36 |
+
The LoRA uses this project's native tensor names (`layers.N.<projection>.lora_a` and `lora_b`). The
|
| 37 |
+
merged text encoder is the recommended artifact for inference; the LoRA is provided for research,
|
| 38 |
+
inspection, and reproducibility.
|
| 39 |
|
| 40 |
## Installation
|
| 41 |
|
|
|
|
| 46 |
```
|
| 47 |
|
| 48 |
Select it in the same Ideogram 4 text-encoder loader used for the stock
|
| 49 |
+
`qwen3vl_8b_fp8_scaled.safetensors`.
|
|
|
|
| 50 |
|
| 51 |
## What was trained
|
| 52 |
|
|
|
|
| 60 |
structured Magic-Prompt JSON -> functional stock conditioning
|
| 61 |
```
|
| 62 |
|
| 63 |
+
Each natural-language prompt therefore has a structured teacher target. Because the natural-language
|
| 64 |
+
and JSON sequences differ in length, training matches their effect at Ideogram's image-side response:
|
|
|
|
| 65 |
|
| 66 |
```text
|
| 67 |
c_T = stock_text_encoder(teacher_json)
|
|
|
|
| 73 |
loss = mean((v_S - v_T) ** 2)
|
| 74 |
```
|
| 75 |
|
| 76 |
+
Ideogram parameters remain frozen while gradients pass through the student conditional forward to the
|
| 77 |
+
text-encoder adapters. The objective is entirely focused on matching the structured teacher response.
|
|
|
|
| 78 |
|
| 79 |
### Stock text-encoder path
|
| 80 |
|
|
|
|
| 95 |
- Teacher conditioning encoded through the exact stock Comfy tokenizer, Qwen taps, and Ideogram ingress
|
| 96 |
projection, stored as float16.
|
| 97 |
|
| 98 |
+
This release contains the trained weights, configuration, checkpoint metadata, and merge provenance.
|
| 99 |
|
| 100 |
### Eight-state trajectory supervision
|
| 101 |
|
|
|
|
| 123 |
- image tokens only;
|
| 124 |
- subtract the nearest-neighbor-resized 8×8 spatial direction, then restore each image token's
|
| 125 |
original L2 norm;
|
| 126 |
+
- the correction is confined to the positive conditional pass.
|
| 127 |
|
| 128 |
Only the first cached target velocity is directly corrected. The later seven target velocities are
|
| 129 |
uncorrected but evaluated on the trajectory produced by the corrected first update. The student always
|
|
|
|
| 135 |
|
| 136 |
## Adapter geometry
|
| 137 |
|
| 138 |
+
This checkpoint uses the following adapter geometry:
|
| 139 |
|
| 140 |
- rank: 64;
|
| 141 |
- alpha: 64;
|
|
|
|
| 145 |
- trainable parameters: 174,587,904;
|
| 146 |
- LoRA tensors: 504.
|
| 147 |
|
| 148 |
+
Per block, the adapter contains 4,849,664 parameters. Adapting all 36 blocks gives every tapped depth a
|
| 149 |
+
trainable path for reshaping the conditioning features consumed by Ideogram.
|
|
|
|
| 150 |
|
| 151 |
## Optimization and numerics
|
| 152 |
|
|
|
|
| 158 |
- Adaptive packing: batch 2 through 1,017 student tokens; longer examples use singleton microbatches.
|
| 159 |
- Gradient clipping: global norm `1.0`.
|
| 160 |
- Optimizer state is held on CPU between updates.
|
| 161 |
+
- The training graph contains the Qwen text transformer with all 36 text blocks.
|
| 162 |
- Initial learning rate: `3.2e-4`.
|
| 163 |
|
| 164 |
Training is quantization-aware for the final deployment format. For each adapted projection, the forward
|
|
|
|
| 166 |
scale to `float8_e4m3fn`. A straight-through estimator supplies LoRA gradients without constructing a
|
| 167 |
dense base-weight gradient.
|
| 168 |
|
| 169 |
+
Learning rate is managed by a transactional loss-curve controller. It evaluates fixed 50-step windows
|
| 170 |
+
every 10 steps, estimates log-loss descent relative to robust residual noise, confirms a failed window
|
| 171 |
+
for 20 additional steps, locates the elbow, restores the complete optimizer/sampler/RNG state at the
|
| 172 |
+
rollback boundary, and continues at half the LR. A rollback requires both deterministic held-out loss
|
| 173 |
+
and interval-averaged training loss to fail their descent tests. Reaching the positive LR floor allows
|
| 174 |
+
training to continue.
|
| 175 |
|
| 176 |
## Step-510 checkpoint metrics
|
| 177 |
|
|
|
|
| 187 |
| Peak reserved VRAM | 24.463 GiB |
|
| 188 |
| Behavior microbatches | 8 batch-2, 0 singleton |
|
| 189 |
|
| 190 |
+
Evaluation combines velocity MSE with fixed-seed stock/teacher/checkpoint image comparisons and owner
|
| 191 |
+
visual review. Standardized perceptual, OCR, and broad safety benchmarking remain future work.
|
|
|
|
| 192 |
|
| 193 |
## Merge and deployment format
|
| 194 |
|
|
|
|
| 200 |
- output tensors: 1,254;
|
| 201 |
- merged projections: 252;
|
| 202 |
- stock per-tensor FP8 scales preserved;
|
| 203 |
+
- tensor inventory remains compatible with the stock text-encoder checkpoint layout.
|
| 204 |
|
| 205 |
The base file is byte-identical to the text encoder published in `Comfy-Org/Qwen3-VL` and
|
| 206 |
`Comfy-Org/Ideogram-4` under the base SHA above.
|
| 207 |
|
| 208 |
## Limitations
|
| 209 |
|
| 210 |
+
- Step 510 is the first public research milestone from a longer training lineage.
|
| 211 |
+
- The learned conditioning targets Ideogram 4's frozen conditional model and Comfy text-encoder ingress
|
| 212 |
+
path; its intended use is image-conditioning inference through that interface.
|
| 213 |
- Training and validation use cached 512×512 trajectories from one fixed sampler contract. Behavior at
|
| 214 |
other resolutions, schedules, samplers, and seeds requires independent evaluation.
|
| 215 |
- Teacher targets can be imperfect or semantically inconsistent despite validation.
|
| 216 |
+
- The corpus deliberately spans adult source-rating groups. Deployment requires an application-level
|
| 217 |
+
safety policy appropriate to the intended use.
|
| 218 |
- Static text conditioning may only approximate a first-step, latent-dependent spatial intervention.
|
| 219 |
+
- FP8 requantization rounds low-magnitude LoRA deltas to the available scaled-FP8 grid; per-projection
|
| 220 |
+
retention metrics are included in the merge manifest.
|
| 221 |
|
| 222 |
## Component and licensing boundary
|
| 223 |
|
| 224 |
+
The release artifacts consist exclusively of modified Qwen3-VL text-encoder parameters. They start
|
| 225 |
+
from the independently published `Comfy-Org/Qwen3-VL` scaled-FP8 checkpoint and are released under its
|
| 226 |
+
Apache-2.0 license.
|
|
|
|
| 227 |
|
| 228 |
+
During training, a separately obtained, frozen Ideogram 4 model served as a differentiable downstream
|
| 229 |
+
evaluator of the conditioning interface. Optimization updated the Qwen adapter parameters, and the
|
| 230 |
+
release files contain the resulting Qwen text-encoder weights.
|
|
|
|
| 231 |
|
| 232 |
+
Ideogram 4 is acquired separately under its own license and usage policy. Users combining the two
|
| 233 |
+
components are responsible for the terms applicable to each. This repository is independently produced
|
| 234 |
+
and has no affiliation with Ideogram, Inc.
|
|
|