README.md

---
license: apache-2.0
base_model: Comfy-Org/Qwen3-VL
pipeline_tag: text-to-image
tags:
  - comfyui
  - ideogram4
  - qwen3-vl
  - text-encoder
  - lora
  - fp8
  - flow-matching
  - experimental
---

# Ideogram 4 Natural-Language Text Encoder — S020 v1

This repository contains an experimental, stock-shaped replacement for the Qwen3-VL 8B text encoder
used by Ideogram 4, plus the matching training LoRA. It is intended to make Ideogram 4 respond to
ordinary natural-language prompts more like the stock model responds to structured Magic-Prompt JSON,
without a runtime prompt rewriter, custom node, companion language model, or patched ComfyUI.

This release is the first strength-0.2 milestone. The nominal 500-step milestone resolved to the first
complete checkpoint in that checkpoint band, **optimizer step 510**.

## Files

| File | Purpose | Size | SHA-256 |
|---|---|---:|---|
| `text_encoders/qwen3vl_8b_ideogram4_nl_s020_v1_step_00000510_fp8_scaled.safetensors` | Standalone drop-in scaled-FP8 text encoder | 10,588,637,800 bytes | `f0d1009af7407b60bda169967bb5e4c3e8ea264042b70f3d7b667bb0c2404ece` |
| `loras/qwen3vl_8b_ideogram4_nl_s020_v1_step_00000510_r64_a64.safetensors` | Matching unmerged research LoRA | 698,404,608 bytes | `85359811e619276f85742072c2b3327915bbb56bcefb874f2f3101a5d1216eef` |
| `manifests/checkpoint_step_00000510.json` | Training checkpoint and controller metadata | — | See `SHA256SUMS` |
| `manifests/merge_step_00000510.json` | Merge inputs, output hash, and per-projection quantization metrics | — | See `SHA256SUMS` |
| `COMPATIBILITY.md` | Component boundary, provenance, and third-party compatibility notice | — | See `SHA256SUMS` |

The LoRA uses this project's native tensor names (`layers.N.<projection>.lora_a` and `lora_b`) and is
not advertised as a PEFT or generic ComfyUI LoRA. Most users should use the merged text encoder.

## Installation

Download the merged file into the normal ComfyUI text-encoder directory:

```text
ComfyUI/models/text_encoders/
```

Select it in the same Ideogram 4 text-encoder loader used for the stock
`qwen3vl_8b_fp8_scaled.safetensors`. No inference-time LoRA loader or source-code modification is
required.

## What was trained

### Task definition

Observed stock behavior motivated the following target:

```text
short natural language       -> weak/broken stock conditioning
detailed natural language    -> weak/broken stock conditioning
structured Magic-Prompt JSON -> functional stock conditioning
```

Each natural-language prompt therefore has a structured teacher target. The adapted text encoder is
not trained to reproduce the teacher's token sequence or Qwen activations. Natural-language and JSON
sequences differ in length, so the student is instead optimized at Ideogram's image-side response:

```text
c_T = stock_text_encoder(teacher_json)
c_S = adapted_text_encoder(natural_language)

v_T = ideogram_conditional(z_t, t, c_T)
v_S = ideogram_conditional(z_t, t, c_S)

loss = mean((v_S - v_T) ** 2)
```

Ideogram parameters are frozen, but gradients remain enabled through the student conditional forward
so the velocity loss can update the text-encoder adapters. There is no stock-natural-language retention
term: preserving stock NL behavior would preserve the behavior this project is replacing.

### Stock text-encoder path

The stock Ideogram path uses Qwen3-VL 8B as a contextual feature extractor. Hidden states from 13 taps
(`0, 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 35`) are concatenated to 53,248 features per text token,
then projected to the 4,608-wide Ideogram DiT. The student preserves this architecture and tensor layout.

### Dataset and teacher

- 4,000 source prompts from public Civitai generation metadata.
- Exactly 1,000 prompts from each source rating group: SFW, R, X, and XXX.
- Exact MaxMin diversity selection in Ideogram's own conditioning space, after literal cleanup,
  provenance filtering, and exclusion of edit/image-input records.
- 3,900 training prompts and 100 deterministic owner-reviewed holdouts (25 per rating group).
- Eight cached behavior states per training prompt: 31,200 training examples.
- Teacher JSON generated by `accounts/fireworks/models/nemotron-3-ultra-nvfp4` through Fireworks,
  temperature 0, high reasoning effort, using the project's Magic-Prompt system instructions.
- Teacher conditioning encoded through the exact stock Comfy tokenizer, Qwen taps, and Ideogram ingress
  projection, stored as float16.

The source prompts, teacher JSON, and adult-content corpus are not distributed in this model repository.

### Eight-state trajectory supervision

Teacher trajectories use the stock `V4_DEFAULT_20` 20-step schedule at 512×512. Eight deterministic
pre-step states are captured at loop indices:

```text
[19, 16, 14, 11, 8, 5, 3, 0]
```

Each cache entry stores the pre-step latent and exact positive conditional teacher velocity. All eight
states are independent velocity-matching examples during training. The unconditional branch is a
separate text-free model and is identical for teacher and student, so conditional velocity matching is
sufficient for the text-encoder objective.

### Strength-0.2 first-step steering target

The teacher trajectory includes one first-conditional-pass spatial correction derived from the
authoritative Ideogram 4 debanner direction:

- correction tensor SHA-256: `5ce873adae5701e9d5f05ebfa8f8b923a1622745c6e9a2bcb3e22fd090ed30c3`;
- Ideogram blocks: 25, 26, 27, and 28;
- distillation strength: `0.2`;
- first denoising step and first positive conditional pass only;
- image tokens only;
- subtract the nearest-neighbor-resized 8×8 spatial direction, then restore each image token's
  original L2 norm;
- unconditional model unchanged.

Only the first cached target velocity is directly corrected. The later seven target velocities are
uncorrected but evaluated on the trajectory produced by the corrected first update. The student always
uses an unpatched stock Ideogram conditional model; the behavior is distilled into the text encoder.

The cache contract fingerprints the correction tensor, metadata, strength, block set, schedule,
resolution, conditional checkpoint, conditioning, and all eight target velocities. Incompatible caches
are rejected before model loading.

## Adapter geometry

The geometry is fixed and was not reduced for this checkpoint:

- rank: 64;
- alpha: 64;
- blocks: all 36 Qwen text blocks (`0–35`);
- projections per block: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`;
- adapted projections: 252;
- trainable parameters: 174,587,904;
- LoRA tensors: 504.

Per block, the adapter contains 4,849,664 parameters. The all-block topology is intentional: the task is
a distributed change in how the 13 tapped Qwen layers express conditioning, not memorization in a small
upper-block adapter.

## Optimization and numerics

- Optimizer: AdamW, FP32 adapter parameters and optimizer state.
- Weight decay: `0.01`.
- Physical batch size: 2.
- Gradient accumulation: 8.
- Effective optimizer batch: 16 behavior examples.
- Adaptive packing: batch 2 through 1,017 student tokens; longer examples use singleton microbatches.
- Gradient clipping: global norm `1.0`.
- Optimizer state is held on CPU between updates.
- Unused multimodal Qwen components are pruned from the training graph; all 36 text blocks remain.
- Initial learning rate: `3.2e-4`.

Training is quantization-aware for the final deployment format. For each adapted projection, the forward
path forms the dequantized stock weight plus the LoRA delta, then requantizes using the stock per-tensor
scale to `float8_e4m3fn`. A straight-through estimator supplies LoRA gradients without constructing a
dense base-weight gradient.

Learning rate is not tied to a fixed step horizon. A transactional controller evaluates fixed 50-step
windows every 10 steps, estimates log-loss descent relative to robust residual noise, confirms a failed
window for 20 additional steps, locates the elbow, restores the complete optimizer/sampler/RNG state at
the rollback boundary, and continues at half the LR. A rollback is issued only when both deterministic
held-out loss and interval-averaged training loss fail their descent tests. The LR floor is positive and
does not terminate training.

## Step-510 checkpoint metrics

| Metric | Value |
|---|---:|
| Optimizer step | 510 |
| Training velocity MSE | 0.0179036569 |
| Held-out velocity MSE | 0.0171197626 |
| Learning rate | `8e-5` |
| LR reductions completed | 2 |
| Gradient norm | 0.0149142 |
| Step time on RTX 5090 | 12.94 s |
| Peak reserved VRAM | 24.463 GiB |
| Behavior microbatches | 8 batch-2, 0 singleton |

Velocity MSE is a training/health proxy, not a complete image-quality metric. The release decision was
made from fixed-seed stock/teacher/checkpoint image comparisons and owner visual review. No standardized
perceptual benchmark, OCR benchmark, or broad safety evaluation is claimed for this checkpoint.

## Merge and deployment format

The standalone file was merged on CPU into the exact stock Comfy scaled-FP8 checkpoint layout:

- stock base SHA-256: `4ba424cf62e51392e4d1a39933e803706f4e823c1065f36aaf149c6453f66bcd`;
- adapter SHA-256: `85359811e619276f85742072c2b3327915bbb56bcefb874f2f3101a5d1216eef`;
- output SHA-256: `f0d1009af7407b60bda169967bb5e4c3e8ea264042b70f3d7b667bb0c2404ece`;
- output tensors: 1,254;
- merged projections: 252;
- stock per-tensor FP8 scales preserved;
- no training-only adapter keys added to the merged file.

The base file is byte-identical to the text encoder published in `Comfy-Org/Qwen3-VL` and
`Comfy-Org/Ideogram-4` under the base SHA above.

## Limitations

- This is an early step-510 research checkpoint, not the end of the longer training lineage.
- The learned conditioning is specific to Ideogram 4's frozen conditional model and Comfy text-encoder
  ingress path. It is not a general-purpose Qwen3-VL chat model.
- Training and validation use cached 512×512 trajectories from one fixed sampler contract. Behavior at
  other resolutions, schedules, samplers, and seeds requires independent evaluation.
- Teacher targets can be imperfect or semantically inconsistent despite validation.
- The corpus deliberately spans adult source-rating groups. The model has not been safety-aligned or
  certified for unrestricted deployment.
- Static text conditioning may only approximate a first-step, latent-dependent spatial intervention.
- FP8 requantization does not retain every low-magnitude LoRA delta exactly; per-projection retention
  metrics are included in the merge manifest.

## Component and licensing boundary

The files in this repository contain only modified Qwen3-VL text-encoder parameters. They start from
the independently published `Comfy-Org/Qwen3-VL` scaled-FP8 checkpoint and are released under its
Apache-2.0 license. They contain no Ideogram DiT, VAE, sampler, tokenizer implementation, source code,
or other Ideogram model parameters.

During training, a separately obtained, frozen Ideogram 4 model was used as a differentiable downstream
evaluator of the conditioning interface. Its weights were never trainable, copied into the adapter, or
included in either release file. The released text encoder cannot generate images or substitute for
Ideogram by itself.

Ideogram 4 is third-party software/modeling infrastructure and is not distributed here. Anyone choosing
to use these Qwen weights with Ideogram must obtain Ideogram separately and is independently responsible
for reviewing and complying with Ideogram's license and usage policy. This repository is not affiliated
with or endorsed by Ideogram, Inc.