README.md

---
license: apache-2.0
base_model: Comfy-Org/Qwen3-VL
pipeline_tag: text-to-image
tags:
  - comfyui
  - ideogram4
  - qwen3-vl
  - text-encoder
  - lora
  - fp8
  - flow-matching
  - experimental
---

# Ideogram 4 Natural-Language Text Encoder — S020 v1

This repository contains an experimental, stock-shaped replacement for the Qwen3-VL 8B text encoder
used by Ideogram 4, plus the matching training LoRA. It is intended to make Ideogram 4 respond to
ordinary natural-language prompts more like the stock model responds to structured Magic-Prompt JSON.
It operates as a direct checkpoint replacement through the stock ComfyUI loader.

This repository publishes the strength-0.2 training lineage at 500-step intervals through optimizer
step 5000. Each milestone contains a merged scaled-FP8 text encoder and its matching rank-64 LoRA. A
milestone resolves to the first complete checkpoint in its 500-step band, so the initial 500 milestone
resolved to **optimizer step 510**. See [`MILESTONES.md`](MILESTONES.md) for the live artifact index.

## Files

| Path | Purpose |
|---|---|
| `text_encoders/*_step_XXXXXXXX_fp8_scaled.safetensors` | Standalone drop-in scaled-FP8 text encoders |
| `loras/*_step_XXXXXXXX_r64_a64.safetensors` | Matching unmerged research LoRAs |
| `manifests/checkpoint_step_XXXXXXXX.json` | Training checkpoint and controller metadata |
| `manifests/merge_step_XXXXXXXX.json` | Merge inputs, output hash, and per-projection quantization metrics |
| `MILESTONES.md` | Human-readable live release index |
| `MILESTONES.json` | Machine-readable live release index |
| `SHA256SUMS` | SHA-256 inventory for every merged encoder and LoRA |
| `COMPATIBILITY.md` | Component boundary, provenance, and third-party compatibility notice |

The LoRA uses this project's native tensor names (`layers.N.<projection>.lora_a` and `lora_b`). The
merged text encoder is the recommended artifact for inference; the LoRA is provided for research,
inspection, and reproducibility.

## Installation

Download the merged file into the normal ComfyUI text-encoder directory:

```text
ComfyUI/models/text_encoders/
```

Select it in the same Ideogram 4 text-encoder loader used for the stock
`qwen3vl_8b_fp8_scaled.safetensors`.

## What was trained

### Task definition

Observed stock behavior motivated the following target:

```text
short natural language       -> weak/broken stock conditioning
detailed natural language    -> weak/broken stock conditioning
structured Magic-Prompt JSON -> functional stock conditioning
```

Each natural-language prompt therefore has a structured teacher target. Because the natural-language
and JSON sequences differ in length, training matches their effect at Ideogram's image-side response:

```text
c_T = stock_text_encoder(teacher_json)
c_S = adapted_text_encoder(natural_language)

v_T = ideogram_conditional(z_t, t, c_T)
v_S = ideogram_conditional(z_t, t, c_S)

loss = mean((v_S - v_T) ** 2)
```

Ideogram parameters remain frozen while gradients pass through the student conditional forward to the
text-encoder adapters. The objective is entirely focused on matching the structured teacher response.

### Stock text-encoder path

The stock Ideogram path uses Qwen3-VL 8B as a contextual feature extractor. Hidden states from 13 taps
(`0, 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 35`) are concatenated to 53,248 features per text token,
then projected to the 4,608-wide Ideogram DiT. The student preserves this architecture and tensor layout.

### Dataset and teacher

- 4,000 source prompts from public Civitai generation metadata.
- Exactly 1,000 prompts from each source rating group: SFW, R, X, and XXX.
- Exact MaxMin diversity selection in Ideogram's own conditioning space, after literal cleanup,
  provenance filtering, and exclusion of edit/image-input records.
- 3,900 training prompts and 100 deterministic owner-reviewed holdouts (25 per rating group).
- Eight cached behavior states per training prompt: 31,200 training examples.
- Teacher JSON generated by `accounts/fireworks/models/nemotron-3-ultra-nvfp4` through Fireworks,
  temperature 0, high reasoning effort, using the project's Magic-Prompt system instructions.
- Teacher conditioning encoded through the exact stock Comfy tokenizer, Qwen taps, and Ideogram ingress
  projection, stored as float16.

This release contains the trained weights, configuration, checkpoint metadata, and merge provenance.

### Eight-state trajectory supervision

Teacher trajectories use the stock `V4_DEFAULT_20` 20-step schedule at 512×512. Eight deterministic
pre-step states are captured at loop indices:

```text
[19, 16, 14, 11, 8, 5, 3, 0]
```

Each cache entry stores the pre-step latent and exact positive conditional teacher velocity. All eight
states are independent velocity-matching examples during training. The unconditional branch is a
separate text-free model and is identical for teacher and student, so conditional velocity matching is
sufficient for the text-encoder objective.

### Strength-0.2 first-step steering target

The teacher trajectory includes one first-conditional-pass spatial correction derived from the
authoritative Ideogram 4 debanner direction:

- correction tensor SHA-256: `5ce873adae5701e9d5f05ebfa8f8b923a1622745c6e9a2bcb3e22fd090ed30c3`;
- Ideogram blocks: 25, 26, 27, and 28;
- distillation strength: `0.2`;
- first denoising step and first positive conditional pass only;
- image tokens only;
- subtract the nearest-neighbor-resized 8×8 spatial direction, then restore each image token's
  original L2 norm;
- the correction is confined to the positive conditional pass.

Only the first cached target velocity is directly corrected. The later seven target velocities are
uncorrected but evaluated on the trajectory produced by the corrected first update. The student always
uses an unpatched stock Ideogram conditional model; the behavior is distilled into the text encoder.

The cache contract fingerprints the correction tensor, metadata, strength, block set, schedule,
resolution, conditional checkpoint, conditioning, and all eight target velocities. Incompatible caches
are rejected before model loading.

## Adapter geometry

This checkpoint uses the following adapter geometry:

- rank: 64;
- alpha: 64;
- blocks: all 36 Qwen text blocks (`0–35`);
- projections per block: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`;
- adapted projections: 252;
- trainable parameters: 174,587,904;
- LoRA tensors: 504.

Per block, the adapter contains 4,849,664 parameters. Adapting all 36 blocks gives every tapped depth a
trainable path for reshaping the conditioning features consumed by Ideogram.

## Optimization and numerics

- Optimizer: AdamW, FP32 adapter parameters and optimizer state.
- Weight decay: `0.01`.
- Physical batch size: 2.
- Gradient accumulation: 8.
- Effective optimizer batch: 16 behavior examples.
- Adaptive packing: batch 2 through 1,017 student tokens; longer examples use singleton microbatches.
- Gradient clipping: global norm `1.0`.
- Optimizer state is held on CPU between updates.
- The training graph contains the Qwen text transformer with all 36 text blocks.
- Initial learning rate: `3.2e-4`.

Training is quantization-aware for the final deployment format. For each adapted projection, the forward
path forms the dequantized stock weight plus the LoRA delta, then requantizes using the stock per-tensor
scale to `float8_e4m3fn`. A straight-through estimator supplies LoRA gradients without constructing a
dense base-weight gradient.

Learning rate is managed by a transactional loss-curve controller. It evaluates fixed 50-step windows
every 10 steps, estimates log-loss descent relative to robust residual noise, confirms a failed window
for 20 additional steps, locates the elbow, restores the complete optimizer/sampler/RNG state at the
rollback boundary, and continues at half the LR. A rollback requires both deterministic held-out loss
and interval-averaged training loss to fail their descent tests. Reaching the positive LR floor allows
training to continue.

## Step-510 checkpoint metrics

| Metric | Value |
|---|---:|
| Optimizer step | 510 |
| Training velocity MSE | 0.0179036569 |
| Held-out velocity MSE | 0.0171197626 |
| Learning rate | `8e-5` |
| LR reductions completed | 2 |
| Gradient norm | 0.0149142 |
| Step time on RTX 5090 | 12.94 s |
| Peak reserved VRAM | 24.463 GiB |
| Behavior microbatches | 8 batch-2, 0 singleton |

Evaluation combines velocity MSE with fixed-seed stock/teacher/checkpoint image comparisons and owner
visual review. Standardized perceptual, OCR, and broad safety benchmarking remain future work.

## Merge and deployment format

The standalone file was merged on CPU into the exact stock Comfy scaled-FP8 checkpoint layout:

- stock base SHA-256: `4ba424cf62e51392e4d1a39933e803706f4e823c1065f36aaf149c6453f66bcd`;
- adapter SHA-256: `85359811e619276f85742072c2b3327915bbb56bcefb874f2f3101a5d1216eef`;
- output SHA-256: `f0d1009af7407b60bda169967bb5e4c3e8ea264042b70f3d7b667bb0c2404ece`;
- output tensors: 1,254;
- merged projections: 252;
- stock per-tensor FP8 scales preserved;
- tensor inventory remains compatible with the stock text-encoder checkpoint layout.

The base file is byte-identical to the text encoder published in `Comfy-Org/Qwen3-VL` and
`Comfy-Org/Ideogram-4` under the base SHA above.

## Limitations

- Step 510 is the first public research milestone from a longer training lineage.
- The learned conditioning targets Ideogram 4's frozen conditional model and Comfy text-encoder ingress
  path; its intended use is image-conditioning inference through that interface.
- Training and validation use cached 512×512 trajectories from one fixed sampler contract. Behavior at
  other resolutions, schedules, samplers, and seeds requires independent evaluation.
- Teacher targets can be imperfect or semantically inconsistent despite validation.
- The corpus deliberately spans adult source-rating groups. Deployment requires an application-level
  safety policy appropriate to the intended use.
- Static text conditioning may only approximate a first-step, latent-dependent spatial intervention.
- FP8 requantization rounds low-magnitude LoRA deltas to the available scaled-FP8 grid; per-projection
  retention metrics are included in the merge manifest.

## Component and licensing boundary

The release artifacts consist exclusively of modified Qwen3-VL text-encoder parameters. They start
from the independently published `Comfy-Org/Qwen3-VL` scaled-FP8 checkpoint and are released under its
Apache-2.0 license.

During training, a separately obtained, frozen Ideogram 4 model served as a differentiable downstream
evaluator of the conditioning interface. Optimization updated the Qwen adapter parameters, and the
release files contain the resulting Qwen text-encoder weights.

Ideogram 4 is acquired separately under its own license and usage policy. Users combining the two
components are responsible for the terms applicable to each. This repository is independently produced
and has no affiliation with Ideogram, Inc.