Hugging Face's logo

mrjackspade
/
Ideogram4-Natural-Language-Text-Encoder

Text-to-Image
comfyui
ideogram4
qwen3-vl
text-encoder
lora
fp8
flow-matching
experimental
Model card Files
xet
 Community
Ideogram4-Natural-Language-Text-Encoder / README.md
mrjackspade's picture
mrjackspade
Add technical model card and step-510 release metadata
a2213cd verified
preview code
|
Raw
History Blame
11.7 kB
metadata 
license: apache-2.0
base_model: Comfy-Org/Qwen3-VL
pipeline_tag: text-to-image
tags:
  - comfyui
  - ideogram4
  - qwen3-vl
  - text-encoder
  - lora
  - fp8
  - flow-matching
  - experimental

Ideogram 4 Natural-Language Text Encoder — S020 v1

This repository contains an experimental, stock-shaped replacement for the Qwen3-VL 8B text encoder used by Ideogram 4, plus the matching training LoRA. It is intended to make Ideogram 4 respond to ordinary natural-language prompts more like the stock model responds to structured Magic-Prompt JSON, without a runtime prompt rewriter, custom node, companion language model, or patched ComfyUI.

This release is the first strength-0.2 milestone. The nominal 500-step milestone resolved to the first complete checkpoint in that checkpoint band, optimizer step 510.

Files

File Purpose Size SHA-256
text_encoders/qwen3vl_8b_ideogram4_nl_s020_v1_step_00000510_fp8_scaled.safetensors Standalone drop-in scaled-FP8 text encoder 10,588,637,800 bytes f0d1009af7407b60bda169967bb5e4c3e8ea264042b70f3d7b667bb0c2404ece
loras/qwen3vl_8b_ideogram4_nl_s020_v1_step_00000510_r64_a64.safetensors Matching unmerged research LoRA 698,404,608 bytes 85359811e619276f85742072c2b3327915bbb56bcefb874f2f3101a5d1216eef
manifests/checkpoint_step_00000510.json Training checkpoint and controller metadata — See SHA256SUMS
manifests/merge_step_00000510.json Merge inputs, output hash, and per-projection quantization metrics — See SHA256SUMS
COMPATIBILITY.md Component boundary, provenance, and third-party compatibility notice — See SHA256SUMS

The LoRA uses this project's native tensor names (layers.N.<projection>.lora_a and lora_b) and is not advertised as a PEFT or generic ComfyUI LoRA. Most users should use the merged text encoder.

Installation

Download the merged file into the normal ComfyUI text-encoder directory:

ComfyUI/models/text_encoders/

Select it in the same Ideogram 4 text-encoder loader used for the stock qwen3vl_8b_fp8_scaled.safetensors. No inference-time LoRA loader or source-code modification is required.

What was trained

Task definition

Observed stock behavior motivated the following target:

short natural language       -> weak/broken stock conditioning
detailed natural language    -> weak/broken stock conditioning
structured Magic-Prompt JSON -> functional stock conditioning

Each natural-language prompt therefore has a structured teacher target. The adapted text encoder is not trained to reproduce the teacher's token sequence or Qwen activations. Natural-language and JSON sequences differ in length, so the student is instead optimized at Ideogram's image-side response:

c_T = stock_text_encoder(teacher_json)
c_S = adapted_text_encoder(natural_language)

v_T = ideogram_conditional(z_t, t, c_T)
v_S = ideogram_conditional(z_t, t, c_S)

loss = mean((v_S - v_T) ** 2)

Ideogram parameters are frozen, but gradients remain enabled through the student conditional forward so the velocity loss can update the text-encoder adapters. There is no stock-natural-language retention term: preserving stock NL behavior would preserve the behavior this project is replacing.

Stock text-encoder path

The stock Ideogram path uses Qwen3-VL 8B as a contextual feature extractor. Hidden states from 13 taps (0, 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 35) are concatenated to 53,248 features per text token, then projected to the 4,608-wide Ideogram DiT. The student preserves this architecture and tensor layout.

Dataset and teacher

  • 4,000 source prompts from public Civitai generation metadata.
  • Exactly 1,000 prompts from each source rating group: SFW, R, X, and XXX.
  • Exact MaxMin diversity selection in Ideogram's own conditioning space, after literal cleanup, provenance filtering, and exclusion of edit/image-input records.
  • 3,900 training prompts and 100 deterministic owner-reviewed holdouts (25 per rating group).
  • Eight cached behavior states per training prompt: 31,200 training examples.
  • Teacher JSON generated by accounts/fireworks/models/nemotron-3-ultra-nvfp4 through Fireworks, temperature 0, high reasoning effort, using the project's Magic-Prompt system instructions.
  • Teacher conditioning encoded through the exact stock Comfy tokenizer, Qwen taps, and Ideogram ingress projection, stored as float16.

The source prompts, teacher JSON, and adult-content corpus are not distributed in this model repository.

Eight-state trajectory supervision

Teacher trajectories use the stock V4_DEFAULT_20 20-step schedule at 512×512. Eight deterministic pre-step states are captured at loop indices:

[19, 16, 14, 11, 8, 5, 3, 0]

Each cache entry stores the pre-step latent and exact positive conditional teacher velocity. All eight states are independent velocity-matching examples during training. The unconditional branch is a separate text-free model and is identical for teacher and student, so conditional velocity matching is sufficient for the text-encoder objective.

Strength-0.2 first-step steering target

The teacher trajectory includes one first-conditional-pass spatial correction derived from the authoritative Ideogram 4 debanner direction:

  • correction tensor SHA-256: 5ce873adae5701e9d5f05ebfa8f8b923a1622745c6e9a2bcb3e22fd090ed30c3;
  • Ideogram blocks: 25, 26, 27, and 28;
  • distillation strength: 0.2;
  • first denoising step and first positive conditional pass only;
  • image tokens only;
  • subtract the nearest-neighbor-resized 8×8 spatial direction, then restore each image token's original L2 norm;
  • unconditional model unchanged.

Only the first cached target velocity is directly corrected. The later seven target velocities are uncorrected but evaluated on the trajectory produced by the corrected first update. The student always uses an unpatched stock Ideogram conditional model; the behavior is distilled into the text encoder.

The cache contract fingerprints the correction tensor, metadata, strength, block set, schedule, resolution, conditional checkpoint, conditioning, and all eight target velocities. Incompatible caches are rejected before model loading.

Adapter geometry

The geometry is fixed and was not reduced for this checkpoint:

  • rank: 64;
  • alpha: 64;
  • blocks: all 36 Qwen text blocks (0–35);
  • projections per block: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj;
  • adapted projections: 252;
  • trainable parameters: 174,587,904;
  • LoRA tensors: 504.

Per block, the adapter contains 4,849,664 parameters. The all-block topology is intentional: the task is a distributed change in how the 13 tapped Qwen layers express conditioning, not memorization in a small upper-block adapter.

Optimization and numerics

  • Optimizer: AdamW, FP32 adapter parameters and optimizer state.
  • Weight decay: 0.01.
  • Physical batch size: 2.
  • Gradient accumulation: 8.
  • Effective optimizer batch: 16 behavior examples.
  • Adaptive packing: batch 2 through 1,017 student tokens; longer examples use singleton microbatches.
  • Gradient clipping: global norm 1.0.
  • Optimizer state is held on CPU between updates.
  • Unused multimodal Qwen components are pruned from the training graph; all 36 text blocks remain.
  • Initial learning rate: 3.2e-4.

Training is quantization-aware for the final deployment format. For each adapted projection, the forward path forms the dequantized stock weight plus the LoRA delta, then requantizes using the stock per-tensor scale to float8_e4m3fn. A straight-through estimator supplies LoRA gradients without constructing a dense base-weight gradient.

Learning rate is not tied to a fixed step horizon. A transactional controller evaluates fixed 50-step windows every 10 steps, estimates log-loss descent relative to robust residual noise, confirms a failed window for 20 additional steps, locates the elbow, restores the complete optimizer/sampler/RNG state at the rollback boundary, and continues at half the LR. A rollback is issued only when both deterministic held-out loss and interval-averaged training loss fail their descent tests. The LR floor is positive and does not terminate training.

Step-510 checkpoint metrics

Metric Value
Optimizer step 510
Training velocity MSE 0.0179036569
Held-out velocity MSE 0.0171197626
Learning rate 8e-5
LR reductions completed 2
Gradient norm 0.0149142
Step time on RTX 5090 12.94 s
Peak reserved VRAM 24.463 GiB
Behavior microbatches 8 batch-2, 0 singleton

Velocity MSE is a training/health proxy, not a complete image-quality metric. The release decision was made from fixed-seed stock/teacher/checkpoint image comparisons and owner visual review. No standardized perceptual benchmark, OCR benchmark, or broad safety evaluation is claimed for this checkpoint.

Merge and deployment format

The standalone file was merged on CPU into the exact stock Comfy scaled-FP8 checkpoint layout:

  • stock base SHA-256: 4ba424cf62e51392e4d1a39933e803706f4e823c1065f36aaf149c6453f66bcd;
  • adapter SHA-256: 85359811e619276f85742072c2b3327915bbb56bcefb874f2f3101a5d1216eef;
  • output SHA-256: f0d1009af7407b60bda169967bb5e4c3e8ea264042b70f3d7b667bb0c2404ece;
  • output tensors: 1,254;
  • merged projections: 252;
  • stock per-tensor FP8 scales preserved;
  • no training-only adapter keys added to the merged file.

The base file is byte-identical to the text encoder published in Comfy-Org/Qwen3-VL and Comfy-Org/Ideogram-4 under the base SHA above.

Limitations

  • This is an early step-510 research checkpoint, not the end of the longer training lineage.
  • The learned conditioning is specific to Ideogram 4's frozen conditional model and Comfy text-encoder ingress path. It is not a general-purpose Qwen3-VL chat model.
  • Training and validation use cached 512×512 trajectories from one fixed sampler contract. Behavior at other resolutions, schedules, samplers, and seeds requires independent evaluation.
  • Teacher targets can be imperfect or semantically inconsistent despite validation.
  • The corpus deliberately spans adult source-rating groups. The model has not been safety-aligned or certified for unrestricted deployment.
  • Static text conditioning may only approximate a first-step, latent-dependent spatial intervention.
  • FP8 requantization does not retain every low-magnitude LoRA delta exactly; per-projection retention metrics are included in the merge manifest.

Component and licensing boundary

The files in this repository contain only modified Qwen3-VL text-encoder parameters. They start from the independently published Comfy-Org/Qwen3-VL scaled-FP8 checkpoint and are released under its Apache-2.0 license. They contain no Ideogram DiT, VAE, sampler, tokenizer implementation, source code, or other Ideogram model parameters.

During training, a separately obtained, frozen Ideogram 4 model was used as a differentiable downstream evaluator of the conditioning interface. Its weights were never trainable, copied into the adapter, or included in either release file. The released text encoder cannot generate images or substitute for Ideogram by itself.

Ideogram 4 is third-party software/modeling infrastructure and is not distributed here. Anyone choosing to use these Qwen weights with Ideogram must obtain Ideogram separately and is independently responsible for reviewing and complying with Ideogram's license and usage policy. This repository is not affiliated with or endorsed by Ideogram, Inc.