VER9 / README.md
honghong3's picture
Update README.md
2dbeb1f verified
|
Raw History Blame Contribute Delete
6.35 kB
metadata
license: mit
library_name: diffusers
pipeline_tag: text-to-image
tags:
  - mmdit
  - text-to-image
  - flow-matching
  - dc-ae
  - t5
language:
  - en
  - ko

Diffusion MMDiT Text2img with DC-AE f32c32

Update β€” 2026.11.20 Sampling: ETA injection equation selection & per-step latent β†’ frame save

update

ETA mode

eta_mode = cos | ln

ln: eta * ln(2 / (1 + x)) / ln 2

graph

Code: src.utilities.predict

  • Eta: 0 is deterministic. The larger it is, the more fresh noise is injected at each step.
  • Eta mode: schedule of the injected noise over the steps. Selectable in all three tabs.

Frame save

Collect the latent at every step, decode all of them into images, and save them as an MP4.

Toggle with Frame save in the Sampling panel.

Batch

Generate 1–8 images at once. When Batch is greater than 1, numbered buttons under the result switch between images. Hover over an image to see its seed.


Detailing & Inpainting

Both tabs share the same mask painter with two layers.

Color Mode Meaning
Red μž¬μƒμ„± (Regen) Area to redraw
Blue 보쑴 (Keep) Area that never changes
– μ§€μš°κ°œ (Erase) Remove paint from both layers
  • Painting one color erases the other under the brush. The later stroke wins, so the layers never overlap.
  • If nothing is painted, the whole image (inpainting) or the whole crop (detailer) is redrawn.
  • Keep overrides Regen. The final redraw area is regen Γ— (1 βˆ’ keep).
  • Masks are resized to the sampling resolution with nearest-neighbor, so they stay binary.
  • Start T (0–1): the point in the schedule where redrawing starts. Higher values change more, lower values stay closer to the original. For a full redraw without a mask, around 0.6 is recommended.
  • Brush: brush size in px.

Inpainting

  1. Enter the image path and press Load.
  2. Paint the Regen / Keep areas on the Original panel.
  3. Press Generate. The result appears on the right and is saved to inpainting/.

The image is fitted so that the long side is 1024 and the short side is snapped to a multiple of patch_size Γ— compression_ratio.

Detailer

  1. Enter the image path and press Load.
  2. Drag a rectangle on the Original panel. The rectangle snaps to multiples of patch_size Γ— compression_ratio, and only sizes that scale cleanly to allowable_size on the long side are allowed.
  3. The selected area appears enlarged in Crop Β· Mask. Paint the Regen / Keep areas there. The label under the crop shows the original size β†’ the enlarged size.
  4. Press Generate. The crop is redrawn at the enlarged size, scaled back down, and pasted into the original image. The result is saved to detailer/.

UI

  • Three tabs: Text2img, Inpainting, Detailer. The last tab is remembered, and #txt2img, #inpainting, #detailer in the URL open that tab directly.
  • Tag Browser: browse and search tags by type, and click to add them to the prompt.
  • Tags autocomplete and History: click a history entry to restore the input plus the expanded tags.
  • κ³ μ • (Fixed) switch: when on, only the entered tags are used and no LLM tag expansion happens.
  • Only one job runs at a time. Generation runs in a background thread and the browser polls for the result, so long jobs do not hit proxy timeouts.

Installation

python -m pip install -r requirements.txt

MMDIT

Architecture

  • Autoencoder: DC-AE f32c32 (32Γ— compression, 32 latent channels)
  • Text Encoder: T5-Large
  • Backbone: MMDiT with Joint Attention
  • Implementation: Custom implementation (no pretrained base model)
  • Parameters: 4B

Train Environment

  • Framework: PyTorch (no custom CUDA kernels)
  • GPU: NVIDIA RTX PRO 6000 (96GB VRAM)
  • Platform: Google Colab

Training

Configuration

  • Batch Size: Micro-batch 4 Γ— Gradient Accumulation 8 = Effective Batch 32
  • Training Speed: ~2.3 s / optimizer step
  • Max Resolution: 1024 Γ— 1024
  • Dataset: 2M images
  • Objective: Flow Matching (shift = 3.0)
  • Precision: BF16
  • Optimizer: AdamW
  • Learning Rate: 1e-4
  • EMA: Enabled

Training Stages

Stage Description Steps Patch Size
Stage 1 Pre-training 600K 2
Stage 2 Fine-tuning 11K 1

Stage 2 Notes:

  • Reset patch embedding weights.
  • Reset output projection weights.

Loss by Noise Level

Noise Level Average Loss
1.0 – 0.9 4.0 ~ 4.3
0.9 – 0.8 2.9 ~ 3.1
0.8 – 0.7 2.4 ~ 2.6
0.7 – 0.6 2.0 ~ 2.1
0.6 – 0.5 1.8 ~ 1.9
0.5 – 0.4 1.6 ~ 1.7
0.4 – 0.3 1.3 ~ 1.6
0.3 – 0.2 1.2 ~ 1.4
0.2 – 0.1 1.0 ~ 1.1

Generation Results

Patch Size 2 + DC-AE f32c32

Patch Size 2

  • Issue: Facial features, particularly the eyes, tend to collapse or become distorted.
  • Observation: Limited spatial detail makes accurate reconstruction of fine facial features challenging.

Patch Size 1 + DC-AE f32c32

Patch Size 1

  • Fine-tuning Result: Fine-tuning with patch size 1 improved facial feature reconstruction.
  • Improvement: Eye and facial distortions were partially resolved, with better preservation of fine details.

Current Limitations

  • Full-body Generation: Full-body generation remains challenging due to insufficient full-body samples in the training dataset.
  • Observation: The current dataset contains insufficient full-body examples to learn consistent full-body composition and anatomical details.
  • Future Work: Increase the proportion and diversity of full-body images in the training dataset.

Train Architecture

  • Use Cache
  • Cpu Multi Threading

inference

Inference Architecture

inference

UI

  • Sampling

sampling

  • Inpainting

inpainting

  • Result

result