AcademiaSD/Ming-Image-NF4-for-LoRA-Training

Ming-Image 0.1 Design NF4 for LoRA Training

Not an official inclusionAI release. This repository holds an NF4 conversion of Ming-Image-0.1-Design made so that the Ming-Image trainer of AcademiaSD LoRAlab Trainer Studio can download it and train LoRAs on consumer NVIDIA GPUs. The prompt encoder and the diffusion transformer are stored in pre-quantized formats used by the trainer; diffusers and ComfyUI do not load them directly. For inference, use inclusionAI's original weights. All credit for the model goes to the inclusionAI team: this repository only changes how the weights are stored and packaged.

You do not need to download this repository by hand: the trainer downloads it on first use.

Contents

Folder	Content	Size
mllm/	BailingMM2 MoE prompt encoder (the "thinker") in pre-quantized NF4 (bitsandbytes, double quantization), with its vision tower and tokenizer. Used only while pre-caching the conditions, never while training	9.35 GB
connector/	6-layer connector that refines the 256 learned query tokens, stored in BF16 (converted from the original FP32)	3.09 GB
mlp/	proj_in, proj_out, proj_directvlm and the 16×16 query_tokens_dict, in FP32, unchanged	0.13 GB
transformer/	6B diffusion transformer: the 238 attention and MLP linear layers of the 30 blocks in NF4 (bitsandbytes, pre-quantized); embedders, adaLN modulation, cap embedder, final layer and norms in BF16	3.38 GB
vae/	Original 16-channel RGBA autoencoder (AutoencoderKLQwenImage), unchanged	0.25 GB
scheduler/, config.json	Original FlowMatchEuler scheduler and BailingMM2 model config, unchanged	<0.01 GB
Total		16.2 GB (original: 52.9 GB)

Format

• mllm/ is a regular transformers checkpoint saved with its quantization_config (bitsandbytes NF4). It loads on GPU or CPU. On GPU, the pre-quantized load produces prompt embeddings that are bit-identical to the BF16 original.
• transformer/ is not a standard diffusers checkpoint. It is a pre-quantized cache: one .safetensors file per layer in weights/ (NF4 weight plus its packed bitsandbytes QuantState, or BF16 for the excluded layers), others.safetensors with the tensors kept in BF16, index.json, and metadata.json listing what was quantized. The trainer rebuilds the DiT from config.json and fills it from this cache, so the BF16 transformer is never downloaded or loaded. 238 layers are NF4 and 38 stay in BF16 (all_x_embedder, all_final_layer, t_embedder, cap_embedder and adaLN_modulation).
• connector/, mlp/, vae/, scheduler/ and the root config.json are the original modules. Only the connector was cast to BF16 (it was FP32) and the transformer/MLLM were quantized; the VAE, scheduler, tokenizer and the small projection modules are unchanged.
• The converter that produced this package is scripts/convert_nf4_mingimage.py in the Trainer Studio repository.

NF4 is meant for training. Measured against the BF16 transformer on the same inputs, the cosine similarity of the predicted velocity is 0.974; the NF4 DiT loads in about 3.4 GB of VRAM. The MLLM peaks at about 11.6 GB while pre-caching. For inference, use inclusionAI's original weights.

What the trainer trains with it

• Ming-Image-0.1-Design is a text-to-image model for text-rich visual design: posters, UI, infographics and other graphic compositions, with RGBA output and transparent backgrounds.
• One image + caption per sample. The official caption format is a Figma-style JSON with canvas_settings and layers; the trigger word goes inside canvas_settings.image_style (for example "TOSTIOK, high-resolution digital illustration, ..."). The trainer generates those captions with the shared Qwen3-VL-8B auto-captioner, which is not part of this repository.
• The model is trained at 1024×1024 or 2048×2048, the resolution buckets it was trained for. 512 and 768 are out of distribution.
• Design (text-to-image) only in this release. Layer decomposition / edit LoRAs are not supported yet.
• LoRA only: there is no turbo or distill adapter for Ming-Image. The MLLM, connector and projection modules are frozen; only the DiT receives LoRA weights, and training uses the latents and conditions produced during pre-caching.

LoRAs trained with it

The trainer exports LoRAs in PEFT/diffusers format with the transformer. prefix, lora_A / lora_B weights and a per-layer alpha, and also supports LoKr. They load with the trainer and with loaders that support the Ming-Image DiT.

Measured on an RTX 5080, rank 8 at 768×768: about 3.5 s per step and 3.4 GB of VRAM for the NF4 DiT.

License

MIT, the same as the original inclusionAI/Ming-Image-0.1-Design (see LICENSE). Copyright (c) 2026 inclusionAI. This repository is provided "as is", without warranties of any kind.

Links

• Trainer: AcademiaSD LoRAlab Trainer Studio
• Original model: inclusionAI/Ming-Image-0.1-Design · GitHub
• ▶ YouTube: youtube.com/@Academia_SD
• 💬 Discord: discord.gg/Syuaduy678
• ☕ Ko-Fi: ko-fi.com/academiasd

---

Resumen en español

Conversión a NF4 de Ming-Image-0.1-Design para el entrenador de Ming-Image del LoRAlab de AcademiaSD, que la descarga solo. Incluye el codificador de prompts (MLLM) en NF4 (9,35 GB), el conector en BF16, el DiT de 6B en NF4 (3,38 GB) y el VAE RGBA original. Total: 16,2 GB frente a 52,9 GB del modelo original. Sirve para entrenar LoRAs de diseño texto-a-imagen (pósters, UI, infografías) a 1024² o 2048², con captions JSON al estilo oficial y la palabra de activación dentro de canvas_settings.image_style. Solo LoRA, sin adaptador turbo. Licencia MIT, la misma del modelo original; todo el mérito es del equipo de inclusionAI.