MOSS-Image-SFT: quantized GGUF (study results)

GGUF quantizations of OpenMOSS-Team/MOSS-Image-SFT (revision f972bd4): the 5.08 B DiT and its 9 B text encoder (stock Qwen3.5-9B). They come from a study that asked one question: what is the smallest model whose images stay as close as possible to the released model's, for the same prompt and seed?

The repo holds all 37 recipes the study produced, the ones that passed and the ones that didn't, with the measured result for each. Most people want one of the two picks:

pick DiT text encoder total what it keeps
Faithful dit/moss_image_sft-dit-w8-gptq.gguf (5.40 GB) text_encoder/moss_image_sft-te-w8-gptq.gguf (8.45 GB) 13.8 GB (53% of 26 GB) Same pictures as the released model. Closer to FP32 than bf16 inference itself is (0.43x its drift).
Equivalent quality dit/moss_image_sft-dit-w8-gptq.gguf (5.40 GB) text_encoder/moss_image_sft-te-cal-mix6.gguf (5.97 GB) 11.4 GB (44%) Same quality and text rendering; a few images on "fork" prompts change composition.

No DiT below 8 bits is faithful. That includes our own Q6_K versions (imatrix and GPTQ) and the community Q6_K and Q4_K_M files. One passes the quality bar: the community Q6_K DiT (molbal's, plain rounding) with our W8 text encoder, 13.0 GB. That is larger than the quality pick. The text encoder can go lower than the DiT.

Same prompt and seed at 1024², 50 steps

1024², 50 steps, seed 0, three of the held-out prompts. Columns: FP32 compute on the released weights (the reference), the released model in bf16 compute, and the two picks. LPIPS to the reference, mean over these three: bf16 0.044, faithful pick 0.013 (0.30x), quality pick 0.031 (0.70x). Every quoted string is rendered in all four.

Where smaller files break

Failure cases at 512², 20 steps

512², 20 steps, held-out prompts, seeds as labelled. LPIPS to the FP32 reference under each image. Below 8 bits in the DiT, or below the cal-mix6 text encoder, the first thing to go is small text. On the calligraphy prompt the red seal should read "墨香"; our Q6_K (imatrix) DiT, the community Q4_K_M DiT and the 9.3 GB pair all turn it into an unreadable stamp, although the rest of the image barely changes (row 2). Q4_K_M also writes the traditional "書" for "书" and redraws whole scenes (rows 4-5). The two-person prompt (row 3) sits near a fork: bf16 itself drifts (0.15). There the quality pick (11.4 GB) draws two children instead of a girl and a man. That is why it is a quality pick and not a faithful one. The faithful pick stays at 0.01-0.03 on every row.

GB means 10⁹ bytes here. The VAE (FLUX.2 small decoder, 0.25 GB) is not included; use the original one.

Using the files

  • ComfyUI with ComfyUI-GGUF: load the DiT with Unet Loader (GGUF) and the text encoder with the GGUF CLIP loader (Qwen3.5 / qwen35 support). The DiT files use the same tensor layout as existing MOSS-Image GGUFs (e.g. molbal's). Files where q/k/v share one type (uniform and rule recipes) have fused qkv; the mixes keep them separate because they differ in type. The text-encoder files use llama.cpp's qwen35 layout with separate attn_qkv / attn_gate / ssm_alpha / ssm_beta, no LM head, and Qwen3.5-9B's tokenizer and metadata.
  • Tensor types are standard ggml types only: Q3_K, Q4_0, Q4_K, Q5_K, Q6_K, Q8_0, BF16, F32. No custom kernels are needed. GPTQ and imatrix change which codes are stored, not the format.
  • Low-VRAM GPUs: run the text encoder once per prompt, then the DiT, then the VAE. On an 8 GB card the faithful DiT (5.40 GB) fits. The 8.45 GB text encoder does not, so run it on the CPU or offload it.

What was and wasn't checked

  • Every file was read back after writing and compared with the quantized weights the study measured: exact for every tensor. Two small exceptions, both listed in the per-file reports: scales of plain round-to-nearest DiT layers are stored as fp16 (relative error ≤ 5.2e-8). In text encoders with K-quant out_proj (ssm_out) tensors, those are re-quantized, because llama.cpp's head order shifts the 256-wide blocks. Their codes differ from the measured ones, at the same error level (e.g. 1.41e-3 vs 1.41e-3 relative). This affects 3-23 tensors per file, including 12 in the quality pick's te-cal-mix6. The faithful pick's te-w8-gptq has none.
  • DiT end to end: images made from the exported dit-w8-gptq and dit-cal-mix6 files are pixel-identical to the ones measured.
  • Quality pick's text encoder end to end: the text encoder was loaded from te-cal-mix6.gguf itself (237 tensors bit-identical to the measured weights, the 12 re-quantized ones taken from the file). Its embeddings are as close to FP32 as the measured ones (hidden-state error 3.326e-3 vs 3.328e-3). They differ from them by 1.1e-4 relative, less than bf16 compute adds. Eight images (seed 0 on all six prompts, plus the two closest calls) stay within 0.001-0.036 LPIPS of the measured ones. All quoted strings are kept, and the "墨香" seal still reads. The composition change on the two-person scene (seed 1) is still there: 0.298 to FP32, measured 0.301, limit 0.301. That is the same picture either side of the line, so the pick's verdicts stand. The other text-encoder files were checked by read-back only.
  • Not checked: running these files inside ComfyUI, or llama.cpp itself. All measurements used a PyTorch harness that dequantizes each weight and computes in FP32. ComfyUI-GGUF also dequantizes on the fly, but computes in bf16/fp16, so its images will differ slightly, as the released model's bf16 images differ from FP32.

How "close" was measured

  • Reference: FP32 compute on the released bf16 weights. This is also exactly how the model runs on GPUs without bf16 (e.g. Pascal). Yardstick: the released model in bf16 compute, i.e. what most people run. Its own drift from FP32 sets the bar.
  • Images: 6 held-out prompts (photo, English sign text, Chinese calligraphy with a seal, watercolour, a JSON layout prompt, a two-person scene) x seeds 0-3 = 24 paired images per variant, 512², 20 steps. The picks were confirmed at 1024², 50 steps. The prompts were written after the recipes were fixed and were never used to tune them.
  • Faithful (set before results): mean LPIPS to the reference ≤ 1.5x bf16's; each image ≤ max(2x bf16's on that image, 0.1); and OCR still finds every quoted string the reference renders. As a scale: the same prompt with another seed is 0.36-0.66 LPIPS; a changed composition ("flip") lands at 0.25-0.33.
  • Equivalent quality (reference-free, as quantization papers report it): ImageReward delta to the reference, lower end of its 95% CI ≥ -0.05, no image below -0.3, and all quoted strings kept (OCR, every miss checked by eye; 3 of 10 were false alarms). CLIPScore was measured too; it separated nothing here.
  • Screening (files marked "screened only" below): the error each variant adds to the DiT's predicted velocity (DiTs), or to the text encoder's hidden states (TEs), relative to what bf16 compute adds. That is a filter, not a verdict: it predicted the clear failures but not which images flip.

Caveats: 24 images on 6 prompts is a small sample, and the per-image limit is fragile on prompts where the model sits near a fork (bf16 itself changes one of the 24 compositions). The rule was never loosened after results came in.

All files

"x bf16" is LPIPS to the reference relative to bf16's (0.048); flips = images with a changed composition out of 24. IR = ImageReward delta, mean ± 95% CI and worst image.

DiT (dit/)

file GB types (linear layers) result
dit-w8-gptq 5.40 Q8_0 (GPTQ codes) Faithful pass with te-w8-gptq: 0.43x bf16, 0 flips, IR -0.007 ± 0.014 (worst -0.11). Community Q8_0 (plain rounding) for comparison: 0.48x, also passes.
dit-rule-w5-w8 4.18 Q5_K body, Q8_0 sensitive layers, small embedders BF16 Screened only: 2.1x bf16's velocity error.
dit-q6k-ik 4.17 Q6_K (imatrix) Fail: with te-w8-gptq 1.58x, 3 flips; the "墨香" seal lost on one image. IR -0.005 ± 0.024. Community Q6_K (plain rounding) for comparison: 1.86x, fails the faithful rule but keeps the seal and passes the quality bar with te-w8-gptq (13.0 GB).
dit-q6k-gptq 4.17 Q6_K (GPTQ codes) Fail: 1.64x, IR worst -0.56. With te-cal-mix6: 1.72x, seal garbled.
dit-mix6.25 3.97 mix Q4_K-Q8_0 Fail: with te-w8-gptq 2.43x, 5 flips, seal lost.
dit-cal-mix6 3.81 mix Q4_0-Q8_0 (calibrated) Fail: with te-cal-mix5.5 (9.3 GB) 2.06x, 4 flips, seal garbled; IR within noise (-0.009 ± 0.037).
dit-mix6 3.81 mix Q4_K-Q8_0 Screened only: 1.2x bf16's velocity error.
dit-mix5.75 3.65 mix Q4_K-Q8_0 Screened only: 1.6x.
dit-rule-w4-w6 3.50 Q4_K body, Q6_K sensitive layers, small embedders BF16 Screened only: 7.2x.
dit-cal-mix5.5 3.50 mix (calibrated) Fail: redraws the sign prompt (seed 0, LPIPS 0.23).
dit-mix5.5 3.49 mix Screened only: 2.0x.
dit-cal-mix5 3.18 mix (calibrated) Fail: redraws the sign prompt (0.36).
dit-mix5 3.18 mix Screened only: 3.5x.
dit-cal-mix4.5 2.86 mix Q3_K-Q8_0, mostly Q4_0 (calibrated) Fail: redraws the sign prompt (0.33).

Text encoder (text_encoder/)

All results below are with the W8 DiT unless noted. The token table counts in the size.

file GB types result
te-mix11 10.92 Q8_0 / BF16 Screened only: 0.45x bf16's hidden-state error.
te-mix10 9.93 Q8_0 / BF16 Screened only: 0.57x.
te-rule-w8-bf16 9.53 Q8_0, sensitive layers BF16 Screened only: 0.78x.
te-mix9 8.94 Q8_0 / BF16 Screened only: 0.73x.
te-mix8.75 8.69 Q8_0 / BF16 / Q6_K Screened only: 1.7x.
te-w8-gptq 8.45 Q8_0 (GPTQ codes), 6 tiny tensors BF16 Faithful pass (see the DiT pick). Hidden-state error below bf16's.
te-uniform-q8 8.45 Q8_0 (plain rounding) Images on 2 prompts only: same pictures. Use te-w8-gptq instead: same size, ~4x lower hidden-state error.
te-mix8.5 8.45 Q8_0 / BF16 / Q6_K Screened only: 1.6x.
te-mix8.25 8.20 Q8_0 / BF16 / Q6_K Screened only: 1.8x.
te-rule-w6-bf16 8.13 Q6_K, sensitive layers BF16 Screened only: 9.3x.
te-mix8 7.95 Q8_0 / Q6_K / BF16 Screened only: 2.6x.
te-cal-mix7 6.96 Q8_0 / Q5_K (calibrated) Quality pass, faithful miss by one image (1.18x, 2 flips; one image at 0.312 vs its limit 0.301). IR +0.010 ± 0.017.
te-mix7 6.96 Q8_0 / Q6_K / BF16 / Q5_K Screened only: 5.9x.
te-cal-mix6.5 6.46 Q8_0 / Q5_K (calibrated) Screened only (through the DiT: between cal-mix6 and cal-mix7).
te-rule-w5-w8 6.28 Q5_K, sensitive layers Q8_0 Screened only: 41x.
te-cal-mix6 5.97 Q5_K / Q8_0 (calibrated) Quality pass, the smallest: 1.30x, 2 flips; misses the faithful rule on two images of the two-person scene. IR +0.006 ± 0.021. Same pictures at 1024².
te-mix6 5.97 Q5_K / Q6_K / Q8_0 Screened only: 23x; failed the through-DiT screen.
te-cal-mix5.5 5.47 Q5_K / Q8_0 / Q4_0 (calibrated) Fail: 1.78x, the "墨香" seal garbled, IR worst -0.43.
te-cal-mix5 4.97 Q4_0 / Q5_K / Q8_0 (calibrated) Fail: with dit-cal-mix6, one image over its limit (seed 0, pose changed).
te-mix5 4.97 Q5_K / Q4_K / ... Screened only: 83x.
te-e8w4 4.99 Q4_0 (GPTQ codes), token table Q8_0 Fail: clean images, but different pictures on every prompt tried (LPIPS 0.33-0.43).
te-cal-mix4.5 4.48 Q4_0 / Q5_K / Q8_0 (calibrated) Fail: 2.42x, 6 flips, seal garbled (IR still +0.041: a smaller TE changes the picture, not its quality).
te-cal-mix4 3.98 Q4_0 / Q3_K / Q5_K / Q8_0 (calibrated) Fail: with dit-cal-mix6, redraws the notebook prompt (seed 0).

Recipe names

  • w8-gptq: every linear W8 GPTQ, group 32 (= Q8_0 layout).
  • mixN: a per-tensor mix from a damage scan, average N bits per weight.
  • cal-mixN: the same, with GPTQ / imatrix-calibrated levels (48 calibration prompts, disjoint from the test prompts).
  • rule-A-B: a fixed rule (most layers A, the sensitive ones B).
  • q6k-ik / q6k-gptq: uniform Q6_K with imatrix or GPTQ codes. e8w4: W4 GPTQ body, 8-bit token table.

Each file's exact per-tensor types are in its GGUF header.

License

Apache-2.0, as the source model (MOSS-Image-SFT; its text encoder is Qwen3.5-9B, also Apache-2.0). These are derived weights. All credit for the model goes to the OpenMOSS team and the Qwen team.

Downloads last month
5,078
GGUF
Model size
5B params
Architecture
moss_image
Hardware compatibility
Log In to add your hardware

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for nynefold/MOSS-Image-SFT-quant-GGUF

Quantized
(58)
this model

Collection including nynefold/MOSS-Image-SFT-quant-GGUF