PotatoForge/moss-image-4b-encoder-adapter

MOSS-Image Smaller Encoder — 4B to 9B Adapter

An experimental, lightweight text-encoder replacement for MOSS-Image. This adapter maps features from Qwen3.5-4B into the conditioning space of MOSS-Image's original Qwen3.5-9B text encoder. The goal is to reduce memory requirements for prompt encoding while retaining useful image-generation quality.

GitHub — ComfyUI node, source code, and installation instructions: bakapotatolord/moss-image-smaller-encoder

Current adapter: fitted on 1,600 clean prompts, with 400 separate validation prompts and 400 held-out test prompts. This is a feature-projection adapter, not a LoRA or a standalone text-to-image model. It requires MOSS-Image, a compatible Qwen3.5-4B encoder, and the adapter node. Equivalent quality to the original 9B encoder is not guaranteed.

How it works

The 4B and 9B encoders remain frozen. The same prompts are encoded using MOSS's prompt template, and token-aligned, final normalized hidden states are collected from both encoders.

A standardized affine projection is fitted using token-weighted ridge regression to approximate the 9B features from the 4B features. Neither encoder nor the diffusion model is fine-tuned.

Component	Details
Student encoder	Qwen3.5-4B
Target encoder	MOSS-Image's original Qwen3.5-9B
Tested encoder checkpoints	INT8 ConvRot versions of both encoders
Projection	2,560 → 4,096 dimensions
Projection parameters	Approximately 10.5 million
Adapter size	Approximately 21 MB / 20 MiB
Stored tensors	FP16 projection matrix and FP32 normalization statistics
Fitting method	Standardized, token-weighted ridge regression
ComfyUI projection	CPU FP32 computation with FP16 output rounding

Only the projection adapter is distributed here. The Qwen encoder and MOSS-Image model weights must be downloaded separately.

Current results — 1,600 fitting prompts

The current adapter uses a 2,400-prompt dataset, divided into:

Split	SFW prompts	Adult/NSFW prompts	Total
Training	800	800	1,600
Validation	200	200	400
Held-out test	200	200	400

Only the 1,600 training prompts were used to fit the projection. Validation selected the ridge regularization strength, and the separate test set measured feature reconstruction. Embedded LoRA and hypernetwork invocation tags were removed before feature collection. The test corpus has been examined in earlier experiments.

The mixed corpus includes prompts from jtatman/stable-diffusion-prompts-stats-full-uncensored. SFW/adult categories are based on source labels and text screening; they are not independently verified semantic annotations. Equal prompt counts also do not imply equal token contributions to the fit.

Feature reconstruction

On the 400-prompt held-out test set:

Metric	Result
Content-token relative L2 error ↓	0.4042
Content-token cosine similarity ↑	0.9126

On these same 400 test prompts, the clean 800-prompt ridge scored 0.4132 relative L2 and 0.9086 cosine similarity. Expanding to 1,600 fitting prompts reduces relative L2 error by 2.17%, with improvements in both SFW and adult-source groups. Adapter size and projection architecture are unchanged.

These metrics measure how closely projected 4B features approximate the original 9B features. They are not image-quality scores, and the modest numerical improvement does not guarantee visibly better images on every prompt.

Image comparisons

The single grid below contains six image pairs, with original 9B on the left and projected 4B using the 1,600-prompt ridge adapter on the right. All displayed projected-4B images use the new adapter.

• Portrait and watercolor boats: two held-out SFW prompts, each shown at seeds 2026101801 and 2026101802. The projected-4B images were freshly rendered; the original 9B references were reused after checking prompt, seed, noise hash, model/LoRA/VAE identity, runtime and image hashes.
• Artichoke railway and glassblower: two newly written SFW prompts, each shown at seed 2026101907. Neither has an exact or substantial substring match in the 2,400-prompt corpus, and each scene's distinctive terms are absent. Generic vocabulary may overlap; absence from encoder pretraining was not assessed.

Every pair uses matching prompts, seeds, initial noise and generation settings:

• Resolution: 640 × 960
• Sampling: 8 steps, Euler / simple
• CFG: 1
• Sampling shift: 5
• MOSS Image 8-step DMD LoRA strength: 0.8

Click the grid to view the full-resolution 1280 × 6240 image. Full prompts and generation details are also available in the GitHub comparison manifest.

The examples retain the main subjects and artistic styles, with differences in composition, geometry, and incidental details. They do not establish quality parity with the original encoder.

Adult prompts were included in fitting and feature evaluation, but adult image-generation quality has not been visually evaluated. Quality with other encoder quantizations, longer prompts, image editing, higher CFG, and other generation settings also remains unestablished.

Models and usage

Original MOSS-Image models

• MOSS-Image-Base
• MOSS-Image-SFT
• ComfyUI-format MOSS-Image files

Smaller encoder

The tested checkpoint is Qwen3.5-4B INT8 ConvRot. Your ComfyUI installation must support its quantization format.

Use Qwen3.5-4B, not Qwen3-VL-4B. These are different model families.

Download the adapter

The current adapter filename is:

moss_qwen35_4b_to_9b_1600_train.safetensors

You can download it from the GitHub release. To use this version from the Hugging Face repository, select the 1,600-prompt filename and verify its checksum below. The 800-prompt file is the earlier version, even if both files are present.

You only need one copy of the adapter.

Adapter SHA-256 checksum

df5ed7fa15f4c628ad6cb3539d9cd16d68f198df9cd1e07740e9bd7b162c954b

ComfyUI setup

1. Install moss-image-smaller-encoder in ComfyUI/custom_nodes/.
2. Place the Qwen3.5-4B checkpoint in ComfyUI/models/text_encoders/.
3. Place the adapter in ComfyUI/models/moss_encoder_adapters/.
4. Restart ComfyUI.
5. Select Qwen3.5-4B in Load CLIP, with type set to moss_image.
6. Add MOSS Smaller Encoder Adapter, select the adapter file, and connect its CLIP output to the normal CLIP Text Encode node.

Load CLIP → Load LoRA (optional) → MOSS Smaller Encoder Adapter → CLIP Text Encode → KSampler

Keep your usual MOSS diffusion-model and VAE connections. The GitHub repository includes a sample workflow and further compatibility details.

The adapter cannot be loaded through a LoRA loader, and it does not replace the complete text encoder.

Development approach

This project is vibe coded-ish: AI assists with research, implementation, and experiments, while I review the proposed steps, verify the work, and inspect the results. The evaluation limits are documented above.

Credits and references

• ComfyUI-ClipProj by nicolab28 — inspiration for replacing a larger text encoder with a smaller encoder and a learned feature projection. See its calibration methodology.
• MOSS-Image by the OpenMOSS team — the original image-generation model.
• Qwen3.5 by the Qwen team — the encoder family used in this experiment.
• jtatman/stable-diffusion-prompts-stats-full-uncensored — a source of prompts for the expanded calibration corpus.

This is an independent community experiment and is not affiliated with the OpenMOSS, Qwen, or ComfyUI-ClipProj developers. Refer to the licenses of the original models, datasets, and software separately.