Adapter for Jina-clip-v2 to be used as a text-encoder for Mugen

Update:

New version: HERE

Also includes finetune of Mugen

Only cross-attn trained, the rest of the model was frozen.

Installation:

Install custom ComfyUI nodes from here
Download jina-clip-v2 and put files under a new folder in ComfyUI/models/LLM/
Download jina-clip-v2-adapter from this repo and put it in ComfyUI/models/llm_adapters/
Download MugenJinaUNET from this repo.
- One thing to note is that the Jina-clip-v2 adapter works with base Mugen too. You don't have to use the finetune of Mugen in this repo.

Details for Usage:

Both natural language and booru-tag prompting. Natural language supports all of the formats found from here Anime-Art-Multicaptions-v5.0. However, booru-styled prompting represented about ~80%+ of the dataset the adapter was trained on.

Recommened Postive prompt : masterpiece, best quality
Recommened Negative prompt: worst quality, low quality, bad anatomy

Artist should be prefixed with @, for example: @someArtistHere

Date Tags:

newest
new
old
oldest

Note: Further details about the dataset are further below

Lora training:

Lora training w/ the adapter as the text encoder is supported in this branch of sd-scripts

You should load any LoRAs not trained with the Jina-clip-v2 adapter with the Lora Loader Model Only node

Training Overview:

First Stage:

~12hours on RTX A6000 distillation training with CLIP-L + CLIP-G from NoobAI

Second Stage:

Standard diffusion loss
Frozen UNeT (Mugen w/LoRa applied) and Jina-clip-v2
Adapter fully unfrozen
Trained at a base resolution of 512.
~18 hours, RTX PRO 6000 Blackwell Server Edition

The LoRa applied was to help Mugen with generations at a base resolution of 512x512. It was created by using the default CLIP text encoder, and training at resolutions ~512x512. Since, Mugen struggles with generating low resolution images, so training the adapter on diffusion loss at low resolution, would result in worse performance, instability, ..etc. I wanted to train at a low resoution, since training at 1024x1024 uses significantly more resources, and takes much much longer.

Third Stage:

Standard diffusion loss
Frozen Jina-clip-v2
Unfroze cross-attention layers in UNet (Base Mugen)
Adapter fully unfrozen
Mainly trained at base resolution of 768. With a small dataset training at 1024.
Added subset of data with increased dropout (0.15)
- Protected all artists from dropout. (Don't dropout tag if it starts with @). Done to help learning artists style, and did significantly, compared to the Second Stage adapter.
~16hours for Adapter, ~28hours for UNeT (used UNeT from previous run Third stage. But reset the adapter back to the end of the second stage.)

Further Dataset Details:

Quality tags:

They are grouped by year, then content rating (Explicit, Questionable, Sensitive, General). Since, different content rating have wildly different requirements for being in the Top 10%. For example, in 2024 the Top 5% of all images is score >= 100, for images with a rating of General its score >= 29, while the Top 5% of images with a rating of Explicit is score >= 258. Which is wildly different based on their content rating, if I didn't do this, then the positive quality tags would be significantly more biased towards more explicit content.

masterpiece: Top 5%
best quality: Top 10%
low quality: Bottom 10%
- Note: low quality also groups in images that are older than 2008.
worst quality: Bottom 5%
bad anatomy: Was never dropped out if the image had this tagged. Recommend to be included in negatives.

Date tags:

newest: 2022-onwards
new: 2017-2022
old: 2013-2017
oldest: 2008-2013

Datasets used:

From: danbooru2024-webp-4MPixel:

booru-styled:

0000-0005 = 48,000
0206-0217 = 96,000
0309-0320 = 96,000
Total = 240,000

Natural Language:

0030-0035 = 21,000
0041-0048 = 28,000
0055-0063 = 33,000
Total = 81,000

Custom curated / filtered by specific rules:

booru-styled

???? = ????

Natural Language

???? = ????

Further Interesting Details:

When using the Jina Text Encode (SDXL) node, I do something sneaky, which is if worst quality is detected in the prompt, I set the padding mode of that prompt to None, no matter what you have it set to. This is disabled for the version under llm_sdxl/jina/advanced/. I did this to only target the negative conditioning (I assume the negative prompt has worst quality in it, therefore I can avoid having two text encode nodes, one for the positive prompt, one for the negative prompt) as it results in much more vibrant generations if the positive conditioning has nearest-77 padding and the negative conditioning uses none. I should have really modified the SDXL cross-attention to take in a mask when I was training it.
- Another sneaky trick I use for the Jina Text Encode (SDXL) node is that when you put an artist tag in the prompt with the @ prefix, the node looks for the artist tag and places it at the start of the prompt instead. It also enforces it format to be @ artist instead of @artist. I did this as when testing I realized that this significantly helped the artist's style come out in the generations. Having it near the end of the prompt results in an unnoticeable effect, but putting it near the start results in a noticeable effect. You can test the difference by using the text encode node under llm_sdxl/jina/advanced and turning format_text to False.

Credits:

DeepGHS: Used danbooru2024-webp-4MPixel many many times. I've used their datasets a lot for many different projects, really appreciate having them easily accessible.
Cabal Research: Creators of Mugen
Minthy: Created T5-gemma adapter, which inspired me, and I used their code for the adapter as a base to work off. Also for Anime-Art-Multicaptions-v5.0 which I used for the natural language dataset.
NeuroSenko: Used their ComfyUI custom node code as a base.