Mugen-Jina-clip-v2
Collection
Collection of models releated to Mugen with Jina-clip-v2 + adapter as a text encoder. • 2 items • Updated
import torch
from diffusers import DiffusionPipeline
# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("TheRemixer/Mugen-Jina-IP-Adapter", dtype=torch.bfloat16, device_map="cuda")
prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt).images[0]IP-Adapter using Jina-clip-v2's vision tower, and to be used on Mugen
Quick Notes:
- 368M parameters
- Trained on Mugen w/Jina with Jina-clip-v2 + Jina-clip-v2 adapter as the text encoder.
- However, it also works on Mugen with the original CLIP-G + CLIP-L text encoders. The outputs are slightly different.
- Trained for 5 epochs with
208,000unbatched steps in total, at a base resolution1024x1024.- Started showing promising results from the 2nd epoch. I have included the previous epochs in this repo if you are interested in testing them.
Installation:
- Install custom ComfyUI nodes from here
- Download the IP-adapter in this repo and place it at the path
ComfyUI/models/ip_adapter/ - Download jina-clip-v2 and put files under a new folder in
ComfyUI/models/LLM/ - Download Mugen Jina (Cross-attn only trained) and Jina-clip-v2 adapter
- Alternatively you can just use Mugen
Usage:
- See the workflow included in this repo on how to use the IP-adapter.
- I mainly tested the IP-adapter as a style transfer, however it can do other things too.
- Try different weighting if the default of
1.0is too much. The default of1.0isn't recommened in all cases.
Credits:
- tencent-ailab: IP-adapter
- DeepGHS: Used danbooru2024-webp-4MPixel
- Cabal Research: Creators of Mugen
- NeuroSenko: Once again, used their ComfyUI custom node code as a base for mine.
- Downloads last month
- 22
Model tree for TheRemixer/Mugen-Jina-IP-Adapter
Base model
Laxhar/noobai-XL-Vpred-1.0 Finetuned
Bluvoll/Experimental_EQ-VAE_NoobAI_tests Finetuned
CabalResearch/Mugen