kentarrito/stable-diffusion-2-kanji-finetune-lora

🧾 Model Card: LoRA Fine-Tuned – kentarrito/stable-diffusion-2-kanji-finetune-lora

🈚 Stable Diffusion 2 – Kanji Fine-Tune (LoRA)

This model is a LoRA fine-tuned version of Stable Diffusion 2, trained to generate kanji-like images based on their English meanings.

LoRA tuning allows lightweight updates to the base model, making it easier to combine with other LoRA modules or revert to base SD2.0.

📦 Usage

from diffusers import StableDiffusionPipeline, UNet2DConditionModel
import torch
from peft import PeftModel

base_model = "stabilityai/stable-diffusion-2"
lora_model = "kentarrito/stable-diffusion-2-kanji-finetune-lora"

pipe = StableDiffusionPipeline.from_pretrained(
    base_model, torch_dtype=torch.float16
).to("cuda")

pipe.unet = PeftModel.from_pretrained(pipe.unet, lora_model)

image = pipe(prompt="peace").images[0]
image.show()

🔧 If using the LoRA manually, merge the LoRA weights into the UNet before generation.

🖼️ Generated Samples

See Github

🧠 Dataset

See kentarrito/kanji_dataset Each image represents a kanji symbol and is paired with a corresponding English word.

🎯 Limitations

• Not suitable for generating real kanji characters or fonts.
• English words that do not map to a trained kanji may result in generic or failed outputs.

🧪 Training

• Training Code Github
• Base: stabilityai/stable-diffusion-2
• Method: LoRA tuning of UNet only
• Framework: Hugging Face diffusers + peft
• LoRA Rank: 4

📜 License

MIT License. Dataset sources follow KanjiVG and KANJIDIC2 licenses.