| # 1to2: Training Multiple-Subject Models using only Single-Subject Data (Experimental) |
|
|
| Updates will be mirrored on both Hugging Face and Civitai. |
|
|
| ## Introduction |
|
|
| [It has been shown that multiple characters can be trained into the model](https://civitai.com/models/23476/the-idolmster-cinderella-girls-starlight-stage-style-90-characters). A harder task is to create a model that can generate multiple characters simultaneously without modifying the generation pipeline. This document describes a simple technique that has been shown to help generating multiple characters in the same image. |
|
|
| ## Method |
|
|
| ``` |
| Requirement: Sets of single-character images |
| Steps: |
| 1. Train a multi-concept model using the original dataset |
| 2. Create an augmentation dataset of joined image pairs from the original dataset |
| 3. Train on the augmentation dataset |
| ``` |
|
|
| ## Experiment |
|
|
|
|
| ### Setup |
|
|
| 3 characters from the game Cinderella Girls are chosen for the experiment. The base model is `anime-final-pruned`. It has been checked that the base model has minimal knowledge of the trained characters. |
|
|
| For the captions of the joined images, the template format `CharLeft/CharRight/COMPOSITE, TagsLeft, TagsRight` is used. |
|
|
| A LoRA (Hadamard product) is trained using the config file below: |
| ``` |
| [model_arguments] |
| v2 = false |
| v_parameterization = false |
| pretrained_model_name_or_path = "Animefull-final-pruned.ckpt" |
| |
| [additional_network_arguments] |
| no_metadata = false |
| unet_lr = 0.0005 |
| text_encoder_lr = 0.0005 |
| network_module = "lycoris.kohya" |
| network_dim = 8 |
| network_alpha = 1 |
| network_args = [ "conv_dim=0", "conv_alpha=16", "algo=loha",] |
| network_train_unet_only = false |
| network_train_text_encoder_only = false |
| |
| [optimizer_arguments] |
| optimizer_type = "AdamW8bit" |
| learning_rate = 0.0005 |
| max_grad_norm = 1.0 |
| lr_scheduler = "cosine" |
| lr_warmup_steps = 0 |
| |
| [dataset_arguments] |
| debug_dataset = false |
| # keep token 1 |
| |
| [training_arguments] |
| output_name = "cg3comp" |
| save_precision = "fp16" |
| save_every_n_epochs = 1 |
| train_batch_size = 2 |
| max_token_length = 225 |
| mem_eff_attn = false |
| xformers = true |
| max_train_epochs = 40 |
| max_data_loader_n_workers = 8 |
| persistent_data_loader_workers = true |
| gradient_checkpointing = false |
| gradient_accumulation_steps = 1 |
| mixed_precision = "fp16" |
| clip_skip = 2 |
| lowram = true |
| |
| [sample_prompt_arguments] |
| sample_every_n_epochs = 1 |
| sample_sampler = "k_euler_a" |
| |
| [saving_arguments] |
| save_model_as = "safetensors" |
| ``` |
| For the second stage of training, the batch size was reduced to 2 while keeping other settings identical. |
| The training took less than 2 hours on a T4 GPU. |
|
|
| ### Results |
| (see preview images) |
|
|
| ## Limitations |
| * This technique doubles the memory/compute requirement |
| * Composites can still be generated despite negative prompting |
| * Cloned characters seem to become the primary failure mode in place of blended characters |
|
|
| ## Related Works |
|
|
| Models been trained on datasets based on anime shows have [demonstrated](https://civitai.com/models/21305/) multi-subject capabilty. |
| Simply using concepts distant enough such as `1girl, 1boy` [has also been shown to be effective](https://civitai.com/models/17640/). |
|
|
| ## Future work |
|
|
| Below is a list of ideas yet to be explored |
| * Synthetic datasets |
| * Regularatization |
| * Joint training instaed of sequential |
