Upload bocchi-the-rock-character with huggingface_hub

bocchi-the-rock-character/README.md

@@ -0,0 +1,96 @@
+---
+license: creativeml-openrail-m
+---
+
+---
+license: creativeml-openrail-m
+---
+
+This is a low-quality bocchi-the-rock (ぼっち・ざ・ろっく!) character model.
+Similar to my [yama-no-susume model](https://huggingface.co/alea31415/yama-no-susume), this model is capable of generating **multi-character scenes** beyond images of a single character.
+Of course, the result is still hit-or-miss, but I with some chance you can get the entire Kessoku Band right in one shot,
+and otherwise, you can always rely on inpainting.
+Here are two examples:
+
+With inpainting
+![4265343062-1047638199](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/with_inpaint/4265343062-1047638199.png)
+
+Without inpainting
+![4265343086-2648280139](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/without_inpaint/4265343086-2648280139.png)
+
+
+### Characters
+
+The model knows 12 characters from bocchi the rock.
+The ressemblance with a character can be improved by a better description of their appearance (for example by adding long wavy hair to ShimizuEliza).
+
+![xy_grid-0028-24](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/grids/xy_grid-0028-24.jpg)
+![xy_grid-0029-24](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/grids/xy_grid-0029-24.jpg)
+![xy_grid-0030-24](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/grids/xy_grid-0030-24.jpg)
+
+
+### Dataset description
+
+The dataset contains around 27K images with the following composition
+- 7024 anime screenshots
+- 1630 fan arts
+- 18519 customized regularization images
+
+The model is trained with a specific weighting scheme to balance between different concepts.
+For example, the above three categories have weights respectively 0.3, 0.25, and 0.45.
+Each category is itself split into many sub-categories in a hierarchical way.
+For more details on the data preparation process please refer to https://github.com/cyber-meow/anime_screenshot_pipeline
+
+
+### Training Details
+
+#### Trainer
+The model is trained using [EveryDream1](https://github.com/victorchall/EveryDream-trainer) as
+EveryDream seems to be the only trainer out there that supports sample weighting (through the use of `multiply.txt`).
+Note that for future training it makes sense to migrate to [EveryDream2](https://github.com/victorchall/EveryDream2trainer).
+
+#### Hardware and cost
+The model is trained on runpod using 3090 and cost me around 15 dollors.
+
+#### Hyperparameter specification
+
+The model is trained for 50000 steps, at batch size 4, lr 1e-6, resolution 512, and conditional dropping rate of 10%.
+
+Note that as a consequence of the weighting scheme which translates into a number of different multiply for each image,
+the count of repeat and epoch has a quite different meaning here.
+For example, depending on the weighting, I have around 300K images (some images are used multiple times) in an epoch,
+and therefore I did not even finish an entire epoch with the 50000 steps at batch size 4.
+
+### Failures
+
+- For the first 24000 steps I use the trigger words `Bfan1` and `Bfan2` for the two fans of Bocchi.
+  However, these two words are too similar and the model fails to different characters for these.
+  Therefore I changed Bfan2 to Bofa2 at step 24000. This seemed to solve the problem.
+- Character blending is always an issue.
+- When prompting the four characters of Kessoku Band we often get side shots.
+  I think this is because of some overfitting to a particular image.
+
+
+### More Example Generations
+
+With inpainting
+![4265343068-2420755431](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/with_inpaint/4265343068-2420755431.png)
+![4265343066-3979275255](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/with_inpaint/4265343066-3979275255.png)
+![4265343022-3534836762](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/with_inpaint/4265343022-3534836762.png)
+
+
+Without inpainting
+![4265343092-803155289](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/without_inpaint/4265343092-803155289.png)
+![4265343053-918713189](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/without_inpaint/4265343053-918713189.png)
+![4265343054-2839948768](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/without_inpaint/4265343054-2839948768.png)
+![4265343096-399054050](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/without_inpaint/4265343096-399054050.png)
+![4265343100-3858388158](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/without_inpaint/4265343100-3858388158.png)
+![4265343016-2842516738](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/without_inpaint/4265343016-2842516738.png)
+![4265343084-3548261345](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/without_inpaint/4265343084-3548261345.png)
+![4265343083-1372779456](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/without_inpaint/4265343083-1372779456.png)
+
+Some failure cases
+![4265343089-2940163958](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/failure/4265343089-2940163958.png)
+![4265343091-129639375](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/failure/4265343091-129639375.png)
+![4265343048-2869643584](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/failure/4265343048-2869643584.png)
+![4265343039-1470057774](https://huggingface.co/alea31415/bocchi-the-rock-character/resolve/main/examples/failure/4265343039-1470057774.png)

bocchi-the-rock-character/bocchitherock24000steps.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:736c25fa26cf3c5f124878a971758fc1b3691696509deaa4ab4e31aa7a75b30a
+size 12126929195

bocchi-the-rock-character/bocchitherock24000stepspruned.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:85bb016f52bad19fe69a40b2f5b6e795fd46a076abe0d5bd189b3ffb5c19dc0e
+size 2132892617

bocchi-the-rock-character/bocchitherock48000steps.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:97279026843d9b7bcaa093a8818f1d4ebf227115d92efebe1d35c3729f350523
+size 12126929195

bocchi-the-rock-character/bocchitherock48000stepspruned.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7b7755321cb0357e2575fdaf2efb20e5640dfa0e8dfe294541842e5cc24fd56c
+size 2132892617

bocchi-the-rock-character/bocchitherock50000stepspruned.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3fc93a9be90bd5d7ef7fb385f20e955fd5dabc15d2fb0064dbb1c6db8f79166
+size 2132892617