Upload README.md

README.md

@@ -0,0 +1,327 @@
+# VQGAN-CLIP Overview
+
+A repo for running VQGAN+CLIP locally. This started out as a Katherine Crowson VQGAN+CLIP derived Google colab notebook.
+
+<a href="https://replicate.ai/nerdyrodent/vqgan-clip"><img src="https://img.shields.io/static/v1?label=Replicate&message=Demo and Docker Image&color=blue"></a>
+
+Original notebook: [![Open In Colab][colab-badge]][colab-notebook]
+
+[colab-notebook]: <https://colab.research.google.com/drive/1ZAus_gn2RhTZWzOWUpPERNC0Q8OhZRTZ>
+[colab-badge]: <https://colab.research.google.com/assets/colab-badge.svg>
+
+Some example images:
+
+<img src="./samples/Cartoon3.png" width="256px"></img><img src="./samples/Cartoon.png" width="256px"></img><img src="./samples/Cartoon2.png" width="256px"></img>
+<img src="./samples/Bedroom.png" width="256px"></img><img src="./samples/DemonBiscuits.png" width="256px"></img><img src="./samples/Football.png" width="256px"></img>
+<img src="./samples/Fractal_Landscape3.png" width="256px"></img><img src="./samples/Games_5.png" width="256px"></img>
+
+Environment:
+
+* Tested on Ubuntu 20.04
+* GPU: Nvidia RTX 3090
+* Typical VRAM requirements:
+  * 24 GB for a 900x900 image
+  * 10 GB for a 512x512 image
+  * 8 GB for a 380x380 image
+
+You may also be interested in [CLIP Guided Diffusion](https://github.com/nerdyrodent/CLIP-Guided-Diffusion)
+
+## Set up
+
+This example uses [Anaconda](https://www.anaconda.com/products/individual#Downloads) to manage virtual Python environments.
+
+Create a new virtual Python environment for VQGAN-CLIP:
+
+```sh
+conda create --name vqgan python=3.9
+conda activate vqgan
+```
+
+Install Pytorch in the new enviroment:
+
+Note: This installs the CUDA version of Pytorch, if you want to use an AMD graphics card, read the [AMD section below](#using-an-amd-graphics-card).
+
+```sh
+pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
+```
+
+Install other required Python packages:
+
+```sh
+pip install ftfy regex tqdm omegaconf pytorch-lightning IPython kornia imageio imageio-ffmpeg einops torch_optimizer
+```
+
+Or use the ```requirements.txt``` file, which includes version numbers.
+
+Clone required repositories:
+
+```sh
+git clone 'https://github.com/nerdyrodent/VQGAN-CLIP'
+cd VQGAN-CLIP
+git clone 'https://github.com/openai/CLIP'
+git clone 'https://github.com/CompVis/taming-transformers'
+```
+
+Note: In my development environment both CLIP and taming-transformers are present in the local directory, and so aren't present in the `requirements.txt` or `vqgan.yml` files.
+
+As an alternative, you can also pip install taming-transformers and CLIP.
+
+You will also need at least 1 VQGAN pretrained model. E.g.
+
+```sh
+mkdir checkpoints
+
+curl -L -o checkpoints/vqgan_imagenet_f16_16384.yaml -C - 'https://heibox.uni-heidelberg.de/d/a7530b09fed84f80a887/files/?p=%2Fconfigs%2Fmodel.yaml&dl=1' #ImageNet 16384
+curl -L -o checkpoints/vqgan_imagenet_f16_16384.ckpt -C - 'https://heibox.uni-heidelberg.de/d/a7530b09fed84f80a887/files/?p=%2Fckpts%2Flast.ckpt&dl=1' #ImageNet 16384
+```
+Note that users of ```curl``` on Microsoft Windows should use double quotes.
+
+The `download_models.sh` script is an optional way to download a number of models. By default, it will download just 1 model.
+
+See <https://github.com/CompVis/taming-transformers#overview-of-pretrained-models> for more information about VQGAN pre-trained models, including download links.
+
+By default, the model .yaml and .ckpt files are expected in the `checkpoints` directory.
+See <https://github.com/CompVis/taming-transformers> for more information on datasets and models.
+
+Video guides are also available:
+* Linux - https://www.youtube.com/watch?v=1Esb-ZjO7tw
+* Windows - https://www.youtube.com/watch?v=XH7ZP0__FXs
+
+### Using an AMD graphics card
+
+Note: This hasn't been tested yet.
+
+ROCm can be used for AMD graphics cards instead of CUDA. You can check if your card is supported here:
+<https://github.com/RadeonOpenCompute/ROCm#supported-gpus>
+
+Install ROCm accordng to the instructions and don't forget to add the user to the video group:
+<https://rocmdocs.amd.com/en/latest/Installation_Guide/Installation-Guide.html>
+
+The usage and set up instructions above are the same, except for the line where you install Pytorch.
+Instead of `pip install torch==1.9.0+cu111 ...`, use the one or two lines which are displayed here (select Pip -> Python-> ROCm):
+<https://pytorch.org/get-started/locally/>
+
+### Using the CPU
+
+If no graphics card can be found, the CPU is automatically used and a warning displayed.
+
+Regardless of an available graphics card, the CPU can also be used by adding this command line argument: `-cd cpu`
+
+This works with the CUDA version of Pytorch, even without CUDA drivers installed, but doesn't seem to work with ROCm as of now.
+
+### Uninstalling
+
+Remove the Python enviroment:
+
+```sh
+conda remove --name vqgan --all
+```
+
+and delete the `VQGAN-CLIP` directory.
+
+## Run
+
+To generate images from text, specify your text prompt as shown in the example below:
+
+```sh
+python generate.py -p "A painting of an apple in a fruit bowl"
+```
+
+<img src="./samples/A_painting_of_an_apple_in_a_fruitbowl.png" width="256px"></img>
+
+## Multiple prompts
+
+Text and image prompts can be split using the pipe symbol in order to allow multiple prompts.
+You can also use a colon followed by a number to set a weight for that prompt. For example:
+
+```sh
+python generate.py -p "A painting of an apple in a fruit bowl | psychedelic | surreal:0.5 | weird:0.25"
+```
+
+<img src="./samples/Apple_weird.png" width="256px"></img>
+
+Image prompts can be split in the same way. For example:
+
+```sh
+python generate.py -p "A picture of a bedroom with a portrait of Van Gogh" -ip "samples/VanGogh.jpg | samples/Bedroom.png"
+```
+
+### Story mode
+
+Sets of text prompts can be created using the caret symbol, in order to generate a sort of story mode. For example:
+
+```sh
+python generate.py -p "A painting of a sunflower|photo:-1 ^ a painting of a rose ^ a painting of a tulip ^ a painting of a daisy flower ^ a photograph of daffodil" -cpe 1500 -zvid -i 6000 -zse 10 -vl 20 -zsc 1.005 -opt Adagrad -lr 0.15 -se 6000
+```
+
+
+## "Style Transfer"
+
+An input image with style text and a low number of iterations can be used create a sort of "style transfer" effect. For example:
+
+```sh
+python generate.py -p "A painting in the style of Picasso" -ii samples/VanGogh.jpg -i 80 -se 10 -opt AdamW -lr 0.25
+```
+
+| Output                                                        | Style       |
+| ------------------------------------------------------------- | ----------- |
+| <img src="./samples/vvg_picasso.png" width="256px"></img>     | Picasso     |
+| <img src="./samples/vvg_sketch.png" width="256px"></img>      | Sketch      |
+| <img src="./samples/vvg_psychedelic.png" width="256px"></img> | Psychedelic |
+
+A video style transfer effect can be achived by specifying a directory of video frames in `video_style_dir`. Output will be saved in the steps directory, using the original video frame filenames. You can also use this as a sort of "batch mode" if you have a directory of images you want to apply a style to. This can also be combined with Story Mode if you don't wish to apply the same style to every images, but instead roll through a list of styles.
+
+## Feedback example
+
+By feeding back the generated images and making slight changes, some interesting effects can be created.
+
+The example `zoom.sh` shows this by applying a zoom and rotate to generated images, before feeding them back in again.
+To use `zoom.sh`, specifying a text prompt, output filename and number of frames. E.g.
+
+```sh
+./zoom.sh "A painting of a red telephone box spinning through a time vortex" Telephone.png 150
+```
+If you don't have ImageMagick installed, you can install it with ```sudo apt install imagemagick```
+
+<img src="./samples/zoom.gif" width="256px"></img>
+
+There is also a simple zoom video creation option available. For example:
+```sh
+python generate.py -p "The inside of a sphere" -zvid -i 4500 -zse 20 -vl 10 -zsc 0.97 -opt Adagrad -lr 0.15 -se 4500
+```
+
+## Random text example
+
+Use `random.sh` to make a batch of images from random text. Edit the text and number of generated images to your taste!
+
+```sh
+./random.sh
+```
+
+## Advanced options
+
+To view the available options, use "-h".
+
+```sh
+python generate.py -h
+```
+
+```sh
+usage: generate.py [-h] [-p PROMPTS] [-ip IMAGE_PROMPTS] [-i MAX_ITERATIONS] [-se DISPLAY_FREQ]
+[-s SIZE SIZE] [-ii INIT_IMAGE] [-in INIT_NOISE] [-iw INIT_WEIGHT] [-m CLIP_MODEL]
+[-conf VQGAN_CONFIG] [-ckpt VQGAN_CHECKPOINT] [-nps [NOISE_PROMPT_SEEDS ...]]
+[-npw [NOISE_PROMPT_WEIGHTS ...]] [-lr STEP_SIZE] [-cuts CUTN] [-cutp CUT_POW] [-sd SEED]
+[-opt {Adam,AdamW,Adagrad,Adamax,DiffGrad,AdamP,RAdam,RMSprop}] [-o OUTPUT] [-vid] [-zvid]
+[-zs ZOOM_START] [-zse ZOOM_FREQUENCY] [-zsc ZOOM_SCALE] [-cpe PROMPT_FREQUENCY]
+[-vl VIDEO_LENGTH] [-ofps OUTPUT_VIDEO_FPS] [-ifps INPUT_VIDEO_FPS] [-d]
+[-aug {Ji,Sh,Gn,Pe,Ro,Af,Et,Ts,Cr,Er,Re} [{Ji,Sh,Gn,Pe,Ro,Af,Et,Ts,Cr,Er,Re} ...]]
+[-cd CUDA_DEVICE]
+```
+
+```sh
+optional arguments:
+  -h, --help            show this help message and exit
+  -p PROMPTS, --prompts PROMPTS
+                        Text prompts
+  -ip IMAGE_PROMPTS, --image_prompts IMAGE_PROMPTS
+                        Image prompts / target image
+  -i MAX_ITERATIONS, --iterations MAX_ITERATIONS
+                        Number of iterations
+  -se DISPLAY_FREQ, --save_every DISPLAY_FREQ
+                        Save image iterations
+  -s SIZE SIZE, --size SIZE SIZE
+                        Image size (width height) (default: [512, 512])
+  -ii INIT_IMAGE, --init_image INIT_IMAGE
+                        Initial image
+  -in INIT_NOISE, --init_noise INIT_NOISE
+                        Initial noise image (pixels or gradient)
+  -iw INIT_WEIGHT, --init_weight INIT_WEIGHT
+                        Initial weight
+  -m CLIP_MODEL, --clip_model CLIP_MODEL
+                        CLIP model (e.g. ViT-B/32, ViT-B/16)
+  -conf VQGAN_CONFIG, --vqgan_config VQGAN_CONFIG
+                        VQGAN config
+  -ckpt VQGAN_CHECKPOINT, --vqgan_checkpoint VQGAN_CHECKPOINT
+                        VQGAN checkpoint
+  -nps [NOISE_PROMPT_SEEDS ...], --noise_prompt_seeds [NOISE_PROMPT_SEEDS ...]
+                        Noise prompt seeds
+  -npw [NOISE_PROMPT_WEIGHTS ...], --noise_prompt_weights [NOISE_PROMPT_WEIGHTS ...]
+                        Noise prompt weights
+  -lr STEP_SIZE, --learning_rate STEP_SIZE
+                        Learning rate
+  -cuts CUTN, --num_cuts CUTN
+                        Number of cuts
+  -cutp CUT_POW, --cut_power CUT_POW
+                        Cut power
+  -sd SEED, --seed SEED
+                        Seed
+  -opt, --optimiser {Adam,AdamW,Adagrad,Adamax,DiffGrad,AdamP,RAdam,RMSprop}
+                        Optimiser
+  -o OUTPUT, --output OUTPUT
+                        Output file
+  -vid, --video         Create video frames?
+  -zvid, --zoom_video   Create zoom video?
+  -zs ZOOM_START, --zoom_start ZOOM_START
+                        Zoom start iteration
+  -zse ZOOM_FREQUENCY, --zoom_save_every ZOOM_FREQUENCY
+                        Save zoom image iterations
+  -zsc ZOOM_SCALE, --zoom_scale ZOOM_SCALE
+                        Zoom scale
+  -cpe PROMPT_FREQUENCY, --change_prompt_every PROMPT_FREQUENCY
+                        Prompt change frequency
+  -vl VIDEO_LENGTH, --video_length VIDEO_LENGTH
+                        Video length in seconds
+  -ofps OUTPUT_VIDEO_FPS, --output_video_fps OUTPUT_VIDEO_FPS
+                        Create an interpolated video (Nvidia GPU only) with this fps (min 10. best set to 30 or 60)
+  -ifps INPUT_VIDEO_FPS, --input_video_fps INPUT_VIDEO_FPS
+                        When creating an interpolated video, use this as the input fps to interpolate from (>0 & <ofps)
+  -d, --deterministic   Enable cudnn.deterministic?
+  -aug, --augments {Ji,Sh,Gn,Pe,Ro,Af,Et,Ts,Cr,Er,Re} [{Ji,Sh,Gn,Pe,Ro,Af,Et,Ts,Cr,Er,Re} ...]
+                        Enabled augments
+  -cd CUDA_DEVICE, --cuda_device CUDA_DEVICE
+                        Cuda device to use
+```
+
+## Troubleshooting
+
+### CUSOLVER_STATUS_INTERNAL_ERROR
+
+For example:
+
+`RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling cusolverDnCreate(handle)`
+
+Make sure you have specified the correct size for the image.
+
+### RuntimeError: CUDA out of memory
+
+For example:
+
+`RuntimeError: CUDA out of memory. Tried to allocate 150.00 MiB (GPU 0; 23.70 GiB total capacity; 21.31 GiB already allocated; 78.56 MiB free; 21.70 GiB reserved in total by PyTorch)`
+
+Your request doesn't fit into your GPU's VRAM. Reduce the image size and/or number of cuts.
+
+
+## Citations
+
+```bibtex
+@misc{unpublished2021clip,
+    title  = {CLIP: Connecting Text and Images},
+    author = {Alec Radford, Ilya Sutskever, Jong Wook Kim, Gretchen Krueger, Sandhini Agarwal},
+    year   = {2021}
+}
+```
+
+```bibtex
+@misc{esser2020taming,
+      title={Taming Transformers for High-Resolution Image Synthesis}, 
+      author={Patrick Esser and Robin Rombach and Björn Ommer},
+      year={2020},
+      eprint={2012.09841},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```
+
+Katherine Crowson - <https://github.com/crowsonkb>
+
+Public Domain images from Open Access Images at the Art Institute of Chicago - <https://www.artic.edu/open-access/open-access-images>