| ---
|
| license: mit
|
| ---
|
|
|
| # 🎨 StyQA: A Quasi-Agent Framework for Vesatile Style Transfer
|
|
|
| <!-- [](https://arxiv.org/abs/2401.00000) -->
|
| [](https://huggingface.co/ReyChiaro/StyQA)
|
|
|
| 
|
|
|
| ## 🔬 Introduction
|
|
|
| StyQA is a unified quasi-agent framework for versatile style transfer tasks (pixel-level, semantic-level and continuous style transfer, etc.). This agent is attempted to be prompted with *style transfer pipeline* and normal style transfer instructions.
|
|
|
| Typically, StyQA will conduct style-analysis $\rightarrow$ style-transfer $\rightarrow$ style-criteria pipeline, and iteratively refines the outputs based on the maximum refinement times. For convenient usage, we provide single stage calling, StyQA can run a single stage or combination of stages.
|
|
|
| ## 🚀 Quick Start
|
|
|
| ### Environment
|
|
|
| StyQA relies on [uv](https://docs.astral.sh/uv/getting-started/) and we provided `pyproject.toml` in this repo. Note that the embedded base model relies on the newest `diffusers` which should be installed with:
|
|
|
| ```sh
|
| uv pip install git+https://github.com/huggingface/diffusers
|
| ```
|
|
|
| > Computational Resource: Make sure your device satisfies the computation requirements:
|
| > - The model with `bfloat16` will comsume about 52G. If LoRA is used, each LoRA module consumes about 1G. So for better inference with at least $1024\times 1024$ pictures, $\geq 80$G is required.
|
|
|
| ### Demos
|
|
|
| `main.py` provides demos for continuous style transfer:
|
|
|
| ```python
|
| @hydra.main(version_base="v1.2", config_path="configs", config_name="agent")
|
| def main(cfgs: OmegaConf):
|
| prompt = "Convert the pixel colors of Picture 1 into the pixel colors of Picture 2 with strength 0.75 and then transfer the semantic style into Picture 3."
|
| cnt_image_path = "demos/content1.jpg"
|
| ref_image_paths = ["demos/pixel.jpg", "demos/semantic1.jpg"]
|
|
|
| agent: StyQA = instantiate(cfgs.agent)
|
| user_input = UserInput(
|
| prompt=prompt,
|
| cnt_image_path=cnt_image_path,
|
| ref_image_paths=ref_image_paths,
|
| )
|
| agent.run_pipeline(user_input)
|
| ```
|
|
|
| Simply run
|
|
|
| ```sh
|
| python main.py
|
| ```
|
|
|
| and you will get the outputs like:
|
|
|
| 
|
|
|
| The agent configurations (`seed, num_inference_steps`, etc.) can be modified in `configs/agent.yaml`, or you can modify them using bash command thanks to [hydra](https://hydra.cc/).
|
|
|
| ```sh
|
| python main.py agent.seed=1234 agent.num_inference_steps=16
|
| ```
|
|
|
| Note that the prompt can points a specific value and style transfer task types explicitly (it is what we recommanded) or with an implicit representation such as
|
|
|
| ```sh
|
| "Using the colors and textures from Picture 2 ... moderate strength ... as if they are in same style category ..."
|
| ```
|
|
|
| ## 🖼️ Visualization
|
|
|
| We provide more visualization performances.
|
|
|
| ### Pixel-level style transfer
|
|
|
| Similar to arbitrary image style transfer, Pixel-level style transfer aims to utilize the color and textures features from low level pixels.
|
|
|
| 
|
|
|
| ### Semantic-level style transfer
|
|
|
| Artistic styles can be devided into different categories, semantic-level style transfer aims to re-generate content image using the same style category, which we call style-centric semantic features.
|
|
|
| 
|
|
|
| ### Continuous style transfer
|
|
|
| Based on different pipeline commands, StyQA will parse them into workflow and conducts style transfer one-by-one, here we provide somes demos.
|
|
|
| 
|
|
|
| ## 📝 TODO
|
|
|
| - [x] Open-source inference code
|
| - [x] Deliver demos
|
| - [ ] Uploads prepared LoRA weights
|
| - [ ] Open-source training/fine-tuning code
|
| - [ ] More experiments |
