File size: 3,876 Bytes
a1d8f51 59102c2 20287e5 59102c2 20287e5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | ---
license: mit
---
# 🎨 StyQA: A Quasi-Agent Framework for Vesatile Style Transfer
<!-- [](https://arxiv.org/abs/2401.00000) -->
[](https://huggingface.co/ReyChiaro/StyQA)
## 🔬 Introduction
StyQA is a unified quasi-agent framework for versatile style transfer tasks (pixel-level, semantic-level and continuous style transfer, etc.). This agent is attempted to be prompted with *style transfer pipeline* and normal style transfer instructions.
Typically, StyQA will conduct style-analysis $\rightarrow$ style-transfer $\rightarrow$ style-criteria pipeline, and iteratively refines the outputs based on the maximum refinement times. For convenient usage, we provide single stage calling, StyQA can run a single stage or combination of stages.
## 🚀 Quick Start
### Environment
StyQA relies on [uv](https://docs.astral.sh/uv/getting-started/) and we provided `pyproject.toml` in this repo. Note that the embedded base model relies on the newest `diffusers` which should be installed with:
```sh
uv pip install git+https://github.com/huggingface/diffusers
```
> Computational Resource: Make sure your device satisfies the computation requirements:
> - The model with `bfloat16` will comsume about 52G. If LoRA is used, each LoRA module consumes about 1G. So for better inference with at least $1024\times 1024$ pictures, $\geq 80$G is required.
### Demos
`main.py` provides demos for continuous style transfer:
```python
@hydra.main(version_base="v1.2", config_path="configs", config_name="agent")
def main(cfgs: OmegaConf):
prompt = "Convert the pixel colors of Picture 1 into the pixel colors of Picture 2 with strength 0.75 and then transfer the semantic style into Picture 3."
cnt_image_path = "demos/content1.jpg"
ref_image_paths = ["demos/pixel.jpg", "demos/semantic1.jpg"]
agent: StyQA = instantiate(cfgs.agent)
user_input = UserInput(
prompt=prompt,
cnt_image_path=cnt_image_path,
ref_image_paths=ref_image_paths,
)
agent.run_pipeline(user_input)
```
Simply run
```sh
python main.py
```
and you will get the outputs like:
The agent configurations (`seed, num_inference_steps`, etc.) can be modified in `configs/agent.yaml`, or you can modify them using bash command thanks to [hydra](https://hydra.cc/).
```sh
python main.py agent.seed=1234 agent.num_inference_steps=16
```
Note that the prompt can points a specific value and style transfer task types explicitly (it is what we recommanded) or with an implicit representation such as
```sh
"Using the colors and textures from Picture 2 ... moderate strength ... as if they are in same style category ..."
```
## 🖼️ Visualization
We provide more visualization performances.
### Pixel-level style transfer
Similar to arbitrary image style transfer, Pixel-level style transfer aims to utilize the color and textures features from low level pixels.
### Semantic-level style transfer
Artistic styles can be devided into different categories, semantic-level style transfer aims to re-generate content image using the same style category, which we call style-centric semantic features.
### Continuous style transfer
Based on different pipeline commands, StyQA will parse them into workflow and conducts style transfer one-by-one, here we provide somes demos.
## 📝 TODO
- [x] Open-source inference code
- [x] Deliver demos
- [ ] Uploads prepared LoRA weights
- [ ] Open-source training/fine-tuning code
- [ ] More experiments |