File size: 3,591 Bytes
c4f02ec 29cea69 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 |
---
license: apache-2.0
datasets:
- ssssmark/AesCoT
metrics:
- spearmanr
- pearsonr
base_model:
- Qwen/Qwen2.5-VL-7B-Instruct
pipeline_tag: reinforcement-learning
---
<div align="center">
# Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization
<a href="https://arxiv.org/pdf/2509.21871" target="_blank">
<img alt="arXiv" src="https://img.shields.io/badge/arXiv-Aes--R1-red?logo=arxiv" height="25" />
</a>
<a href="https://huggingface.co/ssssmark/Aes-R1" target="_blank">
<img alt="HF Model: Aes-R1" src="https://img.shields.io/badge/%F0%9F%A4%97%20Model-Aes--R1-ffc107" height="25" />
</a>
<a href="https://huggingface.co/TianheWu/VisualQuality-R1-7B-preview" target="_blank">
<img alt="HF Dataset : Aes-CoT" src="https://img.shields.io/badge/%F0%9F%A4%97%20Dataset-Aes--CoT-ffc107" height="25" />
</a>
</div>
> A novel and effective reinforcement learning framework designed for Image Aesthetic Assessment and general open-ended preference evaluation.
# 🖥️Training
## Preparation
1. First download the IAA datasets(AVA,TAD66K,AADB,PARA...) and place them all in a single folder.
2. Construct your image-score dataset in the following format:
```json
{
"messages": [
{
"content": "prompt here",
"role": "user"
},
{
"content": "response here",
"role": "assistant"
}
],
"images": "image_path_1"
},
```
we provide an example dataset in `AesR1/data` folder.
3. Download the pre-trained model weights from [here](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) and place them in `AesR1/models`
## Cold-start
We use [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) to train the SFT model.
1. Clone the [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) repository and install the dependencies.
```bash
git clone https://github.com/hiyouga/LLaMA-Factory.git
conda create -n coldstart python=3.11.10
conda activate coldstart
cd LLaMA-Factory
pip install -e ".[torch,metrics]"
```
2. Put your cot dataset info in `LLaMA-Factory/data/dataset_info.json` and move `qwen_aescot.yaml` into `LLaMA-Factory/examples/train_full`
3. Run the following command to train the SFT model.
```bash
llamafactory-cli train examples/train_full/qwen_aescot.yaml
```
## RAPO
First setup the environment for RAPO training.
```bash
conda create -n rapo python=3.11.10
conda activate rapo
bash setup.sh
```
After modification, run the following command to train the RAPO model.
```bash
# For single node training
bash train/rapo/src/open-r1-multimodal/run_scripts/Aes/aes_onenode.sh
# For multi node training
bash train/rapo/src/open-r1-multimodal/run_scripts/Aes/aes_multinode.sh
```
# Inference
After training, you can inference the model by using the scripts in LLaMA-Factory.
```bash
#Install vllm
pip install vllm
#Infer
python scripts/vllm_infer.py \
--model_name_or_path [path/to/your/model] \
--dataset [dataset_name] \
--template qwen2_vl \
--save_name result.jsonl \
--temperature 0.6 \
```
# 📚 Citation
If you find this repo useful, please consider citing our paper as follows:
```
@misc{liu2025unlockingessencebeautyadvanced,
title={Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization},
author={Boyang Liu and Yifan Hu and Senjie Jin and Shihan Dou and Gonglei Shi and Jie Shao and Tao Gui and Xuanjing Huang},
year={2025},
eprint={2509.21871},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2509.21871},
}
``` |