| | --- |
| | license: cc-by-4.0 |
| | language: |
| | - en |
| | pipeline_tag: any-to-any |
| | tags: |
| | - multimodal |
| | library_name: transformers |
| | --- |
| | |
| | # Align-Anything Chameleon 7B Plus |
| |
|
| | ## Introduction |
| |
|
| | Repository for Align-Anything Chameleon 7B Plus, a powerful model for text-image interleaved input and output, with further alignment by [Align-Anything](https://github.com/PKU-Alignment/Align-Anything) algorithm. This model is based on the [Chameleon](https://huggingface.co/facebook/chameleon-7b) model, and is trained and aligned on the [Align-Anything](https://github.com/PKU-Alignment/Align-Anything) framework to further unlock its capability of image generation, and improve the alignment ability towards human preferences. |
| |
|
| | ## Usage |
| |
|
| | To use this model, you can refer to the [Align-Anything](https://github.com/PKU-Alignment) repository for more details, including the training, inference and evaluation: |
| |
|
| | ```bash |
| | git clone https://github.com/PKU-Alignment/align-anything.git |
| | cd align-anything/projects/text_image_to_text_image |
| | ``` |
| |
|
| | Then follow the instructions in the README.md file to set up the environment and run the scripts. |
| |
|
| | Currently, the official Transformer repo does not support Chameleon model with image output (see [this PR](https://github.com/huggingface/transformers/pull/32013) for more details), so we rely on a certain fork of the repo. |
| |
|
| | After installing Align-Anything and correctly set up the envrionment, you can install the forked stable version of the repo by running: |
| |
|
| | ```bash |
| | pip install git+https://github.com/htlou/transformers.git@hantao_stable_cham |
| | ``` |
| |
|
| | If you want to generate image (pure text generation can be directly done by `Transformers`), you can follow the instructions in the [mmsg_chameleon](https://github.com/htlou/mmsg_chameleon) repo to run the inference. |
| |
|
| | ```bash |
| | git clone https://github.com/htlou/mmsg_chameleon.git |
| | cd mmsg_chameleon |
| | ``` |
| |
|
| | Then set up the envrionment using |
| | ```bash |
| | pip install -e . |
| | ``` |
| |
|
| | After setting up the envrioment, set up the correct paths in `scripts/interleaved_gen.sh` and then run |
| | ```bash |
| | bash scripts/interleaved_gen.sh |
| | ``` |
