README.md

---
base_model:
  - black-forest-labs/FLUX.1-schnell
  - black-forest-labs/FLUX.1-Kontext-dev
library_name: peft
license: other
tags:
  - lora
  - flux
  - text-to-image
  - diffusers
  - peft
  - reinforcement-learning
  - face-identity
  - multi-human
language:
  - en
---

# Ar2Can: An Architect and an Artist Leveraging a Canvas for Multi-Human Generation

**Ar2Can** is a two-stage LoRA adapter pipeline for multi-reference face composition in multi-human image generation. It uses two adapters: an **Architect** (on FLUX.1-schnell) for scene layout, and an **Artist** (on FLUX.1-Kontext-dev) for identity-preserving rendering.

[![arXiv](https://img.shields.io/badge/arXiv-2511.22690-b31b1b.svg)](https://arxiv.org/abs/2511.22690)
[![Project Page](https://img.shields.io/badge/Project-Page-blue)](https://qualcomm-ai-research.github.io/ar2can/)
[![GitHub](https://img.shields.io/badge/GitHub-Code-black?logo=github)](https://github.com/Qualcomm-AI-research/ar2can)

![Ar2Can teaser](https://raw.githubusercontent.com/Qualcomm-AI-research/ar2can/main/assets/teaser.png)

---

## Abstract

> Despite recent advances in personalized image generation, existing models consistently fail to produce reliable multi-human scenes, often merging or losing facial identity. We present Ar2Can, a novel two-stage framework that disentangles spatial planning from identity rendering for multi-human generation. The Architect predicts structured layouts, specifying where each person should appear. The Artist then synthesizes photorealistic images, guided by a spatially-grounded face matching reward that combines Hungarian spatial alignment with identity similarity. This approach ensures faces are rendered at correct locations and faithfully preserve reference identities. We develop two Architect variants, seamlessly integrated with our diffusion-based Artist model. This is optimized via Group Relative Policy Optimization (GRPO) using compositional rewards for count accuracy, image quality, and identity matching. Evaluated on the MultiHuman-Testbench, Ar2Can achieves substantial improvements in both count accuracy and identity preservation, while maintaining high perceptual quality. Notably, our method achieves these results using primarily synthetic data, without requiring real multi-human images.

---

## Model Details

This repo contains two LoRA adapters in separate subdirectories:

| Adapter | Subdirectory | Base model | Purpose |
|---|---|---|---|
| Architect | `architect/` | `black-forest-labs/FLUX.1-schnell` | Generates scene layout / probe image |
| Artist | `artist/` | `black-forest-labs/FLUX.1-Kontext-dev` | Renders final identity-preserving composition |

Both adapters share the same LoRA configuration:

| Field | Value |
|---|---|
| Adapter type | LoRA (PEFT) |
| LoRA rank (r) | 64 |
| LoRA alpha | 128 |
| Dropout | 0.0 |
| Weight init | Gaussian |

**Architect target modules:** `attn.to_q`, `attn.to_k`, `attn.to_v`, `attn.to_out.0`, `attn.to_add_out`, `attn.add_q_proj`, `attn.add_k_proj`, `attn.add_v_proj`, `ff.net.0.proj`, `ff.net.2`, `ff_context.net.0.proj`, `ff_context.net.2`

**Artist target modules:** `attn.to_q`, `attn.to_k`, `attn.to_v`, `attn.to_out.0`, `attn.to_add_out`, `attn.add_q_proj`, `attn.add_k_proj`, `attn.add_v_proj`, `ff.net.0.proj`, `ff.net.2`, `ff_context.net.0.proj`, `ff_context.net.2`, `proj_mlp`

---

## Usage

### Option 1: Ar2Can inference repo (recommended)

Clone the repo — weights are pulled automatically from HuggingFace on first run:

```bash
git clone https://github.com/Qualcomm-AI-research/ar2can.git
cd ar2can
```

Run inference:

```bash
python infer.py \
  --faces faces/person_1.webp faces/person_2.webp faces/person_3.webp \
  --prompt "Three people standing side-by-side in a winter landscape, 8K, ultra-realistic, preserve exact facial identity."
```

Or launch the Gradio demo:

```bash
python app.py
# open http://localhost:7865
```

If you prefer locally downloaded weights (e.g. from [GitHub Releases](https://github.com/Qualcomm-AI-research/ar2can/releases)), place them at `loras/architect/adapter_model.safetensors` and `loras/artist/adapter_model.safetensors` — the code will use the local copies automatically.

### Recommended prompts

```
Two people on a sunny beach, ultra-realistic, 8K, preserve exact facial identity.
Three people standing side-by-side in a winter landscape, realistic light rendering, preserve exact facial identity.
Four people in a city plaza at midday, high fidelity, preserve exact facial identity.
```

---

## Hardware Requirements

- **GPU:** 2× NVIDIA A100 40 GB (or equivalent, ≥ 70 GB total VRAM for both models)
- **CUDA:** 12.4
- **Python:** 3.11
- **PyTorch:** 2.6.0

---

## Training Details

Ar2Can is trained with Flow-GRPO, a flow-matching adaptation of Group Relative Policy Optimization (GRPO):

- **Architect base model:** FLUX.1-schnell
- **Artist base model:** FLUX.1-Kontext-dev
- **Training algorithm:** Flow-GRPO (RL via composite reward signal)
- **Reward components:**
  - Spatially-grounded face matching (Hungarian alignment + ArcFace identity similarity)
  - Person count accuracy
  - HPS image quality score
- **Primarily synthetic training data** — no real multi-human images required

---

## Citation

```bibtex
@inproceedings{borse2026ar2can,
  title={Ar2can: An architect and an artist leveraging a canvas for multi-human generation},
  author={Borse, Shubhankar and Pham, Phuc and Farhadzadeh, Farzad and Choi, Seokeon and Nguyen, Phong and Tran, Anh and Yun, Sungrack and Hayat, Munawar and Porikli, Fatih},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={550--560},
  year={2026}
}
```

---

## Notes

> **Disclaimer:** The model weights released here are not identical to the model reported in the paper. They are provided for research and demonstration purposes only. Performance may vary from the results reported in the paper.

---

## License

The Architect LoRA weights are derived from [FLUX.1-schnell](https://huggingface.co/black-forest-labs/FLUX.1-schnell), which is released under the Apache 2.0 License.

The Artist LoRA weights are derived from [FLUX.1-Kontext-dev](https://huggingface.co/black-forest-labs/FLUX.1-Kontext-dev), which is released under a [Non-commercial License](https://huggingface.co/black-forest-labs/FLUX.1-Kontext-dev/blob/main/LICENSE.md). Any use of the Artist LoRA weights must comply with those Non-commercial License terms.

Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries.