Add comprehensive model card for MENTOR

README.md

@@ -0,0 +1,196 @@
+---
+pipeline_tag: text-to-image
+library_name: transformers
+license: mit
+tags:
+- multimodal
+- autoregressive-models
+---
+
+# MENTOR: Efficient Multimodal-Conditioned Tuning for Autoregressive Vision Generation
+
+**MENTOR** is a novel autoregressive (AR) framework for efficient Multimodal-conditioned Tuning for Autoregressive multimodal image generation. It aims to address the limitations of existing text-to-image models in precise visual control, balancing multimodal inputs, and the extensive training required for complex multimodal image generation.
+
+MENTOR combines an AR image generator with a two-stage training paradigm, enabling fine-grained, token-level alignment between multimodal inputs and image outputs without relying on auxiliary adapters or cross-attention modules. The two-stage training consists of: (1) a multimodal alignment stage that establishes robust pixel- and semantic-level alignment, followed by (2) a multimodal instruction tuning stage that balances the integration of multimodal inputs and enhances generation controllability.
+
+Despite modest model size, suboptimal base components, and limited training resources, MENTOR achieves strong performance on the DreamBench++ benchmark, outperforming competitive baselines in concept preservation and prompt following. Additionally, our method delivers superior image reconstruction fidelity, broad task adaptability, and improved training efficiency compared to diffusion-based methods.
+
+- **Paper:** [MENTOR: Efficient Multimodal-Conditioned Tuning for Autoregressive Vision Generation Models](https://huggingface.co/papers/2507.09574)
+- **Project Page:** [https://haozhezhao.github.io/MENTOR.page](https://haozhezhao.github.io/MENTOR.page)
+- **Code:** [https://github.com/HaozheZhao/MENTOR](https://github.com/HaozheZhao/MENTOR)
+
+<p align="center">
+  <img src="https://github.com/HaozheZhao/MENTOR/blob/main/figures/teasarv3.png" width="100%" alt="MENTOR Overview" />
+</p>
+
+### 🏆 Efficient Autoregressive Multimodal Image Generation with 10× Less Data
+
+MENTOR demonstrates competitive multimodal image generation capabilities, achieving superior results with dramatically reduced resources thanks to an efficient tuning paradigm. While competitors like Emu2 require 37 billion parameters and vast datasets, MENTOR surpasses their performance with only 2.3 billion parameters and significantly less training data in an autoregressive vision generation framework.
+
+---
+
+## ✨ Key Features
+
+| Feature | MENTOR | Diffusion-Based Models |
+|:--------|:-------|:-------------------|
+| **Training Efficiency** | ✅ 1.5 days on 8 GPUs | ❌ 3+ days on 256 GPUs |
+| **Deterministic Control** | ✅ Precise AR generation | ❌ Stochastic sampling |
+| **Modality Balance** | ✅ Lowest CP/PF ratio (0.65) | ❌ High imbalance (>1.0) |
+| **Architecture** | ✅ Simple unified transformer | ❌ Complex auxiliary modules |
+
+---
+
+## 📊 Main Results
+
+### 🏅 DreamBench++ Benchmark Leadership
+
+<p align="center">
+<img src="https://github.com/HaozheZhao/MENTOR/blob/main/figures/Figure.png" width="60%" alt="Performance Comparison">
+</p>
+
+| Method | Model Size | Training Data | CP↑ | PF↑ | **CP·PF↑** | **CP/PF↓** |
+|:-------|:----------:|:-------------:|:---:|:---:|:----------:|:------:|
+| DreamEngine | 10.5B | 21M | 0.68 | 0.37 | 0.26 | 1.84 |
+| Kosmos-G | 3B | 200M | 0.54 | 0.51 | 0.28 | 1.06 |
+| Emu2 | 37B | 16M | 0.53 | 0.69 | 0.36 | 0.77 |
+| IP-Adapter ViT-G | 2.5B | 10M | 0.59 | 0.64 | 0.38 | 0.92 |
+| **MENTOR** | **2.3B** | **3M** | 0.55 | 0.84 | **0.47** | **0.65** |
+
+> **CP**: Concept Preservation | **PF**: Prompt Following | **Lower CP/PF = Better Balance**
+
+### 🎨 Superior Image Reconstruction
+
+| Method         | COCO L2↓ | JourneyDB L2↓ | Improvement      |
+|:---------------|:--------:|:-------------:|:----------------:|
+| SeedTokenizer  | 0.5102   | 0.5291        | \          |
+| SEED-X         | 0.4317   | 0.4352        | \          |
+| EMU2-Gen       | 0.3828   | 0.2869        | \           |
+| DreamEngine    | 0.2065   | 0.2052        | Baseline         |
+| **MENTOR**     | **0.1008** | **0.0867**  | **~50% Better** |
+
+---
+
+## 🎯 Usage Examples
+
+This model can be loaded and used with the Hugging Face `transformers` library by setting `trust_remote_code=True`.
+
+### Basic Generation
+
+```python
+import numpy as np
+import torch
+import torchvision.transforms as T
+from PIL import Image
+from torchvision.transforms.functional import InterpolationMode
+from transformers import AutoModel, AutoTokenizer
+
+IMAGENET_MEAN = (0.485, 0.456, 0.406)
+IMAGENET_STD = (0.229, 0.224, 0.225)
+
+def build_transform(input_size):
+    MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
+    transform = T.Compose([
+        T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
+        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
+        T.ToTensor(),
+        T.Normalize(mean=MEAN, std=STD)
+    ])
+    return transform
+
+def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
+    best_ratio_diff = float('inf')
+    best_ratio = (1, 1)
+    area = width * height
+    for ratio in target_ratios:
+        target_aspect_ratio = ratio[0] / ratio[1]
+        ratio_diff = abs(aspect_ratio - target_aspect_ratio)
+        if ratio_diff < best_ratio_diff:
+            best_ratio_diff = ratio_diff
+            best_ratio = ratio
+        elif ratio_diff == best_ratio_diff:
+            if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
+                best_ratio = ratio
+    return best_ratio
+
+def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):
+    orig_width, orig_height = image.size
+    aspect_ratio = orig_width / orig_height
+
+    target_ratios = set(
+        (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
+        i * j <= max_num and i * j >= min_num)
+    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])
+
+    target_aspect_ratio = find_closest_aspect_ratio(
+        aspect_ratio, target_ratios, orig_width, orig_height, image_size)
+
+    target_width = image_size * target_aspect_ratio[0]
+    target_height = image_size * target_aspect_ratio[1]
+    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]
+
+    resized_img = image.resize((target_width, target_height))
+    processed_images = []
+    for i in range(blocks):
+        box = (
+            (i % (target_width // image_size)) * image_size,
+            (i // (target_width // image_size)) * image_size,
+            ((i % (target_width // image_size)) + 1) * image_size,
+            ((i // (target_width // image_size)) + 1) * image_size
+        )
+        split_img = resized_img.crop(box)
+        processed_images.append(split_img)
+    assert len(processed_images) == blocks
+    if use_thumbnail and len(processed_images) != 1:
+        thumbnail_img = image.resize((image_size, image_size))
+        processed_images.append(thumbnail_img)
+    return processed_images
+
+def load_image(image_file, input_size=448, max_num=12):
+    image = Image.open(image_file).convert('RGB')
+    transform = build_transform(input_size=input_size)
+    images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
+    pixel_values = [transform(image) for image in images]
+    pixel_values = torch.stack(pixel_values)
+    return pixel_values
+
+# Load model and tokenizer
+# Note: You may need to download the model checkpoint locally first using
+# `huggingface-cli download BleachNick/Mentor --local-dir Mentor`
+# And specify the path to your downloaded model folder.
+model_name = "BleachNick/Mentor" # Or your local path to the downloaded model
+model = AutoModel.from_pretrained(
+    model_name,
+    torch_dtype=torch.bfloat16,
+    low_cpu_mem_usage=True,
+    trust_remote_code=True
+).eval().cuda()
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+
+# Example usage (Text-to-Image with optional image conditioning)
+# Ensure 'cat.jpg' is available locally (e.g., download from the GitHub repo's figures folder).
+image_path = "./figures/cat.jpg" # Example image from the repo.
+pixel_values = load_image(image_path, max_num=6).to(torch.bfloat16).cuda() if image_path else None
+
+question = "A cat in <image>.
+ A cat in a 16-bit fantasy pixel-art scene"
+generation_config = dict(max_new_tokens=1024, do_sample=True)
+
+response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)
+print(f'User: {question}
+Assistant: {response}')
+```
+
+---
+
+## 📚 Citation
+
+If you find MENTOR useful, please cite our paper:
+
+```bibtex
+@inproceedings{zhao2024mentor,
+  title={MENTOR: Efficient Multimodal-Conditioned Tuning for Autoregressive Vision Generation Models},
+  author={Zhao, Haozhe* and Cai, Zefan* and Si, Shuzheng and Chen, Liang and 
+          Gu, Jiuxiang and Xiao, Wen and Hu, Junjie},
+  year={2024}
+}
+```