README.md

---
base_model:
- meta-llama/Llama-3.2-3B-Instruct
datasets:
- NingLab/MMECInstruct
license: cc-by-4.0
pipeline_tag: image-text-to-text
library_name: transformers
---

# CASLIE-S

This repo contains the models for "[Captions Speak Louder than Images (CASLIE): Generalizing Foundation Models for E-commerce from High-quality Multimodal Instruction Data](https://huggingface.co/papers/2410.17337)".

- 📚 [Paper](https://huggingface.co/papers/2410.17337)
- 🌐 [Project Page](https://ninglab.github.io/CASLIE/)
- 💻 [Code](https://github.com/ninglab/CASLIE)

## Introduction
We introduce [MMECInstruct](https://huggingface.co/datasets/NingLab/MMECInstruct), the first-ever, large-scale, and high-quality multimodal instruction dataset for e-commerce. We also develop CASLIE, a simple, lightweight, yet effective framework for integrating multimodal information. Leveraging MMECInstruct, we fine-tune a series of e-commerce Multimodal Foundation Models (MFMs) within CASLIE.

## CASLIE Models
The CASLIE-S model is instruction-tuned from the small base models [Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct).

## Sample Usage

To conduct multimodal inference with the CASLIE-S model using the Hugging Face `transformers` library, you can follow this example. This snippet demonstrates how to load the model and processor, and perform a basic image-text-to-text generation.

```python
import torch
from transformers import AutoProcessor, AutoModelForCausalLM
from PIL import Image

# Load model and processor
model_path = "NingLab/CASLIE-S"
# The `trust_remote_code=True` is necessary to load custom model and processor definitions.
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.float16, device_map="auto", trust_remote_code=True)

# Example: Image and text input for a product description task
# Replace "image.png" with the actual path to your image file
try:
    image = Image.open("image.png").convert("RGB")
except FileNotFoundError:
    print("Warning: 'image.png' not found. Using a dummy image for demonstration. Please replace with a real image path.")
    # Create a dummy image for demonstration if actual image is not found
    image = Image.new('RGB', (256, 256), color = 'red')

question = "Describe the product in detail."

# Prepare the conversation in a chat template format
# The "<image>" token is a placeholder which the processor handles to embed image features.
messages = [{"role": "user", "content": f"{question} <image>"}]

# Apply the chat template and process inputs (image and text)
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], padding=True, return_tensors="pt").to(model.device)

# Generate response from the model
output_ids = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)
response = processor.decode(output_ids[0], skip_special_tokens=True)

print(f"Question: {question}")
print(f"Response: {response}")

# For more advanced usage, specific tasks, and detailed inference scripts,
# please refer to the project's official GitHub repository:
# https://github.com/ninglab/CASLIE
```

## Citation
```bibtex
@article{ling2024captions,
    title={Captions Speak Louder than Images (CASLIE): Generalizing Foundation Models for E-commerce from High-quality Multimodal Instruction Data},
    author={Ling, Xinyi and Peng, Bo and Du, Hanwen and Zhu, Zhihui and Ning, Xia},
    journal={arXiv preprint arXiv:2410.17337},
    year={2024}
}
```