genloop/llama-3.2-11b-instruct-amazon-product-description-merged-model

Model Card

• Developed by: Genloop.ai
• Funded by: Genloop Labs, Inc.
• Model type: Vision Language Model (VLM)
• Finetuned from model: Meta Llama 3.2 11B Vision Instruct
• Usage: This model is intended for product cataloging, i.e. generating product descriptions from images

How to Get Started with the Model

Make sure to update your transformers installation via pip install --upgrade transformers.

import requests
import torch
from PIL import Image
from transformers import MllamaForConditionalGeneration, AutoProcessor
model_id = "meta-llama/Llama-3.2-11B-Vision-Instruct"
model = MllamaForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)
url = "insert_your_image_link_here"
image = Image.open(requests.get(url, stream=True).raw)
user_prompt= """Create a SHORT Product description based on the provided a given ##PRODUCT NAME## and a ##CATEGORY## and an image of the product. 
Only return description. The description should be SEO optimized and for a better mobile search experience.

##PRODUCT NAME##: {product_name}
##CATEGORY##: {prod_category}"""

product_name = "insert_your_product_name_here"
product_category = "insert_your_product_category_here"
messages = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": user_prompt.format(product_name = product_name, product_category = product_category)}
    ]}
]
input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(
    image,
    input_text,
    add_special_tokens=False,
    return_tensors="pt"
).to(model.device)
output = model.generate(**inputs, max_new_tokens=30)
print(processor.decode(output[0]))

Training Details

This model has been finetuned on the Amazon-Product-Descriptions dataset. The reference descriptions were generated using Gemini Flash.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 2
• seed: 3407
• gradient_accumulation_steps: 4
• gradient_checkpointing: True
• total_train_batch_size: 8
• lr_scheduler_type: linear
• num_epochs: 3

Results

MODEL	FINETUNED OR NOT	INFERENCE LATENCY	METEOR Score
Llama-3.2-11B-Vision-Instruct	Not Finetuned	1.68	0.38
Llama-3.2-11B-Vision-Instruct	Finetuned	1.68	0.53