transformers/pr_37082/en/model_doc/idefics3.md

Idefics3

Overview

The Idefics3 model was proposed in Building and better understanding vision-language models: insights and future directions by Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon.

Idefics3 is an adaptation of the Idefics2 model with three main differences:

• It uses Llama3 for the text model.
• It uses an updated processing logic for the images.
• It removes the perceiver.

The abstract from the paper is the following:

The field of vision-language models (VLMs), which take images and texts as inputs and output texts, is rapidly evolving and has yet to reach consensus on several key aspects of the development pipeline, including data, architecture, and training methods. This paper can be seen as a tutorial for building a VLM. We begin by providing a comprehensive overview of the current state-of-the-art approaches, highlighting the strengths and weaknesses of each, addressing the major challenges in the field, and suggesting promising research directions for underexplored areas. We then walk through the practical steps to build Idefics3-8B, a powerful VLM that significantly outperforms its predecessor Idefics2-8B, while being trained efficiently, exclusively on open datasets, and using a straightforward pipeline. These steps include the creation of Docmatix, a dataset for improving document understanding capabilities, which is 240 times larger than previously available datasets. We release the model along with the datasets created for its training.

Usage tips

Input images are processed either by upsampling (if resizing is enabled) or at their original resolution. The resizing behavior depends on two parameters: do_resize and size.

If do_resize is set to True, the model resizes images so that the longest edge is 4*364 pixels by default. The default resizing behavior can be customized by passing a dictionary to the size parameter. For example, {"longest_edge": 4 * 364} is the default, but you can change it to a different value if needed.

Here’s how to control resizing and set a custom size:

image_processor = Idefics3ImageProcessor(do_resize=True, size={"longest_edge": 2 * 364}, max_image_size=364)

Additionally, the max_image_size parameter, which controls the size of each square patch the image is decomposed into, is set to 364 by default but can be adjusted as needed. After resizing (if applicable), the image processor decomposes the images into square patches based on the max_image_size parameter.

This model was contributed by amyeroberts and andimarafioti.

Idefics3Config[[transformers.Idefics3Config]]

transformers.Idefics3Config[[transformers.Idefics3Config]]

Source

This is the configuration class to store the configuration of a Idefics3Model. It is used to instantiate a Idefics3 model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the model of the Idefics3 HuggingFaceM4/Idefics3-8B-Llama3 architecture.

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

Example:

>>> from transformers import Idefics3Model, Idefics3Config
>>> # Initializing configuration
>>> configuration = Idefics3Config()
>>> # Initializing a model from the configuration
>>> model = Idefics3Model(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config

Parameters:

use_cache (bool, optional, defaults to True) : Whether or not the model should cache the key/value pairs of the attention mechanism. Only relevant if config.is_decoder=True.

image_token_id (int, optional, defaults to 128257) : The id of the "image" token.

tie_word_embeddings (bool, optional, defaults to False) : Whether or not to tie the word embeddings with the token embeddings.

vision_config (IdeficsVisionConfig or dict, optional, defaults to IdeficsVisionConfig) : Custom vision config or dict for the vision tower

text_config (PreTrainedConfig or dict, optional, defaults to LlamaConfig) : Custom text config or dict for the text model

scale_factor (int, optional, defaults to 2) : The scale factor for the image encoder.

pad_token_id (int, optional, defaults to 128002) : The id of the padding token.

Idefics3VisionConfig[[transformers.Idefics3VisionConfig]]

transformers.Idefics3VisionConfig[[transformers.Idefics3VisionConfig]]

Source

This is the configuration class to store the configuration of a Idefics3VisionModel. It is used to instantiate a Idefics3 vision encoder according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the SigLIP checkpoint google/siglip-base-patch16-224 used in the Idefics3 model HuggingFaceM4/Idefics3-8B-Llama3.

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

Example:

>>> from transformers.models.idefics3.modeling_idefics3 import Idefics3VisionTransformer
>>> from transformers.models.idefics3.configuration_idefics3 import Idefics3VisionConfig

>>> # Initializing a Idefics3VisionConfig with google/siglip-base-patch16-224 style configuration
>>> configuration = Idefics3VisionConfig()

>>> # Initializing a Idefics3VisionTransformer (with random weights) from the google/siglip-base-patch16-224 style configuration
>>> model = Idefics3VisionTransformer(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config

Parameters:

hidden_size (int, optional, defaults to 1152) : Dimensionality of the encoder layers and the pooler layer.

intermediate_size (int, optional, defaults to 3072) : Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.

num_hidden_layers (int, optional, defaults to 12) : Number of hidden layers in the Transformer encoder.

num_attention_heads (int, optional, defaults to 16) : Number of attention heads for each attention layer in the Transformer encoder.

num_channels (int, optional, defaults to 3) : Number of channels in the input images.

image_size (int, optional, defaults to 224) : The size (resolution) of each image.

patch_size (int, optional, defaults to 32) : The size (resolution) of each patch.

hidden_act (str or function, optional, defaults to "gelu_pytorch_tanh") : The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu", "selu" and "gelu_new" "quick_gelu" are supported.

layer_norm_eps (float, optional, defaults to 1e-06) : The epsilon used by the layer normalization layers.

attention_dropout (float, optional, defaults to 0.0) : The dropout ratio for the attention probabilities.

initializer_range (float, optional, defaults to 0.02) : The standard deviation of the truncated_normal_initializer for initializing all weight matrices.

Idefics3VisionTransformer[[transformers.Idefics3VisionTransformer]]

transformers.Idefics3VisionTransformer[[transformers.Idefics3VisionTransformer]]

Source

The Idefics3 Vision Transformer Model outputting raw image embedding.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

Parameters:

config (Idefics3VisionConfig) : Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

Idefics3Model[[transformers.Idefics3Model]]

transformers.Idefics3Model[[transformers.Idefics3Model]]

Source

Idefics3 model consisting of a SIGLIP vision encoder and Llama3 language decoder

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forwardtransformers.Idefics3Model.forwardhttps://github.com/huggingface/transformers/blob/vr_37082/src/transformers/models/idefics3/modeling_idefics3.py#L665[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "position_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "past_key_values", "val": ": typing.Optional[transformers.cache_utils.Cache] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "pixel_attention_mask", "val": ": typing.Optional[torch.BoolTensor] = None"}, {"name": "image_hidden_states", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "use_cache", "val": ": typing.Optional[bool] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "cache_position", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}, {"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.modeling_flash_attention_utils.FlashAttentionKwargs]"}]- input_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) -- Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

What are input IDs?

• attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) -- Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:

  • 1 for tokens that are not masked,
  • 0 for tokens that are masked.

  What are attention masks?

• position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) -- Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.n_positions - 1].

  What are position IDs?

• past_key_values (~cache_utils.Cache, optional) -- Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the past_key_values returned by the model at a previous stage of decoding, when use_cache=True or config.use_cache=True.

  Only Cache instance is allowed as input, see our kv cache guide. If no past_key_values are passed, DynamicCache will be initialized by default.

  The model will output the same cache format that is fed as input.

  If past_key_values are used, the user is expected to input only unprocessed input_ids (those that don't have their past key value states given to this model) of shape (batch_size, unprocessed_length) instead of all input_ids of shape (batch_size, sequence_length).

• inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) -- Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model's internal embedding lookup matrix.

• pixel_values (torch.FloatTensor of shape (batch_size, num_channels, image_size, image_size), optional) -- The tensors corresponding to the input images. Pixel values can be obtained using Idefics3ImageProcessor. See Idefics3ImageProcessor.call() for details (Idefics3Processor uses Idefics3ImageProcessor for processing images).

• pixel_attention_mask (torch.Tensor of shape (batch_size, image_size, image_size), optional) -- Mask to avoid performing attention on padding pixel indices.

• image_hidden_states (torch.FloatTensor of shape (batch_size, num_channels, image_size, image_size)) -- The hidden states of the image encoder after modality projection.

• use_cache (bool, optional) -- If set to True, past_key_values key value states are returned and can be used to speed up decoding (see past_key_values).

• output_attentions (bool, optional) -- Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.

• output_hidden_states (bool, optional) -- Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.

• cache_position (torch.LongTensor of shape (sequence_length), optional) -- Indices depicting the position of the input sequence tokens in the sequence. Contrarily to position_ids, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length.

• return_dict (bool, optional) -- Whether or not to return a ModelOutput instead of a plain tuple.0transformers.models.idefics3.modeling_idefics3.Idefics3BaseModelOutputWithPast or tuple(torch.FloatTensor)A transformers.models.idefics3.modeling_idefics3.Idefics3BaseModelOutputWithPast or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (Idefics3Config) and inputs.

• last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) -- Sequence of hidden-states at the output of the last layer of the model. If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.

• past_key_values (Cache, optional, returned when use_cache=True is passed or when config.use_cache=True) -- It is a Cache instance. For more details, see our kv cache guide.

  Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if config.is_encoder_decoder=True in the cross-attention blocks) that can be used (see past_key_values input) to speed up sequential decoding.

• hidden_states (tuple[torch.FloatTensor], optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) -- Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

• attentions (tuple[torch.FloatTensor], optional, returned when output_attentions=True is passed or when config.output_attentions=True) -- Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

• image_hidden_states (tuple(torch.FloatTensor), optional) -- Tuple of torch.FloatTensor (one for the output of the image embeddings, (batch_size, num_images, sequence_length, hidden_size). image_hidden_states of the model produced by the vision encoder

Inputs fed to the model can have an arbitrary number of images. To account for this, pixel_values fed to the model have image padding -> (batch_size, max_num_images, 3, max_heights, max_widths) where max_num_images is the maximum number of images among the batch_size samples in the batch. Padding images are not needed beyond padding the pixel_values at the entrance of the model. For efficiency, we only pass through the vision_model's forward the real images by discarding the padding images i.e. pixel_values of size (image_batch_size, 3, height, width) where image_batch_size would be 7 when num_images_per_sample=[1, 3, 1, 2] and max_num_images would be 3.

Parameters:

config (Idefics3Config) : Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

Returns:

transformers.models.idefics3.modeling_idefics3.Idefics3BaseModelOutputWithPast` or `tuple(torch.FloatTensor)

A transformers.models.idefics3.modeling_idefics3.Idefics3BaseModelOutputWithPast or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (Idefics3Config) and inputs.

• last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) -- Sequence of hidden-states at the output of the last layer of the model. If past_key_values is used only the last hidden-state of the sequences of shape (batch_size, 1, hidden_size) is output.

• past_key_values (Cache, optional, returned when use_cache=True is passed or when config.use_cache=True) -- It is a Cache instance. For more details, see our kv cache guide.

  Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if config.is_encoder_decoder=True in the cross-attention blocks) that can be used (see past_key_values input) to speed up sequential decoding.

• hidden_states (tuple[torch.FloatTensor], optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) -- Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

• attentions (tuple[torch.FloatTensor], optional, returned when output_attentions=True is passed or when config.output_attentions=True) -- Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

• image_hidden_states (tuple(torch.FloatTensor), optional) -- Tuple of torch.FloatTensor (one for the output of the image embeddings, (batch_size, num_images, sequence_length, hidden_size). image_hidden_states of the model produced by the vision encoder

Idefics3ForConditionalGeneration[[transformers.Idefics3ForConditionalGeneration]]

transformers.Idefics3ForConditionalGeneration[[transformers.Idefics3ForConditionalGeneration]]

Source

The Idefics3 Model with a language modeling head. It is made up a SigLIP vision encoder, with a language modeling head on top.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forwardtransformers.Idefics3ForConditionalGeneration.forwardhttps://github.com/huggingface/transformers/blob/vr_37082/src/transformers/models/idefics3/modeling_idefics3.py#L819[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "position_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "past_key_values", "val": ": typing.Optional[transformers.cache_utils.Cache] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "pixel_attention_mask", "val": ": typing.Optional[torch.BoolTensor] = None"}, {"name": "image_hidden_states", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "labels", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "use_cache", "val": ": typing.Optional[bool] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "cache_position", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}, {"name": "logits_to_keep", "val": ": typing.Union[int, torch.Tensor] = 0"}, {"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.utils.generic.TransformersKwargs]"}]- input_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) -- Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

What are input IDs?

• attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) -- Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:

  • 1 for tokens that are not masked,
  • 0 for tokens that are masked.

  What are attention masks?

• position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) -- Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.n_positions - 1].

  What are position IDs?

• past_key_values (~cache_utils.Cache, optional) -- Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the past_key_values returned by the model at a previous stage of decoding, when use_cache=True or config.use_cache=True.

  Only Cache instance is allowed as input, see our kv cache guide. If no past_key_values are passed, DynamicCache will be initialized by default.

  The model will output the same cache format that is fed as input.

  If past_key_values are used, the user is expected to input only unprocessed input_ids (those that don't have their past key value states given to this model) of shape (batch_size, unprocessed_length) instead of all input_ids of shape (batch_size, sequence_length).

• inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) -- Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model's internal embedding lookup matrix.

• pixel_values (torch.FloatTensor of shape (batch_size, num_channels, image_size, image_size), optional) -- The tensors corresponding to the input images. Pixel values can be obtained using Idefics3ImageProcessor. See Idefics3ImageProcessor.call() for details (Idefics3Processor uses Idefics3ImageProcessor for processing images).

• pixel_attention_mask (torch.Tensor of shape (batch_size, image_size, image_size), optional) -- Mask to avoid performing attention on padding pixel indices.

• image_hidden_states (torch.FloatTensor of shape (batch_size, num_channels, image_size, image_size)) -- The hidden states of the image encoder after modality projection.

• labels (torch.LongTensor of shape (batch_size, sequence_length), optional) -- Labels for computing the masked language modeling loss. Indices should either be in [0, ..., config.vocab_size] or model.image_token_id (where model is your instance of Idefics3ForConditionalGeneration). Tokens with indices set to model.image_token_id are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size].

• use_cache (bool, optional) -- If set to True, past_key_values key value states are returned and can be used to speed up decoding (see past_key_values).

• output_attentions (bool, optional) -- Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.

• output_hidden_states (bool, optional) -- Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.

• cache_position (torch.LongTensor of shape (sequence_length), optional) -- Indices depicting the position of the input sequence tokens in the sequence. Contrarily to position_ids, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length.

• return_dict (bool, optional) -- Whether or not to return a ModelOutput instead of a plain tuple.

• logits_to_keep (Union[int, torch.Tensor], defaults to 0) -- If an int, compute logits for the last logits_to_keep tokens. If 0, calculate logits for all input_ids (special case). Only last token logits are needed for generation, and calculating them only for that token can save memory, which becomes pretty significant for long sequences or large vocabulary size. If a torch.Tensor, must be 1D corresponding to the indices to keep in the sequence length dimension. This is useful when using packed tensor format (single dimension for batch and sequence length).0transformers.models.idefics3.modeling_idefics3.Idefics3CausalLMOutputWithPast or tuple(torch.FloatTensor)A transformers.models.idefics3.modeling_idefics3.Idefics3CausalLMOutputWithPast or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (Idefics3Config) and inputs.

• loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) -- Language modeling loss (for next-token prediction).

• logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) -- Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

• past_key_values (Cache, optional, returned when use_cache=True is passed or when config.use_cache=True) -- It is a Cache instance. For more details, see our kv cache guide.

  Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see past_key_values input) to speed up sequential decoding.

• hidden_states (tuple[torch.FloatTensor], optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) -- Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

• attentions (tuple[torch.FloatTensor], optional, returned when output_attentions=True is passed or when config.output_attentions=True) -- Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

• image_hidden_states (tuple(torch.FloatTensor), optional) -- Tuple of torch.FloatTensor (one for the output of the image embeddings, (batch_size, num_images, sequence_length, hidden_size). image_hidden_states of the model produced by the vision encoder The Idefics3ForConditionalGeneration forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

>>> import requests
>>> import torch
>>> from PIL import Image
>>> from io import BytesIO

>>> from transformers import AutoProcessor, AutoModelForImageTextToText
>>> from transformers.image_utils import load_image

>>> # Note that passing the image urls (instead of the actual pil images) to the processor is also possible
>>> image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg")
>>> image2 = load_image("https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg")
>>> image3 = load_image("https://cdn.britannica.com/68/170868-050-8DDE8263/Golden-Gate-Bridge-San-Francisco.jpg")

>>> processor = AutoProcessor.from_pretrained("HuggingFaceM4/Idefics3-8B-Llama3")
>>> model = AutoModelForImageTextToText.from_pretrained("HuggingFaceM4/Idefics3-8B-Llama3", dtype=torch.bfloat16, device_map="auto")

>>> # Create inputs
>>> messages = [
...     {
...         "role": "user",
...         "content": [
...             {"type": "image"},
...             {"type": "text", "text": "In this image, we can see the city of New York, and more specifically the Statue of Liberty."},
...             {"type": "image"},
...             {"type": "text", "text": "What can we see in this image?"},
...         ]
...     },
...     {
...         "role": "user",
...         "content": [
...             {"type": "image"},
...             {"type": "text", "text": "In which city is that bridge located?"},
...         ]
...     }
... ]

>>> prompts = [processor.apply_chat_template([message], add_generation_prompt=True) for message in messages]
>>> images = [[image1, image2], [image3]]
>>> inputs = processor(text=prompts, images=images, padding=True, return_tensors="pt").to(model.device)

>>> # Generate
>>> generated_ids = model.generate(**inputs, max_new_tokens=256)
>>> generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)

>>> print(generated_texts[0])
Assistant: There are buildings, trees, lights, and water visible in this image.

>>> print(generated_texts[1])
Assistant: The bridge is in San Francisco.

Parameters:

config (Idefics3ForConditionalGeneration) : Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

Returns:

transformers.models.idefics3.modeling_idefics3.Idefics3CausalLMOutputWithPast` or `tuple(torch.FloatTensor)

A transformers.models.idefics3.modeling_idefics3.Idefics3CausalLMOutputWithPast or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (Idefics3Config) and inputs.

• loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) -- Language modeling loss (for next-token prediction).

• logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) -- Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

• past_key_values (Cache, optional, returned when use_cache=True is passed or when config.use_cache=True) -- It is a Cache instance. For more details, see our kv cache guide.

  Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see past_key_values input) to speed up sequential decoding.

• hidden_states (tuple[torch.FloatTensor], optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) -- Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

• attentions (tuple[torch.FloatTensor], optional, returned when output_attentions=True is passed or when config.output_attentions=True) -- Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

• image_hidden_states (tuple(torch.FloatTensor), optional) -- Tuple of torch.FloatTensor (one for the output of the image embeddings, (batch_size, num_images, sequence_length, hidden_size). image_hidden_states of the model produced by the vision encoder

Idefics3ImageProcessor[[transformers.Idefics3ImageProcessor]]

transformers.Idefics3ImageProcessor[[transformers.Idefics3ImageProcessor]]

Source

Constructs a Idefics3 image processor.

preprocesstransformers.Idefics3ImageProcessor.preprocesshttps://github.com/huggingface/transformers/blob/vr_37082/src/transformers/models/idefics3/image_processing_idefics3.py#L621[{"name": "images", "val": ": typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']]"}, {"name": "do_convert_rgb", "val": ": typing.Optional[bool] = None"}, {"name": "do_resize", "val": ": typing.Optional[bool] = None"}, {"name": "size", "val": ": typing.Optional[dict[str, int]] = None"}, {"name": "resample", "val": ": typing.Optional[PIL.Image.Resampling] = None"}, {"name": "do_image_splitting", "val": ": typing.Optional[bool] = None"}, {"name": "do_rescale", "val": ": typing.Optional[bool] = None"}, {"name": "max_image_size", "val": ": typing.Optional[dict[str, int]] = None"}, {"name": "rescale_factor", "val": ": typing.Optional[float] = None"}, {"name": "do_normalize", "val": ": typing.Optional[bool] = None"}, {"name": "image_mean", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "image_std", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "do_pad", "val": ": typing.Optional[bool] = None"}, {"name": "return_tensors", "val": ": typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None"}, {"name": "return_row_col_info", "val": ": bool = False"}, {"name": "data_format", "val": ": typing.Optional[transformers.image_utils.ChannelDimension] = "}, {"name": "input_data_format", "val": ": typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None"}]- images (ImageInput) -- A list of images to preprocess.

• do_convert_rgb (bool, optional, defaults to self.do_convert_rgb) -- Whether to convert the image to RGB.
• do_resize (bool, optional, defaults to self.do_resize) -- Whether to resize the image.
• size (dict[str, int], optional, defaults to self.size) -- Size of the image after resizing. With the longest edge resized to keep the input aspect ratio.
• resample (int, optional, defaults to self.resample) -- Resampling filter to use if resizing the image. This can be one of the enum PILImageResampling. Only has an effect if do_resize is set to True.
• do_image_splitting (bool, optional, defaults to self.do_image_splitting) -- Whether to split the image into sub-images concatenated with the original image. They are split into patches such that each patch has a size of max_image_size["height"] x max_image_size["width"].
• max_image_size (Dict, optional, defaults to self.max_image_size) -- Maximum resolution of the images. If the image is larger than this size, the image is split into patches.
• do_rescale (bool, optional, defaults to self.do_rescale) -- Whether to rescale the image.
• rescale_factor (float, optional, defaults to self.rescale_factor) -- Rescale factor to rescale the image by if do_rescale is set to True.
• do_normalize (bool, optional, defaults to self.do_normalize) -- Whether to normalize the image.
• image_mean (float or list[float], optional, defaults to self.image_mean) -- Image mean to use for normalization. Only has an effect if do_normalize is set to True.
• image_std (float or list[float], optional, defaults to self.image_std) -- Image standard deviation to use for normalization. Only has an effect if do_normalize is set to True.
• do_pad (bool, optional, defaults to self.do_pad) -- Whether or not to pad the images to the largest height and width in the batch.
• return_tensors (str or TensorType, optional) -- The type of tensors to return. Can be one of:
  • Unset: Return a list of np.ndarray.
  • TensorType.PYTORCH or 'pt': Return a batch of type torch.Tensor.
  • TensorType.NUMPY or 'np': Return a batch of type np.ndarray.
• return_row_col_info (bool, optional, default to False) -- Whether to return the number of rows and columns of the split images. This is used for the Idefics3Processor to generate prompt strings based on the number of rows and columns.
• data_format (ChannelDimension or str, optional, defaults to ChannelDimension.FIRST) -- The channel dimension format for the output image. Can be one of:
  • "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
  • "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
  • Unset: Use the channel dimension format of the input image.
• input_data_format (ChannelDimension or str, optional) -- The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of:
  • "channels_first" or ChannelDimension.FIRST: image in (num_channels, height, width) format.
  • "channels_last" or ChannelDimension.LAST: image in (height, width, num_channels) format.
  • "none" or ChannelDimension.NONE: image in (height, width) format.0

Preprocess a batch of images.

Parameters:

do_convert_rgb (bool, optional, defaults to True) : Whether to convert the image to RGB. This is useful if the input image is of a different format e.g. RGBA. Only has an effect if the input image is in the PIL format.

do_resize (bool, optional, defaults to True) : Whether to resize the image. The longest edge of the image is resized to be >> import requests

from transformers import Idefics3Processor from transformers.image_utils import load_image

processor = Idefics3Processor.from_pretrained("HuggingFaceM4/Idefics3-8B-Llama3") processor.image_processor.do_image_splitting = False # Force as False to simplify the example

url1 = "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" url2 = "https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg"

image1, image2 = load_image(url1), load_image(url2) images = [[image1], [image2]]

text = [ ... "In this image, we see", ... "bla bla bla", ... ] outputs = processor(images=images, text=text, return_tensors="pt", padding=True) input_ids = outputs.input_ids input_tokens = processor.tokenizer.batch_decode(input_ids) print(input_tokens) ['(()*169) In this image, we see', 'bla bla bla(()*169)']

**Parameters:**

image_processor (`Idefics3ImageProcessor`) : An instance of [Idefics3ImageProcessor](/docs/transformers/pr_37082/en/model_doc/idefics3#transformers.Idefics3ImageProcessor). The image processor is a required input.

tokenizer (`PreTrainedTokenizerBase`, *optional*) : An instance of [PreTrainedTokenizerBase](/docs/transformers/pr_37082/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase). This should correspond with the model's text model. The tokenizer is a required input.

image_seq_len (`int`, *optional*, defaults to 169) : The length of the image sequence i.e. the number of  tokens per image in the input. This parameter is used to build the string from the input prompt and image tokens and should match the value the model used. It is computed as: image_seq_len = int(((image_size // patch_size) ** 2) / (scale_factor**2))

chat_template (`str`, *optional*) : A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string.