Buckets:
| # ViLT | |
| <div class="flex flex-wrap space-x-1"> | |
| <img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white"> | |
| </div> | |
| ## Overview | |
| The ViLT model was proposed in [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://huggingface.co/papers/2102.03334) | |
| by Wonjae Kim, Bokyung Son, Ildoo Kim. ViLT incorporates text embeddings into a Vision Transformer (ViT), allowing it to have a minimal design | |
| for Vision-and-Language Pre-training (VLP). | |
| The abstract from the paper is the following: | |
| *Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. | |
| Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision | |
| (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we | |
| find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more | |
| computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive | |
| power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, | |
| Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically | |
| simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of | |
| times faster than previous VLP models, yet with competitive or better downstream task performance.* | |
| <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vilt_architecture.jpg" | |
| alt="drawing" width="600"/> | |
| <small> ViLT architecture. Taken from the <a href="https://huggingface.co/papers/2102.03334">original paper</a>. </small> | |
| This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/dandelin/ViLT). | |
| ## Usage tips | |
| - The quickest way to get started with ViLT is by checking the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ViLT) | |
| (which showcase both inference and fine-tuning on custom data). | |
| - ViLT is a model that takes both `pixel_values` and `input_ids` as input. One can use [ViltProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltProcessor) to prepare data for the model. | |
| This processor wraps a image processor (for the image modality) and a tokenizer (for the language modality) into one. | |
| - ViLT is trained with images of various sizes: the authors resize the shorter edge of input images to 384 and limit the longer edge to | |
| under 640 while preserving the aspect ratio. To make batching of images possible, the authors use a `pixel_mask` that indicates | |
| which pixel values are real and which are padding. [ViltProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltProcessor) automatically creates this for you. | |
| - The design of ViLT is very similar to that of a standard Vision Transformer (ViT). The only difference is that the model includes | |
| additional embedding layers for the language modality. | |
| - The PyTorch version of this model is only available in torch 1.10 and higher. | |
| ## ViltConfig[[transformers.ViltConfig]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.ViltConfig</name><anchor>transformers.ViltConfig</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/configuration_vilt.py#L24</source><parameters>[{"name": "vocab_size", "val": " = 30522"}, {"name": "type_vocab_size", "val": " = 2"}, {"name": "modality_type_vocab_size", "val": " = 2"}, {"name": "max_position_embeddings", "val": " = 40"}, {"name": "hidden_size", "val": " = 768"}, {"name": "num_hidden_layers", "val": " = 12"}, {"name": "num_attention_heads", "val": " = 12"}, {"name": "intermediate_size", "val": " = 3072"}, {"name": "hidden_act", "val": " = 'gelu'"}, {"name": "hidden_dropout_prob", "val": " = 0.0"}, {"name": "attention_probs_dropout_prob", "val": " = 0.0"}, {"name": "initializer_range", "val": " = 0.02"}, {"name": "layer_norm_eps", "val": " = 1e-12"}, {"name": "image_size", "val": " = 384"}, {"name": "patch_size", "val": " = 32"}, {"name": "num_channels", "val": " = 3"}, {"name": "qkv_bias", "val": " = True"}, {"name": "max_image_length", "val": " = -1"}, {"name": "tie_word_embeddings", "val": " = False"}, {"name": "num_images", "val": " = -1"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **vocab_size** (`int`, *optional*, defaults to 30522) -- | |
| Vocabulary size of the text part of the model. Defines the number of different tokens that can be | |
| represented by the `inputs_ids` passed when calling [ViltModel](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltModel). | |
| - **type_vocab_size** (`int`, *optional*, defaults to 2) -- | |
| The vocabulary size of the `token_type_ids` passed when calling [ViltModel](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltModel). This is used when encoding | |
| text. | |
| - **modality_type_vocab_size** (`int`, *optional*, defaults to 2) -- | |
| The vocabulary size of the modalities passed when calling [ViltModel](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltModel). This is used after concatenating the | |
| embeddings of the text and image modalities. | |
| - **max_position_embeddings** (`int`, *optional*, defaults to 40) -- | |
| The maximum sequence length that this model might ever be used with. | |
| - **hidden_size** (`int`, *optional*, defaults to 768) -- | |
| Dimensionality of the encoder layers and the pooler layer. | |
| - **num_hidden_layers** (`int`, *optional*, defaults to 12) -- | |
| Number of hidden layers in the Transformer encoder. | |
| - **num_attention_heads** (`int`, *optional*, defaults to 12) -- | |
| Number of attention heads for each attention layer in the Transformer encoder. | |
| - **intermediate_size** (`int`, *optional*, defaults to 3072) -- | |
| Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. | |
| - **hidden_act** (`str` or `function`, *optional*, defaults to `"gelu"`) -- | |
| The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, | |
| `"relu"`, `"selu"` and `"gelu_new"` are supported. | |
| - **hidden_dropout_prob** (`float`, *optional*, defaults to 0.0) -- | |
| The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. | |
| - **attention_probs_dropout_prob** (`float`, *optional*, defaults to 0.0) -- | |
| The dropout ratio for the attention probabilities. | |
| - **initializer_range** (`float`, *optional*, defaults to 0.02) -- | |
| The standard deviation of the truncated_normal_initializer for initializing all weight matrices. | |
| - **layer_norm_eps** (`float`, *optional*, defaults to 1e-12) -- | |
| The epsilon used by the layer normalization layers. | |
| - **image_size** (`int`, *optional*, defaults to 384) -- | |
| The size (resolution) of each image. | |
| - **patch_size** (`int`, *optional*, defaults to 32) -- | |
| The size (resolution) of each patch. | |
| - **num_channels** (`int`, *optional*, defaults to 3) -- | |
| The number of input channels. | |
| - **qkv_bias** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to add a bias to the queries, keys and values. | |
| - **max_image_length** (`int`, *optional*, defaults to -1) -- | |
| The maximum number of patches to take as input for the Transformer encoder. If set to a positive integer, | |
| the encoder will sample `max_image_length` patches at maximum. If set to -1, will not be taken into | |
| account. | |
| - **num_images** (`int`, *optional*, defaults to -1) -- | |
| The number of images to use for natural language visual reasoning. If set to a positive integer, will be | |
| used by [ViltForImagesAndTextClassification](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltForImagesAndTextClassification) for defining the classifier head.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| This is the configuration class to store the configuration of a `ViLTModel`. It is used to instantiate an ViLT | |
| model according to the specified arguments, defining the model architecture. Instantiating a configuration with the | |
| defaults will yield a similar configuration to that of the ViLT | |
| [dandelin/vilt-b32-mlm](https://huggingface.co/dandelin/vilt-b32-mlm) architecture. | |
| Configuration objects inherit from [PreTrainedConfig](/docs/transformers/pr_33962/en/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the | |
| documentation from [PreTrainedConfig](/docs/transformers/pr_33962/en/main_classes/configuration#transformers.PreTrainedConfig) for more information. | |
| <ExampleCodeBlock anchor="transformers.ViltConfig.example"> | |
| Example: | |
| ```python | |
| >>> from transformers import ViLTModel, ViLTConfig | |
| >>> # Initializing a ViLT dandelin/vilt-b32-mlm style configuration | |
| >>> configuration = ViLTConfig() | |
| >>> # Initializing a model from the dandelin/vilt-b32-mlm style configuration | |
| >>> model = ViLTModel(configuration) | |
| >>> # Accessing the model configuration | |
| >>> configuration = model.config | |
| ``` | |
| </ExampleCodeBlock> | |
| </div> | |
| ## ViltImageProcessor[[transformers.ViltImageProcessor]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.ViltImageProcessor</name><anchor>transformers.ViltImageProcessor</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/image_processing_vilt.py#L129</source><parameters>[{"name": "do_resize", "val": ": bool = True"}, {"name": "size", "val": ": typing.Optional[dict[str, int]] = None"}, {"name": "size_divisor", "val": ": int = 32"}, {"name": "resample", "val": ": Resampling = <Resampling.BICUBIC: 3>"}, {"name": "do_rescale", "val": ": bool = True"}, {"name": "rescale_factor", "val": ": typing.Union[int, float] = 0.00392156862745098"}, {"name": "do_normalize", "val": ": bool = True"}, {"name": "image_mean", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "image_std", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "do_pad", "val": ": bool = True"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **do_resize** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to resize the image's (height, width) dimensions to the specified `size`. Can be overridden by the | |
| `do_resize` parameter in the `preprocess` method. | |
| - **size** (`dict[str, int]` *optional*, defaults to `{"shortest_edge" -- 384}`): | |
| Resize the shorter side of the input to `size["shortest_edge"]`. The longer side will be limited to under | |
| `int((1333 / 800) * size["shortest_edge"])` while preserving the aspect ratio. Only has an effect if | |
| `do_resize` is set to `True`. Can be overridden by the `size` parameter in the `preprocess` method. | |
| - **size_divisor** (`int`, *optional*, defaults to 32) -- | |
| The size by which to make sure both the height and width can be divided. Only has an effect if `do_resize` | |
| is set to `True`. Can be overridden by the `size_divisor` parameter in the `preprocess` method. | |
| - **resample** (`PILImageResampling`, *optional*, defaults to `Resampling.BICUBIC`) -- | |
| Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`. Can be | |
| overridden by the `resample` parameter in the `preprocess` method. | |
| - **do_rescale** (`bool`, *optional*, defaults to `True`) -- | |
| Wwhether to rescale the image by the specified scale `rescale_factor`. Can be overridden by the | |
| `do_rescale` parameter in the `preprocess` method. | |
| - **rescale_factor** (`int` or `float`, *optional*, defaults to `1/255`) -- | |
| Scale factor to use if rescaling the image. Only has an effect if `do_rescale` is set to `True`. Can be | |
| overridden by the `rescale_factor` parameter in the `preprocess` method. | |
| - **do_normalize** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to normalize the image. Can be overridden by the `do_normalize` parameter in the `preprocess` | |
| method. Can be overridden by the `do_normalize` parameter in the `preprocess` method. | |
| - **image_mean** (`float` or `list[float]`, *optional*, defaults to `IMAGENET_STANDARD_MEAN`) -- | |
| Mean to use if normalizing the image. This is a float or list of floats the length of the number of | |
| channels in the image. Can be overridden by the `image_mean` parameter in the `preprocess` method. Can be | |
| overridden by the `image_mean` parameter in the `preprocess` method. | |
| - **image_std** (`float` or `list[float]`, *optional*, defaults to `IMAGENET_STANDARD_STD`) -- | |
| Standard deviation to use if normalizing the image. This is a float or list of floats the length of the | |
| number of channels in the image. Can be overridden by the `image_std` parameter in the `preprocess` method. | |
| Can be overridden by the `image_std` parameter in the `preprocess` method. | |
| - **do_pad** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to pad the image to the `(max_height, max_width)` of the images in the batch. Can be overridden by | |
| the `do_pad` parameter in the `preprocess` method.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| Constructs a ViLT image processor. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>preprocess</name><anchor>transformers.ViltImageProcessor.preprocess</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/image_processing_vilt.py#L344</source><parameters>[{"name": "images", "val": ": typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']]"}, {"name": "do_resize", "val": ": typing.Optional[bool] = None"}, {"name": "size", "val": ": typing.Optional[dict[str, int]] = None"}, {"name": "size_divisor", "val": ": typing.Optional[int] = None"}, {"name": "resample", "val": ": typing.Optional[PIL.Image.Resampling] = None"}, {"name": "do_rescale", "val": ": typing.Optional[bool] = None"}, {"name": "rescale_factor", "val": ": typing.Optional[float] = None"}, {"name": "do_normalize", "val": ": typing.Optional[bool] = None"}, {"name": "image_mean", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "image_std", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "do_pad", "val": ": typing.Optional[bool] = None"}, {"name": "return_tensors", "val": ": typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None"}, {"name": "data_format", "val": ": ChannelDimension = <ChannelDimension.FIRST: 'channels_first'>"}, {"name": "input_data_format", "val": ": typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None"}]</parameters><paramsdesc>- **images** (`ImageInput`) -- | |
| Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If | |
| passing in images with pixel values between 0 and 1, set `do_rescale=False`. | |
| - **do_resize** (`bool`, *optional*, defaults to `self.do_resize`) -- | |
| Whether to resize the image. | |
| - **size** (`dict[str, int]`, *optional*, defaults to `self.size`) -- | |
| Controls the size of the image after `resize`. The shortest edge of the image is resized to | |
| `size["shortest_edge"]` whilst preserving the aspect ratio. If the longest edge of this resized image | |
| is > `int(size["shortest_edge"] * (1333 / 800))`, then the image is resized again to make the longest | |
| edge equal to `int(size["shortest_edge"] * (1333 / 800))`. | |
| - **size_divisor** (`int`, *optional*, defaults to `self.size_divisor`) -- | |
| The image is resized to a size that is a multiple of this value. | |
| - **resample** (`PILImageResampling`, *optional*, defaults to `self.resample`) -- | |
| Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`. | |
| - **do_rescale** (`bool`, *optional*, defaults to `self.do_rescale`) -- | |
| Whether to rescale the image values between [0 - 1]. | |
| - **rescale_factor** (`float`, *optional*, defaults to `self.rescale_factor`) -- | |
| Rescale factor to rescale the image by if `do_rescale` is set to `True`. | |
| - **do_normalize** (`bool`, *optional*, defaults to `self.do_normalize`) -- | |
| Whether to normalize the image. | |
| - **image_mean** (`float` or `list[float]`, *optional*, defaults to `self.image_mean`) -- | |
| Image mean to normalize the image by if `do_normalize` is set to `True`. | |
| - **image_std** (`float` or `list[float]`, *optional*, defaults to `self.image_std`) -- | |
| Image standard deviation to normalize the image by if `do_normalize` is set to `True`. | |
| - **do_pad** (`bool`, *optional*, defaults to `self.do_pad`) -- | |
| Whether to pad the image to the (max_height, max_width) in the batch. If `True`, a pixel mask is also | |
| created and returned. | |
| - **return_tensors** (`str` or `TensorType`, *optional*) -- | |
| The type of tensors to return. Can be one of: | |
| - Unset: Return a list of `np.ndarray`. | |
| - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. | |
| - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. | |
| - **data_format** (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`) -- | |
| The channel dimension format for the output image. Can be one of: | |
| - `ChannelDimension.FIRST`: image in (num_channels, height, width) format. | |
| - `ChannelDimension.LAST`: image in (height, width, num_channels) format. | |
| - **input_data_format** (`ChannelDimension` or `str`, *optional*) -- | |
| The channel dimension format for the input image. If unset, the channel dimension format is inferred | |
| from the input image. Can be one of: | |
| - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. | |
| - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. | |
| - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| Preprocess an image or batch of images. | |
| </div></div> | |
| ## ViltImageProcessorFast[[transformers.ViltImageProcessorFast]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.ViltImageProcessorFast</name><anchor>transformers.ViltImageProcessorFast</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/image_processing_vilt_fast.py#L43</source><parameters>[{"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.processing_utils.ImagesKwargs]"}]</parameters></docstring> | |
| Constructs a fast Vilt image processor. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>preprocess</name><anchor>transformers.ViltImageProcessorFast.preprocess</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/image_processing_utils_fast.py#L710</source><parameters>[{"name": "images", "val": ": typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']]"}, {"name": "*args", "val": ""}, {"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.processing_utils.ImagesKwargs]"}]</parameters><paramsdesc>- **images** (`Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']]`) -- | |
| Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If | |
| passing in images with pixel values between 0 and 1, set `do_rescale=False`. | |
| - **do_convert_rgb** (`bool`, *optional*) -- | |
| Whether to convert the image to RGB. | |
| - **do_resize** (`bool`, *optional*) -- | |
| Whether to resize the image. | |
| - **size** (`Annotated[Union[int, list[int], tuple[int, ...], dict[str, int], NoneType], None]`) -- | |
| Describes the maximum input dimensions to the model. | |
| - **crop_size** (`Annotated[Union[int, list[int], tuple[int, ...], dict[str, int], NoneType], None]`) -- | |
| Size of the output image after applying `center_crop`. | |
| - **resample** (`Annotated[Union[PILImageResampling, int, NoneType], None]`) -- | |
| Resampling filter to use if resizing the image. This can be one of the enum `PILImageResampling`. Only | |
| has an effect if `do_resize` is set to `True`. | |
| - **do_rescale** (`bool`, *optional*) -- | |
| Whether to rescale the image. | |
| - **rescale_factor** (`float`, *optional*) -- | |
| Rescale factor to rescale the image by if `do_rescale` is set to `True`. | |
| - **do_normalize** (`bool`, *optional*) -- | |
| Whether to normalize the image. | |
| - **image_mean** (`Union[float, list[float], tuple[float, ...], NoneType]`) -- | |
| Image mean to use for normalization. Only has an effect if `do_normalize` is set to `True`. | |
| - **image_std** (`Union[float, list[float], tuple[float, ...], NoneType]`) -- | |
| Image standard deviation to use for normalization. Only has an effect if `do_normalize` is set to | |
| `True`. | |
| - **do_pad** (`bool`, *optional*) -- | |
| Whether to pad the image. Padding is done either to the largest size in the batch | |
| or to a fixed square size per image. The exact padding strategy depends on the model. | |
| - **pad_size** (`Annotated[Union[int, list[int], tuple[int, ...], dict[str, int], NoneType], None]`) -- | |
| The size in `{"height": int, "width" int}` to pad the images to. Must be larger than any image size | |
| provided for preprocessing. If `pad_size` is not provided, images will be padded to the largest | |
| height and width in the batch. Applied only when `do_pad=True.` | |
| - **do_center_crop** (`bool`, *optional*) -- | |
| Whether to center crop the image. | |
| - **data_format** (`Union[str, ~image_utils.ChannelDimension, NoneType]`) -- | |
| Only `ChannelDimension.FIRST` is supported. Added for compatibility with slow processors. | |
| - **input_data_format** (`Union[str, ~image_utils.ChannelDimension, NoneType]`) -- | |
| The channel dimension format for the input image. If unset, the channel dimension format is inferred | |
| from the input image. Can be one of: | |
| - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. | |
| - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. | |
| - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. | |
| - **device** (`Annotated[str, None]`, *optional*) -- | |
| The device to process the images on. If unset, the device is inferred from the input images. | |
| - **return_tensors** (`Annotated[Union[str, ~utils.generic.TensorType, NoneType], None]`) -- | |
| Returns stacked tensors if set to `pt, otherwise returns a list of tensors. | |
| - **disable_grouping** (`bool`, *optional*) -- | |
| Whether to disable grouping of images by size to process them individually and not in batches. | |
| If None, will be set to True if the images are on CPU, and False otherwise. This choice is based on | |
| empirical observations, as detailed here: https://github.com/huggingface/transformers/pull/38157</paramsdesc><paramgroups>0</paramgroups><rettype>`<class 'transformers.image_processing_base.BatchFeature'>`</rettype><retdesc>- **data** (`dict`) -- Dictionary of lists/arrays/tensors returned by the __call__ method ('pixel_values', etc.). | |
| - **tensor_type** (`Union[None, str, TensorType]`, *optional*) -- You can give a tensor_type here to convert the lists of integers in PyTorch/Numpy Tensors at | |
| initialization.</retdesc></docstring> | |
| </div></div> | |
| ## ViltProcessor[[transformers.ViltProcessor]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.ViltProcessor</name><anchor>transformers.ViltProcessor</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/processing_vilt.py#L37</source><parameters>[{"name": "image_processor", "val": " = None"}, {"name": "tokenizer", "val": " = None"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **image_processor** (`ViltImageProcessor`, *optional*) -- | |
| An instance of [ViltImageProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltImageProcessor). The image processor is a required input. | |
| - **tokenizer** (`BertTokenizerFast`, *optional*) -- | |
| An instance of ['BertTokenizerFast`]. The tokenizer is a required input.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| Constructs a ViLT processor which wraps a BERT tokenizer and ViLT image processor into a single processor. | |
| [ViltProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltProcessor) offers all the functionalities of [ViltImageProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltImageProcessor) and [BertTokenizerFast](/docs/transformers/pr_33962/en/model_doc/bert#transformers.BertTokenizerFast). See the | |
| docstring of [__call__()](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltProcessor.__call__) and [decode()](/docs/transformers/pr_33962/en/main_classes/processors#transformers.ProcessorMixin.decode) for more information. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>__call__</name><anchor>transformers.ViltProcessor.__call__</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/processing_utils.py#L574</source><parameters>[{"name": "images", "val": ": typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor'], NoneType] = None"}, {"name": "text", "val": ": typing.Union[str, list[str], list[list[str]], NoneType] = None"}, {"name": "videos", "val": ": typing.Union[list['PIL.Image.Image'], numpy.ndarray, ForwardRef('torch.Tensor'), list[numpy.ndarray], list['torch.Tensor'], list[list['PIL.Image.Image']], list[list[numpy.ndarray]], list[list['torch.Tensor']], transformers.video_utils.URL, list[transformers.video_utils.URL], list[list[transformers.video_utils.URL]], transformers.video_utils.Path, list[transformers.video_utils.Path], list[list[transformers.video_utils.Path]], NoneType] = None"}, {"name": "audio", "val": ": typing.Union[numpy.ndarray, ForwardRef('torch.Tensor'), collections.abc.Sequence[numpy.ndarray], collections.abc.Sequence['torch.Tensor'], NoneType] = None"}, {"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.processing_utils.ProcessingKwargs]"}]</parameters><paramsdesc>- **images** (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`) -- | |
| The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch | |
| tensor. Both channels-first and channels-last formats are supported. | |
| - **text** (`TextInput`, `PreTokenizedInput`, `list[TextInput]`, `list[PreTokenizedInput]`, *optional*) -- | |
| The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings | |
| (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set | |
| `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). | |
| - **videos** (`np.ndarray`, `torch.Tensor`, `List[np.ndarray]`, `List[torch.Tensor]`) -- | |
| The video or batch of videos to be prepared. Each video can be a 4D NumPy array or PyTorch | |
| tensor, or a nested list of 3D frames. Both channels-first and channels-last formats are supported. | |
| - **audio** (`np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`) -- | |
| The audio or batch of audio to be prepared. Each audio can be a NumPy array or PyTorch | |
| tensor. | |
| - **return_tensors** (`str` or [TensorType](/docs/transformers/pr_33962/en/internal/file_utils#transformers.TensorType), *optional*) -- | |
| If set, will return tensors of a particular framework. Acceptable values are: | |
| - `'pt'`: Return PyTorch `torch.Tensor` objects. | |
| - `'np'`: Return NumPy `np.ndarray` objects.</paramsdesc><paramgroups>0</paramgroups><rettype>[BatchFeature](/docs/transformers/pr_33962/en/main_classes/image_processor#transformers.BatchFeature)</rettype><retdesc>A [BatchFeature](/docs/transformers/pr_33962/en/main_classes/image_processor#transformers.BatchFeature) object with processed inputs in a dict format.</retdesc></docstring> | |
| Main method to prepare for model inputs. This method forwards the each modality argument to its own processor | |
| along with `kwargs`. Please refer to the docstring of the each processor attributes for more information. | |
| </div></div> | |
| ## ViltModel[[transformers.ViltModel]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.ViltModel</name><anchor>transformers.ViltModel</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/modeling_vilt.py#L534</source><parameters>[{"name": "config", "val": ""}, {"name": "add_pooling_layer", "val": " = True"}]</parameters><paramsdesc>- **config** ([ViltModel](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltModel)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. | |
| - **add_pooling_layer** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to add a pooling layer</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| The bare Vilt Model outputting raw hidden-states without any specific head on top. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.ViltModel.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/modeling_vilt.py#L558</source><parameters>[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "token_type_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "pixel_mask", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "image_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "image_token_type_idx", "val": ": typing.Optional[int] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **token_type_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: | |
| - 0 corresponds to a *sentence A* token, | |
| - 1 corresponds to a *sentence B* token. | |
| [What are token type IDs?](../glossary#token-type-ids) | |
| - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [ViltImageProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltImageProcessor). See [ViltImageProcessor.__call__()](/docs/transformers/pr_33962/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([ViltProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltProcessor) uses | |
| [ViltImageProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltImageProcessor) for processing images). | |
| - **pixel_mask** (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*) -- | |
| Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: | |
| - 1 for pixels that are real (i.e. **not masked**), | |
| - 0 for pixels that are padding (i.e. **masked**). | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This | |
| is useful if you want more control over how to convert `input_ids` indices into associated vectors than the | |
| model's internal embedding lookup matrix. | |
| - **image_embeds** (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. | |
| This is useful if you want more control over how to convert `pixel_values` into patch embeddings. | |
| - **image_token_type_idx** (`int`, *optional*) -- | |
| - The token type ids for images. | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*) -- | |
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | |
| more detail. | |
| - **return_dict** (`bool`, *optional*) -- | |
| Whether or not to return a [ModelOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple.</paramsdesc><paramgroups>0</paramgroups><rettype>[transformers.modeling_outputs.BaseModelOutputWithPooling](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or `tuple(torch.FloatTensor)`</rettype><retdesc>A [transformers.modeling_outputs.BaseModelOutputWithPooling](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([ViltConfig](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltConfig)) and inputs. | |
| - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model. | |
| - **pooler_output** (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) -- Last layer hidden-state of the first token of the sequence (classification token) after further processing | |
| through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns | |
| the classification token after processing through a linear layer and a tanh activation function. The linear | |
| layer weights are trained from the next sentence prediction (classification) objective during pretraining. | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads.</retdesc></docstring> | |
| The [ViltModel](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltModel) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.ViltModel.forward.example"> | |
| Examples: | |
| ```python | |
| >>> from transformers import ViltProcessor, ViltModel | |
| >>> from PIL import Image | |
| >>> import requests | |
| >>> # prepare image and text | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> image = Image.open(requests.get(url, stream=True).raw) | |
| >>> text = "hello world" | |
| >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") | |
| >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") | |
| >>> inputs = processor(image, text, return_tensors="pt") | |
| >>> outputs = model(**inputs) | |
| >>> last_hidden_states = outputs.last_hidden_state | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| ## ViltForMaskedLM[[transformers.ViltForMaskedLM]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.ViltForMaskedLM</name><anchor>transformers.ViltForMaskedLM</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/modeling_vilt.py#L689</source><parameters>[{"name": "config", "val": ""}]</parameters><paramsdesc>- **config** ([ViltForMaskedLM](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltForMaskedLM)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| ViLT Model with a language modeling head on top as done during pretraining. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.ViltForMaskedLM.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/modeling_vilt.py#L708</source><parameters>[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "token_type_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "pixel_mask", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "image_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "labels", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **token_type_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: | |
| - 0 corresponds to a *sentence A* token, | |
| - 1 corresponds to a *sentence B* token. | |
| [What are token type IDs?](../glossary#token-type-ids) | |
| - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [ViltImageProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltImageProcessor). See [ViltImageProcessor.__call__()](/docs/transformers/pr_33962/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([ViltProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltProcessor) uses | |
| [ViltImageProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltImageProcessor) for processing images). | |
| - **pixel_mask** (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*) -- | |
| Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: | |
| - 1 for pixels that are real (i.e. **not masked**), | |
| - 0 for pixels that are padding (i.e. **masked**). | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This | |
| is useful if you want more control over how to convert `input_ids` indices into associated vectors than the | |
| model's internal embedding lookup matrix. | |
| - **image_embeds** (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. | |
| This is useful if you want more control over how to convert `pixel_values` into patch embeddings. | |
| - **labels** (`*torch.LongTensor*` of shape *(batch_size, sequence_length)*, *optional*) -- | |
| Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., | |
| config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the | |
| loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*) -- | |
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | |
| more detail. | |
| - **return_dict** (`bool`, *optional*) -- | |
| Whether or not to return a [ModelOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple.</paramsdesc><paramgroups>0</paramgroups><rettype>[transformers.modeling_outputs.MaskedLMOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.MaskedLMOutput) or `tuple(torch.FloatTensor)`</rettype><retdesc>A [transformers.modeling_outputs.MaskedLMOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.MaskedLMOutput) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([ViltConfig](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltConfig)) and inputs. | |
| - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Masked language modeling (MLM) loss. | |
| - **logits** (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) -- Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads.</retdesc></docstring> | |
| The [ViltForMaskedLM](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltForMaskedLM) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.ViltForMaskedLM.forward.example"> | |
| Examples: | |
| ```python | |
| >>> from transformers import ViltProcessor, ViltForMaskedLM | |
| >>> import requests | |
| >>> from PIL import Image | |
| >>> import re | |
| >>> import torch | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> image = Image.open(requests.get(url, stream=True).raw) | |
| >>> text = "a bunch of [MASK] laying on a [MASK]." | |
| >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") | |
| >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") | |
| >>> # prepare inputs | |
| >>> encoding = processor(image, text, return_tensors="pt") | |
| >>> # forward pass | |
| >>> outputs = model(**encoding) | |
| >>> tl = len(re.findall("\[MASK\]", text)) | |
| >>> inferred_token = [text] | |
| >>> # gradually fill in the MASK tokens, one by one | |
| >>> with torch.no_grad(): | |
| ... for i in range(tl): | |
| ... encoded = processor.tokenizer(inferred_token) | |
| ... input_ids = torch.tensor(encoded.input_ids) | |
| ... encoded = encoded["input_ids"][0][1:-1] | |
| ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) | |
| ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) | |
| ... # only take into account text features (minus CLS and SEP token) | |
| ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] | |
| ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) | |
| ... # only take into account text | |
| ... mlm_values[torch.tensor(encoded) != 103] = 0 | |
| ... select = mlm_values.argmax().item() | |
| ... encoded[select] = mlm_ids[select].item() | |
| ... inferred_token = [processor.decode(encoded)] | |
| >>> selected_token = "" | |
| >>> encoded = processor.tokenizer(inferred_token) | |
| >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) | |
| >>> print(output) | |
| a bunch of cats laying on a couch. | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| ## ViltForQuestionAnswering[[transformers.ViltForQuestionAnswering]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.ViltForQuestionAnswering</name><anchor>transformers.ViltForQuestionAnswering</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/modeling_vilt.py#L866</source><parameters>[{"name": "config", "val": ""}]</parameters><paramsdesc>- **config** ([ViltForQuestionAnswering](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltForQuestionAnswering)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] | |
| token) for visual question answering, e.g. for VQAv2. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.ViltForQuestionAnswering.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/modeling_vilt.py#L884</source><parameters>[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "token_type_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "pixel_mask", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "image_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "labels", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **token_type_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: | |
| - 0 corresponds to a *sentence A* token, | |
| - 1 corresponds to a *sentence B* token. | |
| [What are token type IDs?](../glossary#token-type-ids) | |
| - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [ViltImageProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltImageProcessor). See [ViltImageProcessor.__call__()](/docs/transformers/pr_33962/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([ViltProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltProcessor) uses | |
| [ViltImageProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltImageProcessor) for processing images). | |
| - **pixel_mask** (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*) -- | |
| Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: | |
| - 1 for pixels that are real (i.e. **not masked**), | |
| - 0 for pixels that are padding (i.e. **masked**). | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This | |
| is useful if you want more control over how to convert `input_ids` indices into associated vectors than the | |
| model's internal embedding lookup matrix. | |
| - **image_embeds** (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. | |
| This is useful if you want more control over how to convert `pixel_values` into patch embeddings. | |
| - **labels** (`torch.FloatTensor` of shape `(batch_size, num_labels)`, *optional*) -- | |
| Labels for computing the visual question answering loss. This tensor must be either a one-hot encoding of | |
| all answers that are applicable for a given example in the batch, or a soft encoding indicating which | |
| answers are applicable, where 1.0 is the highest score. | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*) -- | |
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | |
| more detail. | |
| - **return_dict** (`bool`, *optional*) -- | |
| Whether or not to return a [ModelOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple.</paramsdesc><paramgroups>0</paramgroups><rettype>[transformers.modeling_outputs.SequenceClassifierOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.SequenceClassifierOutput) or `tuple(torch.FloatTensor)`</rettype><retdesc>A [transformers.modeling_outputs.SequenceClassifierOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.SequenceClassifierOutput) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([ViltConfig](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltConfig)) and inputs. | |
| - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Classification (or regression if config.num_labels==1) loss. | |
| - **logits** (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`) -- Classification (or regression if config.num_labels==1) scores (before SoftMax). | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads.</retdesc></docstring> | |
| The [ViltForQuestionAnswering](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltForQuestionAnswering) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.ViltForQuestionAnswering.forward.example"> | |
| Examples: | |
| ```python | |
| >>> from transformers import ViltProcessor, ViltForQuestionAnswering | |
| >>> import requests | |
| >>> from PIL import Image | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> image = Image.open(requests.get(url, stream=True).raw) | |
| >>> text = "How many cats are there?" | |
| >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") | |
| >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") | |
| >>> # prepare inputs | |
| >>> encoding = processor(image, text, return_tensors="pt") | |
| >>> # forward pass | |
| >>> outputs = model(**encoding) | |
| >>> logits = outputs.logits | |
| >>> idx = logits.argmax(-1).item() | |
| >>> print("Predicted answer:", model.config.id2label[idx]) | |
| Predicted answer: 2 | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| ## ViltForImagesAndTextClassification[[transformers.ViltForImagesAndTextClassification]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.ViltForImagesAndTextClassification</name><anchor>transformers.ViltForImagesAndTextClassification</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/modeling_vilt.py#L1072</source><parameters>[{"name": "config", "val": ""}]</parameters><paramsdesc>- **config** ([ViltForImagesAndTextClassification](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltForImagesAndTextClassification)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| Vilt Model transformer with a classifier head on top for natural language visual reasoning, e.g. NLVR2. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.ViltForImagesAndTextClassification.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/modeling_vilt.py#L1091</source><parameters>[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "token_type_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "pixel_mask", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "image_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "labels", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **token_type_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: | |
| - 0 corresponds to a *sentence A* token, | |
| - 1 corresponds to a *sentence B* token. | |
| [What are token type IDs?](../glossary#token-type-ids) | |
| - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [ViltImageProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltImageProcessor). See [ViltImageProcessor.__call__()](/docs/transformers/pr_33962/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([ViltProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltProcessor) uses | |
| [ViltImageProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltImageProcessor) for processing images). | |
| - **pixel_mask** (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*) -- | |
| Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: | |
| - 1 for pixels that are real (i.e. **not masked**), | |
| - 0 for pixels that are padding (i.e. **masked**). | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This | |
| is useful if you want more control over how to convert `input_ids` indices into associated vectors than the | |
| model's internal embedding lookup matrix. | |
| - **image_embeds** (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. | |
| This is useful if you want more control over how to convert `pixel_values` into patch embeddings. | |
| - **labels** (`torch.LongTensor` of shape `(batch_size,)`, *optional*) -- | |
| Binary classification labels. | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*) -- | |
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | |
| more detail. | |
| - **return_dict** (`bool`, *optional*) -- | |
| Whether or not to return a [ModelOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple.</paramsdesc><paramgroups>0</paramgroups><rettype>`transformers.models.vilt.modeling_vilt.ViltForImagesAndTextClassificationOutput` or `tuple(torch.FloatTensor)`</rettype><retdesc>A `transformers.models.vilt.modeling_vilt.ViltForImagesAndTextClassificationOutput` or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([ViltConfig](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltConfig)) and inputs. | |
| - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Classification (or regression if config.num_labels==1) loss. | |
| - **logits** (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`) -- Classification (or regression if config.num_labels==1) scores (before SoftMax). | |
| - **hidden_states** (`list[tuple(torch.FloatTensor)]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the output of | |
| the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the initial embedding outputs. | |
| - **attentions** (`list[tuple[torch.FloatTensor]]`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads.</retdesc></docstring> | |
| The [ViltForImagesAndTextClassification](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltForImagesAndTextClassification) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.ViltForImagesAndTextClassification.forward.example"> | |
| Examples: | |
| ```python | |
| >>> from transformers import ViltProcessor, ViltForImagesAndTextClassification | |
| >>> import requests | |
| >>> from PIL import Image | |
| >>> image1 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_0.jpg", stream=True).raw) | |
| >>> image2 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_1.jpg", stream=True).raw) | |
| >>> text = "The left image contains twice the number of dogs as the right image." | |
| >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") | |
| >>> model = ViltForImagesAndTextClassification.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") | |
| >>> # prepare inputs | |
| >>> encoding = processor([image1, image2], text, return_tensors="pt") | |
| >>> # forward pass | |
| >>> outputs = model(input_ids=encoding.input_ids, pixel_values=encoding.pixel_values.unsqueeze(0)) | |
| >>> logits = outputs.logits | |
| >>> idx = logits.argmax(-1).item() | |
| >>> print("Predicted answer:", model.config.id2label[idx]) | |
| Predicted answer: True | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| ## ViltForImageAndTextRetrieval[[transformers.ViltForImageAndTextRetrieval]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.ViltForImageAndTextRetrieval</name><anchor>transformers.ViltForImageAndTextRetrieval</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/modeling_vilt.py#L976</source><parameters>[{"name": "config", "val": ""}]</parameters><paramsdesc>- **config** ([ViltForImageAndTextRetrieval](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltForImageAndTextRetrieval)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] | |
| token) for image-to-text or text-to-image retrieval, e.g. MSCOCO and F30K. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.ViltForImageAndTextRetrieval.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/modeling_vilt.py#L988</source><parameters>[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "token_type_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "pixel_mask", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "image_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "labels", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **token_type_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: | |
| - 0 corresponds to a *sentence A* token, | |
| - 1 corresponds to a *sentence B* token. | |
| [What are token type IDs?](../glossary#token-type-ids) | |
| - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [ViltImageProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltImageProcessor). See [ViltImageProcessor.__call__()](/docs/transformers/pr_33962/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([ViltProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltProcessor) uses | |
| [ViltImageProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltImageProcessor) for processing images). | |
| - **pixel_mask** (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*) -- | |
| Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: | |
| - 1 for pixels that are real (i.e. **not masked**), | |
| - 0 for pixels that are padding (i.e. **masked**). | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This | |
| is useful if you want more control over how to convert `input_ids` indices into associated vectors than the | |
| model's internal embedding lookup matrix. | |
| - **image_embeds** (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. | |
| This is useful if you want more control over how to convert `pixel_values` into patch embeddings. | |
| - **labels** (`torch.LongTensor` of shape `(batch_size,)`, *optional*) -- | |
| Labels are currently not supported. | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*) -- | |
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | |
| more detail. | |
| - **return_dict** (`bool`, *optional*) -- | |
| Whether or not to return a [ModelOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple.</paramsdesc><paramgroups>0</paramgroups><rettype>[transformers.modeling_outputs.SequenceClassifierOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.SequenceClassifierOutput) or `tuple(torch.FloatTensor)`</rettype><retdesc>A [transformers.modeling_outputs.SequenceClassifierOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.SequenceClassifierOutput) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([ViltConfig](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltConfig)) and inputs. | |
| - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Classification (or regression if config.num_labels==1) loss. | |
| - **logits** (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`) -- Classification (or regression if config.num_labels==1) scores (before SoftMax). | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads.</retdesc></docstring> | |
| The [ViltForImageAndTextRetrieval](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltForImageAndTextRetrieval) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.ViltForImageAndTextRetrieval.forward.example"> | |
| Examples: | |
| ```python | |
| >>> from transformers import ViltProcessor, ViltForImageAndTextRetrieval | |
| >>> import requests | |
| >>> from PIL import Image | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> image = Image.open(requests.get(url, stream=True).raw) | |
| >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] | |
| >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-coco") | |
| >>> model = ViltForImageAndTextRetrieval.from_pretrained("dandelin/vilt-b32-finetuned-coco") | |
| >>> # forward pass | |
| >>> scores = dict() | |
| >>> for text in texts: | |
| ... # prepare inputs | |
| ... encoding = processor(image, text, return_tensors="pt") | |
| ... outputs = model(**encoding) | |
| ... scores[text] = outputs.logits[0, :].item() | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| ## ViltForTokenClassification[[transformers.ViltForTokenClassification]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.ViltForTokenClassification</name><anchor>transformers.ViltForTokenClassification</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/modeling_vilt.py#L1206</source><parameters>[{"name": "config", "val": ""}]</parameters><paramsdesc>- **config** ([ViltForTokenClassification](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltForTokenClassification)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| The Vilt transformer with a token classification head on top (a linear layer on top of the hidden-states | |
| output) e.g. for Named-Entity-Recognition (NER) tasks. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.ViltForTokenClassification.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/vilt/modeling_vilt.py#L1219</source><parameters>[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "token_type_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "pixel_mask", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "image_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "labels", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **token_type_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: | |
| - 0 corresponds to a *sentence A* token, | |
| - 1 corresponds to a *sentence B* token. | |
| [What are token type IDs?](../glossary#token-type-ids) | |
| - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [ViltImageProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltImageProcessor). See [ViltImageProcessor.__call__()](/docs/transformers/pr_33962/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([ViltProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltProcessor) uses | |
| [ViltImageProcessor](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltImageProcessor) for processing images). | |
| - **pixel_mask** (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*) -- | |
| Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: | |
| - 1 for pixels that are real (i.e. **not masked**), | |
| - 0 for pixels that are padding (i.e. **masked**). | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This | |
| is useful if you want more control over how to convert `input_ids` indices into associated vectors than the | |
| model's internal embedding lookup matrix. | |
| - **image_embeds** (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. | |
| This is useful if you want more control over how to convert `pixel_values` into patch embeddings. | |
| - **labels** (`torch.LongTensor` of shape `(batch_size, text_sequence_length)`, *optional*) -- | |
| Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*) -- | |
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | |
| more detail. | |
| - **return_dict** (`bool`, *optional*) -- | |
| Whether or not to return a [ModelOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple.</paramsdesc><paramgroups>0</paramgroups><rettype>[transformers.modeling_outputs.TokenClassifierOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.TokenClassifierOutput) or `tuple(torch.FloatTensor)`</rettype><retdesc>A [transformers.modeling_outputs.TokenClassifierOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.TokenClassifierOutput) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([ViltConfig](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltConfig)) and inputs. | |
| - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Classification loss. | |
| - **logits** (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_labels)`) -- Classification scores (before SoftMax). | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads.</retdesc></docstring> | |
| The [ViltForTokenClassification](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltForTokenClassification) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.ViltForTokenClassification.forward.example"> | |
| Example: | |
| ```python | |
| >>> from transformers import AutoTokenizer, ViltForTokenClassification | |
| >>> import torch | |
| >>> tokenizer = AutoTokenizer.from_pretrained("dandelin/vilt-b32-mlm") | |
| >>> model = ViltForTokenClassification.from_pretrained("dandelin/vilt-b32-mlm") | |
| >>> inputs = tokenizer( | |
| ... "HuggingFace is a company based in Paris and New York", add_special_tokens=False, return_tensors="pt" | |
| ... ) | |
| >>> with torch.no_grad(): | |
| ... logits = model(**inputs).logits | |
| >>> predicted_token_class_ids = logits.argmax(-1) | |
| >>> # Note that tokens are classified rather then input words which means that | |
| >>> # there might be more predicted token classes than words. | |
| >>> # Multiple token classes might account for the same word | |
| >>> predicted_tokens_classes = [model.config.id2label[t.item()] for t in predicted_token_class_ids[0]] | |
| >>> predicted_tokens_classes | |
| ... | |
| >>> labels = predicted_token_class_ids | |
| >>> loss = model(**inputs, labels=labels).loss | |
| >>> round(loss.item(), 2) | |
| ... | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| <EditOnGithub source="https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/vilt.md" /> |
Xet Storage Details
- Size:
- 89.7 kB
- Xet hash:
- 1dcd44a7744734bc75d82d6d5201c27f70eb87ea0946afe5a339b7dfa8b46d16
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.