Buckets:
| # BridgeTower | |
| <div class="flex flex-wrap space-x-1"> | |
| <img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white"> | |
| </div> | |
| ## Overview | |
| The BridgeTower model was proposed in [BridgeTower: Building Bridges Between Encoders in Vision-Language Representative Learning](https://huggingface.co/papers/2206.08657) by Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, Wanxiang Che, Nan Duan. The goal of this model is to build a | |
| bridge between each uni-modal encoder and the cross-modal encoder to enable comprehensive and detailed interaction at each layer of the cross-modal encoder thus achieving remarkable performance on various downstream tasks with almost negligible additional performance and computational costs. | |
| This paper has been accepted to the [AAAI'23](https://aaai.org/Conferences/AAAI-23/) conference. | |
| The abstract from the paper is the following: | |
| *Vision-Language (VL) models with the TWO-TOWER architecture have dominated visual-language representation learning in recent years. | |
| Current VL models either use lightweight uni-modal encoders and learn to extract, align and fuse both modalities simultaneously in a deep cross-modal encoder, or feed the last-layer uni-modal representations from the deep pre-trained uni-modal encoders into the top cross-modal encoder. | |
| Both approaches potentially restrict vision-language representation learning and limit model performance. In this paper, we propose BRIDGETOWER, which introduces multiple bridge layers that build a connection between the top layers of uni-modal encoders and each layer of the crossmodal encoder. | |
| This enables effective bottom-up cross-modal alignment and fusion between visual and textual representations of different semantic levels of pre-trained uni-modal encoders in the cross-modal encoder. Pre-trained with only 4M images, BRIDGETOWER achieves state-of-the-art performance on various downstream vision-language tasks. | |
| In particular, on the VQAv2 test-std set, BRIDGETOWER achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data and almost negligible additional parameters and computational costs. | |
| Notably, when further scaling the model, BRIDGETOWER achieves an accuracy of 81.15%, surpassing models that are pre-trained on orders-of-magnitude larger datasets.* | |
| <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/bridgetower_architecture%20.jpg" | |
| alt="drawing" width="600"/> | |
| <small> BridgeTower architecture. Taken from the <a href="https://huggingface.co/papers/2206.08657">original paper.</a> </small> | |
| This model was contributed by [Anahita Bhiwandiwalla](https://huggingface.co/anahita-b), [Tiep Le](https://huggingface.co/Tile) and [Shaoyen Tseng](https://huggingface.co/shaoyent). The original code can be found [here](https://github.com/microsoft/BridgeTower). | |
| ## Usage tips and examples | |
| BridgeTower consists of a visual encoder, a textual encoder and cross-modal encoder with multiple lightweight bridge layers. | |
| The goal of this approach was to build a bridge between each uni-modal encoder and the cross-modal encoder to enable comprehensive and detailed interaction at each layer of the cross-modal encoder. | |
| In principle, one can apply any visual, textual or cross-modal encoder in the proposed architecture. | |
| The [BridgeTowerProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerProcessor) wraps [RobertaTokenizer](/docs/transformers/pr_33962/en/model_doc/roberta#transformers.RobertaTokenizer) and [BridgeTowerImageProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerImageProcessor) into a single instance to both | |
| encode the text and prepare the images respectively. | |
| The following example shows how to run contrastive learning using [BridgeTowerProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerProcessor) and [BridgeTowerForContrastiveLearning](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerForContrastiveLearning). | |
| ```python | |
| >>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning | |
| >>> import requests | |
| >>> from PIL import Image | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> image = Image.open(requests.get(url, stream=True).raw) | |
| >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] | |
| >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") | |
| >>> model = BridgeTowerForContrastiveLearning.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") | |
| >>> # forward pass | |
| >>> scores = dict() | |
| >>> for text in texts: | |
| ... # prepare inputs | |
| ... encoding = processor(image, text, return_tensors="pt") | |
| ... outputs = model(**encoding) | |
| ... scores[text] = outputs | |
| ``` | |
| The following example shows how to run image-text retrieval using [BridgeTowerProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerProcessor) and [BridgeTowerForImageAndTextRetrieval](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerForImageAndTextRetrieval). | |
| ```python | |
| >>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval | |
| >>> import requests | |
| >>> from PIL import Image | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> image = Image.open(requests.get(url, stream=True).raw) | |
| >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] | |
| >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") | |
| >>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") | |
| >>> # forward pass | |
| >>> scores = dict() | |
| >>> for text in texts: | |
| ... # prepare inputs | |
| ... encoding = processor(image, text, return_tensors="pt") | |
| ... outputs = model(**encoding) | |
| ... scores[text] = outputs.logits[0, 1].item() | |
| ``` | |
| The following example shows how to run masked language modeling using [BridgeTowerProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerProcessor) and [BridgeTowerForMaskedLM](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerForMaskedLM). | |
| ```python | |
| >>> from transformers import BridgeTowerProcessor, BridgeTowerForMaskedLM | |
| >>> from PIL import Image | |
| >>> import requests | |
| >>> url = "http://images.cocodataset.org/val2017/000000360943.jpg" | |
| >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") | |
| >>> text = "a <mask> looking out of the window" | |
| >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") | |
| >>> model = BridgeTowerForMaskedLM.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") | |
| >>> # prepare inputs | |
| >>> encoding = processor(image, text, return_tensors="pt") | |
| >>> # forward pass | |
| >>> outputs = model(**encoding) | |
| >>> results = processor.decode(outputs.logits.argmax(dim=-1).squeeze(0).tolist()) | |
| >>> print(results) | |
| .a cat looking out of the window. | |
| ``` | |
| Tips: | |
| - This implementation of BridgeTower uses [RobertaTokenizer](/docs/transformers/pr_33962/en/model_doc/roberta#transformers.RobertaTokenizer) to generate text embeddings and OpenAI's CLIP/ViT model to compute visual embeddings. | |
| - Checkpoints for pre-trained [bridgeTower-base](https://huggingface.co/BridgeTower/bridgetower-base) and [bridgetower masked language modeling and image text matching](https://huggingface.co/BridgeTower/bridgetower-base-itm-mlm) are released. | |
| - Please refer to [Table 5](https://huggingface.co/papers/2206.08657) for BridgeTower's performance on Image Retrieval and other down stream tasks. | |
| - The PyTorch version of this model is only available in torch 1.10 and higher. | |
| ## BridgeTowerConfig[[transformers.BridgeTowerConfig]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.BridgeTowerConfig</name><anchor>transformers.BridgeTowerConfig</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/configuration_bridgetower.py#L197</source><parameters>[{"name": "share_cross_modal_transformer_layers", "val": " = True"}, {"name": "hidden_act", "val": " = 'gelu'"}, {"name": "hidden_size", "val": " = 768"}, {"name": "initializer_factor", "val": " = 1"}, {"name": "layer_norm_eps", "val": " = 1e-05"}, {"name": "share_link_tower_layers", "val": " = False"}, {"name": "link_tower_type", "val": " = 'add'"}, {"name": "num_attention_heads", "val": " = 12"}, {"name": "num_hidden_layers", "val": " = 6"}, {"name": "tie_word_embeddings", "val": " = False"}, {"name": "init_layernorm_from_vision_encoder", "val": " = False"}, {"name": "text_config", "val": " = None"}, {"name": "vision_config", "val": " = None"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **share_cross_modal_transformer_layers** (`bool`, *optional*, defaults to `True`) -- | |
| Whether cross modal transformer layers are shared. | |
| - **hidden_act** (`str` or `function`, *optional*, defaults to `"gelu"`) -- | |
| The non-linear activation function (function or string) in the encoder and pooler. | |
| - **hidden_size** (`int`, *optional*, defaults to 768) -- | |
| Dimensionality of the encoder layers and the pooler layer. | |
| - **initializer_factor** (`float`, *optional*, defaults to 1) -- | |
| A factor for initializing all weight matrices (should be kept to 1, used internally for initialization | |
| testing). | |
| - **layer_norm_eps** (`float`, *optional*, defaults to 1e-05) -- | |
| The epsilon used by the layer normalization layers. | |
| - **share_link_tower_layers** (`bool`, *optional*, defaults to `False`) -- | |
| Whether the bride/link tower layers are shared. | |
| - **link_tower_type** (`str`, *optional*, defaults to `"add"`) -- | |
| Type of the bridge/link layer. | |
| - **num_attention_heads** (`int`, *optional*, defaults to 12) -- | |
| Number of attention heads for each attention layer in the Transformer encoder. | |
| - **num_hidden_layers** (`int`, *optional*, defaults to 6) -- | |
| Number of hidden layers in the Transformer encoder. | |
| - **tie_word_embeddings** (`bool`, *optional*, defaults to `False`) -- | |
| Whether to tie input and output embeddings. | |
| - **init_layernorm_from_vision_encoder** (`bool`, *optional*, defaults to `False`) -- | |
| Whether to init LayerNorm from the vision encoder. | |
| - **text_config** (`dict`, *optional*) -- | |
| Dictionary of configuration options used to initialize [BridgeTowerTextConfig](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerTextConfig). | |
| - **vision_config** (`dict`, *optional*) -- | |
| Dictionary of configuration options used to initialize [BridgeTowerVisionConfig](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerVisionConfig).</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| This is the configuration class to store the configuration of a [BridgeTowerModel](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerModel). It is used to instantiate a | |
| BridgeTower model according to the specified arguments, defining the model architecture. Instantiating a | |
| configuration with the defaults will yield a similar configuration to that of the bridgetower-base | |
| [BridgeTower/bridgetower-base](https://huggingface.co/BridgeTower/bridgetower-base/) architecture. | |
| Configuration objects inherit from [PreTrainedConfig](/docs/transformers/pr_33962/en/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the | |
| documentation from [PreTrainedConfig](/docs/transformers/pr_33962/en/main_classes/configuration#transformers.PreTrainedConfig) for more information. | |
| <ExampleCodeBlock anchor="transformers.BridgeTowerConfig.example"> | |
| Example: | |
| ```python | |
| >>> from transformers import BridgeTowerModel, BridgeTowerConfig | |
| >>> # Initializing a BridgeTower BridgeTower/bridgetower-base style configuration | |
| >>> configuration = BridgeTowerConfig() | |
| >>> # Initializing a model from the BridgeTower/bridgetower-base style configuration | |
| >>> model = BridgeTowerModel(configuration) | |
| >>> # Accessing the model configuration | |
| >>> configuration = model.config | |
| ``` | |
| </ExampleCodeBlock> | |
| </div> | |
| ## BridgeTowerTextConfig[[transformers.BridgeTowerTextConfig]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.BridgeTowerTextConfig</name><anchor>transformers.BridgeTowerTextConfig</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/configuration_bridgetower.py#L97</source><parameters>[{"name": "vocab_size", "val": " = 50265"}, {"name": "hidden_size", "val": " = 768"}, {"name": "num_hidden_layers", "val": " = 12"}, {"name": "num_attention_heads", "val": " = 12"}, {"name": "initializer_factor", "val": " = 1"}, {"name": "intermediate_size", "val": " = 3072"}, {"name": "hidden_act", "val": " = 'gelu'"}, {"name": "hidden_dropout_prob", "val": " = 0.1"}, {"name": "attention_probs_dropout_prob", "val": " = 0.1"}, {"name": "max_position_embeddings", "val": " = 514"}, {"name": "type_vocab_size", "val": " = 1"}, {"name": "layer_norm_eps", "val": " = 1e-05"}, {"name": "pad_token_id", "val": " = 1"}, {"name": "bos_token_id", "val": " = 0"}, {"name": "eos_token_id", "val": " = 2"}, {"name": "use_cache", "val": " = True"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **vocab_size** (`int`, *optional*, defaults to 50265) -- | |
| Vocabulary size of the text part of the model. Defines the number of different tokens that can be | |
| represented by the `inputs_ids` passed when calling [BridgeTowerModel](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerModel). | |
| - **hidden_size** (`int`, *optional*, defaults to 768) -- | |
| Dimensionality of the encoder layers and the pooler layer. | |
| - **num_hidden_layers** (`int`, *optional*, defaults to 12) -- | |
| Number of hidden layers in the Transformer encoder. | |
| - **num_attention_heads** (`int`, *optional*, defaults to 12) -- | |
| Number of attention heads for each attention layer in the Transformer encoder. | |
| - **intermediate_size** (`int`, *optional*, defaults to 3072) -- | |
| Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder. | |
| - **hidden_act** (`str` or `Callable`, *optional*, defaults to `"gelu"`) -- | |
| The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, | |
| `"relu"`, `"silu"` and `"gelu_new"` are supported. | |
| - **hidden_dropout_prob** (`float`, *optional*, defaults to 0.1) -- | |
| The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. | |
| - **attention_probs_dropout_prob** (`float`, *optional*, defaults to 0.1) -- | |
| The dropout ratio for the attention probabilities. | |
| - **max_position_embeddings** (`int`, *optional*, defaults to 514) -- | |
| The maximum sequence length that this model might ever be used with. Typically set this to something large | |
| just in case (e.g., 512 or 1024 or 2048). | |
| - **type_vocab_size** (`int`, *optional*, defaults to 2) -- | |
| The vocabulary size of the `token_type_ids`. | |
| - **initializer_factor** (`float`, *optional*, defaults to 1) -- | |
| A factor for initializing all weight matrices (should be kept to 1, used internally for initialization | |
| testing). | |
| - **layer_norm_eps** (`float`, *optional*, defaults to 1e-05) -- | |
| The epsilon used by the layer normalization layers. | |
| - **is_decoder** (`bool`, *optional*, defaults to `False`) -- | |
| Whether the model is used as a decoder or not. If `False`, the model is used as an encoder. | |
| - **use_cache** (`bool`, *optional*, defaults to `True`) -- | |
| Whether or not the model should return the last key/values attentions (not used by all models). Only | |
| relevant if `config.is_decoder=True`.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| This is the configuration class to store the text configuration of a [BridgeTowerModel](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerModel). The default values here | |
| are copied from RoBERTa. Instantiating a configuration with the defaults will yield a similar configuration to that | |
| of the bridgetower-base [BridegTower/bridgetower-base](https://huggingface.co/BridgeTower/bridgetower-base/) | |
| architecture. | |
| Configuration objects inherit from [PreTrainedConfig](/docs/transformers/pr_33962/en/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the | |
| documentation from [PreTrainedConfig](/docs/transformers/pr_33962/en/main_classes/configuration#transformers.PreTrainedConfig) for more information. | |
| <ExampleCodeBlock anchor="transformers.BridgeTowerTextConfig.example"> | |
| Example: | |
| ```python | |
| >>> from transformers import BridgeTowerTextConfig | |
| >>> # Initializing a BridgeTower BridgeTower/bridgetower-base style configuration for the text model | |
| >>> configuration = BridgeTowerTextConfig() | |
| >>> # Accessing the configuration | |
| >>> configuration | |
| ``` | |
| </ExampleCodeBlock> | |
| </div> | |
| ## BridgeTowerVisionConfig[[transformers.BridgeTowerVisionConfig]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.BridgeTowerVisionConfig</name><anchor>transformers.BridgeTowerVisionConfig</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/configuration_bridgetower.py#L24</source><parameters>[{"name": "hidden_size", "val": " = 768"}, {"name": "num_hidden_layers", "val": " = 12"}, {"name": "num_channels", "val": " = 3"}, {"name": "patch_size", "val": " = 16"}, {"name": "image_size", "val": " = 288"}, {"name": "initializer_factor", "val": " = 1"}, {"name": "layer_norm_eps", "val": " = 1e-05"}, {"name": "stop_gradient", "val": " = False"}, {"name": "share_layernorm", "val": " = True"}, {"name": "remove_last_layer", "val": " = False"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **hidden_size** (`int`, *optional*, defaults to 768) -- | |
| Dimensionality of the encoder layers and the pooler layer. | |
| - **num_hidden_layers** (`int`, *optional*, defaults to 12) -- | |
| Number of hidden layers in visual encoder model. | |
| - **patch_size** (`int`, *optional*, defaults to 16) -- | |
| The size (resolution) of each patch. | |
| - **image_size** (`int`, *optional*, defaults to 288) -- | |
| The size (resolution) of each image. | |
| - **initializer_factor** (`float`, *optional*, defaults to 1) -- | |
| A factor for initializing all weight matrices (should be kept to 1, used internally for initialization | |
| testing). | |
| - **layer_norm_eps** (`float`, *optional*, defaults to 1e-05) -- | |
| The epsilon used by the layer normalization layers. | |
| - **stop_gradient** (`bool`, *optional*, defaults to `False`) -- | |
| Whether to stop gradient for training. | |
| - **share_layernorm** (`bool`, *optional*, defaults to `True`) -- | |
| Whether LayerNorm layers are shared. | |
| - **remove_last_layer** (`bool`, *optional*, defaults to `False`) -- | |
| Whether to remove the last layer from the vision encoder.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| This is the configuration class to store the vision configuration of a [BridgeTowerModel](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerModel). Instantiating a | |
| configuration with the defaults will yield a similar configuration to that of the bridgetower-base | |
| [BridgeTower/bridgetower-base](https://huggingface.co/BridgeTower/bridgetower-base/) architecture. | |
| Configuration objects inherit from [PreTrainedConfig](/docs/transformers/pr_33962/en/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the | |
| documentation from [PreTrainedConfig](/docs/transformers/pr_33962/en/main_classes/configuration#transformers.PreTrainedConfig) for more information. | |
| <ExampleCodeBlock anchor="transformers.BridgeTowerVisionConfig.example"> | |
| Example: | |
| ```python | |
| >>> from transformers import BridgeTowerVisionConfig | |
| >>> # Initializing a BridgeTower BridgeTower/bridgetower-base style configuration for the vision model | |
| >>> configuration = BridgeTowerVisionConfig() | |
| >>> # Accessing the configuration | |
| >>> configuration | |
| ``` | |
| </ExampleCodeBlock> | |
| </div> | |
| ## BridgeTowerImageProcessor[[transformers.BridgeTowerImageProcessor]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.BridgeTowerImageProcessor</name><anchor>transformers.BridgeTowerImageProcessor</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/image_processing_bridgetower.py#L130</source><parameters>[{"name": "do_resize", "val": ": bool = True"}, {"name": "size", "val": ": typing.Optional[dict[str, int]] = None"}, {"name": "size_divisor", "val": ": int = 32"}, {"name": "resample", "val": ": Resampling = <Resampling.BICUBIC: 3>"}, {"name": "do_rescale", "val": ": bool = True"}, {"name": "rescale_factor", "val": ": typing.Union[int, float] = 0.00392156862745098"}, {"name": "do_normalize", "val": ": bool = True"}, {"name": "image_mean", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "image_std", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "do_center_crop", "val": ": bool = True"}, {"name": "crop_size", "val": ": typing.Optional[dict[str, int]] = None"}, {"name": "do_pad", "val": ": bool = True"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **do_resize** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to resize the image's (height, width) dimensions to the specified `size`. Can be overridden by the | |
| `do_resize` parameter in the `preprocess` method. | |
| - **size** (`dict[str, int]` *optional*, defaults to `{'shortest_edge' -- 288}`): | |
| Resize the shorter side of the input to `size["shortest_edge"]`. The longer side will be limited to under | |
| `int((1333 / 800) * size["shortest_edge"])` while preserving the aspect ratio. Only has an effect if | |
| `do_resize` is set to `True`. Can be overridden by the `size` parameter in the `preprocess` method. | |
| - **size_divisor** (`int`, *optional*, defaults to 32) -- | |
| The size by which to make sure both the height and width can be divided. Only has an effect if `do_resize` | |
| is set to `True`. Can be overridden by the `size_divisor` parameter in the `preprocess` method. | |
| - **resample** (`PILImageResampling`, *optional*, defaults to `Resampling.BICUBIC`) -- | |
| Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`. Can be | |
| overridden by the `resample` parameter in the `preprocess` method. | |
| - **do_rescale** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by the `do_rescale` | |
| parameter in the `preprocess` method. | |
| - **rescale_factor** (`int` or `float`, *optional*, defaults to `1/255`) -- | |
| Scale factor to use if rescaling the image. Only has an effect if `do_rescale` is set to `True`. Can be | |
| overridden by the `rescale_factor` parameter in the `preprocess` method. | |
| - **do_normalize** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to normalize the image. Can be overridden by the `do_normalize` parameter in the `preprocess` | |
| method. Can be overridden by the `do_normalize` parameter in the `preprocess` method. | |
| - **image_mean** (`float` or `list[float]`, *optional*, defaults to `IMAGENET_STANDARD_MEAN`) -- | |
| Mean to use if normalizing the image. This is a float or list of floats the length of the number of | |
| channels in the image. Can be overridden by the `image_mean` parameter in the `preprocess` method. Can be | |
| overridden by the `image_mean` parameter in the `preprocess` method. | |
| - **image_std** (`float` or `list[float]`, *optional*, defaults to `IMAGENET_STANDARD_STD`) -- | |
| Standard deviation to use if normalizing the image. This is a float or list of floats the length of the | |
| number of channels in the image. Can be overridden by the `image_std` parameter in the `preprocess` method. | |
| Can be overridden by the `image_std` parameter in the `preprocess` method. | |
| - **do_center_crop** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to center crop the image. Can be overridden by the `do_center_crop` parameter in the `preprocess` | |
| method. | |
| - **crop_size** (`dict[str, int]`, *optional*) -- | |
| Desired output size when applying center-cropping. Only has an effect if `do_center_crop` is set to `True`. | |
| Can be overridden by the `crop_size` parameter in the `preprocess` method. If unset defaults to `size`, | |
| - **do_pad** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to pad the image to the `(max_height, max_width)` of the images in the batch. Can be overridden by | |
| the `do_pad` parameter in the `preprocess` method.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| Constructs a BridgeTower image processor. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>preprocess</name><anchor>transformers.BridgeTowerImageProcessor.preprocess</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/image_processing_bridgetower.py#L378</source><parameters>[{"name": "images", "val": ": typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']]"}, {"name": "do_resize", "val": ": typing.Optional[bool] = None"}, {"name": "size", "val": ": typing.Optional[dict[str, int]] = None"}, {"name": "size_divisor", "val": ": typing.Optional[int] = None"}, {"name": "resample", "val": ": typing.Optional[PIL.Image.Resampling] = None"}, {"name": "do_rescale", "val": ": typing.Optional[bool] = None"}, {"name": "rescale_factor", "val": ": typing.Optional[float] = None"}, {"name": "do_normalize", "val": ": typing.Optional[bool] = None"}, {"name": "image_mean", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "image_std", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "do_pad", "val": ": typing.Optional[bool] = None"}, {"name": "do_center_crop", "val": ": typing.Optional[bool] = None"}, {"name": "crop_size", "val": ": typing.Optional[dict[str, int]] = None"}, {"name": "return_tensors", "val": ": typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None"}, {"name": "data_format", "val": ": ChannelDimension = <ChannelDimension.FIRST: 'channels_first'>"}, {"name": "input_data_format", "val": ": typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None"}]</parameters><paramsdesc>- **images** (`ImageInput`) -- | |
| Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If | |
| passing in images with pixel values between 0 and 1, set `do_rescale=False`. | |
| - **do_resize** (`bool`, *optional*, defaults to `self.do_resize`) -- | |
| Whether to resize the image. | |
| - **size** (`dict[str, int]`, *optional*, defaults to `self.size`) -- | |
| Controls the size of the image after `resize`. The shortest edge of the image is resized to | |
| `size["shortest_edge"]` whilst preserving the aspect ratio. If the longest edge of this resized image | |
| is > `int(size["shortest_edge"] * (1333 / 800))`, then the image is resized again to make the longest | |
| edge equal to `int(size["shortest_edge"] * (1333 / 800))`. | |
| - **size_divisor** (`int`, *optional*, defaults to `self.size_divisor`) -- | |
| The image is resized to a size that is a multiple of this value. | |
| - **resample** (`PILImageResampling`, *optional*, defaults to `self.resample`) -- | |
| Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`. | |
| - **do_rescale** (`bool`, *optional*, defaults to `self.do_rescale`) -- | |
| Whether to rescale the image values between [0 - 1]. | |
| - **rescale_factor** (`float`, *optional*, defaults to `self.rescale_factor`) -- | |
| Rescale factor to rescale the image by if `do_rescale` is set to `True`. | |
| - **do_normalize** (`bool`, *optional*, defaults to `self.do_normalize`) -- | |
| Whether to normalize the image. | |
| - **image_mean** (`float` or `list[float]`, *optional*, defaults to `self.image_mean`) -- | |
| Image mean to normalize the image by if `do_normalize` is set to `True`. | |
| - **image_std** (`float` or `list[float]`, *optional*, defaults to `self.image_std`) -- | |
| Image standard deviation to normalize the image by if `do_normalize` is set to `True`. | |
| - **do_pad** (`bool`, *optional*, defaults to `self.do_pad`) -- | |
| Whether to pad the image to the (max_height, max_width) in the batch. If `True`, a pixel mask is also | |
| created and returned. | |
| - **do_center_crop** (`bool`, *optional*, defaults to `self.do_center_crop`) -- | |
| Whether to center crop the image. If the input size is smaller than `crop_size` along any edge, the | |
| image is padded with 0's and then center cropped. | |
| - **crop_size** (`dict[str, int]`, *optional*, defaults to `self.crop_size`) -- | |
| Size of the image after center crop. If one edge the image is smaller than `crop_size`, it will be | |
| padded with zeros and then cropped | |
| - **return_tensors** (`str` or `TensorType`, *optional*) -- | |
| The type of tensors to return. Can be one of: | |
| - Unset: Return a list of `np.ndarray`. | |
| - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. | |
| - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. | |
| - **data_format** (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`) -- | |
| The channel dimension format for the output image. Can be one of: | |
| - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. | |
| - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. | |
| - Unset: Use the channel dimension format of the input image. | |
| - **input_data_format** (`ChannelDimension` or `str`, *optional*) -- | |
| The channel dimension format for the input image. If unset, the channel dimension format is inferred | |
| from the input image. Can be one of: | |
| - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. | |
| - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. | |
| - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| Preprocess an image or batch of images. | |
| </div></div> | |
| ## BridgeTowerImageProcessorFast[[transformers.BridgeTowerImageProcessorFast]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.BridgeTowerImageProcessorFast</name><anchor>transformers.BridgeTowerImageProcessorFast</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/image_processing_bridgetower_fast.py#L89</source><parameters>[{"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.models.bridgetower.image_processing_bridgetower.BridgeTowerImageProcessorKwargs]"}]</parameters></docstring> | |
| Constructs a fast Bridgetower image processor. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>preprocess</name><anchor>transformers.BridgeTowerImageProcessorFast.preprocess</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/image_processing_bridgetower_fast.py#L108</source><parameters>[{"name": "images", "val": ": typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']]"}, {"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.models.bridgetower.image_processing_bridgetower.BridgeTowerImageProcessorKwargs]"}]</parameters><paramsdesc>- **images** (`Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']]`) -- | |
| Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If | |
| passing in images with pixel values between 0 and 1, set `do_rescale=False`. | |
| - **do_convert_rgb** (`bool`, *optional*) -- | |
| Whether to convert the image to RGB. | |
| - **do_resize** (`bool`, *optional*) -- | |
| Whether to resize the image. | |
| - **size** (`Annotated[Union[int, list[int], tuple[int, ...], dict[str, int], NoneType], None]`) -- | |
| Describes the maximum input dimensions to the model. | |
| - **crop_size** (`Annotated[Union[int, list[int], tuple[int, ...], dict[str, int], NoneType], None]`) -- | |
| Size of the output image after applying `center_crop`. | |
| - **resample** (`Annotated[Union[PILImageResampling, int, NoneType], None]`) -- | |
| Resampling filter to use if resizing the image. This can be one of the enum `PILImageResampling`. Only | |
| has an effect if `do_resize` is set to `True`. | |
| - **do_rescale** (`bool`, *optional*) -- | |
| Whether to rescale the image. | |
| - **rescale_factor** (`float`, *optional*) -- | |
| Rescale factor to rescale the image by if `do_rescale` is set to `True`. | |
| - **do_normalize** (`bool`, *optional*) -- | |
| Whether to normalize the image. | |
| - **image_mean** (`Union[float, list[float], tuple[float, ...], NoneType]`) -- | |
| Image mean to use for normalization. Only has an effect if `do_normalize` is set to `True`. | |
| - **image_std** (`Union[float, list[float], tuple[float, ...], NoneType]`) -- | |
| Image standard deviation to use for normalization. Only has an effect if `do_normalize` is set to | |
| `True`. | |
| - **do_pad** (`bool`, *optional*) -- | |
| Whether to pad the image. Padding is done either to the largest size in the batch | |
| or to a fixed square size per image. The exact padding strategy depends on the model. | |
| - **pad_size** (`Annotated[Union[int, list[int], tuple[int, ...], dict[str, int], NoneType], None]`) -- | |
| The size in `{"height": int, "width" int}` to pad the images to. Must be larger than any image size | |
| provided for preprocessing. If `pad_size` is not provided, images will be padded to the largest | |
| height and width in the batch. Applied only when `do_pad=True.` | |
| - **do_center_crop** (`bool`, *optional*) -- | |
| Whether to center crop the image. | |
| - **data_format** (`Union[str, ~image_utils.ChannelDimension, NoneType]`) -- | |
| Only `ChannelDimension.FIRST` is supported. Added for compatibility with slow processors. | |
| - **input_data_format** (`Union[str, ~image_utils.ChannelDimension, NoneType]`) -- | |
| The channel dimension format for the input image. If unset, the channel dimension format is inferred | |
| from the input image. Can be one of: | |
| - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. | |
| - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. | |
| - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. | |
| - **device** (`Annotated[str, None]`, *optional*) -- | |
| The device to process the images on. If unset, the device is inferred from the input images. | |
| - **return_tensors** (`Annotated[Union[str, ~utils.generic.TensorType, NoneType], None]`) -- | |
| Returns stacked tensors if set to `pt, otherwise returns a list of tensors. | |
| - **disable_grouping** (`bool`, *optional*) -- | |
| Whether to disable grouping of images by size to process them individually and not in batches. | |
| If None, will be set to True if the images are on CPU, and False otherwise. This choice is based on | |
| empirical observations, as detailed here: https://github.com/huggingface/transformers/pull/38157 | |
| - **size_divisor** (`<class 'int'>.size_divisor`) -- | |
| The size by which to make sure both the height and width can be divided.</paramsdesc><paramgroups>0</paramgroups><rettype>`<class 'transformers.image_processing_base.BatchFeature'>`</rettype><retdesc>- **data** (`dict`) -- Dictionary of lists/arrays/tensors returned by the __call__ method ('pixel_values', etc.). | |
| - **tensor_type** (`Union[None, str, TensorType]`, *optional*) -- You can give a tensor_type here to convert the lists of integers in PyTorch/Numpy Tensors at | |
| initialization.</retdesc></docstring> | |
| </div></div> | |
| ## BridgeTowerProcessor[[transformers.BridgeTowerProcessor]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.BridgeTowerProcessor</name><anchor>transformers.BridgeTowerProcessor</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/processing_bridgetower.py#L41</source><parameters>[{"name": "image_processor", "val": ""}, {"name": "tokenizer", "val": ""}]</parameters><paramsdesc>- **image_processor** (`BridgeTowerImageProcessor`) -- | |
| An instance of [BridgeTowerImageProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerImageProcessor). The image processor is a required input. | |
| - **tokenizer** (`RobertaTokenizerFast`) -- | |
| An instance of ['RobertaTokenizerFast`]. The tokenizer is a required input.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| Constructs a BridgeTower processor which wraps a Roberta tokenizer and BridgeTower image processor into a single | |
| processor. | |
| [BridgeTowerProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerProcessor) offers all the functionalities of [BridgeTowerImageProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerImageProcessor) and | |
| [RobertaTokenizerFast](/docs/transformers/pr_33962/en/model_doc/roberta#transformers.RobertaTokenizerFast). See the docstring of [__call__()](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltProcessor.__call__) and | |
| [decode()](/docs/transformers/pr_33962/en/main_classes/processors#transformers.ProcessorMixin.decode) for more information. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>__call__</name><anchor>transformers.BridgeTowerProcessor.__call__</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/processing_utils.py#L574</source><parameters>[{"name": "images", "val": ": typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor'], NoneType] = None"}, {"name": "text", "val": ": typing.Union[str, list[str], list[list[str]], NoneType] = None"}, {"name": "videos", "val": ": typing.Union[list['PIL.Image.Image'], numpy.ndarray, ForwardRef('torch.Tensor'), list[numpy.ndarray], list['torch.Tensor'], list[list['PIL.Image.Image']], list[list[numpy.ndarray]], list[list['torch.Tensor']], transformers.video_utils.URL, list[transformers.video_utils.URL], list[list[transformers.video_utils.URL]], transformers.video_utils.Path, list[transformers.video_utils.Path], list[list[transformers.video_utils.Path]], NoneType] = None"}, {"name": "audio", "val": ": typing.Union[numpy.ndarray, ForwardRef('torch.Tensor'), collections.abc.Sequence[numpy.ndarray], collections.abc.Sequence['torch.Tensor'], NoneType] = None"}, {"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.processing_utils.ProcessingKwargs]"}]</parameters><paramsdesc>- **images** (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`) -- | |
| The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch | |
| tensor. Both channels-first and channels-last formats are supported. | |
| - **text** (`TextInput`, `PreTokenizedInput`, `list[TextInput]`, `list[PreTokenizedInput]`, *optional*) -- | |
| The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings | |
| (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set | |
| `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). | |
| - **videos** (`np.ndarray`, `torch.Tensor`, `List[np.ndarray]`, `List[torch.Tensor]`) -- | |
| The video or batch of videos to be prepared. Each video can be a 4D NumPy array or PyTorch | |
| tensor, or a nested list of 3D frames. Both channels-first and channels-last formats are supported. | |
| - **audio** (`np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`) -- | |
| The audio or batch of audio to be prepared. Each audio can be a NumPy array or PyTorch | |
| tensor. | |
| - **return_tensors** (`str` or [TensorType](/docs/transformers/pr_33962/en/internal/file_utils#transformers.TensorType), *optional*) -- | |
| If set, will return tensors of a particular framework. Acceptable values are: | |
| - `'pt'`: Return PyTorch `torch.Tensor` objects. | |
| - `'np'`: Return NumPy `np.ndarray` objects.</paramsdesc><paramgroups>0</paramgroups><rettype>[BatchFeature](/docs/transformers/pr_33962/en/main_classes/image_processor#transformers.BatchFeature)</rettype><retdesc>A [BatchFeature](/docs/transformers/pr_33962/en/main_classes/image_processor#transformers.BatchFeature) object with processed inputs in a dict format.</retdesc></docstring> | |
| Main method to prepare for model inputs. This method forwards the each modality argument to its own processor | |
| along with `kwargs`. Please refer to the docstring of the each processor attributes for more information. | |
| </div></div> | |
| ## BridgeTowerModel[[transformers.BridgeTowerModel]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.BridgeTowerModel</name><anchor>transformers.BridgeTowerModel</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/modeling_bridgetower.py#L1142</source><parameters>[{"name": "config", "val": ""}]</parameters><paramsdesc>- **config** ([BridgeTowerModel](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerModel)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| The bare BridgeTower Model transformer outputting BridgeTowerModelOutput object without any specific head on | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.BridgeTowerModel.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/modeling_bridgetower.py#L1205</source><parameters>[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "token_type_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "pixel_mask", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "image_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "image_token_type_idx", "val": ": typing.Optional[int] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}, {"name": "labels", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "interpolate_pos_encoding", "val": ": bool = False"}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **token_type_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: | |
| - 0 corresponds to a *sentence A* token, | |
| - 1 corresponds to a *sentence B* token. | |
| [What are token type IDs?](../glossary#token-type-ids) | |
| - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [BridgeTowerImageProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerImageProcessor). See [BridgeTowerImageProcessor.__call__()](/docs/transformers/pr_33962/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([BridgeTowerProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerProcessor) uses | |
| [BridgeTowerImageProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerImageProcessor) for processing images). | |
| - **pixel_mask** (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*) -- | |
| Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: | |
| - 1 for pixels that are real (i.e. **not masked**), | |
| - 0 for pixels that are padding (i.e. **masked**). | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This | |
| is useful if you want more control over how to convert `input_ids` indices into associated vectors than the | |
| model's internal embedding lookup matrix. | |
| - **image_embeds** (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. | |
| This is useful if you want more control over how to convert `pixel_values` into patch embeddings. | |
| - **image_token_type_idx** (`int`, *optional*) -- | |
| - The token type ids for images. | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*) -- | |
| If set to `True`, hidden states are returned as a list containing the hidden states of text, image, and | |
| cross-modal components respectively. i.e. `(hidden_states_text, hidden_states_image, | |
| hidden_states_cross_modal)` where each element is a list of the hidden states of the corresponding | |
| modality. `hidden_states_txt/img` are a list of tensors corresponding to unimodal hidden states and | |
| `hidden_states_cross_modal` is a list of tuples containing `cross_modal_text_hidden_states` and | |
| `cross_modal_image_hidden_states` of each brdige layer. | |
| - **return_dict** (`bool`, *optional*) -- | |
| Whether or not to return a [ModelOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple. | |
| - **labels** (`torch.LongTensor` of shape `(batch_size,)`, *optional*) -- | |
| Labels are currently not supported. | |
| - **interpolate_pos_encoding** (`bool`, defaults to `False`) -- | |
| Whether to interpolate the pre-trained position encodings.</paramsdesc><paramgroups>0</paramgroups><rettype>`transformers.models.bridgetower.modeling_bridgetower.BridgeTowerModelOutput` or `tuple(torch.FloatTensor)`</rettype><retdesc>A `transformers.models.bridgetower.modeling_bridgetower.BridgeTowerModelOutput` or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([BridgeTowerConfig](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerConfig)) and inputs. | |
| - **text_features** (`torch.FloatTensor` of shape `(batch_size, text_sequence_length, hidden_size)`) -- Sequence of hidden-states at the text output of the last layer of the model. | |
| - **image_features** (`torch.FloatTensor` of shape `(batch_size, image_sequence_length, hidden_size)`) -- Sequence of hidden-states at the image output of the last layer of the model. | |
| - **pooler_output** (`torch.FloatTensor` of shape `(batch_size, hidden_size x 2)`) -- Concatenation of last layer hidden-state of the first token of the text and image sequence (classification | |
| token), respectively, after further processing through layers used for auxiliary pretraining tasks. | |
| - **hidden_states** (`tuple[torch.FloatTensor]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple[torch.FloatTensor]`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads.</retdesc></docstring> | |
| The [BridgeTowerModel](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerModel) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.BridgeTowerModel.forward.example"> | |
| Examples: | |
| ```python | |
| >>> from transformers import BridgeTowerProcessor, BridgeTowerModel | |
| >>> from PIL import Image | |
| >>> import requests | |
| >>> # prepare image and text | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> image = Image.open(requests.get(url, stream=True).raw) | |
| >>> text = "hello world" | |
| >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base") | |
| >>> model = BridgeTowerModel.from_pretrained("BridgeTower/bridgetower-base") | |
| >>> inputs = processor(image, text, return_tensors="pt") | |
| >>> outputs = model(**inputs) | |
| >>> outputs.keys() | |
| odict_keys(['text_features', 'image_features', 'pooler_output']) | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| ## BridgeTowerForContrastiveLearning[[transformers.BridgeTowerForContrastiveLearning]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.BridgeTowerForContrastiveLearning</name><anchor>transformers.BridgeTowerForContrastiveLearning</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/modeling_bridgetower.py#L1712</source><parameters>[{"name": "config", "val": ""}]</parameters><paramsdesc>- **config** ([BridgeTowerForContrastiveLearning](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerForContrastiveLearning)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| BridgeTower Model with a image-text contrastive head on top computing image-text contrastive loss. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.BridgeTowerForContrastiveLearning.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/modeling_bridgetower.py#L1726</source><parameters>[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "token_type_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "pixel_mask", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "image_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = True"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}, {"name": "return_loss", "val": ": typing.Optional[bool] = None"}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **token_type_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: | |
| - 0 corresponds to a *sentence A* token, | |
| - 1 corresponds to a *sentence B* token. | |
| [What are token type IDs?](../glossary#token-type-ids) | |
| - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [BridgeTowerImageProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerImageProcessor). See [BridgeTowerImageProcessor.__call__()](/docs/transformers/pr_33962/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([BridgeTowerProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerProcessor) uses | |
| [BridgeTowerImageProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerImageProcessor) for processing images). | |
| - **pixel_mask** (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*) -- | |
| Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: | |
| - 1 for pixels that are real (i.e. **not masked**), | |
| - 0 for pixels that are padding (i.e. **masked**). | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This | |
| is useful if you want more control over how to convert `input_ids` indices into associated vectors than the | |
| model's internal embedding lookup matrix. | |
| - **image_embeds** (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. | |
| This is useful if you want more control over how to convert `pixel_values` into patch embeddings. | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*, defaults to `True`) -- | |
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | |
| more detail. | |
| - **return_dict** (`bool`, *optional*) -- | |
| Whether or not to return a [ModelOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple. | |
| - **return_loss** (`bool`, *optional*) -- | |
| Whether or not to return the contrastive loss.</paramsdesc><paramgroups>0</paramgroups><rettype>`transformers.models.bridgetower.modeling_bridgetower.BridgeTowerContrastiveOutput` or `tuple(torch.FloatTensor)`</rettype><retdesc>A `transformers.models.bridgetower.modeling_bridgetower.BridgeTowerContrastiveOutput` or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([BridgeTowerConfig](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerConfig)) and inputs. | |
| - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`) -- Image-text contrastive loss. | |
| - **logits** (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) -- Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). | |
| - **text_embeds** (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`) -- The text embeddings obtained by applying the projection layer to the pooler_output. | |
| - **image_embeds** (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`) -- The image embeddings obtained by applying the projection layer to the pooler_output. | |
| - **cross_embeds** (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`) -- The text-image cross-modal embeddings obtained by applying the projection layer to the pooler_output. | |
| - **hidden_states** (`tuple[torch.FloatTensor]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`.</retdesc></docstring> | |
| The [BridgeTowerForContrastiveLearning](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerForContrastiveLearning) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.BridgeTowerForContrastiveLearning.forward.example"> | |
| Examples: | |
| ```python | |
| >>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning | |
| >>> import requests | |
| >>> from PIL import Image | |
| >>> import torch | |
| >>> image_urls = [ | |
| ... "https://farm4.staticflickr.com/3395/3428278415_81c3e27f15_z.jpg", | |
| ... "http://images.cocodataset.org/val2017/000000039769.jpg", | |
| ... ] | |
| >>> texts = ["two dogs in a car", "two cats sleeping on a couch"] | |
| >>> images = [Image.open(requests.get(url, stream=True).raw) for url in image_urls] | |
| >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") | |
| >>> model = BridgeTowerForContrastiveLearning.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") | |
| >>> inputs = processor(images, texts, padding=True, return_tensors="pt") | |
| >>> loss = model(**inputs, return_loss=True).loss | |
| >>> inputs = processor(images, texts[::-1], padding=True, return_tensors="pt") | |
| >>> loss_swapped = model(**inputs, return_loss=True).loss | |
| >>> print("Loss", round(loss.item(), 4)) | |
| Loss 0.0019 | |
| >>> print("Loss with swapped images", round(loss_swapped.item(), 4)) | |
| Loss with swapped images 2.126 | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| ## BridgeTowerForMaskedLM[[transformers.BridgeTowerForMaskedLM]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.BridgeTowerForMaskedLM</name><anchor>transformers.BridgeTowerForMaskedLM</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/modeling_bridgetower.py#L1496</source><parameters>[{"name": "config", "val": ""}]</parameters><paramsdesc>- **config** ([BridgeTowerForMaskedLM](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerForMaskedLM)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| BridgeTower Model with a language modeling head on top as done during pretraining. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.BridgeTowerForMaskedLM.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/modeling_bridgetower.py#L1514</source><parameters>[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "token_type_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "pixel_mask", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "image_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}, {"name": "labels", "val": ": typing.Optional[torch.LongTensor] = None"}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **token_type_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: | |
| - 0 corresponds to a *sentence A* token, | |
| - 1 corresponds to a *sentence B* token. | |
| [What are token type IDs?](../glossary#token-type-ids) | |
| - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [BridgeTowerImageProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerImageProcessor). See [BridgeTowerImageProcessor.__call__()](/docs/transformers/pr_33962/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([BridgeTowerProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerProcessor) uses | |
| [BridgeTowerImageProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerImageProcessor) for processing images). | |
| - **pixel_mask** (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*) -- | |
| Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: | |
| - 1 for pixels that are real (i.e. **not masked**), | |
| - 0 for pixels that are padding (i.e. **masked**). | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This | |
| is useful if you want more control over how to convert `input_ids` indices into associated vectors than the | |
| model's internal embedding lookup matrix. | |
| - **image_embeds** (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. | |
| This is useful if you want more control over how to convert `pixel_values` into patch embeddings. | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*) -- | |
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | |
| more detail. | |
| - **return_dict** (`bool`, *optional*) -- | |
| Whether or not to return a [ModelOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple. | |
| - **labels** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., | |
| config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the | |
| loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`</paramsdesc><paramgroups>0</paramgroups><rettype>[transformers.modeling_outputs.MaskedLMOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.MaskedLMOutput) or `tuple(torch.FloatTensor)`</rettype><retdesc>A [transformers.modeling_outputs.MaskedLMOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.MaskedLMOutput) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([BridgeTowerConfig](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerConfig)) and inputs. | |
| - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Masked language modeling (MLM) loss. | |
| - **logits** (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) -- Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads.</retdesc></docstring> | |
| The [BridgeTowerForMaskedLM](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerForMaskedLM) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.BridgeTowerForMaskedLM.forward.example"> | |
| Examples: | |
| ```python | |
| >>> from transformers import BridgeTowerProcessor, BridgeTowerForMaskedLM | |
| >>> from PIL import Image | |
| >>> import requests | |
| >>> url = "http://images.cocodataset.org/val2017/000000360943.jpg" | |
| >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") | |
| >>> text = "a <mask> looking out of the window" | |
| >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") | |
| >>> model = BridgeTowerForMaskedLM.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") | |
| >>> # prepare inputs | |
| >>> encoding = processor(image, text, return_tensors="pt") | |
| >>> # forward pass | |
| >>> outputs = model(**encoding) | |
| >>> results = processor.decode(outputs.logits.argmax(dim=-1).squeeze(0).tolist()) | |
| >>> print(results) | |
| .a cat looking out of the window. | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| ## BridgeTowerForImageAndTextRetrieval[[transformers.BridgeTowerForImageAndTextRetrieval]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.BridgeTowerForImageAndTextRetrieval</name><anchor>transformers.BridgeTowerForImageAndTextRetrieval</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/modeling_bridgetower.py#L1603</source><parameters>[{"name": "config", "val": ""}]</parameters><paramsdesc>- **config** ([BridgeTowerForImageAndTextRetrieval](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerForImageAndTextRetrieval)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| BridgeTower Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the | |
| [CLS] token) for image-to-text matching. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.BridgeTowerForImageAndTextRetrieval.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/bridgetower/modeling_bridgetower.py#L1614</source><parameters>[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "token_type_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "pixel_mask", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "inputs_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "image_embeds", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict", "val": ": typing.Optional[bool] = None"}, {"name": "labels", "val": ": typing.Optional[torch.LongTensor] = None"}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **token_type_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: | |
| - 0 corresponds to a *sentence A* token, | |
| - 1 corresponds to a *sentence B* token. | |
| [What are token type IDs?](../glossary#token-type-ids) | |
| - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [BridgeTowerImageProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerImageProcessor). See [BridgeTowerImageProcessor.__call__()](/docs/transformers/pr_33962/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([BridgeTowerProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerProcessor) uses | |
| [BridgeTowerImageProcessor](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerImageProcessor) for processing images). | |
| - **pixel_mask** (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*) -- | |
| Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: | |
| - 1 for pixels that are real (i.e. **not masked**), | |
| - 0 for pixels that are padding (i.e. **masked**). | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This | |
| is useful if you want more control over how to convert `input_ids` indices into associated vectors than the | |
| model's internal embedding lookup matrix. | |
| - **image_embeds** (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. | |
| This is useful if you want more control over how to convert `pixel_values` into patch embeddings. | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*) -- | |
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | |
| more detail. | |
| - **return_dict** (`bool`, *optional*) -- | |
| Whether or not to return a [ModelOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.utils.ModelOutput) instead of a plain tuple. | |
| - **labels** (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*) -- | |
| Labels for computing the image-text matching loss. 0 means the pairs don't match and 1 means they match. | |
| The pairs with 0 will be skipped for calculation.</paramsdesc><paramgroups>0</paramgroups><rettype>[transformers.modeling_outputs.SequenceClassifierOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.SequenceClassifierOutput) or `tuple(torch.FloatTensor)`</rettype><retdesc>A [transformers.modeling_outputs.SequenceClassifierOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.SequenceClassifierOutput) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([BridgeTowerConfig](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerConfig)) and inputs. | |
| - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Classification (or regression if config.num_labels==1) loss. | |
| - **logits** (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`) -- Classification (or regression if config.num_labels==1) scores (before SoftMax). | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads.</retdesc></docstring> | |
| The [BridgeTowerForImageAndTextRetrieval](/docs/transformers/pr_33962/en/model_doc/bridgetower#transformers.BridgeTowerForImageAndTextRetrieval) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.BridgeTowerForImageAndTextRetrieval.forward.example"> | |
| Examples: | |
| ```python | |
| >>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval | |
| >>> import requests | |
| >>> from PIL import Image | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> image = Image.open(requests.get(url, stream=True).raw) | |
| >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] | |
| >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") | |
| >>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") | |
| >>> # forward pass | |
| >>> scores = dict() | |
| >>> for text in texts: | |
| ... # prepare inputs | |
| ... encoding = processor(image, text, return_tensors="pt") | |
| ... outputs = model(**encoding) | |
| ... scores[text] = outputs.logits[0, 1].item() | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| <EditOnGithub source="https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/bridgetower.md" /> |
Xet Storage Details
- Size:
- 83.3 kB
- Xet hash:
- 91bfb977c48a323789cdf6068a69fc380e81341a6251fa847733215f89a7f851
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.