Buckets:
| # CLIP | |
| [CLIP](https://huggingface.co/papers/2103.00020) is a is a multimodal vision and language model motivated by overcoming the fixed number of object categories when training a computer vision model. CLIP learns about images directly from raw text by jointly training on 400M (image, text) pairs. Pretraining on this scale enables zero-shot transfer to downstream tasks. CLIP uses an image encoder and text encoder to get visual features and text features. Both features are projected to a latent space with the same number of dimensions and their dot product gives a similarity score. | |
| You can find all the original CLIP checkpoints under the [OpenAI](https://huggingface.co/openai?search_models=clip) organization. | |
| > [!TIP] | |
| > Click on the CLIP models in the right sidebar for more examples of how to apply CLIP to different image and language tasks. | |
| The example below demonstrates how to calculate similarity scores between multiple text descriptions and an image with [Pipeline](/docs/transformers/pr_40546/en/main_classes/pipelines#transformers.Pipeline) or the [AutoModel](/docs/transformers/pr_40546/en/model_doc/auto#transformers.AutoModel) class. | |
| ```python | |
| from transformers import pipeline | |
| clip = pipeline( | |
| task="zero-shot-image-classification", | |
| model="openai/clip-vit-base-patch32", | |
| device=0 | |
| ) | |
| labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"] | |
| clip("http://images.cocodataset.org/val2017/000000039769.jpg", candidate_labels=labels) | |
| ``` | |
| ```python | |
| import requests | |
| from PIL import Image | |
| from transformers import AutoModel, AutoProcessor | |
| model = AutoModel.from_pretrained("openai/clip-vit-base-patch32", attn_implementation="sdpa", device_map="auto") | |
| processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") | |
| url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| image = Image.open(requests.get(url, stream=True).raw) | |
| labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"] | |
| inputs = processor(text=labels, images=image, return_tensors="pt", padding=True).to(model.device) | |
| outputs = model(**inputs) | |
| logits_per_image = outputs.logits_per_image | |
| probs = logits_per_image.softmax(dim=1) | |
| most_likely_idx = probs.argmax(dim=1).item() | |
| most_likely_label = labels[most_likely_idx] | |
| print(f"Most likely label: {most_likely_label} with probability: {probs[0][most_likely_idx].item():.3f}") | |
| ``` | |
| ## Notes | |
| - Use [CLIPImageProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPImageProcessor) to resize (or rescale) and normalizes images for the model. | |
| ## CLIPConfig[[transformers.CLIPConfig]] | |
| - **text_config** (`dict`, *optional*) -- | |
| Dictionary of configuration options used to initialize [CLIPTextConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPTextConfig). | |
| - **vision_config** (`dict`, *optional*) -- | |
| Dictionary of configuration options used to initialize [CLIPVisionConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPVisionConfig). | |
| - **projection_dim** (`int`, *optional*, defaults to `512`) -- | |
| Dimensionality of text and vision projection layers. | |
| - **logit_scale_init_value** (`float | int`, *optional*, defaults to 2.6592) -- | |
| The initial value of the *logit_scale* parameter. Default is used as per the original CLIP implementation. | |
| - **initializer_factor** (`float`, *optional*, defaults to `1.0`) -- | |
| A factor for initializing all weight matrices (should be kept to 1, used internally for initialization | |
| testing). | |
| This is the configuration class to store the configuration of a CLIPModel. It is used to instantiate a Clip | |
| model according to the specified arguments, defining the model architecture. Instantiating a configuration with the | |
| defaults will yield a similar configuration to that of the [openai/clip-vit-base-patch32](https://huggingface.co/openai/clip-vit-base-patch32) | |
| Configuration objects inherit from [PreTrainedConfig](/docs/transformers/pr_40546/en/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the | |
| documentation from [PreTrainedConfig](/docs/transformers/pr_40546/en/main_classes/configuration#transformers.PreTrainedConfig) for more information. | |
| Example: | |
| ```python | |
| >>> from transformers import CLIPConfig, CLIPModel | |
| >>> # Initializing a CLIPConfig with openai/clip-vit-base-patch32 style configuration | |
| >>> configuration = CLIPConfig() | |
| >>> # Initializing a CLIPModel (with random weights) from the openai/clip-vit-base-patch32 style configuration | |
| >>> model = CLIPModel(configuration) | |
| >>> # Accessing the model configuration | |
| >>> configuration = model.config | |
| >>> # We can also initialize a CLIPConfig from a CLIPTextConfig and a CLIPVisionConfig | |
| >>> from transformers import CLIPTextConfig, CLIPVisionConfig | |
| >>> # Initializing a CLIPText and CLIPVision configuration | |
| >>> config_text = CLIPTextConfig() | |
| >>> config_vision = CLIPVisionConfig() | |
| >>> config = CLIPConfig(text_config=config_text, vision_config=config_vision) | |
| ``` | |
| ## CLIPTextConfig[[transformers.CLIPTextConfig]] | |
| - **vocab_size** (`int`, *optional*, defaults to `49408`) -- | |
| Vocabulary size of the model. Defines the number of different tokens that can be represented by the `input_ids`. | |
| - **hidden_size** (`int`, *optional*, defaults to `512`) -- | |
| Dimension of the hidden representations. | |
| - **intermediate_size** (`int`, *optional*, defaults to `2048`) -- | |
| Dimension of the MLP representations. | |
| - **projection_dim** (`int`, *optional*, defaults to `512`) -- | |
| Dimensionality of text and vision projection layers. | |
| - **num_hidden_layers** (`int`, *optional*, defaults to `12`) -- | |
| Number of hidden layers in the Transformer decoder. | |
| - **num_attention_heads** (`int`, *optional*, defaults to `8`) -- | |
| Number of attention heads for each attention layer in the Transformer decoder. | |
| - **max_position_embeddings** (`int`, *optional*, defaults to `77`) -- | |
| The maximum sequence length that this model might ever be used with. | |
| - **hidden_act** (`str`, *optional*, defaults to `quick_gelu`) -- | |
| The non-linear activation function (function or string) in the decoder. For example, `"gelu"`, | |
| `"relu"`, `"silu"`, etc. | |
| - **layer_norm_eps** (`float`, *optional*, defaults to `1e-05`) -- | |
| The epsilon used by the layer normalization layers. | |
| - **attention_dropout** (`Union[int, float]`, *optional*, defaults to `0.0`) -- | |
| The dropout ratio for the attention probabilities. | |
| - **initializer_range** (`float`, *optional*, defaults to `0.02`) -- | |
| The standard deviation of the truncated_normal_initializer for initializing all weight matrices. | |
| - **initializer_factor** (`float`, *optional*, defaults to `1.0`) -- | |
| A factor for initializing all weight matrices (should be kept to 1, used internally for initialization | |
| testing). | |
| - **pad_token_id** (`int`, *optional*, defaults to `1`) -- | |
| Token id used for padding in the vocabulary. | |
| - **bos_token_id** (`int`, *optional*, defaults to `49406`) -- | |
| Token id used for beginning-of-stream in the vocabulary. | |
| - **eos_token_id** (`Union[int, list[int]]`, *optional*, defaults to `49407`) -- | |
| Token id used for end-of-stream in the vocabulary. | |
| This is the configuration class to store the configuration of a CLIPModel. It is used to instantiate a Clip | |
| model according to the specified arguments, defining the model architecture. Instantiating a configuration with the | |
| defaults will yield a similar configuration to that of the [openai/clip-vit-base-patch32](https://huggingface.co/openai/clip-vit-base-patch32) | |
| Configuration objects inherit from [PreTrainedConfig](/docs/transformers/pr_40546/en/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the | |
| documentation from [PreTrainedConfig](/docs/transformers/pr_40546/en/main_classes/configuration#transformers.PreTrainedConfig) for more information. | |
| Example: | |
| ```python | |
| >>> from transformers import CLIPTextConfig, CLIPTextModel | |
| >>> # Initializing a CLIPTextConfig with openai/clip-vit-base-patch32 style configuration | |
| >>> configuration = CLIPTextConfig() | |
| >>> # Initializing a CLIPTextModel (with random weights) from the openai/clip-vit-base-patch32 style configuration | |
| >>> model = CLIPTextModel(configuration) | |
| >>> # Accessing the model configuration | |
| >>> configuration = model.config | |
| ``` | |
| ## CLIPVisionConfig[[transformers.CLIPVisionConfig]] | |
| - **hidden_size** (`int`, *optional*, defaults to `768`) -- | |
| Dimension of the hidden representations. | |
| - **intermediate_size** (`int`, *optional*, defaults to `3072`) -- | |
| Dimension of the MLP representations. | |
| - **projection_dim** (`int`, *optional*, defaults to `512`) -- | |
| Dimensionality of text and vision projection layers. | |
| - **num_hidden_layers** (`int`, *optional*, defaults to `12`) -- | |
| Number of hidden layers in the Transformer decoder. | |
| - **num_attention_heads** (`int`, *optional*, defaults to `12`) -- | |
| Number of attention heads for each attention layer in the Transformer decoder. | |
| - **num_channels** (`int`, *optional*, defaults to `3`) -- | |
| The number of input channels. | |
| - **image_size** (`Union[int, list[int], tuple[int, int]]`, *optional*, defaults to `224`) -- | |
| The size (resolution) of each image. | |
| - **patch_size** (`Union[int, list[int], tuple[int, int]]`, *optional*, defaults to `32`) -- | |
| The size (resolution) of each patch. | |
| - **hidden_act** (`str`, *optional*, defaults to `quick_gelu`) -- | |
| The non-linear activation function (function or string) in the decoder. For example, `"gelu"`, | |
| `"relu"`, `"silu"`, etc. | |
| - **layer_norm_eps** (`float`, *optional*, defaults to `1e-05`) -- | |
| The epsilon used by the layer normalization layers. | |
| - **attention_dropout** (`Union[int, float]`, *optional*, defaults to `0.0`) -- | |
| The dropout ratio for the attention probabilities. | |
| - **initializer_range** (`float`, *optional*, defaults to `0.02`) -- | |
| The standard deviation of the truncated_normal_initializer for initializing all weight matrices. | |
| - **initializer_factor** (`float`, *optional*, defaults to `1.0`) -- | |
| A factor for initializing all weight matrices (should be kept to 1, used internally for initialization | |
| testing). | |
| This is the configuration class to store the configuration of a CLIPModel. It is used to instantiate a Clip | |
| model according to the specified arguments, defining the model architecture. Instantiating a configuration with the | |
| defaults will yield a similar configuration to that of the [openai/clip-vit-base-patch32](https://huggingface.co/openai/clip-vit-base-patch32) | |
| Configuration objects inherit from [PreTrainedConfig](/docs/transformers/pr_40546/en/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the | |
| documentation from [PreTrainedConfig](/docs/transformers/pr_40546/en/main_classes/configuration#transformers.PreTrainedConfig) for more information. | |
| Example: | |
| ```python | |
| >>> from transformers import CLIPVisionConfig, CLIPVisionModel | |
| >>> # Initializing a CLIPVisionConfig with openai/clip-vit-base-patch32 style configuration | |
| >>> configuration = CLIPVisionConfig() | |
| >>> # Initializing a CLIPVisionModel (with random weights) from the openai/clip-vit-base-patch32 style configuration | |
| >>> model = CLIPVisionModel(configuration) | |
| >>> # Accessing the model configuration | |
| >>> configuration = model.config | |
| ``` | |
| ## CLIPTokenizer[[transformers.CLIPTokenizer]] | |
| '"}, {"name": "bos_token", "val": ": str = '<|startoftext|>'"}, {"name": "eos_token", "val": ": str = '<|endoftext|>'"}, {"name": "pad_token", "val": ": str = '<|endoftext|>'"}, {"name": "**kwargs", "val": ""}]}> | |
| - **vocab** (`str`, `dict` or `list`, *optional*) -- | |
| Vocabulary dict to use for the tokenizer. | |
| - **merges** (`str` or `list`, *optional*) -- | |
| Merges list to use for the BPE tokenizer. | |
| - **unk_token** (`str`, *optional*, defaults to `"<|endoftext|>"`) -- | |
| The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this | |
| token instead. | |
| - **bos_token** (`str`, *optional*, defaults to `"<|startoftext|>"`) -- | |
| The beginning of sequence token. | |
| - **eos_token** (`str`, *optional*, defaults to `"<|endoftext|>"`) -- | |
| The end of sequence token. | |
| - **pad_token** (`str`, *optional*, defaults to `"<|endoftext|>"`) -- | |
| The token used for padding, for example when batching sequences of different lengths. | |
| Construct a CLIP tokenizer (backed by HuggingFace's *tokenizers* library). Based on byte-level | |
| Byte-Pair-Encoding. | |
| This tokenizer inherits from [TokenizersBackend](/docs/transformers/pr_40546/en/main_classes/tokenizer#transformers.TokenizersBackend) which contains most of the main methods. Users should | |
| refer to this superclass for more information regarding those methods. | |
| - **token_ids_0** -- List of IDs for the (possibly already formatted) sequence. | |
| - **token_ids_1** -- Unused when `already_has_special_tokens=True`. Must be None in that case. | |
| - **already_has_special_tokens** -- Whether the sequence is already formatted with special tokens.A list of integers in the range [0, 1]1 for a special token, 0 for a sequence token. | |
| Retrieve sequence ids from a token list that has no special tokens added. | |
| For fast tokenizers, data collators call this with `already_has_special_tokens=True` to build a mask over an | |
| already-formatted sequence. In that case, we compute the mask by checking membership in `all_special_ids`. | |
| ## CLIPTokenizerFast[[transformers.CLIPTokenizer]] | |
| '"}, {"name": "bos_token", "val": ": str = '<|startoftext|>'"}, {"name": "eos_token", "val": ": str = '<|endoftext|>'"}, {"name": "pad_token", "val": ": str = '<|endoftext|>'"}, {"name": "**kwargs", "val": ""}]}> | |
| - **vocab** (`str`, `dict` or `list`, *optional*) -- | |
| Vocabulary dict to use for the tokenizer. | |
| - **merges** (`str` or `list`, *optional*) -- | |
| Merges list to use for the BPE tokenizer. | |
| - **unk_token** (`str`, *optional*, defaults to `"<|endoftext|>"`) -- | |
| The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this | |
| token instead. | |
| - **bos_token** (`str`, *optional*, defaults to `"<|startoftext|>"`) -- | |
| The beginning of sequence token. | |
| - **eos_token** (`str`, *optional*, defaults to `"<|endoftext|>"`) -- | |
| The end of sequence token. | |
| - **pad_token** (`str`, *optional*, defaults to `"<|endoftext|>"`) -- | |
| The token used for padding, for example when batching sequences of different lengths. | |
| Construct a CLIP tokenizer (backed by HuggingFace's *tokenizers* library). Based on byte-level | |
| Byte-Pair-Encoding. | |
| This tokenizer inherits from [TokenizersBackend](/docs/transformers/pr_40546/en/main_classes/tokenizer#transformers.TokenizersBackend) which contains most of the main methods. Users should | |
| refer to this superclass for more information regarding those methods. | |
| ## CLIPImageProcessor[[transformers.CLIPImageProcessor]] | |
| - ****kwargs** ([ImagesKwargs](/docs/transformers/pr_40546/en/main_classes/processors#transformers.ImagesKwargs), *optional*) -- | |
| Additional image preprocessing options. Model-specific kwargs are listed above; see the TypedDict class | |
| for the complete list of supported arguments. | |
| Constructs a CLIPImageProcessor image processor. | |
| - **images** (`Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, list[PIL.Image.Image], list[numpy.ndarray], list[torch.Tensor]]`) -- | |
| Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If | |
| passing in images with pixel values between 0 and 1, set `do_rescale=False`. | |
| - **return_tensors** (`str` or [TensorType](/docs/transformers/pr_40546/en/internal/file_utils#transformers.TensorType), *optional*) -- | |
| Returns stacked tensors if set to `'pt'`, otherwise returns a list of tensors. | |
| - ****kwargs** ([ImagesKwargs](/docs/transformers/pr_40546/en/main_classes/processors#transformers.ImagesKwargs), *optional*) -- | |
| Additional image preprocessing options. Model-specific kwargs are listed above; see the TypedDict class | |
| for the complete list of supported arguments.`~image_processing_base.BatchFeature`- **data** (`dict`) -- Dictionary of lists/arrays/tensors returned by the __call__ method ('pixel_values', etc.). | |
| - **tensor_type** (`Union[None, str, TensorType]`, *optional*) -- You can give a tensor_type here to convert the lists of integers in PyTorch/Numpy Tensors at | |
| initialization. | |
| ## CLIPImageProcessorPil[[transformers.CLIPImageProcessorPil]] | |
| - ****kwargs** ([ImagesKwargs](/docs/transformers/pr_40546/en/main_classes/processors#transformers.ImagesKwargs), *optional*) -- | |
| Additional image preprocessing options. Model-specific kwargs are listed above; see the TypedDict class | |
| for the complete list of supported arguments. | |
| Constructs a CLIPImageProcessor image processor. | |
| - **images** (`Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, list[PIL.Image.Image], list[numpy.ndarray], list[torch.Tensor]]`) -- | |
| Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If | |
| passing in images with pixel values between 0 and 1, set `do_rescale=False`. | |
| - **return_tensors** (`str` or [TensorType](/docs/transformers/pr_40546/en/internal/file_utils#transformers.TensorType), *optional*) -- | |
| Returns stacked tensors if set to `'pt'`, otherwise returns a list of tensors. | |
| - ****kwargs** ([ImagesKwargs](/docs/transformers/pr_40546/en/main_classes/processors#transformers.ImagesKwargs), *optional*) -- | |
| Additional image preprocessing options. Model-specific kwargs are listed above; see the TypedDict class | |
| for the complete list of supported arguments.`~image_processing_base.BatchFeature`- **data** (`dict`) -- Dictionary of lists/arrays/tensors returned by the __call__ method ('pixel_values', etc.). | |
| - **tensor_type** (`Union[None, str, TensorType]`, *optional*) -- You can give a tensor_type here to convert the lists of integers in PyTorch/Numpy Tensors at | |
| initialization. | |
| ## CLIPProcessor[[transformers.CLIPProcessor]] | |
| - **image_processor** (`CLIPImageProcessor`) -- | |
| The image processor is a required input. | |
| - **tokenizer** (`CLIPTokenizer`) -- | |
| The tokenizer is a required input. | |
| Constructs a CLIPProcessor which wraps a image processor and a tokenizer into a single processor. | |
| [CLIPProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPProcessor) offers all the functionalities of [CLIPImageProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPImageProcessor) and [CLIPTokenizer](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPTokenizer). See the | |
| [~CLIPImageProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPImageProcessor) and [~CLIPTokenizer](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPTokenizer) for more information. | |
| - **images** (`Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, list[PIL.Image.Image], list[numpy.ndarray], list[torch.Tensor]]`, *optional*) -- | |
| Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If | |
| passing in images with pixel values between 0 and 1, set `do_rescale=False`. | |
| - **text** (`Union[str, list[str], list[list[str]]]`, *optional*) -- | |
| The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings | |
| (pretokenized string). If you pass a pretokenized input, set `is_split_into_words=True` to avoid ambiguity with batched inputs. | |
| - **videos** (`Union[list[PIL.Image.Image], numpy.ndarray, torch.Tensor, list[numpy.ndarray], list[torch.Tensor], list[list[PIL.Image.Image]], list[list[numpy.ndarray]], list[list[torch.Tensor]], ~video_utils.URL, list[~video_utils.URL], list[list[~video_utils.URL]], ~video_utils.Path, list[~video_utils.Path], list[list[~video_utils.Path]]]`, *optional*) -- | |
| Video to preprocess. Expects a single or batch of videos with pixel values ranging from 0 to 255. If | |
| passing in videos with pixel values between 0 and 1, set `do_rescale=False`. | |
| - **audio** (`Union[numpy.ndarray, torch.Tensor, collections.abc.Sequence[numpy.ndarray], collections.abc.Sequence[torch.Tensor]]`, *optional*) -- | |
| The audio or batch of audios to be prepared. Each audio can be a NumPy array or PyTorch tensor. | |
| In case of a NumPy array/PyTorch tensor, each audio should be of shape (C, T), where C is a number of channels, | |
| and T is the sample length of the audio. | |
| - **return_tensors** (`str` or [TensorType](/docs/transformers/pr_40546/en/internal/file_utils#transformers.TensorType), *optional*) -- | |
| If set, will return tensors of a particular framework. Acceptable values are: | |
| - `'pt'`: Return PyTorch `torch.Tensor` objects. | |
| - `'np'`: Return NumPy `np.ndarray` objects. | |
| - ****kwargs** ([ProcessingKwargs](/docs/transformers/pr_40546/en/main_classes/processors#transformers.ProcessingKwargs), *optional*) -- | |
| Additional processing options for each modality (text, images, videos, audio). Model-specific parameters | |
| are listed above; see the TypedDict class for the complete list of supported arguments. | |
| ## CLIPModel[[transformers.CLIPModel]] | |
| - **config** ([CLIPConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPConfig)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. | |
| The bare Clip Model outputting raw hidden-states without any specific head on top. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| - **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_40546/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_40546/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_40546/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [CLIPImageProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPImageProcessor). See `CLIPImageProcessor.__call__()` for details ([CLIPProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPProcessor) uses | |
| [CLIPImageProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPImageProcessor) for processing images). | |
| - **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **position_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. | |
| [What are position IDs?](../glossary#position-ids) | |
| - **return_loss** (`bool`, *optional*) -- | |
| Whether or not to return the contrastive loss. | |
| - **interpolate_pos_encoding** (`bool`, *optional*, defaults to `False`) -- | |
| Whether to interpolate the pre-trained position encodings.`CLIPOutput` or `tuple(torch.FloatTensor)`A `CLIPOutput` or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([CLIPConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPConfig)) and inputs. | |
| The [CLIPModel](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPModel) forward method, overrides the `__call__` special method. | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`) -- Contrastive loss for image-text similarity. | |
| - **logits_per_image** (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`) -- The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text | |
| similarity scores. | |
| - **logits_per_text** (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`) -- The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image | |
| similarity scores. | |
| - **text_embeds** (`torch.FloatTensor` of shape `(batch_size, output_dim`) -- The text embeddings obtained by applying the projection layer to the pooled output of [CLIPTextModel](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPTextModel). | |
| - **image_embeds** (`torch.FloatTensor` of shape `(batch_size, output_dim`) -- The image embeddings obtained by applying the projection layer to the pooled output of [CLIPVisionModel](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPVisionModel). | |
| - **text_model_output** (`~modeling_outputs.BaseModelOutputWithPooling`, *optional*) -- The output of the [CLIPTextModel](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPTextModel). | |
| - **vision_model_output** (`~modeling_outputs.BaseModelOutputWithPooling`, *optional*) -- The output of the [CLIPVisionModel](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPVisionModel). | |
| Examples: | |
| ```python | |
| >>> import torch | |
| >>> from transformers import AutoProcessor, CLIPModel | |
| >>> from transformers.image_utils import load_image | |
| >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> image = load_image(url) | |
| >>> inputs = processor( | |
| ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True | |
| ... ) | |
| >>> with torch.inference_mode(): | |
| ... outputs = model(**inputs) | |
| >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score | |
| >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities | |
| ``` | |
| - **input_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_40546/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_40546/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_40546/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **position_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. | |
| [What are position IDs?](../glossary#position-ids)[BaseModelOutputWithPooling](/docs/transformers/pr_40546/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or `tuple(torch.FloatTensor)`A [BaseModelOutputWithPooling](/docs/transformers/pr_40546/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([CLIPConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPConfig)) and inputs. | |
| - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model. | |
| - **pooler_output** (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) -- Last layer hidden-state of the first token of the sequence (classification token) after further processing | |
| through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns | |
| the classification token after processing through a linear layer and a tanh activation function. The linear | |
| layer weights are trained from the next sentence prediction (classification) objective during pretraining. | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads. | |
| Examples: | |
| ```python | |
| >>> import torch | |
| >>> from transformers import AutoTokenizer, CLIPModel | |
| >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") | |
| >>> with torch.inference_mode(): | |
| ... text_features = model.get_text_features(**inputs) | |
| ``` | |
| - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [CLIPImageProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPImageProcessor). See `CLIPImageProcessor.__call__()` for details ([CLIPProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPProcessor) uses | |
| [CLIPImageProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPImageProcessor) for processing images). | |
| - **interpolate_pos_encoding** (`bool`, *optional*, defaults to `False`) -- | |
| Whether to interpolate the pre-trained position encodings.[BaseModelOutputWithPooling](/docs/transformers/pr_40546/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or `tuple(torch.FloatTensor)`A [BaseModelOutputWithPooling](/docs/transformers/pr_40546/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([CLIPConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPConfig)) and inputs. | |
| - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model. | |
| - **pooler_output** (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) -- Last layer hidden-state of the first token of the sequence (classification token) after further processing | |
| through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns | |
| the classification token after processing through a linear layer and a tanh activation function. The linear | |
| layer weights are trained from the next sentence prediction (classification) objective during pretraining. | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads. | |
| Examples: | |
| ```python | |
| >>> import torch | |
| >>> from transformers import AutoProcessor, CLIPModel | |
| >>> from transformers.image_utils import load_image | |
| >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> image = load_image(url) | |
| >>> inputs = processor(images=image, return_tensors="pt") | |
| >>> with torch.inference_mode(): | |
| ... image_features = model.get_image_features(**inputs) | |
| ``` | |
| ## CLIPTextModel[[transformers.CLIPTextModel]] | |
| - **config** ([CLIPTextConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPTextConfig)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. | |
| The text model from CLIP without any head or projection on top. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| - **input_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_40546/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_40546/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_40546/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **position_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. | |
| [What are position IDs?](../glossary#position-ids)[BaseModelOutputWithPooling](/docs/transformers/pr_40546/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or `tuple(torch.FloatTensor)`A [BaseModelOutputWithPooling](/docs/transformers/pr_40546/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([CLIPConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPConfig)) and inputs. | |
| The [CLIPTextModel](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPTextModel) forward method, overrides the `__call__` special method. | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model. | |
| - **pooler_output** (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) -- Last layer hidden-state of the first token of the sequence (classification token) after further processing | |
| through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns | |
| the classification token after processing through a linear layer and a tanh activation function. The linear | |
| layer weights are trained from the next sentence prediction (classification) objective during pretraining. | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads. | |
| Examples: | |
| ```python | |
| >>> from transformers import AutoTokenizer, CLIPTextModel | |
| >>> model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") | |
| >>> outputs = model(**inputs) | |
| >>> last_hidden_state = outputs.last_hidden_state | |
| >>> pooled_output = outputs.pooler_output # pooled (EOS token) states | |
| ``` | |
| ## CLIPTextModelWithProjection[[transformers.CLIPTextModelWithProjection]] | |
| - **config** ([CLIPTextConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPTextConfig)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. | |
| The Clip Model with a projection layer on top (a linear layer on top of the pooled output). | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| - **input_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_40546/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_40546/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_40546/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **position_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. | |
| [What are position IDs?](../glossary#position-ids)`CLIPTextModelOutput` or `tuple(torch.FloatTensor)`A `CLIPTextModelOutput` or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([CLIPConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPConfig)) and inputs. | |
| The [CLIPTextModelWithProjection](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPTextModelWithProjection) forward method, overrides the `__call__` special method. | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| - **text_embeds** (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`) -- The text embeddings obtained by applying the projection layer to the pooler_output. | |
| - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- Sequence of hidden-states at the output of the last layer of the model. | |
| - **hidden_states** (`tuple[torch.FloatTensor, ...]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple[torch.FloatTensor, ...]`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads. | |
| Examples: | |
| ```python | |
| >>> import torch | |
| >>> from transformers import AutoTokenizer, CLIPTextModelWithProjection | |
| >>> model = CLIPTextModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") | |
| >>> with torch.inference_mode(): | |
| ... outputs = model(**inputs) | |
| >>> text_embeds = outputs.text_embeds | |
| ``` | |
| ## CLIPVisionModelWithProjection[[transformers.CLIPVisionModelWithProjection]] | |
| - **config** ([CLIPVisionConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPVisionConfig)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. | |
| The Clip Model with a projection layer on top (a linear layer on top of the pooled output). | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [CLIPImageProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPImageProcessor). See `CLIPImageProcessor.__call__()` for details ([CLIPProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPProcessor) uses | |
| [CLIPImageProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPImageProcessor) for processing images). | |
| - **interpolate_pos_encoding** (`bool`, *optional*, defaults to `False`) -- | |
| Whether to interpolate the pre-trained position encodings.`CLIPVisionModelOutput` or `tuple(torch.FloatTensor)`A `CLIPVisionModelOutput` or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([CLIPConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPConfig)) and inputs. | |
| The [CLIPVisionModelWithProjection](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPVisionModelWithProjection) forward method, overrides the `__call__` special method. | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| - **image_embeds** (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`) -- The image embeddings obtained by applying the projection layer to the pooler_output. | |
| - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- Sequence of hidden-states at the output of the last layer of the model. | |
| - **hidden_states** (`tuple[torch.FloatTensor, ...]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple[torch.FloatTensor, ...]`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads. | |
| Examples: | |
| ```python | |
| >>> import torch | |
| >>> from transformers import AutoProcessor, CLIPVisionModelWithProjection | |
| >>> from transformers.image_utils import load_image | |
| >>> model = CLIPVisionModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> image = load_image(url) | |
| >>> inputs = processor(images=image, return_tensors="pt") | |
| >>> with torch.inference_mode(): | |
| ... outputs = model(**inputs) | |
| >>> image_embeds = outputs.image_embeds | |
| ``` | |
| ## CLIPVisionModel[[transformers.CLIPVisionModel]] | |
| - **config** ([CLIPVisionConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPVisionConfig)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. | |
| The vision model from CLIP without any head or projection on top. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [CLIPImageProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPImageProcessor). See `CLIPImageProcessor.__call__()` for details ([CLIPProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPProcessor) uses | |
| [CLIPImageProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPImageProcessor) for processing images). | |
| - **interpolate_pos_encoding** (`bool`, *optional*, defaults to `False`) -- | |
| Whether to interpolate the pre-trained position encodings.[BaseModelOutputWithPooling](/docs/transformers/pr_40546/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or `tuple(torch.FloatTensor)`A [BaseModelOutputWithPooling](/docs/transformers/pr_40546/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([CLIPConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPConfig)) and inputs. | |
| The [CLIPVisionModel](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPVisionModel) forward method, overrides the `__call__` special method. | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model. | |
| - **pooler_output** (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) -- Last layer hidden-state of the first token of the sequence (classification token) after further processing | |
| through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns | |
| the classification token after processing through a linear layer and a tanh activation function. The linear | |
| layer weights are trained from the next sentence prediction (classification) objective during pretraining. | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads. | |
| Example: | |
| ```python | |
| >>> from PIL import Image | |
| >>> import httpx | |
| >>> from io import BytesIO | |
| >>> from transformers import AutoProcessor, CLIPVisionModel | |
| >>> model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> with httpx.stream("GET", url) as response: | |
| ... image = Image.open(BytesIO(response.read())) | |
| >>> inputs = processor(images=image, return_tensors="pt") | |
| >>> outputs = model(**inputs) | |
| >>> last_hidden_state = outputs.last_hidden_state | |
| >>> pooled_output = outputs.pooler_output # pooled CLS states | |
| ``` | |
| ## CLIPForImageClassification[[transformers.CLIPForImageClassification]] | |
| - **config** ([CLIPConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPConfig)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. | |
| CLIP vision encoder with an image classification head on top (a linear layer on top of the pooled final hidden states of | |
| the patch tokens) e.g. for ImageNet. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| - **pixel_values** (`torch.Tensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [CLIPImageProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPImageProcessor). See `CLIPImageProcessor.__call__()` for details ([CLIPProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPProcessor) uses | |
| [CLIPImageProcessor](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPImageProcessor) for processing images). | |
| - **labels** (`torch.LongTensor` of shape `(batch_size,)`, *optional*) -- | |
| Labels for computing the image classification/regression loss. Indices should be in `[0, ..., | |
| config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If | |
| `config.num_labels > 1` a classification loss is computed (Cross-Entropy).[ImageClassifierOutput](/docs/transformers/pr_40546/en/main_classes/output#transformers.modeling_outputs.ImageClassifierOutput) or `tuple(torch.FloatTensor)`A [ImageClassifierOutput](/docs/transformers/pr_40546/en/main_classes/output#transformers.modeling_outputs.ImageClassifierOutput) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([CLIPConfig](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPConfig)) and inputs. | |
| The [CLIPForImageClassification](/docs/transformers/pr_40546/en/model_doc/clip#transformers.CLIPForImageClassification) forward method, overrides the `__call__` special method. | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Classification (or regression if config.num_labels==1) loss. | |
| - **logits** (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`) -- Classification (or regression if config.num_labels==1) scores (before SoftMax). | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states | |
| (also called feature maps) of the model at the output of each stage. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, patch_size, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads. | |
| Example: | |
| ```python | |
| >>> from transformers import AutoImageProcessor, CLIPForImageClassification | |
| >>> import torch | |
| >>> from datasets import load_dataset | |
| >>> dataset = load_dataset("huggingface/cats-image") | |
| >>> image = dataset["test"]["image"][0] | |
| >>> image_processor = AutoImageProcessor.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> model = CLIPForImageClassification.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> inputs = image_processor(image, return_tensors="pt") | |
| >>> with torch.no_grad(): | |
| ... logits = model(**inputs).logits | |
| >>> # model predicts one of the 1000 ImageNet classes | |
| >>> predicted_label = logits.argmax(-1).item() | |
| >>> print(model.config.id2label[predicted_label]) | |
| ... | |
| ``` | |
Xet Storage Details
- Size:
- 59.2 kB
- Xet hash:
- c4ba90975c32de0213508fc940d6beba22f2177adf354c40f7e3b2ba55f59f5c
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.