Buckets:
| # CLIP | |
| [CLIP](https://huggingface.co/papers/2103.00020) is a is a multimodal vision and language model motivated by overcoming the fixed number of object categories when training a computer vision model. CLIP learns about images directly from raw text by jointly training on 400M (image, text) pairs. Pretraining on this scale enables zero-shot transfer to downstream tasks. CLIP uses an image encoder and text encoder to get visual features and text features. Both features are projected to a latent space with the same number of dimensions and their dot product gives a similarity score. | |
| You can find all the original CLIP checkpoints under the [OpenAI](https://huggingface.co/openai?search_models=clip) organization. | |
| > [!TIP] | |
| > Click on the CLIP models in the right sidebar for more examples of how to apply CLIP to different image and language tasks. | |
| The example below demonstrates how to calculate similarity scores between multiple text descriptions and an image with [Pipeline](/docs/transformers/pr_33962/en/main_classes/pipelines#transformers.Pipeline) or the [AutoModel](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoModel) class. | |
| <hfoptions id="usage"> | |
| <hfoption id="Pipeline"> | |
| ```py | |
| import torch | |
| from transformers import pipeline | |
| clip = pipeline( | |
| task="zero-shot-image-classification", | |
| model="openai/clip-vit-base-patch32", | |
| dtype=torch.bfloat16, | |
| device=0 | |
| ) | |
| labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"] | |
| clip("http://images.cocodataset.org/val2017/000000039769.jpg", candidate_labels=labels) | |
| ``` | |
| </hfoption> | |
| <hfoption id="AutoModel"> | |
| ```py | |
| import requests | |
| import torch | |
| from PIL import Image | |
| from transformers import AutoProcessor, AutoModel | |
| model = AutoModel.from_pretrained("openai/clip-vit-base-patch32", dtype=torch.bfloat16, attn_implementation="sdpa") | |
| processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") | |
| url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| image = Image.open(requests.get(url, stream=True).raw) | |
| labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"] | |
| inputs = processor(text=labels, images=image, return_tensors="pt", padding=True) | |
| outputs = model(**inputs) | |
| logits_per_image = outputs.logits_per_image | |
| probs = logits_per_image.softmax(dim=1) | |
| most_likely_idx = probs.argmax(dim=1).item() | |
| most_likely_label = labels[most_likely_idx] | |
| print(f"Most likely label: {most_likely_label} with probability: {probs[0][most_likely_idx].item():.3f}") | |
| ``` | |
| </hfoption> | |
| </hfoptions> | |
| ## Notes | |
| - Use [CLIPImageProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPImageProcessor) to resize (or rescale) and normalizes images for the model. | |
| ## CLIPConfig[[transformers.CLIPConfig]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.CLIPConfig</name><anchor>transformers.CLIPConfig</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/configuration_clip.py#L226</source><parameters>[{"name": "text_config", "val": " = None"}, {"name": "vision_config", "val": " = None"}, {"name": "projection_dim", "val": " = 512"}, {"name": "logit_scale_init_value", "val": " = 2.6592"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **text_config** (`dict`, *optional*) -- | |
| Dictionary of configuration options used to initialize [CLIPTextConfig](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPTextConfig). | |
| - **vision_config** (`dict`, *optional*) -- | |
| Dictionary of configuration options used to initialize [CLIPVisionConfig](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPVisionConfig). | |
| - **projection_dim** (`int`, *optional*, defaults to 512) -- | |
| Dimensionality of text and vision projection layers. | |
| - **logit_scale_init_value** (`float`, *optional*, defaults to 2.6592) -- | |
| The initial value of the *logit_scale* parameter. Default is used as per the original CLIP implementation. | |
| - **kwargs** (*optional*) -- | |
| Dictionary of keyword arguments.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| [CLIPConfig](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPConfig) is the configuration class to store the configuration of a [CLIPModel](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPModel). It is used to instantiate | |
| a CLIP model according to the specified arguments, defining the text model and vision model configs. Instantiating | |
| a configuration with the defaults will yield a similar configuration to that of the CLIP | |
| [openai/clip-vit-base-patch32](https://huggingface.co/openai/clip-vit-base-patch32) architecture. | |
| Configuration objects inherit from [PreTrainedConfig](/docs/transformers/pr_33962/en/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the | |
| documentation from [PreTrainedConfig](/docs/transformers/pr_33962/en/main_classes/configuration#transformers.PreTrainedConfig) for more information. | |
| <ExampleCodeBlock anchor="transformers.CLIPConfig.example"> | |
| Example: | |
| ```python | |
| >>> from transformers import CLIPConfig, CLIPModel | |
| >>> # Initializing a CLIPConfig with openai/clip-vit-base-patch32 style configuration | |
| >>> configuration = CLIPConfig() | |
| >>> # Initializing a CLIPModel (with random weights) from the openai/clip-vit-base-patch32 style configuration | |
| >>> model = CLIPModel(configuration) | |
| >>> # Accessing the model configuration | |
| >>> configuration = model.config | |
| >>> # We can also initialize a CLIPConfig from a CLIPTextConfig and a CLIPVisionConfig | |
| >>> from transformers import CLIPTextConfig, CLIPVisionConfig | |
| >>> # Initializing a CLIPText and CLIPVision configuration | |
| >>> config_text = CLIPTextConfig() | |
| >>> config_vision = CLIPVisionConfig() | |
| >>> config = CLIPConfig(text_config=config_text, vision_config=config_vision) | |
| ``` | |
| </ExampleCodeBlock> | |
| </div> | |
| ## CLIPTextConfig[[transformers.CLIPTextConfig]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.CLIPTextConfig</name><anchor>transformers.CLIPTextConfig</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/configuration_clip.py#L33</source><parameters>[{"name": "vocab_size", "val": " = 49408"}, {"name": "hidden_size", "val": " = 512"}, {"name": "intermediate_size", "val": " = 2048"}, {"name": "projection_dim", "val": " = 512"}, {"name": "num_hidden_layers", "val": " = 12"}, {"name": "num_attention_heads", "val": " = 8"}, {"name": "max_position_embeddings", "val": " = 77"}, {"name": "hidden_act", "val": " = 'quick_gelu'"}, {"name": "layer_norm_eps", "val": " = 1e-05"}, {"name": "attention_dropout", "val": " = 0.0"}, {"name": "initializer_range", "val": " = 0.02"}, {"name": "initializer_factor", "val": " = 1.0"}, {"name": "pad_token_id", "val": " = 1"}, {"name": "bos_token_id", "val": " = 49406"}, {"name": "eos_token_id", "val": " = 49407"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **vocab_size** (`int`, *optional*, defaults to 49408) -- | |
| Vocabulary size of the CLIP text model. Defines the number of different tokens that can be represented by | |
| the `inputs_ids` passed when calling [CLIPModel](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPModel). | |
| - **hidden_size** (`int`, *optional*, defaults to 512) -- | |
| Dimensionality of the encoder layers and the pooler layer. | |
| - **intermediate_size** (`int`, *optional*, defaults to 2048) -- | |
| Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. | |
| - **projection_dim** (`int`, *optional*, defaults to 512) -- | |
| Dimensionality of text and vision projection layers. | |
| - **num_hidden_layers** (`int`, *optional*, defaults to 12) -- | |
| Number of hidden layers in the Transformer encoder. | |
| - **num_attention_heads** (`int`, *optional*, defaults to 8) -- | |
| Number of attention heads for each attention layer in the Transformer encoder. | |
| - **max_position_embeddings** (`int`, *optional*, defaults to 77) -- | |
| The maximum sequence length that this model might ever be used with. Typically set this to something large | |
| just in case (e.g., 512 or 1024 or 2048). | |
| - **hidden_act** (`str` or `function`, *optional*, defaults to `"quick_gelu"`) -- | |
| The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, | |
| `"relu"`, `"selu"` and `"gelu_new"` `"quick_gelu"` are supported. | |
| - **layer_norm_eps** (`float`, *optional*, defaults to 1e-05) -- | |
| The epsilon used by the layer normalization layers. | |
| - **attention_dropout** (`float`, *optional*, defaults to 0.0) -- | |
| The dropout ratio for the attention probabilities. | |
| - **initializer_range** (`float`, *optional*, defaults to 0.02) -- | |
| The standard deviation of the truncated_normal_initializer for initializing all weight matrices. | |
| - **initializer_factor** (`float`, *optional*, defaults to 1.0) -- | |
| A factor for initializing all weight matrices (should be kept to 1, used internally for initialization | |
| testing). | |
| - **pad_token_id** (`int`, *optional*, defaults to 1) -- | |
| Padding token id. | |
| - **bos_token_id** (`int`, *optional*, defaults to 49406) -- | |
| Beginning of stream token id. | |
| - **eos_token_id** (`int`, *optional*, defaults to 49407) -- | |
| End of stream token id.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| This is the configuration class to store the configuration of a [CLIPTextModel](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPTextModel). It is used to instantiate a CLIP | |
| text encoder according to the specified arguments, defining the model architecture. Instantiating a configuration | |
| with the defaults will yield a similar configuration to that of the text encoder of the CLIP | |
| [openai/clip-vit-base-patch32](https://huggingface.co/openai/clip-vit-base-patch32) architecture. | |
| Configuration objects inherit from [PreTrainedConfig](/docs/transformers/pr_33962/en/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the | |
| documentation from [PreTrainedConfig](/docs/transformers/pr_33962/en/main_classes/configuration#transformers.PreTrainedConfig) for more information. | |
| <ExampleCodeBlock anchor="transformers.CLIPTextConfig.example"> | |
| Example: | |
| ```python | |
| >>> from transformers import CLIPTextConfig, CLIPTextModel | |
| >>> # Initializing a CLIPTextConfig with openai/clip-vit-base-patch32 style configuration | |
| >>> configuration = CLIPTextConfig() | |
| >>> # Initializing a CLIPTextModel (with random weights) from the openai/clip-vit-base-patch32 style configuration | |
| >>> model = CLIPTextModel(configuration) | |
| >>> # Accessing the model configuration | |
| >>> configuration = model.config | |
| ``` | |
| </ExampleCodeBlock> | |
| </div> | |
| ## CLIPVisionConfig[[transformers.CLIPVisionConfig]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.CLIPVisionConfig</name><anchor>transformers.CLIPVisionConfig</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/configuration_clip.py#L134</source><parameters>[{"name": "hidden_size", "val": " = 768"}, {"name": "intermediate_size", "val": " = 3072"}, {"name": "projection_dim", "val": " = 512"}, {"name": "num_hidden_layers", "val": " = 12"}, {"name": "num_attention_heads", "val": " = 12"}, {"name": "num_channels", "val": " = 3"}, {"name": "image_size", "val": " = 224"}, {"name": "patch_size", "val": " = 32"}, {"name": "hidden_act", "val": " = 'quick_gelu'"}, {"name": "layer_norm_eps", "val": " = 1e-05"}, {"name": "attention_dropout", "val": " = 0.0"}, {"name": "initializer_range", "val": " = 0.02"}, {"name": "initializer_factor", "val": " = 1.0"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **hidden_size** (`int`, *optional*, defaults to 768) -- | |
| Dimensionality of the encoder layers and the pooler layer. | |
| - **intermediate_size** (`int`, *optional*, defaults to 3072) -- | |
| Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. | |
| - **projection_dim** (`int`, *optional*, defaults to 512) -- | |
| Dimensionality of text and vision projection layers. | |
| - **num_hidden_layers** (`int`, *optional*, defaults to 12) -- | |
| Number of hidden layers in the Transformer encoder. | |
| - **num_attention_heads** (`int`, *optional*, defaults to 12) -- | |
| Number of attention heads for each attention layer in the Transformer encoder. | |
| - **num_channels** (`int`, *optional*, defaults to 3) -- | |
| The number of input channels. | |
| - **image_size** (`int`, *optional*, defaults to 224) -- | |
| The size (resolution) of each image. | |
| - **patch_size** (`int`, *optional*, defaults to 32) -- | |
| The size (resolution) of each patch. | |
| - **hidden_act** (`str` or `function`, *optional*, defaults to `"quick_gelu"`) -- | |
| The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, | |
| `"relu"`, `"selu"` and `"gelu_new"` `"quick_gelu"` are supported. | |
| - **layer_norm_eps** (`float`, *optional*, defaults to 1e-05) -- | |
| The epsilon used by the layer normalization layers. | |
| - **attention_dropout** (`float`, *optional*, defaults to 0.0) -- | |
| The dropout ratio for the attention probabilities. | |
| - **initializer_range** (`float`, *optional*, defaults to 0.02) -- | |
| The standard deviation of the truncated_normal_initializer for initializing all weight matrices. | |
| - **initializer_factor** (`float`, *optional*, defaults to 1.0) -- | |
| A factor for initializing all weight matrices (should be kept to 1, used internally for initialization | |
| testing).</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| This is the configuration class to store the configuration of a [CLIPVisionModel](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPVisionModel). It is used to instantiate a | |
| CLIP vision encoder according to the specified arguments, defining the model architecture. Instantiating a | |
| configuration with the defaults will yield a similar configuration to that of the vision encoder of the CLIP | |
| [openai/clip-vit-base-patch32](https://huggingface.co/openai/clip-vit-base-patch32) architecture. | |
| Configuration objects inherit from [PreTrainedConfig](/docs/transformers/pr_33962/en/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the | |
| documentation from [PreTrainedConfig](/docs/transformers/pr_33962/en/main_classes/configuration#transformers.PreTrainedConfig) for more information. | |
| <ExampleCodeBlock anchor="transformers.CLIPVisionConfig.example"> | |
| Example: | |
| ```python | |
| >>> from transformers import CLIPVisionConfig, CLIPVisionModel | |
| >>> # Initializing a CLIPVisionConfig with openai/clip-vit-base-patch32 style configuration | |
| >>> configuration = CLIPVisionConfig() | |
| >>> # Initializing a CLIPVisionModel (with random weights) from the openai/clip-vit-base-patch32 style configuration | |
| >>> model = CLIPVisionModel(configuration) | |
| >>> # Accessing the model configuration | |
| >>> configuration = model.config | |
| ``` | |
| </ExampleCodeBlock> | |
| </div> | |
| ## CLIPTokenizer[[transformers.CLIPTokenizer]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.CLIPTokenizer</name><anchor>transformers.CLIPTokenizer</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/tokenization_clip.py#L254</source><parameters>[{"name": "vocab_file", "val": ""}, {"name": "merges_file", "val": ""}, {"name": "errors", "val": " = 'replace'"}, {"name": "unk_token", "val": " = '<|endoftext|>'"}, {"name": "bos_token", "val": " = '<|startoftext|>'"}, {"name": "eos_token", "val": " = '<|endoftext|>'"}, {"name": "pad_token", "val": " = '<|endoftext|>'"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **vocab_file** (`str`) -- | |
| Path to the vocabulary file. | |
| - **merges_file** (`str`) -- | |
| Path to the merges file. | |
| - **errors** (`str`, *optional*, defaults to `"replace"`) -- | |
| Paradigm to follow when decoding bytes to UTF-8. See | |
| [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information. | |
| - **unk_token** (`str`, *optional*, defaults to `"<|endoftext|>"`) -- | |
| The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this | |
| token instead. | |
| - **bos_token** (`str`, *optional*, defaults to `"<|startoftext|>"`) -- | |
| The beginning of sequence token. | |
| - **eos_token** (`str`, *optional*, defaults to `"<|endoftext|>"`) -- | |
| The end of sequence token. | |
| - **pad_token** (`str`, *optional*, defaults to `"<|endoftext|>"`) -- | |
| The token used for padding, for example when batching sequences of different lengths.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| Construct a CLIP tokenizer. Based on byte-level Byte-Pair-Encoding. | |
| This tokenizer inherits from [PreTrainedTokenizer](/docs/transformers/pr_33962/en/main_classes/tokenizer#transformers.PreTrainedTokenizer) which contains most of the main methods. Users should refer to | |
| this superclass for more information regarding those methods. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>build_inputs_with_special_tokens</name><anchor>transformers.CLIPTokenizer.build_inputs_with_special_tokens</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/tokenization_clip.py#L339</source><parameters>[{"name": "token_ids_0", "val": ": list"}, {"name": "token_ids_1", "val": ": typing.Optional[list[int]] = None"}]</parameters><paramsdesc>- **token_ids_0** (`list[int]`) -- | |
| List of IDs to which the special tokens will be added. | |
| - **token_ids_1** (`list[int]`, *optional*) -- | |
| Optional second list of IDs for sequence pairs.</paramsdesc><paramgroups>0</paramgroups><rettype>`list[int]`</rettype><retdesc>List of [input IDs](../glossary#input-ids) with the appropriate special tokens.</retdesc></docstring> | |
| Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and | |
| adding special tokens. A CLIP sequence has the following format: | |
| - single sequence: `<|startoftext|> X <|endoftext|>` | |
| Pairs of sequences are not the expected use case, but they will be handled without a separator. | |
| </div> | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>get_special_tokens_mask</name><anchor>transformers.CLIPTokenizer.get_special_tokens_mask</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/tokenization_clip.py#L366</source><parameters>[{"name": "token_ids_0", "val": ": list"}, {"name": "token_ids_1", "val": ": typing.Optional[list[int]] = None"}, {"name": "already_has_special_tokens", "val": ": bool = False"}]</parameters><paramsdesc>- **token_ids_0** (`list[int]`) -- | |
| List of IDs. | |
| - **token_ids_1** (`list[int]`, *optional*) -- | |
| Optional second list of IDs for sequence pairs. | |
| - **already_has_special_tokens** (`bool`, *optional*, defaults to `False`) -- | |
| Whether or not the token list is already formatted with special tokens for the model.</paramsdesc><paramgroups>0</paramgroups><rettype>`list[int]`</rettype><retdesc>A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.</retdesc></docstring> | |
| Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding | |
| special tokens using the tokenizer `prepare_for_model` method. | |
| </div> | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>create_token_type_ids_from_sequences</name><anchor>transformers.CLIPTokenizer.create_token_type_ids_from_sequences</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/tokenization_clip.py#L394</source><parameters>[{"name": "token_ids_0", "val": ": list"}, {"name": "token_ids_1", "val": ": typing.Optional[list[int]] = None"}]</parameters><paramsdesc>- **token_ids_0** (`list[int]`) -- | |
| List of IDs. | |
| - **token_ids_1** (`list[int]`, *optional*) -- | |
| Optional second list of IDs for sequence pairs.</paramsdesc><paramgroups>0</paramgroups><rettype>`list[int]`</rettype><retdesc>List of zeros.</retdesc></docstring> | |
| Create a mask from the two sequences passed. CLIP does not make use of token type ids, therefore a list of | |
| zeros is returned. | |
| </div> | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>save_vocabulary</name><anchor>transformers.CLIPTokenizer.save_vocabulary</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/tokenization_clip.py#L489</source><parameters>[{"name": "save_directory", "val": ": str"}, {"name": "filename_prefix", "val": ": typing.Optional[str] = None"}]</parameters></docstring> | |
| </div></div> | |
| ## CLIPTokenizerFast[[transformers.CLIPTokenizerFast]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.CLIPTokenizerFast</name><anchor>transformers.CLIPTokenizerFast</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/tokenization_clip_fast.py#L31</source><parameters>[{"name": "vocab_file", "val": " = None"}, {"name": "merges_file", "val": " = None"}, {"name": "tokenizer_file", "val": " = None"}, {"name": "unk_token", "val": " = '<|endoftext|>'"}, {"name": "bos_token", "val": " = '<|startoftext|>'"}, {"name": "eos_token", "val": " = '<|endoftext|>'"}, {"name": "pad_token", "val": " = '<|endoftext|>'"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **vocab_file** (`str`, *optional*) -- | |
| Path to the vocabulary file. | |
| - **merges_file** (`str`, *optional*) -- | |
| Path to the merges file. | |
| - **tokenizer_file** (`str`, *optional*) -- | |
| The path to a tokenizer file to use instead of the vocab file. | |
| - **unk_token** (`str`, *optional*, defaults to `"<|endoftext|>"`) -- | |
| The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this | |
| token instead. | |
| - **bos_token** (`str`, *optional*, defaults to `"<|startoftext|>"`) -- | |
| The beginning of sequence token. | |
| - **eos_token** (`str`, *optional*, defaults to `"<|endoftext|>"`) -- | |
| The end of sequence token. | |
| - **pad_token** (`str`, *optional*, defaults to `"<|endoftext|>"`) -- | |
| The token used for padding, for example when batching sequences of different lengths.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| Construct a "fast" CLIP tokenizer (backed by HuggingFace's *tokenizers* library). Based on byte-level | |
| Byte-Pair-Encoding. | |
| This tokenizer inherits from [PreTrainedTokenizerFast](/docs/transformers/pr_33962/en/main_classes/tokenizer#transformers.PreTrainedTokenizerFast) which contains most of the main methods. Users should | |
| refer to this superclass for more information regarding those methods. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>build_inputs_with_special_tokens</name><anchor>transformers.CLIPTokenizerFast.build_inputs_with_special_tokens</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/tokenization_clip_fast.py#L109</source><parameters>[{"name": "token_ids_0", "val": ": list"}, {"name": "token_ids_1", "val": ": typing.Optional[list[int]] = None"}]</parameters><paramsdesc>- **token_ids_0** (`list[int]`) -- | |
| List of IDs to which the special tokens will be added. | |
| - **token_ids_1** (`list[int]`, *optional*) -- | |
| Optional second list of IDs for sequence pairs.</paramsdesc><paramgroups>0</paramgroups><rettype>`list[int]`</rettype><retdesc>List of [input IDs](../glossary#input-ids) with the appropriate special tokens.</retdesc></docstring> | |
| Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and | |
| adding special tokens. A CLIP sequence has the following format: | |
| - single sequence: `<|startoftext|> X <|endoftext|>` | |
| Pairs of sequences are not the expected use case, but they will be handled without a separator. | |
| </div> | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>create_token_type_ids_from_sequences</name><anchor>transformers.CLIPTokenizerFast.create_token_type_ids_from_sequences</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/tokenization_clip_fast.py#L136</source><parameters>[{"name": "token_ids_0", "val": ": list"}, {"name": "token_ids_1", "val": ": typing.Optional[list[int]] = None"}]</parameters><paramsdesc>- **token_ids_0** (`list[int]`) -- | |
| List of IDs. | |
| - **token_ids_1** (`list[int]`, *optional*) -- | |
| Optional second list of IDs for sequence pairs.</paramsdesc><paramgroups>0</paramgroups><rettype>`list[int]`</rettype><retdesc>List of zeros.</retdesc></docstring> | |
| Create a mask from the two sequences passed. CLIP does not make use of token type ids, therefore a list of | |
| zeros is returned. | |
| </div></div> | |
| ## CLIPImageProcessor[[transformers.CLIPImageProcessor]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.CLIPImageProcessor</name><anchor>transformers.CLIPImageProcessor</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/image_processing_clip.py#L54</source><parameters>[{"name": "do_resize", "val": ": bool = True"}, {"name": "size", "val": ": typing.Optional[dict[str, int]] = None"}, {"name": "resample", "val": ": Resampling = <Resampling.BICUBIC: 3>"}, {"name": "do_center_crop", "val": ": bool = True"}, {"name": "crop_size", "val": ": typing.Optional[dict[str, int]] = None"}, {"name": "do_rescale", "val": ": bool = True"}, {"name": "rescale_factor", "val": ": typing.Union[int, float] = 0.00392156862745098"}, {"name": "do_normalize", "val": ": bool = True"}, {"name": "image_mean", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "image_std", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "do_convert_rgb", "val": ": bool = True"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **do_resize** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to resize the image's (height, width) dimensions to the specified `size`. Can be overridden by | |
| `do_resize` in the `preprocess` method. | |
| - **size** (`dict[str, int]` *optional*, defaults to `{"shortest_edge" -- 224}`): | |
| Size of the image after resizing. The shortest edge of the image is resized to size["shortest_edge"], with | |
| the longest edge resized to keep the input aspect ratio. Can be overridden by `size` in the `preprocess` | |
| method. | |
| - **resample** (`PILImageResampling`, *optional*, defaults to `Resampling.BICUBIC`) -- | |
| Resampling filter to use if resizing the image. Can be overridden by `resample` in the `preprocess` method. | |
| - **do_center_crop** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to center crop the image to the specified `crop_size`. Can be overridden by `do_center_crop` in the | |
| `preprocess` method. | |
| - **crop_size** (`dict[str, int]` *optional*, defaults to 224) -- | |
| Size of the output image after applying `center_crop`. Can be overridden by `crop_size` in the `preprocess` | |
| method. | |
| - **do_rescale** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by `do_rescale` in | |
| the `preprocess` method. | |
| - **rescale_factor** (`int` or `float`, *optional*, defaults to `1/255`) -- | |
| Scale factor to use if rescaling the image. Can be overridden by `rescale_factor` in the `preprocess` | |
| method. | |
| - **do_normalize** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to normalize the image. Can be overridden by `do_normalize` in the `preprocess` method. | |
| - **image_mean** (`float` or `list[float]`, *optional*, defaults to `[0.48145466, 0.4578275, 0.40821073]`) -- | |
| Mean to use if normalizing the image. This is a float or list of floats the length of the number of | |
| channels in the image. Can be overridden by the `image_mean` parameter in the `preprocess` method. | |
| - **image_std** (`float` or `list[float]`, *optional*, defaults to `[0.26862954, 0.26130258, 0.27577711]`) -- | |
| Standard deviation to use if normalizing the image. This is a float or list of floats the length of the | |
| number of channels in the image. Can be overridden by the `image_std` parameter in the `preprocess` method. | |
| Can be overridden by the `image_std` parameter in the `preprocess` method. | |
| - **do_convert_rgb** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to convert the image to RGB.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| Constructs a CLIP image processor. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>preprocess</name><anchor>transformers.CLIPImageProcessor.preprocess</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/image_processing_clip.py#L202</source><parameters>[{"name": "images", "val": ": typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']]"}, {"name": "do_resize", "val": ": typing.Optional[bool] = None"}, {"name": "size", "val": ": typing.Optional[dict[str, int]] = None"}, {"name": "resample", "val": ": typing.Optional[PIL.Image.Resampling] = None"}, {"name": "do_center_crop", "val": ": typing.Optional[bool] = None"}, {"name": "crop_size", "val": ": typing.Optional[int] = None"}, {"name": "do_rescale", "val": ": typing.Optional[bool] = None"}, {"name": "rescale_factor", "val": ": typing.Optional[float] = None"}, {"name": "do_normalize", "val": ": typing.Optional[bool] = None"}, {"name": "image_mean", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "image_std", "val": ": typing.Union[float, list[float], NoneType] = None"}, {"name": "do_convert_rgb", "val": ": typing.Optional[bool] = None"}, {"name": "return_tensors", "val": ": typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None"}, {"name": "data_format", "val": ": typing.Optional[transformers.image_utils.ChannelDimension] = <ChannelDimension.FIRST: 'channels_first'>"}, {"name": "input_data_format", "val": ": typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **images** (`ImageInput`) -- | |
| Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If | |
| passing in images with pixel values between 0 and 1, set `do_rescale=False`. | |
| - **do_resize** (`bool`, *optional*, defaults to `self.do_resize`) -- | |
| Whether to resize the image. | |
| - **size** (`dict[str, int]`, *optional*, defaults to `self.size`) -- | |
| Size of the image after resizing. Shortest edge of the image is resized to size["shortest_edge"], with | |
| the longest edge resized to keep the input aspect ratio. | |
| - **resample** (`int`, *optional*, defaults to `self.resample`) -- | |
| Resampling filter to use if resizing the image. This can be one of the enum `PILImageResampling`. Only | |
| has an effect if `do_resize` is set to `True`. | |
| - **do_center_crop** (`bool`, *optional*, defaults to `self.do_center_crop`) -- | |
| Whether to center crop the image. | |
| - **crop_size** (`dict[str, int]`, *optional*, defaults to `self.crop_size`) -- | |
| Size of the center crop. Only has an effect if `do_center_crop` is set to `True`. | |
| - **do_rescale** (`bool`, *optional*, defaults to `self.do_rescale`) -- | |
| Whether to rescale the image. | |
| - **rescale_factor** (`float`, *optional*, defaults to `self.rescale_factor`) -- | |
| Rescale factor to rescale the image by if `do_rescale` is set to `True`. | |
| - **do_normalize** (`bool`, *optional*, defaults to `self.do_normalize`) -- | |
| Whether to normalize the image. | |
| - **image_mean** (`float` or `list[float]`, *optional*, defaults to `self.image_mean`) -- | |
| Image mean to use for normalization. Only has an effect if `do_normalize` is set to `True`. | |
| - **image_std** (`float` or `list[float]`, *optional*, defaults to `self.image_std`) -- | |
| Image standard deviation to use for normalization. Only has an effect if `do_normalize` is set to | |
| `True`. | |
| - **do_convert_rgb** (`bool`, *optional*, defaults to `self.do_convert_rgb`) -- | |
| Whether to convert the image to RGB. | |
| - **return_tensors** (`str` or `TensorType`, *optional*) -- | |
| The type of tensors to return. Can be one of: | |
| - Unset: Return a list of `np.ndarray`. | |
| - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. | |
| - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. | |
| - **data_format** (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`) -- | |
| The channel dimension format for the output image. Can be one of: | |
| - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. | |
| - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. | |
| - Unset: Use the channel dimension format of the input image. | |
| - **input_data_format** (`ChannelDimension` or `str`, *optional*) -- | |
| The channel dimension format for the input image. If unset, the channel dimension format is inferred | |
| from the input image. Can be one of: | |
| - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. | |
| - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. | |
| - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| Preprocess an image or batch of images. | |
| </div></div> | |
| ## CLIPImageProcessorFast[[transformers.CLIPImageProcessorFast]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.CLIPImageProcessorFast</name><anchor>transformers.CLIPImageProcessorFast</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/image_processing_clip_fast.py#L23</source><parameters>[{"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.processing_utils.ImagesKwargs]"}]</parameters></docstring> | |
| Constructs a fast Clip image processor. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>preprocess</name><anchor>transformers.CLIPImageProcessorFast.preprocess</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/image_processing_utils_fast.py#L710</source><parameters>[{"name": "images", "val": ": typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']]"}, {"name": "*args", "val": ""}, {"name": "**kwargs", "val": ": typing_extensions.Unpack[transformers.processing_utils.ImagesKwargs]"}]</parameters><paramsdesc>- **images** (`Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']]`) -- | |
| Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If | |
| passing in images with pixel values between 0 and 1, set `do_rescale=False`. | |
| - **do_convert_rgb** (`bool`, *optional*) -- | |
| Whether to convert the image to RGB. | |
| - **do_resize** (`bool`, *optional*) -- | |
| Whether to resize the image. | |
| - **size** (`Annotated[Union[int, list[int], tuple[int, ...], dict[str, int], NoneType], None]`) -- | |
| Describes the maximum input dimensions to the model. | |
| - **crop_size** (`Annotated[Union[int, list[int], tuple[int, ...], dict[str, int], NoneType], None]`) -- | |
| Size of the output image after applying `center_crop`. | |
| - **resample** (`Annotated[Union[PILImageResampling, int, NoneType], None]`) -- | |
| Resampling filter to use if resizing the image. This can be one of the enum `PILImageResampling`. Only | |
| has an effect if `do_resize` is set to `True`. | |
| - **do_rescale** (`bool`, *optional*) -- | |
| Whether to rescale the image. | |
| - **rescale_factor** (`float`, *optional*) -- | |
| Rescale factor to rescale the image by if `do_rescale` is set to `True`. | |
| - **do_normalize** (`bool`, *optional*) -- | |
| Whether to normalize the image. | |
| - **image_mean** (`Union[float, list[float], tuple[float, ...], NoneType]`) -- | |
| Image mean to use for normalization. Only has an effect if `do_normalize` is set to `True`. | |
| - **image_std** (`Union[float, list[float], tuple[float, ...], NoneType]`) -- | |
| Image standard deviation to use for normalization. Only has an effect if `do_normalize` is set to | |
| `True`. | |
| - **do_pad** (`bool`, *optional*) -- | |
| Whether to pad the image. Padding is done either to the largest size in the batch | |
| or to a fixed square size per image. The exact padding strategy depends on the model. | |
| - **pad_size** (`Annotated[Union[int, list[int], tuple[int, ...], dict[str, int], NoneType], None]`) -- | |
| The size in `{"height": int, "width" int}` to pad the images to. Must be larger than any image size | |
| provided for preprocessing. If `pad_size` is not provided, images will be padded to the largest | |
| height and width in the batch. Applied only when `do_pad=True.` | |
| - **do_center_crop** (`bool`, *optional*) -- | |
| Whether to center crop the image. | |
| - **data_format** (`Union[str, ~image_utils.ChannelDimension, NoneType]`) -- | |
| Only `ChannelDimension.FIRST` is supported. Added for compatibility with slow processors. | |
| - **input_data_format** (`Union[str, ~image_utils.ChannelDimension, NoneType]`) -- | |
| The channel dimension format for the input image. If unset, the channel dimension format is inferred | |
| from the input image. Can be one of: | |
| - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. | |
| - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. | |
| - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. | |
| - **device** (`Annotated[str, None]`, *optional*) -- | |
| The device to process the images on. If unset, the device is inferred from the input images. | |
| - **return_tensors** (`Annotated[Union[str, ~utils.generic.TensorType, NoneType], None]`) -- | |
| Returns stacked tensors if set to `pt, otherwise returns a list of tensors. | |
| - **disable_grouping** (`bool`, *optional*) -- | |
| Whether to disable grouping of images by size to process them individually and not in batches. | |
| If None, will be set to True if the images are on CPU, and False otherwise. This choice is based on | |
| empirical observations, as detailed here: https://github.com/huggingface/transformers/pull/38157</paramsdesc><paramgroups>0</paramgroups><rettype>`<class 'transformers.image_processing_base.BatchFeature'>`</rettype><retdesc>- **data** (`dict`) -- Dictionary of lists/arrays/tensors returned by the __call__ method ('pixel_values', etc.). | |
| - **tensor_type** (`Union[None, str, TensorType]`, *optional*) -- You can give a tensor_type here to convert the lists of integers in PyTorch/Numpy Tensors at | |
| initialization.</retdesc></docstring> | |
| </div></div> | |
| ## CLIPProcessor[[transformers.CLIPProcessor]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.CLIPProcessor</name><anchor>transformers.CLIPProcessor</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/processing_clip.py#L22</source><parameters>[{"name": "image_processor", "val": " = None"}, {"name": "tokenizer", "val": " = None"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **image_processor** ([CLIPImageProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPImageProcessor), *optional*) -- | |
| The image processor is a required input. | |
| - **tokenizer** ([AutoTokenizer](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoTokenizer), *optional*) -- | |
| The tokenizer is a required input.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| Constructs a CLIP processor which wraps a CLIP image processor and a CLIP tokenizer into a single processor. | |
| [CLIPProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPProcessor) offers all the functionalities of [CLIPImageProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPImageProcessor) and [CLIPTokenizerFast](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPTokenizerFast). See the | |
| [__call__()](/docs/transformers/pr_33962/en/model_doc/vilt#transformers.ViltProcessor.__call__) and [decode()](/docs/transformers/pr_33962/en/main_classes/processors#transformers.ProcessorMixin.decode) for more information. | |
| </div> | |
| ## CLIPModel[[transformers.CLIPModel]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.CLIPModel</name><anchor>transformers.CLIPModel</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/modeling_clip.py#L822</source><parameters>[{"name": "config", "val": ": CLIPConfig"}]</parameters><paramsdesc>- **config** ([CLIPConfig](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPConfig)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| The bare Clip Model outputting raw hidden-states without any specific head on top. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.CLIPModel.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/modeling_clip.py#L938</source><parameters>[{"name": "input_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "position_ids", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "return_loss", "val": ": typing.Optional[bool] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "interpolate_pos_encoding", "val": ": bool = False"}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [CLIPImageProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPImageProcessor). See [CLIPImageProcessor.__call__()](/docs/transformers/pr_33962/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([CLIPProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPProcessor) uses | |
| [CLIPImageProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPImageProcessor) for processing images). | |
| - **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **position_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. | |
| [What are position IDs?](../glossary#position-ids) | |
| - **return_loss** (`bool`, *optional*) -- | |
| Whether or not to return the contrastive loss. | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*) -- | |
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | |
| more detail. | |
| - **interpolate_pos_encoding** (`bool`, defaults to `False`) -- | |
| Whether to interpolate the pre-trained position encodings.</paramsdesc><paramgroups>0</paramgroups><rettype>`transformers.models.clip.modeling_clip.CLIPOutput` or `tuple(torch.FloatTensor)`</rettype><retdesc>A `transformers.models.clip.modeling_clip.CLIPOutput` or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([CLIPConfig](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPConfig)) and inputs. | |
| - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`) -- Contrastive loss for image-text similarity. | |
| - **logits_per_image** (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`) -- The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text | |
| similarity scores. | |
| - **logits_per_text** (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`) -- The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image | |
| similarity scores. | |
| - **text_embeds** (`torch.FloatTensor` of shape `(batch_size, output_dim`) -- The text embeddings obtained by applying the projection layer to the pooled output of [CLIPTextModel](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPTextModel). | |
| - **image_embeds** (`torch.FloatTensor` of shape `(batch_size, output_dim`) -- The image embeddings obtained by applying the projection layer to the pooled output of [CLIPVisionModel](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPVisionModel). | |
| - **text_model_output** (`<class '~modeling_outputs.BaseModelOutputWithPooling'>.text_model_output`, defaults to `None`) -- The output of the [CLIPTextModel](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPTextModel). | |
| - **vision_model_output** (`<class '~modeling_outputs.BaseModelOutputWithPooling'>.vision_model_output`, defaults to `None`) -- The output of the [CLIPVisionModel](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPVisionModel).</retdesc></docstring> | |
| The [CLIPModel](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPModel) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.CLIPModel.forward.example"> | |
| Examples: | |
| ```python | |
| >>> import torch | |
| >>> from transformers import AutoProcessor, CLIPModel | |
| >>> from transformers.image_utils import load_image | |
| >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> image = load_image(url) | |
| >>> inputs = processor( | |
| ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True | |
| ... ) | |
| >>> with torch.inference_mode(): | |
| ... outputs = model(**inputs) | |
| >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score | |
| >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities | |
| ``` | |
| </ExampleCodeBlock> | |
| </div> | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>get_text_features</name><anchor>transformers.CLIPModel.get_text_features</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/modeling_clip.py#L862</source><parameters>[{"name": "input_ids", "val": ": Tensor"}, {"name": "attention_mask", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "position_ids", "val": ": typing.Optional[torch.Tensor] = None"}]</parameters><paramsdesc>- **input_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **position_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. | |
| [What are position IDs?](../glossary#position-ids)</paramsdesc><paramgroups>0</paramgroups><rettype>text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)</rettype><retdesc>The text embeddings obtained by | |
| applying the projection layer to the pooled output of [CLIPTextModel](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPTextModel).</retdesc></docstring> | |
| <ExampleCodeBlock anchor="transformers.CLIPModel.get_text_features.example"> | |
| Examples: | |
| ```python | |
| >>> import torch | |
| >>> from transformers import AutoTokenizer, CLIPModel | |
| >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") | |
| >>> with torch.inference_mode(): | |
| ... text_features = model.get_text_features(**inputs) | |
| ``` | |
| </ExampleCodeBlock> | |
| </div> | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>get_image_features</name><anchor>transformers.CLIPModel.get_image_features</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/modeling_clip.py#L899</source><parameters>[{"name": "pixel_values", "val": ": FloatTensor"}, {"name": "interpolate_pos_encoding", "val": ": bool = False"}]</parameters><paramsdesc>- **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [CLIPImageProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPImageProcessor). See [CLIPImageProcessor.__call__()](/docs/transformers/pr_33962/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([CLIPProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPProcessor) uses | |
| [CLIPImageProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPImageProcessor) for processing images). | |
| - **interpolate_pos_encoding** (`bool`, defaults to `False`) -- | |
| Whether to interpolate the pre-trained position encodings.</paramsdesc><paramgroups>0</paramgroups><rettype>image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)</rettype><retdesc>The image embeddings obtained by | |
| applying the projection layer to the pooled output of [CLIPVisionModel](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPVisionModel).</retdesc></docstring> | |
| <ExampleCodeBlock anchor="transformers.CLIPModel.get_image_features.example"> | |
| Examples: | |
| ```python | |
| >>> import torch | |
| >>> from transformers import AutoProcessor, CLIPModel | |
| >>> from transformers.image_utils import load_image | |
| >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> image = load_image(url) | |
| >>> inputs = processor(images=image, return_tensors="pt") | |
| >>> with torch.inference_mode(): | |
| ... image_features = model.get_image_features(**inputs) | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| ## CLIPTextModel[[transformers.CLIPTextModel]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.CLIPTextModel</name><anchor>transformers.CLIPTextModel</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/modeling_clip.py#L663</source><parameters>[{"name": "config", "val": ": CLIPTextConfig"}]</parameters><paramsdesc>- **config** ([CLIPTextConfig](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPTextConfig)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| The text model from CLIP without any head or projection on top. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.CLIPTextModel.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/modeling_clip.py#L681</source><parameters>[{"name": "input_ids", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "position_ids", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}]</parameters><paramsdesc>- **input_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **position_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. | |
| [What are position IDs?](../glossary#position-ids) | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*) -- | |
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | |
| more detail.</paramsdesc><paramgroups>0</paramgroups><rettype>[transformers.modeling_outputs.BaseModelOutputWithPooling](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or `tuple(torch.FloatTensor)`</rettype><retdesc>A [transformers.modeling_outputs.BaseModelOutputWithPooling](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([CLIPConfig](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPConfig)) and inputs. | |
| - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model. | |
| - **pooler_output** (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) -- Last layer hidden-state of the first token of the sequence (classification token) after further processing | |
| through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns | |
| the classification token after processing through a linear layer and a tanh activation function. The linear | |
| layer weights are trained from the next sentence prediction (classification) objective during pretraining. | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads.</retdesc></docstring> | |
| The [CLIPTextModel](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPTextModel) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.CLIPTextModel.forward.example"> | |
| Examples: | |
| ```python | |
| >>> from transformers import AutoTokenizer, CLIPTextModel | |
| >>> model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") | |
| >>> outputs = model(**inputs) | |
| >>> last_hidden_state = outputs.last_hidden_state | |
| >>> pooled_output = outputs.pooler_output # pooled (EOS token) states | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| ## CLIPTextModelWithProjection[[transformers.CLIPTextModelWithProjection]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.CLIPTextModelWithProjection</name><anchor>transformers.CLIPTextModelWithProjection</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/modeling_clip.py#L1030</source><parameters>[{"name": "config", "val": ": CLIPTextConfig"}]</parameters><paramsdesc>- **config** ([CLIPTextConfig](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPTextConfig)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| The Clip Model with a projection layer on top (a linear layer on top of the pooled output). | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.CLIPTextModelWithProjection.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/modeling_clip.py#L1053</source><parameters>[{"name": "input_ids", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "position_ids", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}]</parameters><paramsdesc>- **input_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_33962/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_33962/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **position_ids** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. | |
| [What are position IDs?](../glossary#position-ids) | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*) -- | |
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | |
| more detail.</paramsdesc><paramgroups>0</paramgroups><rettype>`transformers.models.clip.modeling_clip.CLIPTextModelOutput` or `tuple(torch.FloatTensor)`</rettype><retdesc>A `transformers.models.clip.modeling_clip.CLIPTextModelOutput` or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([CLIPConfig](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPConfig)) and inputs. | |
| - **text_embeds** (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`) -- The text embeddings obtained by applying the projection layer to the pooler_output. | |
| - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*, defaults to `None`) -- Sequence of hidden-states at the output of the last layer of the model. | |
| - **hidden_states** (`tuple[torch.FloatTensor, ...]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple[torch.FloatTensor, ...]`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads.</retdesc></docstring> | |
| The [CLIPTextModelWithProjection](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPTextModelWithProjection) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.CLIPTextModelWithProjection.forward.example"> | |
| Examples: | |
| ```python | |
| >>> import torch | |
| >>> from transformers import AutoTokenizer, CLIPTextModelWithProjection | |
| >>> model = CLIPTextModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") | |
| >>> with torch.inference_mode(): | |
| ... outputs = model(**inputs) | |
| >>> text_embeds = outputs.text_embeds | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| ## CLIPVisionModelWithProjection[[transformers.CLIPVisionModelWithProjection]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.CLIPVisionModelWithProjection</name><anchor>transformers.CLIPVisionModelWithProjection</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/modeling_clip.py#L1099</source><parameters>[{"name": "config", "val": ": CLIPVisionConfig"}]</parameters><paramsdesc>- **config** ([CLIPVisionConfig](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPVisionConfig)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| The Clip Model with a projection layer on top (a linear layer on top of the pooled output). | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.CLIPVisionModelWithProjection.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/modeling_clip.py#L1117</source><parameters>[{"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "interpolate_pos_encoding", "val": ": bool = False"}]</parameters><paramsdesc>- **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [CLIPImageProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPImageProcessor). See [CLIPImageProcessor.__call__()](/docs/transformers/pr_33962/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([CLIPProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPProcessor) uses | |
| [CLIPImageProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPImageProcessor) for processing images). | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*) -- | |
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | |
| more detail. | |
| - **interpolate_pos_encoding** (`bool`, defaults to `False`) -- | |
| Whether to interpolate the pre-trained position encodings.</paramsdesc><paramgroups>0</paramgroups><rettype>`transformers.models.clip.modeling_clip.CLIPVisionModelOutput` or `tuple(torch.FloatTensor)`</rettype><retdesc>A `transformers.models.clip.modeling_clip.CLIPVisionModelOutput` or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([CLIPConfig](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPConfig)) and inputs. | |
| - **image_embeds** (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`) -- The image embeddings obtained by applying the projection layer to the pooler_output. | |
| - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*, defaults to `None`) -- Sequence of hidden-states at the output of the last layer of the model. | |
| - **hidden_states** (`tuple[torch.FloatTensor, ...]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple[torch.FloatTensor, ...]`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads.</retdesc></docstring> | |
| The [CLIPVisionModelWithProjection](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPVisionModelWithProjection) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.CLIPVisionModelWithProjection.forward.example"> | |
| Examples: | |
| ```python | |
| >>> import torch | |
| >>> from transformers import AutoProcessor, CLIPVisionModelWithProjection | |
| >>> from transformers.image_utils import load_image | |
| >>> model = CLIPVisionModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> image = load_image(url) | |
| >>> inputs = processor(images=image, return_tensors="pt") | |
| >>> with torch.inference_mode(): | |
| ... outputs = model(**inputs) | |
| >>> image_embeds = outputs.image_embeds | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| ## CLIPVisionModel[[transformers.CLIPVisionModel]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.CLIPVisionModel</name><anchor>transformers.CLIPVisionModel</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/modeling_clip.py#L769</source><parameters>[{"name": "config", "val": ": CLIPVisionConfig"}]</parameters><paramsdesc>- **config** ([CLIPVisionConfig](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPVisionConfig)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| The vision model from CLIP without any head or projection on top. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.CLIPVisionModel.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/modeling_clip.py#L783</source><parameters>[{"name": "pixel_values", "val": ": typing.Optional[torch.FloatTensor] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "interpolate_pos_encoding", "val": ": bool = False"}]</parameters><paramsdesc>- **pixel_values** (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [CLIPImageProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPImageProcessor). See [CLIPImageProcessor.__call__()](/docs/transformers/pr_33962/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([CLIPProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPProcessor) uses | |
| [CLIPImageProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPImageProcessor) for processing images). | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*) -- | |
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | |
| more detail. | |
| - **interpolate_pos_encoding** (`bool`, defaults to `False`) -- | |
| Whether to interpolate the pre-trained position encodings.</paramsdesc><paramgroups>0</paramgroups><rettype>[transformers.modeling_outputs.BaseModelOutputWithPooling](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or `tuple(torch.FloatTensor)`</rettype><retdesc>A [transformers.modeling_outputs.BaseModelOutputWithPooling](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([CLIPConfig](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPConfig)) and inputs. | |
| - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model. | |
| - **pooler_output** (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) -- Last layer hidden-state of the first token of the sequence (classification token) after further processing | |
| through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns | |
| the classification token after processing through a linear layer and a tanh activation function. The linear | |
| layer weights are trained from the next sentence prediction (classification) objective during pretraining. | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads.</retdesc></docstring> | |
| The [CLIPVisionModel](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPVisionModel) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.CLIPVisionModel.forward.example"> | |
| Example: | |
| ```python | |
| >>> from PIL import Image | |
| >>> import requests | |
| >>> from transformers import AutoProcessor, CLIPVisionModel | |
| >>> model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" | |
| >>> image = Image.open(requests.get(url, stream=True).raw) | |
| >>> inputs = processor(images=image, return_tensors="pt") | |
| >>> outputs = model(**inputs) | |
| >>> last_hidden_state = outputs.last_hidden_state | |
| >>> pooled_output = outputs.pooler_output # pooled CLS states | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| ## CLIPForImageClassification[[transformers.CLIPForImageClassification]] | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>class transformers.CLIPForImageClassification</name><anchor>transformers.CLIPForImageClassification</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/modeling_clip.py#L1170</source><parameters>[{"name": "config", "val": ": CLIPConfig"}]</parameters><paramsdesc>- **config** ([CLIPConfig](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPConfig)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.</paramsdesc><paramgroups>0</paramgroups></docstring> | |
| CLIP vision encoder with an image classification head on top (a linear layer on top of the pooled final hidden states of | |
| the patch tokens) e.g. for ImageNet. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_33962/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| <div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8"> | |
| <docstring><name>forward</name><anchor>transformers.CLIPForImageClassification.forward</anchor><source>https://github.com/huggingface/transformers/blob/vr_33962/src/transformers/models/clip/modeling_clip.py#L1188</source><parameters>[{"name": "pixel_values", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "labels", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}]</parameters><paramsdesc>- **pixel_values** (`torch.Tensor` of shape `(batch_size, num_channels, image_size, image_size)`, *optional*) -- | |
| The tensors corresponding to the input images. Pixel values can be obtained using | |
| [CLIPImageProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPImageProcessor). See [CLIPImageProcessor.__call__()](/docs/transformers/pr_33962/en/model_doc/fuyu#transformers.FuyuImageProcessor.__call__) for details ([CLIPProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPProcessor) uses | |
| [CLIPImageProcessor](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPImageProcessor) for processing images). | |
| - **labels** (`torch.LongTensor` of shape `(batch_size,)`, *optional*) -- | |
| Labels for computing the image classification/regression loss. Indices should be in `[0, ..., | |
| config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If | |
| `config.num_labels > 1` a classification loss is computed (Cross-Entropy). | |
| - **output_attentions** (`bool`, *optional*) -- | |
| Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned | |
| tensors for more detail. | |
| - **output_hidden_states** (`bool`, *optional*) -- | |
| Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for | |
| more detail.</paramsdesc><paramgroups>0</paramgroups><rettype>[transformers.modeling_outputs.ImageClassifierOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.ImageClassifierOutput) or `tuple(torch.FloatTensor)`</rettype><retdesc>A [transformers.modeling_outputs.ImageClassifierOutput](/docs/transformers/pr_33962/en/main_classes/output#transformers.modeling_outputs.ImageClassifierOutput) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([CLIPConfig](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPConfig)) and inputs. | |
| - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Classification (or regression if config.num_labels==1) loss. | |
| - **logits** (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`) -- Classification (or regression if config.num_labels==1) scores (before SoftMax). | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states | |
| (also called feature maps) of the model at the output of each stage. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, patch_size, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads.</retdesc></docstring> | |
| The [CLIPForImageClassification](/docs/transformers/pr_33962/en/model_doc/clip#transformers.CLIPForImageClassification) forward method, overrides the `__call__` special method. | |
| <Tip> | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| </Tip> | |
| <ExampleCodeBlock anchor="transformers.CLIPForImageClassification.forward.example"> | |
| Example: | |
| ```python | |
| >>> from transformers import AutoImageProcessor, CLIPForImageClassification | |
| >>> import torch | |
| >>> from datasets import load_dataset | |
| >>> dataset = load_dataset("huggingface/cats-image") | |
| >>> image = dataset["test"]["image"][0] | |
| >>> image_processor = AutoImageProcessor.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> model = CLIPForImageClassification.from_pretrained("openai/clip-vit-base-patch32") | |
| >>> inputs = image_processor(image, return_tensors="pt") | |
| >>> with torch.no_grad(): | |
| ... logits = model(**inputs).logits | |
| >>> # model predicts one of the 1000 ImageNet classes | |
| >>> predicted_label = logits.argmax(-1).item() | |
| >>> print(model.config.id2label[predicted_label]) | |
| ... | |
| ``` | |
| </ExampleCodeBlock> | |
| </div></div> | |
| <EditOnGithub source="https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/clip.md" /> |
Xet Storage Details
- Size:
- 88.7 kB
- Xet hash:
- e26dc731a6a661068470bd0554eb50aa3039a8c59861911e039c3a5dc723661e
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.