Buckets:
| # Zamba2 | |
| [Zamba2](https://huggingface.co/papers/2411.15242) is a large language model (LLM) trained by Zyphra, and made available under an Apache 2.0 license. Please see the [Zyphra Hugging Face](https://huggingface.co/collections/zyphra/) repository for model weights. | |
| This model was contributed by [pglo](https://huggingface.co/pglo). | |
| ## Model details | |
| [Zamba2-1.2B](https://www.zyphra.com/post/zamba2-mini), [Zamba2-2.7B](https://www.zyphra.com/post/zamba2-small) and [Zamba2-7B](https://www.zyphra.com/post/zamba2-7b) are hybrid models combining state-space models (Specifically [Mamba2](https://github.com/state-spaces/mamba)) and transformer, and were trained using next-token prediction. Zamba2 uses shared transformer layers after every 6 mamba blocks. It uses the [Mistral v0.1 tokenizer](https://huggingface.co/mistralai/Mistral-7B-v0.1). We came to this architecture after a series of ablations at small scales. Zamba2-1.2B, Zamba2-2.7B and Zamba2-7B were pre-trained on 2T and 3T tokens, respectively. | |
| ## Quick start | |
| ### Presequities | |
| Zamba2 requires you use `transformers` version 4.48.0 or higher: | |
| ```bash | |
| pip install transformers>=4.48.0 | |
| ``` | |
| ## Inference | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba2-7B") | |
| model = AutoModelForCausalLM.from_pretrained("Zyphra/Zamba2-7B", device_map="auto") | |
| input_text = "What factors contributed to the fall of the Roman Empire?" | |
| input_ids = tokenizer(input_text, return_tensors="pt").to(model.device) | |
| outputs = model.generate(**input_ids, max_new_tokens=100) | |
| print(tokenizer.decode(outputs[0])) | |
| ``` | |
| ## Model card | |
| The model cards can be found at: | |
| * [Zamba2-1.2B](https://huggingface.co/Zyphra/Zamba2-1.2B) | |
| * [Zamba2-2.7B](https://huggingface.co/Zyphra/Zamba2-2.7B) | |
| * [Zamba2-7B](https://huggingface.co/Zyphra/Zamba2-7B) | |
| ## Issues | |
| For issues with model output, or community discussion, please use the Hugging Face community [forum](https://huggingface.co/Zyphra/Zamba2-7B/discussions) | |
| ## License | |
| The model weights are open-sourced via an Apache 2.0 license. | |
| ## Zamba2Config[[transformers.Zamba2Config]] | |
| - **vocab_size** (`int`, *optional*, defaults to `32000`) -- | |
| Vocabulary size of the model. Defines the number of different tokens that can be represented by the `input_ids`. | |
| - **max_position_embeddings** (`int`, *optional*, defaults to `4096`) -- | |
| The maximum sequence length that this model might ever be used with. | |
| - **hidden_size** (`int`, *optional*, defaults to `2560`) -- | |
| Dimension of the hidden representations. | |
| - **num_hidden_layers** (`int`, *optional*, defaults to `54`) -- | |
| Number of hidden layers in the Transformer decoder. | |
| - **layers_block_type** (`list[str]`, *optional*) -- | |
| A list that explicitly maps each layer index with its layer type. If not provided, it will be automatically | |
| generated based on config values. | |
| - **mamba_d_state** (`int`, *optional*, defaults to `64`) -- | |
| Size of the SSM state (latent state dimension) in the Mamba layers. | |
| - **mamba_d_conv** (`int`, *optional*, defaults to `4`) -- | |
| The size of the mamba convolution kernel | |
| - **mamba_expand** (`int`, *optional*, defaults to `2`) -- | |
| Expanding factor (relative to hidden_size) used to determine the mamba intermediate size | |
| - **mamba_ngroups** (`int`, *optional*, defaults to 1) -- | |
| Number of groups for the evolution matrices of mamba 2. | |
| - **time_step_min** (`float`, *optional*, defaults to `0.001`) -- | |
| Minimum `time_step` used to bound `dt_proj.bias`. | |
| - **time_step_max** (`float`, *optional*, defaults to `0.1`) -- | |
| Maximum `time_step` used to bound `dt_proj.bias`. | |
| - **time_step_floor** (`float`, *optional*, defaults to `0.0001`) -- | |
| Minimum allowed value for the discrete time step delta after softplus activation. | |
| - **n_mamba_heads** (`int`, *optional*, defaults to 8) -- | |
| Number of heads for the evolution matrices of mamba 2. | |
| - **use_mamba_kernels** (`bool`, *optional*, defaults to `True`) -- | |
| Flag indicating whether or not to use the fast mamba kernels. | |
| - **use_conv_bias** (`bool`, *optional*, defaults to `True`) -- | |
| Whether or not to use bias in the convolution layer of the mixer block. | |
| - **chunk_size** (`int`, *optional*, defaults to 256) -- | |
| Size of the chunks that will comprise the sequence. | |
| - **use_mem_eff_path** (`bool`, *optional*, defaults to `False`) -- | |
| Whether or not to use the fused conv1d and scan in mamba2 layers. | |
| - **add_bias_linear** (`bool`, *optional*, defaults to `False`) -- | |
| Flag indicating whether or not to use bias in various layers | |
| - **intermediate_size** (`int`, *optional*) -- | |
| Dimension of the MLP representations. | |
| - **hidden_act** (`str`, *optional*, defaults to `gelu`) -- | |
| The non-linear activation function (function or string) in the decoder. For example, `"gelu"`, | |
| `"relu"`, `"silu"`, etc. | |
| - **num_attention_heads** (`int`, *optional*, defaults to `32`) -- | |
| Number of attention heads for each attention layer in the Transformer decoder. | |
| - **num_key_value_heads** (`int`, *optional*) -- | |
| This is the number of key_value heads that should be used to implement Grouped Query Attention. If | |
| `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if | |
| `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When | |
| converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed | |
| by meanpooling all the original heads within that group. For more details, check out [this | |
| paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to | |
| `num_attention_heads`. | |
| - **attention_dropout** (`Union[float, int]`, *optional*, defaults to `0.0`) -- | |
| The dropout ratio for the attention probabilities. | |
| - **num_mem_blocks** (`int`, *optional*, defaults to 1) -- | |
| Number of unshared transformer blocks. | |
| - **use_shared_attention_adapter** (`bool`, *optional*, defaults to `False`) -- | |
| If True, unshared adapters (formally the same as LoRA but used in the base model) will be added to the q, k, v projectors in the shared attention layers. | |
| - **adapter_rank** (`int`, *optional*, defaults to 128) -- | |
| Rank of the adapter in the shared MLP and shared attention layers. | |
| - **use_mem_rope** (`bool`, *optional*, defaults to `False`) -- | |
| If True, includes RoPE in the shared attention layers. | |
| - **rope_parameters** (`Union[~modeling_rope_utils.RopeParameters, dict]`, *optional*) -- | |
| Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain | |
| a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE | |
| with longer `max_position_embeddings`. | |
| - **initializer_range** (`float`, *optional*, defaults to `0.02`) -- | |
| The standard deviation of the truncated_normal_initializer for initializing all weight matrices. | |
| - **rms_norm_eps** (`float`, *optional*, defaults to `1e-05`) -- | |
| The epsilon used by the rms normalization layers. | |
| - **use_cache** (`bool`, *optional*, defaults to `True`) -- | |
| Whether or not the model should return the last key/values attentions (not used by all models). Only | |
| relevant if `config.is_decoder=True` or when the model is a decoder-only generative model. | |
| - **num_logits_to_keep** (`int` or `None`, *optional*, defaults to 1) -- | |
| Number of prompt logits to calculate during generation. If `None`, all logits will be calculated. If an | |
| integer value, only last `num_logits_to_keep` logits will be calculated. Default is 1 because only the | |
| logits of the last prompt token are needed for generation. For long sequences, the logits for the entire | |
| sequence may use a lot of memory so, setting `num_logits_to_keep=1` will reduce memory footprint | |
| significantly. | |
| - **pad_token_id** (`int`, *optional*, defaults to `0`) -- | |
| Token id used for padding in the vocabulary. | |
| - **bos_token_id** (`int`, *optional*, defaults to `1`) -- | |
| Token id used for beginning-of-stream in the vocabulary. | |
| - **eos_token_id** (`Union[int, list[int]]`, *optional*, defaults to `2`) -- | |
| Token id used for end-of-stream in the vocabulary. | |
| - **use_long_context** (`bool`, *optional*, defaults to `False`) -- | |
| Activates the context-extended version of Zamba by modifying RoPE. | |
| - **tie_word_embeddings** (`bool`, *optional*, defaults to `True`) -- | |
| Whether to tie weight embeddings according to model's `tied_weights_keys` mapping. | |
| This is the configuration class to store the configuration of a Zamba2Model. It is used to instantiate a Zamba2 | |
| model according to the specified arguments, defining the model architecture. Instantiating a configuration with the | |
| defaults will yield a similar configuration to that of the [Zyphra/Zamba2-2.7B](https://huggingface.co/Zyphra/Zamba2-2.7B) | |
| Configuration objects inherit from [PreTrainedConfig](/docs/transformers/pr_40546/en/main_classes/configuration#transformers.PreTrainedConfig) and can be used to control the model outputs. Read the | |
| documentation from [PreTrainedConfig](/docs/transformers/pr_40546/en/main_classes/configuration#transformers.PreTrainedConfig) for more information. | |
| Example: | |
| ```python | |
| >>> from transformers import Zamba2Model, Zamba2Config | |
| >>> # Initializing a Zamba2-2.7B style configuration | |
| >>> configuration = Zamba2Config() | |
| >>> # Initializing a model from the Zamba2-2.7B style configuration | |
| >>> model = Zamba2Model(configuration) | |
| >>> # Accessing the model configuration | |
| >>> configuration = model.config | |
| ``` | |
| ## Zamba2Model[[transformers.Zamba2Model]] | |
| - **config** ([Zamba2Config](/docs/transformers/pr_40546/en/model_doc/zamba2#transformers.Zamba2Config)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. | |
| The bare Zamba2 Model outputting raw hidden-states without any specific head on top. | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| - **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_40546/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_40546/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_40546/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **position_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. | |
| [What are position IDs?](../glossary#position-ids) | |
| - **past_key_values** (`~cache_utils.Cache`, *optional*) -- | |
| Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention | |
| blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` | |
| returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. | |
| Only [Cache](/docs/transformers/pr_40546/en/internal/generation_utils#transformers.Cache) instance is allowed as input, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). | |
| If no `past_key_values` are passed, [DynamicCache](/docs/transformers/pr_40546/en/internal/generation_utils#transformers.DynamicCache) will be initialized by default. | |
| The model will output the same cache format that is fed as input. | |
| If `past_key_values` are used, the user is expected to input only unprocessed `input_ids` (those that don't | |
| have their past key value states given to this model) of shape `(batch_size, unprocessed_length)` instead of all `input_ids` | |
| of shape `(batch_size, sequence_length)`. | |
| - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This | |
| is useful if you want more control over how to convert `input_ids` indices into associated vectors than the | |
| model's internal embedding lookup matrix. | |
| - **use_cache** (`bool`, *optional*) -- | |
| If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see | |
| `past_key_values`).[BaseModelOutputWithPast](/docs/transformers/pr_40546/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPast) or `tuple(torch.FloatTensor)`A [BaseModelOutputWithPast](/docs/transformers/pr_40546/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPast) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([Zamba2Config](/docs/transformers/pr_40546/en/model_doc/zamba2#transformers.Zamba2Config)) and inputs. | |
| The [Zamba2Model](/docs/transformers/pr_40546/en/model_doc/zamba2#transformers.Zamba2Model) forward method, overrides the `__call__` special method. | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| - **last_hidden_state** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) -- Sequence of hidden-states at the output of the last layer of the model. | |
| If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, | |
| hidden_size)` is output. | |
| - **past_key_values** (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`) -- It is a [Cache](/docs/transformers/pr_40546/en/internal/generation_utils#transformers.Cache) instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). | |
| Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if | |
| `config.is_encoder_decoder=True` in the cross-attention blocks) that can be used (see `past_key_values` | |
| input) to speed up sequential decoding. | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads. | |
| ## Zamba2ForCausalLM[[transformers.Zamba2ForCausalLM]] | |
| - **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_40546/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_40546/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_40546/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **position_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. | |
| [What are position IDs?](../glossary#position-ids) | |
| - **past_key_values** (`~cache_utils.Cache`, *optional*) -- | |
| Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention | |
| blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` | |
| returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. | |
| Only [Cache](/docs/transformers/pr_40546/en/internal/generation_utils#transformers.Cache) instance is allowed as input, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). | |
| If no `past_key_values` are passed, [DynamicCache](/docs/transformers/pr_40546/en/internal/generation_utils#transformers.DynamicCache) will be initialized by default. | |
| The model will output the same cache format that is fed as input. | |
| If `past_key_values` are used, the user is expected to input only unprocessed `input_ids` (those that don't | |
| have their past key value states given to this model) of shape `(batch_size, unprocessed_length)` instead of all `input_ids` | |
| of shape `(batch_size, sequence_length)`. | |
| - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This | |
| is useful if you want more control over how to convert `input_ids` indices into associated vectors than the | |
| model's internal embedding lookup matrix. | |
| - **labels** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., | |
| config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored | |
| (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. | |
| - **use_cache** (`bool`, *optional*) -- | |
| If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see | |
| `past_key_values`). | |
| - **logits_to_keep** (`Union[int, torch.Tensor]`, *optional*, defaults to `0`) -- | |
| If an `int`, compute logits for the last `logits_to_keep` tokens. If `0`, calculate logits for all | |
| `input_ids` (special case). Only last token logits are needed for generation, and calculating them only for that | |
| token can save memory, which becomes pretty significant for long sequences or large vocabulary size. | |
| If a `torch.Tensor`, must be 1D corresponding to the indices to keep in the sequence length dimension. | |
| This is useful when using packed tensor format (single dimension for batch and sequence length).[CausalLMOutputWithPast](/docs/transformers/pr_40546/en/main_classes/output#transformers.modeling_outputs.CausalLMOutputWithPast) or `tuple(torch.FloatTensor)`A [CausalLMOutputWithPast](/docs/transformers/pr_40546/en/main_classes/output#transformers.modeling_outputs.CausalLMOutputWithPast) or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([Zamba2Config](/docs/transformers/pr_40546/en/model_doc/zamba2#transformers.Zamba2Config)) and inputs. | |
| The [Zamba2ForCausalLM](/docs/transformers/pr_40546/en/model_doc/zamba2#transformers.Zamba2ForCausalLM) forward method, overrides the `__call__` special method. | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Language modeling loss (for next-token prediction). | |
| - **logits** (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) -- Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). | |
| - **past_key_values** (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`) -- It is a [Cache](/docs/transformers/pr_40546/en/internal/generation_utils#transformers.Cache) instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). | |
| Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see | |
| `past_key_values` input) to speed up sequential decoding. | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads. | |
| Example: | |
| ```python | |
| >>> from transformers import AutoTokenizer, Zamba2ForCausalLM | |
| >>> model = Zamba2ForCausalLM.from_pretrained("Zyphra/Zamba2-7B-v1") | |
| >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba2-7B-v1") | |
| >>> prompt = "Hey, are you conscious? Can you talk to me?" | |
| >>> inputs = tokenizer(prompt, return_tensors="pt") | |
| >>> # Generate | |
| >>> generate_ids = model.generate(inputs.input_ids, max_length=30) | |
| >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] | |
| "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." | |
| ``` | |
| ## Zamba2ForSequenceClassification[[transformers.Zamba2ForSequenceClassification]] | |
| - **config** ([Zamba2Config](/docs/transformers/pr_40546/en/model_doc/zamba2#transformers.Zamba2Config)) -- | |
| Model configuration class with all the parameters of the model. Initializing with a config file does not | |
| load the weights associated with the model, only the configuration. Check out the | |
| [from_pretrained()](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights. | |
| The Zamba2 Model with a sequence classification head on top (linear layer). | |
| [Zamba2ForSequenceClassification](/docs/transformers/pr_40546/en/model_doc/zamba2#transformers.Zamba2ForSequenceClassification) uses the last token in order to do the classification, as other causal models | |
| (e.g. GPT-2) do. | |
| Since it does classification on the last token, it requires to know the position of the last token. If a | |
| `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If | |
| no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the | |
| padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in | |
| each row of the batch). | |
| This model inherits from [PreTrainedModel](/docs/transformers/pr_40546/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the | |
| library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads | |
| etc.) | |
| This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. | |
| Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage | |
| and behavior. | |
| - **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of input sequence tokens in the vocabulary. Padding will be ignored by default. | |
| Indices can be obtained using [AutoTokenizer](/docs/transformers/pr_40546/en/model_doc/auto#transformers.AutoTokenizer). See [PreTrainedTokenizer.encode()](/docs/transformers/pr_40546/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.encode) and | |
| [PreTrainedTokenizer.__call__()](/docs/transformers/pr_40546/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.__call__) for details. | |
| [What are input IDs?](../glossary#input-ids) | |
| - **attention_mask** (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: | |
| - 1 for tokens that are **not masked**, | |
| - 0 for tokens that are **masked**. | |
| [What are attention masks?](../glossary#attention-mask) | |
| - **position_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) -- | |
| Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. | |
| [What are position IDs?](../glossary#position-ids) | |
| - **past_key_values** (`~cache_utils.Cache`, *optional*) -- | |
| Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention | |
| blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` | |
| returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. | |
| Only [Cache](/docs/transformers/pr_40546/en/internal/generation_utils#transformers.Cache) instance is allowed as input, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). | |
| If no `past_key_values` are passed, [DynamicCache](/docs/transformers/pr_40546/en/internal/generation_utils#transformers.DynamicCache) will be initialized by default. | |
| The model will output the same cache format that is fed as input. | |
| If `past_key_values` are used, the user is expected to input only unprocessed `input_ids` (those that don't | |
| have their past key value states given to this model) of shape `(batch_size, unprocessed_length)` instead of all `input_ids` | |
| of shape `(batch_size, sequence_length)`. | |
| - **inputs_embeds** (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) -- | |
| Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This | |
| is useful if you want more control over how to convert `input_ids` indices into associated vectors than the | |
| model's internal embedding lookup matrix. | |
| - **labels** (`torch.LongTensor` of shape `(batch_size,)`, *optional*) -- | |
| Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., | |
| config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If | |
| `config.num_labels > 1` a classification loss is computed (Cross-Entropy). | |
| - **use_cache** (`bool`, *optional*) -- | |
| If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see | |
| `past_key_values`). | |
| - **logits_to_keep** (`Union[int, torch.Tensor]`, *optional*, defaults to `0`) -- | |
| If an `int`, compute logits for the last `logits_to_keep` tokens. If `0`, calculate logits for all | |
| `input_ids` (special case). Only last token logits are needed for generation, and calculating them only for that | |
| token can save memory, which becomes pretty significant for long sequences or large vocabulary size. | |
| If a `torch.Tensor`, must be 1D corresponding to the indices to keep in the sequence length dimension. | |
| This is useful when using packed tensor format (single dimension for batch and sequence length).`SequenceClassifierOutputWithPast` or `tuple(torch.FloatTensor)`A `SequenceClassifierOutputWithPast` or a tuple of | |
| `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various | |
| elements depending on the configuration ([Zamba2Config](/docs/transformers/pr_40546/en/model_doc/zamba2#transformers.Zamba2Config)) and inputs. | |
| The [Zamba2ForSequenceClassification](/docs/transformers/pr_40546/en/model_doc/zamba2#transformers.Zamba2ForSequenceClassification) forward method, overrides the `__call__` special method. | |
| Although the recipe for forward pass needs to be defined within this function, one should call the `Module` | |
| instance afterwards instead of this since the former takes care of running the pre and post processing steps while | |
| the latter silently ignores them. | |
| - **loss** (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) -- Classification (or regression if config.num_labels==1) loss. | |
| - **logits** (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`) -- Classification (or regression if config.num_labels==1) scores (before SoftMax). | |
| - **past_key_values** (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`) -- It is a [Cache](/docs/transformers/pr_40546/en/internal/generation_utils#transformers.Cache) instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). | |
| Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see | |
| `past_key_values` input) to speed up sequential decoding. | |
| - **hidden_states** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) -- Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + | |
| one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. | |
| Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. | |
| - **attentions** (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) -- Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, | |
| sequence_length)`. | |
| Attentions weights after the attention softmax, used to compute the weighted average in the self-attention | |
| heads. | |
| Example of single-label classification: | |
| ```python | |
| >>> import torch | |
| >>> from transformers import AutoTokenizer, Zamba2ForSequenceClassification | |
| >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba2-2.7B") | |
| >>> model = Zamba2ForSequenceClassification.from_pretrained("Zyphra/Zamba2-2.7B") | |
| >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") | |
| >>> with torch.no_grad(): | |
| ... logits = model(**inputs).logits | |
| >>> predicted_class_id = logits.argmax().item() | |
| >>> model.config.id2label[predicted_class_id] | |
| ... | |
| >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` | |
| >>> num_labels = len(model.config.id2label) | |
| >>> model = Zamba2ForSequenceClassification.from_pretrained("Zyphra/Zamba2-2.7B", num_labels=num_labels) | |
| >>> labels = torch.tensor([1]) | |
| >>> loss = model(**inputs, labels=labels).loss | |
| >>> round(loss.item(), 2) | |
| ... | |
| ``` | |
| Example of multi-label classification: | |
| ```python | |
| >>> import torch | |
| >>> from transformers import AutoTokenizer, Zamba2ForSequenceClassification | |
| >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba2-2.7B") | |
| >>> model = Zamba2ForSequenceClassification.from_pretrained("Zyphra/Zamba2-2.7B", problem_type="multi_label_classification") | |
| >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") | |
| >>> with torch.no_grad(): | |
| ... logits = model(**inputs).logits | |
| >>> predicted_class_ids = torch.arange(0, logits.shape[-1])[torch.sigmoid(logits).squeeze(dim=0) > 0.5] | |
| >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` | |
| >>> num_labels = len(model.config.id2label) | |
| >>> model = Zamba2ForSequenceClassification.from_pretrained( | |
| ... "Zyphra/Zamba2-2.7B", num_labels=num_labels, problem_type="multi_label_classification" | |
| ... ) | |
| >>> labels = torch.sum( | |
| ... torch.nn.functional.one_hot(predicted_class_ids[None, :].clone(), num_classes=num_labels), dim=1 | |
| ... ).to(torch.float) | |
| >>> loss = model(**inputs, labels=labels).loss | |
| ``` | |
Xet Storage Details
- Size:
- 33.6 kB
- Xet hash:
- 9bbd3d3c5c84a59cf647761faba8f5570e8b968146e7fe1fd446b0bb717cc79d
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.