Upload config

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

config.json

@@ -0,0 +1,99 @@
+{
+  "_attn_implementation_autoset": true,
+  "auto_map": {
+    "AutoConfig": "configuration_meralion3.MERaLiON3Config",
+    "AutoModelForSpeechSeq2Seq": "modeling_meralion3.MERaLiON3ForConditionalGeneration",
+    "AutoProcessor": "processing_meralion3.MERaLiON3Processor"
+  },
+  "cache_implementation": "hybrid",
+  "fixed_speech_embeds_length": 300,
+  "head_dim": 256,
+  "hidden_size": 3584,
+  "intermediate_size": 14336,
+  "model_type": "meralion3",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 42,
+  "num_key_value_heads": 8,
+  "sliding_window": 4096,
+  "speech_config": {
+    "_attn_implementation_autoset": true,
+    "_name_or_path": "/data/projects/13003558/lewiswon/models/meralion_whisper_v3_normed_cleaned",
+    "activation_dropout": 0.0,
+    "activation_function": "gelu",
+    "apply_spec_augment": true,
+    "architectures": [
+      "WhisperForConditionalGeneration"
+    ],
+    "attention_dropout": 0.0,
+    "begin_suppress_tokens": null,
+    "bos_token_id": 50257,
+    "classifier_proj_size": 256,
+    "d_model": 1280,
+    "decoder_attention_heads": 20,
+    "decoder_ffn_dim": 5120,
+    "decoder_layerdrop": 0.0,
+    "decoder_layers": 32,
+    "decoder_start_token_id": 50258,
+    "dropout": 0.0,
+    "encoder_attention_heads": 20,
+    "encoder_ffn_dim": 5120,
+    "encoder_layerdrop": 0.0,
+    "encoder_layers": 32,
+    "eos_token_id": 50257,
+    "init_std": 0.02,
+    "mask_feature_length": 10,
+    "mask_feature_min_masks": 0,
+    "mask_feature_prob": 0.1,
+    "mask_time_length": 20,
+    "mask_time_min_masks": 2,
+    "mask_time_prob": 0.1,
+    "max_length": null,
+    "max_source_positions": 1500,
+    "max_target_positions": 448,
+    "median_filter_width": 7,
+    "model_type": "whisper",
+    "num_hidden_layers": 32,
+    "num_mel_bins": 128,
+    "scale_embedding": false,
+    "torch_dtype": "bfloat16",
+    "use_cache": true,
+    "use_weighted_layer_sum": true,
+    "vocab_size": 51866
+  },
+  "speech_mlp_scale_factor": 5,
+  "speech_mlp_use_projection": true,
+  "speech_token_index": 255999,
+  "text_config": {
+    "_attn_implementation_autoset": true,
+    "_name_or_path": "/home/models/gemma2_9b-sg-inst",
+    "architectures": [
+      "Gemma2ForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_logit_softcapping": 50.0,
+    "cache_implementation": "hybrid",
+    "eos_token_id": 107,
+    "final_logit_softcapping": 30.0,
+    "head_dim": 256,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 3584,
+    "initializer_range": 0.02,
+    "intermediate_size": 14336,
+    "max_position_embeddings": 8192,
+    "model_type": "gemma2",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 42,
+    "num_key_value_heads": 8,
+    "query_pre_attn_scalar": 256,
+    "rms_norm_eps": 1e-06,
+    "rope_theta": 10000.0,
+    "sliding_window": 4096,
+    "sliding_window_size": 4096,
+    "torch_dtype": "bfloat16",
+    "use_cache": true,
+    "vocab_size": 256000
+  },
+  "transformers_version": "4.51.3"
+}

configuration_meralion3.py

@@ -0,0 +1,98 @@
+"""MERaLiON2 model configuration"""
+
+from transformers import Gemma2Config, WhisperConfig
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+
+logger = logging.get_logger(__name__)
+
+
+class MERaLiON3Config(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`MERaLiON3ForConditionalGeneration`]. It is used to instantiate an
+    MERaLiON3 model according to the specified arguments, defining the model architecture. Instantiating a configuration
+    with the defaults will yield a similar configuration to that of the MERaLiON3.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+    Args:
+        audio_config (`Union[AutoConfig, dict]`,  *optional*, defaults to `CLIPVisionConfig`):
+            The config object or dictionary of the audio backbone.
+        text_config (`Union[AutoConfig, dict]`, *optional*, defaults to `LlamaConfig`):
+            The config object or dictionary of the text backbone.
+        audio_token_index (`int`, *optional*, defaults to 151646):
+            The image token index to encode the image prompt.
+    """
+
+    model_type = "meralion3"
+    is_composition = False
+
+    def __init__(
+        self,
+        speech_config=None,
+        text_config=None,
+        speech_mlp_use_projection=True,
+        speech_mlp_scale_factor=5,
+        speech_token_index=255999,
+        fixed_speech_embeds_length=None,
+        **kwargs,
+    ):
+        # Set as instance attribute so it's serialized into config.json,
+        # enabling trust_remote_code=True loading without manual registration.
+        self.auto_map = {
+            "AutoConfig": "configuration_meralion3.MERaLiON3Config",
+            "AutoModelForSpeechSeq2Seq": "modeling_meralion3.MERaLiON3ForConditionalGeneration",
+            "AutoProcessor": "processing_meralion3.MERaLiON3Processor",
+        }
+
+        if isinstance(speech_config, dict):
+            speech_config = WhisperConfig(**speech_config)
+        elif speech_config is None:
+            speech_config = WhisperConfig(
+                d_model=1280,
+                encoder_attention_heads=20,
+                encoder_ffn_dim=5120,
+                encoder_layerdrop=0.0,
+                encoder_layers=32,
+                num_mel_bins=128,
+                max_source_positions=1500,
+                scale_embedding=False,
+                activation_function="gelu",
+            )
+
+        self.speech_config = speech_config
+
+        if isinstance(text_config, dict):
+            text_config = Gemma2Config(**text_config)
+        elif text_config is None:
+            text_config = Gemma2Config()
+
+        self.text_config = text_config
+
+        self.speech_mlp_use_projection = speech_mlp_use_projection
+        self.speech_mlp_scale_factor = speech_mlp_scale_factor
+        self.speech_token_index = speech_token_index
+        # Whisper encoder outputs 1500 tokens (3000 mel frames / stride 2).
+        # The adapter reduces by speech_mlp_scale_factor, so the number of
+        # speech embedding tokens per chunk is 1500 // scale_factor.
+        if fixed_speech_embeds_length is None:
+            fixed_speech_embeds_length = 1500 // speech_mlp_scale_factor
+        self.fixed_speech_embeds_length = fixed_speech_embeds_length
+        
+        self.sliding_window = self.text_config.sliding_window
+        self.hidden_size = self.text_config.hidden_size
+        self.num_attention_heads = self.text_config.num_attention_heads
+        self.num_hidden_layers = self.text_config.num_hidden_layers
+        self.num_key_value_heads = self.text_config.num_key_value_heads
+        self.head_dim = self.text_config.head_dim
+        self.intermediate_size = self.text_config.intermediate_size
+        # Gemma2 requires HybridCache for correct sliding window attention
+        # on alternating layers. Without this, GenerationMixin defaults to
+        # DynamicCache which disables sliding window and causes repetition.
+        self.cache_implementation = getattr(
+            self.text_config, 'cache_implementation', 'hybrid'
+        )
+        
+        super().__init__(**kwargs)