Initial model files and config for custom ELECTRA Base classifier for sentiment

README.md

@@ -1,3 +1,208 @@
 ---
 license: mit
+tags:
+- sentiment-analysis
+- text-classification
+- electra
+- pytorch
+- transformers
 ---
+
+# Electra Base Classifier for Sentiment Analysis
+
+This model is a fine-tuned [ELECTRA base model](https://huggingface.co/google/electra-base-discriminator) for sentiment analysis, enhanced with custom pooling and a SwishGLU activation function. It classifies text into three sentiment categories: negative, neutral, and positive.
+
+## Model Architecture
+
+- **Base Model**: ELECTRA base (`google/electra-base-discriminator`)
+- **Custom Components**:
+  - **Pooling Layer**: Custom pooling layer supporting 'cls', 'mean', and 'max' pooling types.
+  - **Activation Function**: Custom SwishGLU activation function.
+  - **Classifier**: Custom classifier with configurable hidden dimensions, number of layers, and dropout rate.
+
+## Labels
+
+The model predicts the following labels:
+
+- `0`: negative
+- `1`: neutral
+- `2`: positive
+
+## How to Use
+
+To use this model, you need to download the `electra_base_classifier_sentiment.py` file from this repository and place it in your working directory. This file contains the custom classes required to load and use the model.
+
+```python
+import torch
+from transformers import AutoTokenizer
+from electra_base_classifier_sentiment import ElectraBaseClassifierSentiment  # Ensure this file is in your working directory
+
+model_name = "jbeno/electra-base-classifier-sentiment"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = ElectraBaseClassifierSentiment.from_pretrained(model_name)
+model.eval()
+
+# Example usage
+text = "I love this product!"
+inputs = tokenizer(text, return_tensors="pt")
+with torch.no_grad():
+    logits = model(**inputs)
+    predicted_class_id = torch.argmax(logits, dim=1).item()
+    predicted_label = model.config.id2label[str(predicted_class_id)]
+    print(f"Predicted label: {predicted_label}")
+```
+
+## Requirements
+- Python 3.7+
+- PyTorch
+- Transformers library
+- Additional Files:
+    - Download **electra_base_classifier_sentiment.py** from this repository and place it in your working directory.
+
+## Custom Model Components
+
+### SwishGLU Activation Function
+
+The SwishGLU activation function combines the Swish activation with a Gated Linear Unit (GLU). It enhances the model's ability to capture complex patterns in the data.
+
+```python
+class SwishGLU(nn.Module):
+    def __init__(self, input_dim: int, output_dim: int):
+        super(SwishGLU, self).__init__()
+        self.projection = nn.Linear(input_dim, 2 * output_dim)
+        self.activation = nn.SiLU()
+
+    def forward(self, x):
+        x_proj_gate = self.projection(x)
+        projected, gate = x_proj_gate.tensor_split(2, dim=-1)
+        return projected * self.activation(gate)
+```
+
+### PoolingLayer
+
+The PoolingLayer class allows you to choose between different pooling strategies:
+
+- `cls`: Uses the representation of the \[CLS\] token.
+- `mean`: Calculates the mean of the token embeddings.
+- `max`: Takes the maximum value across token embeddings.
+
+**'mean'** pooling was used in the fine-tuned model.
+
+```python
+class PoolingLayer(nn.Module):
+    def __init__(self, pooling_type='cls'):
+        super().__init__()
+        self.pooling_type = pooling_type
+
+    def forward(self, last_hidden_state, attention_mask):
+        if self.pooling_type == 'cls':
+            return last_hidden_state[:, 0, :]
+        elif self.pooling_type == 'mean':
+            return (last_hidden_state * attention_mask.unsqueeze(-1)).sum(1) / attention_mask.sum(-1).unsqueeze(-1)
+        elif self.pooling_type == 'max':
+            return torch.max(last_hidden_state * attention_mask.unsqueeze(-1), dim=1)[0]
+        else:
+            raise ValueError(f"Unknown pooling method: {self.pooling_type}")
+```
+
+### Classifier
+
+The Classifier class is a customizable feed-forward neural network used for the final classification.
+
+The fine-tuned model had:
+
+- `input_dim`: 768
+- `num_layers`: 2
+- `hidden_dim`: 1024
+- `hidden_activation`: SwishGLU
+- `dropout_rate`: 0.3
+- `n_classes`: 3
+
+```python
+class Classifier(nn.Module):
+    def __init__(self, input_dim, hidden_dim, hidden_activation, num_layers, n_classes, dropout_rate=0.0):
+        super().__init__()
+        layers = []
+        layers.append(nn.Linear(input_dim, hidden_dim))
+        layers.append(hidden_activation)
+        if dropout_rate > 0:
+            layers.append(nn.Dropout(dropout_rate))
+
+        for _ in range(num_layers - 1):
+            layers.append(nn.Linear(hidden_dim, hidden_dim))
+            layers.append(hidden_activation)
+            if dropout_rate > 0:
+                layers.append(nn.Dropout(dropout_rate))
+
+        layers.append(nn.Linear(hidden_dim, n_classes))
+        self.layers = nn.Sequential(*layers)
+```
+
+## Model Configuration
+
+The model's configuration (config.json) includes custom parameters:
+
+- `hidden_dim`: Size of the hidden layers in the classifier.
+- `hidden_activation`: Activation function used in the classifier ('SwishGLU').
+- `num_layers`: Number of layers in the classifier.
+- `dropout_rate`: Dropout rate used in the classifier.
+- `pooling`: Pooling strategy used ('mean').
+
+## Training Details
+
+### Dataset
+
+The model was trained on the [Sentiment Merged](https://huggingface.co/datasets/jbeno/sentiment_merged) dataset, which is a mix of Stanford Sentiment Treebank (SST-3), DynaSent Round 1, and DynaSent Round 2.
+
+### Code
+
+The code used to train the model can be found on GitHub: [jbeno/sentiment](https://github.com/jbeno/sentiment)
+
+### Research Paper
+
+The research paper can be found here: [ELECTRA and GPT-4o: Cost-Effective Partners for Sentiment Analysis](https://github.com/jbeno/sentiment/research_paper.pdf)
+
+### Performance
+
+- **Merged Dataset**
+    - Macro Average F1: **79.29**
+    - Accuracy: **79.69**
+- **DynaSent R1**
+    - Macro Average F1: **82.10**
+    - Accuracy: **82.14**
+- **DynaSent R2**
+    - Macro Average F1: **71.83**
+    - Accuracy: **71.94**
+- **SST-3**
+    - Macro Average F1: **69.95**
+    - Accuracy: **78.24**
+
+## License
+
+This model is licensed under the MIT License.
+
+## Citation
+
+If you use this model in your work, please consider citing it:
+
+```bibtex
+@misc{beno-2024-electra_base_classifier_sentiment,
+  title={Electra Base Classifier for Sentiment Analysis},
+  author={Jim Beno},
+  year={2024},
+  publisher={Hugging Face},
+  howpublished={\url{https://huggingface.co/jbeno/electra-base-classifier-sentiment}},
+}
+```
+
+## Contact
+
+For questions or comments, please open an issue on the repository or contact [Jim Beno](https://huggingface.co/jbeno).
+
+## Acknowledgments
+
+- The [Hugging Face Transformers library](https://github.com/huggingface/transformers) for providing powerful tools for model development.
+- The creators of the [ELECTRA model](https://arxiv.org/abs/2003.10555) for their foundational work.
+- The authors of the datasets used: [Stanford Sentiment Treebank](https://huggingface.co/datasets/stanfordnlp/sst), [DynaSent](https://huggingface.co/datasets/dynabench/dynasent).
+- [Stanford Engineering CGOE](https://cgoe.stanford.edu), [Chris Potts](https://stanford.edu/~cgpotts/), and the Course Facilitators of [XCS224U](https://online.stanford.edu/courses/xcs224u-natural-language-understanding)
+

__init__.py

@@ -0,0 +1 @@
+from .electra_base_classifier_sentiment import ElectraBaseClassifierSentiment

config.json

@@ -0,0 +1,44 @@
+{
+  "architectures": [
+    "ElectraBaseClassifierSentiment"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "dropout_rate": 0.3,
+  "embedding_size": 768,
+  "hidden_act": "gelu",
+  "hidden_activation": "SwishGLU",
+  "hidden_dim": 1024,
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "negative",
+    "1": "neutral",
+    "2": "positive"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "negative": 0,
+    "neutral": 1,
+    "positive": 2
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "electra",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_layers": 2,
+  "pad_token_id": 0,
+  "pooling": "mean",
+  "position_embedding_type": "absolute",
+  "summary_activation": "gelu",
+  "summary_last_dropout": 0.1,
+  "summary_type": "first",
+  "summary_use_proj": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.37.1",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

electra_base_classifier_sentiment.py

@@ -0,0 +1,95 @@
+import torch
+import torch.nn as nn
+from transformers import ElectraPreTrainedModel, ElectraModel
+
+class SwishGLU(nn.Module):
+    def __init__(self, input_dim: int, output_dim: int):
+        super(SwishGLU, self).__init__()
+        # Linear projection to 2 * output_dim to split for gate and projection
+        self.projection = nn.Linear(input_dim, 2 * output_dim)
+        self.activation = nn.SiLU()  # Swish activation
+
+    def forward(self, x):
+        # Apply linear projection
+        x_proj_gate = self.projection(x)
+        # Split the projection into two parts
+        projected, gate = x_proj_gate.tensor_split(2, dim=-1)
+        # Apply Swish activation and multiply
+        return projected * self.activation(gate)
+
+    
+class PoolingLayer(nn.Module):
+    def __init__(self, pooling_type='cls'):
+        super().__init__()
+        self.pooling_type = pooling_type
+
+    def forward(self, last_hidden_state, attention_mask):
+        if self.pooling_type == 'cls':
+            return last_hidden_state[:, 0, :]
+        elif self.pooling_type == 'mean':
+            # Mean pooling over the token embeddings
+            return (last_hidden_state * attention_mask.unsqueeze(-1)).sum(1) / attention_mask.sum(-1).unsqueeze(-1)
+        elif self.pooling_type == 'max':
+            # Max pooling over the token embeddings
+            return torch.max(last_hidden_state * attention_mask.unsqueeze(-1), dim=1)[0]
+        else:
+            raise ValueError(f"Unknown pooling method: {self.pooling_type}")
+
+class Classifier(nn.Module):
+    def __init__(self, input_dim, hidden_dim, hidden_activation, num_layers, n_classes, dropout_rate=0.0):
+        super().__init__()
+        layers = []
+        layers.append(nn.Linear(input_dim, hidden_dim))
+        layers.append(hidden_activation)
+        if dropout_rate > 0:
+            layers.append(nn.Dropout(dropout_rate))
+
+        for _ in range(num_layers - 1):
+            layers.append(nn.Linear(hidden_dim, hidden_dim))
+            layers.append(hidden_activation)
+            if dropout_rate > 0:
+                layers.append(nn.Dropout(dropout_rate))
+
+        layers.append(nn.Linear(hidden_dim, n_classes))
+        self.layers = nn.Sequential(*layers)
+
+    def forward(self, x):
+        return self.layers(x)
+
+class ElectraBaseClassifierSentiment(ElectraPreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.electra = ElectraModel(config)
+
+        if hasattr(self.electra, 'pooler'):
+            self.electra.pooler = None
+
+        self.pooling = PoolingLayer(pooling_type=config.pooling)
+
+        # Handle custom activation functions
+        activation_name = config.hidden_activation
+        if activation_name == 'SwishGLU':
+            hidden_activation = SwishGLU(
+                input_dim=config.hidden_dim,
+                output_dim=config.hidden_dim
+            )
+        else:
+            activation_class = getattr(nn, activation_name)
+            hidden_activation = activation_class()
+
+        self.classifier = Classifier(
+            input_dim=config.hidden_size,
+            hidden_dim=config.hidden_dim,
+            hidden_activation=hidden_activation,
+            num_layers=config.num_layers,
+            n_classes=config.num_labels,
+            dropout_rate=config.dropout_rate
+        )
+        self.init_weights()
+
+
+    def forward(self, input_ids=None, attention_mask=None, **kwargs):
+        outputs = self.electra(input_ids, attention_mask=attention_mask, **kwargs)
+        pooled_output = self.pooling(outputs.last_hidden_state, attention_mask)
+        logits = self.classifier(pooled_output)
+        return logits

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:777d8bf4e1c5f72c736ac5adc523639c22caf43cf2ba274245a69b2fd0f7e2ca
+size 451348484

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bf240ff1218ce49a92f220aa4964c8723729c74734db31a56a92d5d52d8aeb82
+size 451391998

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,55 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "ElectraTokenizer",
+  "unk_token": "[UNK]"
+}