Upload folder using huggingface_hub

README.md

@@ -0,0 +1,237 @@
+# Vivian - YouTube Bot Comment Detector
+
+This model detects bot comments on YouTube videos using a fine-tuned KcELECTRA model with custom classification layers.
+
+## Model Description
+
+Vivian is a specialized model for identifying bot-generated comments on YouTube. It leverages the KcELECTRA base model with a custom architecture optimized for handling the class imbalance inherent in bot detection tasks.
+
+### Model Architecture
+
+- **Base Model**: [beomi/KcELECTRA-base](https://huggingface.co/beomi/KcELECTRA-base) - A Korean-focused ELECTRA model
+- **Modifications**:
+  - Frozen initial transformer layers to prevent overfitting
+  - Custom classification layers with dropout for regularization
+  - Combined CLS token and mean pooling for improved feature representation
+  - Focal Loss implementation to handle class imbalance
+
+### Key Features
+
+- Effective on Korean YouTube comments
+- Robust against class imbalance (few bot comments vs. many human comments)
+- Optimized for both precision and recall in bot detection
+
+## Intended Uses
+
+This model is designed for:
+- Content moderation on YouTube videos
+- Automated filtering of bot comments
+- Research on bot behavior in social media
+
+## Training Data
+
+The model was trained on the [MisileLab/youtube-bot-comments](https://huggingface.co/datasets/MisileLab/youtube-bot-comments) dataset, which contains:
+- YouTube comments collected from popular Korean videos
+- Manual annotations for bot vs. human comments
+- A 70/20/10 train/test/validation split
+
+## Performance
+
+The model achieves:
+- High precision in bot detection to minimize false positives
+- Good recall to catch the majority of bot comments
+- Balanced performance across different comment lengths and styles
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, ElectraModel
+import torch
+import torch.nn as nn
+
+# Load the tokenizer
+tokenizer = AutoTokenizer.from_pretrained("beomi/KcELECTRA-base")
+
+# Define the model architecture (same as in training)
+class SpamUserClassificationLayer(nn.Module):
+    def __init__(self, encoder: ElectraModel):
+        super().__init__()
+
+        self.encoder = encoder
+
+        # Classification network optimized for imbalanced datasets
+        # Changed input dimension from 768 to 1536 (CLS + mean pooling)
+        self.dense1 = nn.Linear(1536, 512)
+        self.layernorm1 = nn.LayerNorm(512)
+        self.gelu1 = nn.GELU()
+        self.dropout1 = nn.Dropout(0.4)
+
+        self.dense2 = nn.Linear(512, 256)
+        self.layernorm2 = nn.LayerNorm(256)
+        self.gelu2 = nn.GELU()
+        self.dropout2 = nn.Dropout(0.3)
+
+    def forward(self, input_ids, attention_mask=None, token_type_ids=None):
+        # Get encoder outputs
+        outputs = self.encoder(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            output_attentions=True
+        )
+
+        # CLS token representation
+        cls_output = outputs.last_hidden_state[:, 0, :]  # [batch, 768]
+
+        # Mean pooling with proper attention masking
+        token_embeddings = outputs.last_hidden_state  # [batch, seq_len, 768]
+        input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
+        sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
+        sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
+        mean_pooled = sum_embeddings / sum_mask  # [batch, 768]
+
+        # Concatenate CLS + mean pooling
+        combined_output = torch.cat([cls_output, mean_pooled], dim=1)  # [batch, 1536]
+
+        # Pass through classification network
+        x = self.dense1(combined_output)
+        x = self.layernorm1(x)
+        x = self.gelu1(x)
+        x = self.dropout1(x)
+
+        x = self.dense2(x)
+        x = self.layernorm2(x)
+        x = self.gelu2(x)
+        x = self.dropout2(x)
+
+        return x
+
+class SpamUserClassifier(nn.Module):
+    def __init__(self, pretrained_model_name="beomi/kcelectra-base"):
+        super().__init__()
+
+        self.encoder = ElectraModel.from_pretrained(pretrained_model_name)
+
+        # Freeze first 2 layers for imbalanced dataset scenario
+        for i, layer in enumerate(self.encoder.encoder.layer):
+            if i < 2:
+                for param in layer.parameters():
+                    param.requires_grad = False
+
+        self.nameLayer = SpamUserClassificationLayer(self.encoder)
+        self.contentLayer = SpamUserClassificationLayer(self.encoder)
+
+        self.dense = nn.Linear(512, 256)
+        self.layernorm = nn.LayerNorm(256)
+        self.gelu = nn.GELU()
+        self.dropout = nn.Dropout(0.3)
+
+        self.output_layer = nn.Linear(256, 1)
+        self.sigmoid = nn.Sigmoid()
+
+    def forward(self, name_input_ids, content_input_ids, name_attention_mask=None, name_token_type_ids=None,
+               content_attention_mask=None, content_token_type_ids=None, return_logits=False, return_probs=True):
+
+        namePrediction = self.nameLayer(name_input_ids, name_attention_mask, name_token_type_ids)
+        contentPrediction = self.contentLayer(content_input_ids, content_attention_mask, content_token_type_ids)
+
+        # Pass through classification network
+        x = self.dense(torch.cat([namePrediction, contentPrediction], dim=1))
+        x = self.layernorm(x)
+        x = self.gelu(x)
+        x = self.dropout(x)
+
+        logits = self.output_layer(x)
+
+        if return_logits:
+            return logits
+        else:
+            # Apply sigmoid and return probabilities or predictions
+            probs = self.sigmoid(logits)
+            # Return class predictions: 0 (not bot) or 1 (bot)
+            return probs if return_probs else (probs > 0.9).long().squeeze(-1)
+
+# Load the model
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+model = SpamUserClassifier()
+model.load_state_dict(torch.load("model.pth", map_location=device))
+model.to(device)
+model.eval()
+
+# Example inference
+def classify_comment(author_name, comment_text, threshold=0.9):
+    # Tokenize author name
+    name_encoding = tokenizer(
+        author_name,
+        truncation=True,
+        padding="max_length",
+        max_length=128,
+        return_tensors="pt"
+    )
+    name_input_ids = name_encoding["input_ids"].to(device)
+    name_attention_mask = name_encoding["attention_mask"].to(device)
+
+    # Tokenize content
+    content_encoding = tokenizer(
+        comment_text,
+        truncation=True,
+        padding="max_length",
+        max_length=128,
+        return_tensors="pt"
+    )
+    content_input_ids = content_encoding["input_ids"].to(device)
+    content_attention_mask = content_encoding["attention_mask"].to(device)
+
+    # Get prediction
+    with torch.no_grad():
+        probs = model(
+            name_input_ids=name_input_ids,
+            content_input_ids=content_input_ids,
+            name_attention_mask=name_attention_mask,
+            content_attention_mask=content_attention_mask,
+            return_logits=False,
+            return_probs=True
+        )
+    
+    # Get probability and prediction
+    probability = probs.item()
+    is_bot = probability > threshold
+    
+    return {
+        "probability": probability,
+        "is_bot": is_bot
+    }
+
+# Example usage
+result = classify_comment(
+    author_name="SpamBot2023",
+    comment_text="Check out my channel for free gift cards!"
+)
+print(f"Bot probability: {result['probability']:.4f}")
+print(f"Is bot comment: {result['is_bot']}")
+```
+
+## Limitations
+
+- Primarily optimized for Korean YouTube comments
+- May have reduced performance on other languages or platforms
+- Cannot detect sophisticated bots that closely mimic human writing patterns
+- Limited to text-based features (doesn't consider user history or behavior patterns)
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```
+@misc{vivian-youtube-bot-detector,
+  author = {MisileLab},
+  title = {Vivian: YouTube Bot Comment Detection System},
+  year = {2025},
+  publisher = {Hugging Face},
+  howpublished = {\url{https://huggingface.co/MisileLab/vivian}}
+}
+```
+
+## Contact
+
+For questions, issues, or feedback, please open an issue on the [GitHub repository](https://github.com/misilelab/h3).

model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e98c216e4144dc3e557763bf3bdd1cd8d33e9948704e4aaddf047442256227f
+size 442019527