pajola/hate-speech-detection-lr-tfidf

Hate Speech Detection Model

Model Description

This is a hate speech detection model trained on social media text data. The model classifies text into three categories:

• Hate Speech: Content that attacks or uses pejorative language against a person or group
• Offensive Language: Content that is offensive but not specifically targeting a group
• Neither: Content that is neither hate speech nor offensive

Model Architecture

• Framework: scikit-learn
• Algorithm: Logistic Regression with TF-IDF vectorization
• Features: N-grams (1-3) with TF-IDF weighting
• Max Features: 10000
• N-gram Range: (1, 3)

Performance

• Test Accuracy: 0.8543
• Cross-validation F1-macro: 0.7201
• Best Parameters: {'classifier__C': 1, 'classifier__class_weight': 'balanced'}

Usage

from inference import HateSpeechDetector

# Initialize the detector
detector = HateSpeechDetector()

# Make a prediction
text = "Your text here"
result = detector.predict(text)

print(f"Predicted class: {result['predicted_class']}")
print(f"Probabilities: {result['probabilities']}")

Files

• model.pkl: The trained scikit-learn pipeline
• config.json: Model configuration and metadata
• preprocessing.py: Text preprocessing functions
• inference.py: Inference class for making predictions
• README.md: This documentation

Training Data

The model was trained on labeled social media data containing tweets labeled for hate speech detection.

Preprocessing

The following preprocessing steps are applied:

1. Convert text to lowercase
2. Remove URLs, mentions (@), and hashtags (#)
3. Remove retweet indicators (RT)
4. Remove punctuation except spaces
5. Remove extra whitespace

Limitations

• The model is trained on social media text and may not perform well on other text types
• Performance may vary across different demographics and contexts
• The model should be used responsibly and results should be interpreted carefully

Ethical Considerations

This model is intended for research and educational purposes. When using for content moderation:

• Human review is recommended for final decisions
• Be aware of potential biases in the training data
• Consider the context and cultural aspects of the content
• Regularly evaluate model performance across different groups

Citation

If you use this model, please cite appropriately and consider the ethical implications of automated hate speech detection.