README.md

---
language: en
license: apache-2.0
tags:
- toxicity-detection
- text-classification
- dendritic-optimization
- perforated-backpropagation
- bert-tiny
- efficient-ml
datasets:
- jigsaw-toxic-comment-classification-challenge
metrics:
- f1
- accuracy
- precision
- recall
model-index:
- name: dendritic-bert-tiny-toxicity
  results:
  - task:
      type: text-classification
      name: Toxicity Detection
    dataset:
      name: Civil Comments
      type: jigsaw_toxicity_pred
    metrics:
    - type: f1
      value: 0.358
      name: F1 Score (Toxic Class)
    - type: accuracy
      value: 0.918
      name: Accuracy
    - type: inference_time
      value: 2.25
      name: Inference Latency (ms)
---

# Dendritic BERT-Tiny for Toxicity Detection

## Model Description

This model applies **Perforated Backpropagation with Dendritic Optimization** to enhance a compact BERT-Tiny model (4.8M parameters) for toxicity classification. It achieves performance comparable to BERT-Base (109M parameters) while maintaining **17.8x faster inference speed**.

**Key Features:**
-  **22.8x smaller** than BERT-Base (4.8M vs 109M parameters)
-  **17.8x faster** inference (2.25ms vs 40ms)
-  **Superior toxic detection** (F1=0.358 vs BERT-Base F1=0.05)
-  **Dendritic optimization** using PerforatedAI
-  **Edge-ready** for real-time deployment

## Model Details

- **Model Type:** BERT-Tiny with Dendritic Optimization
- **Language:** English
- **Task:** Binary Text Classification (Toxic/Non-Toxic)
- **Base Model:** [prajjwal1/bert-tiny](https://huggingface.co/prajjwal1/bert-tiny)
- **Framework:** PyTorch 2.9.1 + PerforatedAI 3.0.7
- **Parameters:** 4,798,468
- **Model Size:** 19.3 MB

## Performance Metrics

| Metric | Dendritic BERT-Tiny | BERT-Base | Improvement |
|--------|---------------------|-----------|-------------|
| **Parameters** | 4.8M | 109M | 22.8x smaller |
| **F1 Score (Toxic)** | 0.358 | 0.050 | 7.16x better |
| **Accuracy** | 91.8% | 91.0% | +0.8% |
| **Inference Time** | 2.25ms | 40.1ms | 17.8x faster |
| **Throughput** | 444 samples/s | 25 samples/s | 17.8x higher |

## Intended Use

### Primary Use Cases
- Real-time content moderation in online forums and social media
- Edge device deployment for resource-constrained environments
- High-throughput toxicity screening systems
- Research in efficient NLP and dendritic optimization

### Limitations
- Trained on Civil Comments dataset (English only)
- May not generalize to all forms of toxicity or cultural contexts
- Class imbalance handling (94% non-toxic samples)
- Best suited for binary toxicity detection

## How to Use

### Installation

```bash
pip install transformers torch perforatedai
```

### Quick Start

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model and tokenizer
model_name = "your-username/dendritic-bert-tiny-toxicity"
tokenizer = AutoTokenizer.from_pretrained("prajjwal1/bert-tiny")
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Prepare input
text = "This is a sample comment to classify"
inputs = tokenizer(text, return_tensors="pt", max_length=128, truncation=True, padding=True)

# Inference
model.eval()
with torch.no_grad():
    outputs = model(**inputs)
    logits = outputs.logits
    prediction = torch.argmax(logits, dim=-1)
    
# Get result
label = "toxic" if prediction.item() == 1 else "non-toxic"
confidence = torch.softmax(logits, dim=-1)[0][prediction].item()
print(f"Prediction: {label} (confidence: {confidence:.2%})")
```

### Batch Processing for High Throughput

```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_name = "your-username/dendritic-bert-tiny-toxicity"
tokenizer = AutoTokenizer.from_pretrained("prajjwal1/bert-tiny")
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Example batch of comments
comments = [
    "This is a great discussion!",
    "You are absolutely terrible",
    "I disagree but respect your opinion",
]

# Tokenize batch
inputs = tokenizer(comments, return_tensors="pt", max_length=128, 
                   truncation=True, padding=True)

# Batch inference
model.eval()
with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.argmax(outputs.logits, dim=-1)
    probabilities = torch.softmax(outputs.logits, dim=-1)

# Display results
for comment, pred, probs in zip(comments, predictions, probabilities):
    label = "toxic" if pred.item() == 1 else "non-toxic"
    confidence = probs[pred].item()
    print(f"'{comment}' -> {label} ({confidence:.2%})")
```

### Edge Deployment with Quantization

```python
import torch
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    "your-username/dendritic-bert-tiny-toxicity"
)

# Dynamic quantization for edge devices
quantized_model = torch.quantization.quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8
)

# Model is now ~75% smaller and faster on CPU
```

## Training Details

### Training Dataset
- **Source:** Civil Comments / Jigsaw Toxicity Dataset
- **Train samples:** 5,000 (4.54% toxic)
- **Validation samples:** 1,000 (6.60% toxic)
- **Test samples:** 1,000 (9.00% toxic)

### Training Procedure
- **Optimizer:** AdamW (lr=2e-5, weight_decay=0.01)
- **Epochs:** 10
- **Batch size:** 32
- **Max sequence length:** 128 tokens
- **Loss function:** Cross-entropy with class weights (1.0, 21.0)
- **Warmup steps:** 500
- **Early stopping:** Patience 3 epochs

### Dendritic Optimization
This model uses **Perforated Backpropagation** with dendritic nodes to enhance learning capacity:
- Correlation threshold: 0.95
- Perforated AI version: 3.0.7
- Configuration: 3D tensor output dimensions for transformer layers

## Technical Architecture

```
Input Text (max 128 tokens)
    ↓
BERT-Tiny Tokenizer
    ↓
Embedding Layer (vocab_size: 30522, hidden: 128)
    ↓
Transformer Layer 1 (2 attention heads) + Dendritic Nodes
    ↓
Transformer Layer 2 (2 attention heads) + Dendritic Nodes
    ↓
Pooler Layer (CLS token)
    ↓
Classification Head (128 → 2)
    ↓
Output: [non-toxic_logit, toxic_logit]
```

## Ethical Considerations

### Bias and Fairness
- Model may inherit biases from the Civil Comments dataset
- Performance may vary across demographic groups and cultural contexts
- Should not be the sole decision-maker in content moderation

### Recommended Practices
- Use as part of a human-in-the-loop moderation system
- Regularly evaluate for fairness across user demographics
- Combine with other signals for critical decisions
- Provide appeal mechanisms for users

## Citation

If you use this model in your research, please cite:

```bibtex
@misc{dendritic-bert-tiny-2026,
  title={Giant-Killer NLP: Dendritic Optimization for Toxicity Classification},
  author={PROJECT-Z Team},
  year={2026},
  publisher={HuggingFace Hub},
  howpublished={\url{https://huggingface.co/your-username/dendritic-bert-tiny-toxicity}},
  note={PyTorch Dendritic Optimization Hackathon}
}
```

## Acknowledgments

- **Base Model:** BERT-Tiny by [prajjwal1](https://huggingface.co/prajjwal1)
- **Optimization Framework:** [PerforatedAI](https://perforatedai.com/)
- **Dataset:** Jigsaw/Google Civil Comments
- **Hackathon:** PyTorch Dendritic Optimization Challenge

## License

Apache 2.0

## Contact

For questions, issues, or collaboration opportunities, please open an issue on the [GitHub repository](https://github.com/your-username/dendritic-bert-tiny-toxicity).

---

**Developed for the PyTorch Dendritic Optimization Hackathon - January 2026**