File size: 5,263 Bytes
59a99db | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 | ---
language: en
license: apache-2.0
library_name: pytorch
tags:
- distilbert
- text-classification
- multi-task
- incident-response
- sre
- sentinelops
pipeline_tag: text-classification
datasets:
- ayushgupta07xx/sentinelops-corpus
base_model: distilbert-base-uncased
---
# SentinelOps Incident Classifier
Multi-task DistilBERT classifier that predicts **severity** (P0/P1/P2/P3) and **category** (networking, database, deploy, capacity, auth, other) for SRE incident postmortems. Non-generative baseline for the [SentinelOps](https://github.com/ayushgupta07xx/sentinelops) flagship project, benchmarked against a fine-tuned Mistral-7B generation model.
## Intended use
Given an incident summary, returns `{severity, category}` predictions for routing, prioritization, or retrieval filtering inside the SentinelOps agent.
**Not intended for**: standalone production incident triage, compliance decisions, or any use case where a miscategorization has safety or financial impact.
## Training data
- **Corpus**: 2,000+ public postmortems scraped from `danluu/post-mortems`, Cloudflare blog, GitHub status (Atom feed), and AWS post-event summaries.
- **Labeled subset**: ~270 manually labeled for severity (4 classes) and category (6 classes). Remaining examples weakly labeled via regex + keyword rules.
- **Preprocessing**: HTML→text; MinHash dedup (threshold 0.85, 5-word shingles, 128 permutations); PII scrub (regex for emails/IPs/tokens + Presidio for names/phones).
Dataset: https://huggingface.co/datasets/ayushgupta07xx/sentinelops-corpus
## Training
- **Base**: `distilbert-base-uncased`
- **Architecture**: Shared DistilBERT encoder with two classification heads (severity + category), joint loss.
- **Hardware**: Kaggle T4 GPU (PyTorch).
- **Optimizer**: AdamW with cosine LR schedule.
- **Tracking**: Weights & Biases — project `sentinelops`, job_type `classifier_pt`.
Training code: https://github.com/ayushgupta07xx/sentinelops/blob/main/training/classifier/train_pt.py
## Evaluation
### ⚠️ Test set is 27 examples
This is a **small held-out set** — per-class F1 numbers have high variance and a single misclassification moves a 9-example class F1 by ~0.1. Treat these numbers as directional indicators, not population estimates. The limit reflects the labeled-data budget (~270 manually labeled examples, standard 80/10/10 split) which is the realistic ceiling for a solo 5-week project.
### Headline metrics
| Head | Accuracy | Macro-F1 | Weighted-F1 |
|----------|----------|----------|-------------|
| Severity | 0.48 | 0.36 | 0.52 |
| Category | 0.30 | 0.36 | 0.24 |
Full per-class precision/recall/F1 is in `eval_report.json`. Confusion matrices: `assets/confusion_matrix_severity.png`, `assets/confusion_matrix_category.png`.
## Limitations
1. **Tiny test set (27)** — confidence intervals on per-class F1 are wide. Do not use these numbers to claim SOTA anything.
2. **Class imbalance** — some classes (e.g., `auth`) had fewer than 10 training examples; the model underperforms there.
3. **Weak labels** — the majority of training data uses regex/keyword rules, so the model partially learns those rules rather than semantic features. Performance on incidents whose vocabulary doesn't match the rules is likely worse.
4. **Domain drift** — trained on public postmortems from hyperscalers and open-source infra. Generalization to internal enterprise incidents without further fine-tuning is unverified.
5. **English only** — all training data is English.
## Bias considerations
- Overrepresentation of hyperscaler incidents (AWS, GitHub, Cloudflare) vs. small-org or on-prem incidents.
- "Severity" labels reflect the reporting org's conventions, which differ across companies — the model learns an average that may not match your org's definitions.
- Weak labels encode my prior about what keywords indicate each category, which may bias category boundaries.
## Files
- `model.pt` — PyTorch state dict (~253 MB)
- `tokenizer/` — HF tokenizer files
- `config.json` — model hyperparameters
- `label_mappings.json` — `severity` and `category` label lists
- `eval_report.json` — full classification reports
- `assets/confusion_matrix_{severity,category}.png` — confusion matrices
## Loading
```python
import json, torch
from transformers import DistilBertTokenizerFast
from huggingface_hub import snapshot_download
local = snapshot_download("ayushgupta07xx/sentinelops-classifier")
# Requires the DistilBertMultiTask class from:
# https://github.com/ayushgupta07xx/sentinelops/blob/main/training/classifier/model_pt.py
from model_pt import DistilBertMultiTask, Config
with open(f"{local}/label_mappings.json") as f:
lm = json.load(f)
cfg = Config(num_severity=len(lm["severity"]), num_category=len(lm["category"]))
model = DistilBertMultiTask(cfg)
model.load_state_dict(torch.load(f"{local}/model.pt", map_location="cpu"))
model.eval()
tokenizer = DistilBertTokenizerFast.from_pretrained(f"{local}/tokenizer")
```
## Citation
Built as part of [SentinelOps](https://github.com/ayushgupta07xx/sentinelops). No paper — this is engineering, not research.
## Changelog
- **2026-04 v0.1** — Initial release; 27-example test set.
|