File size: 4,717 Bytes
813b852 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 | ---
language:
- en
- zh
tags:
- text-classification
- safety
- education
- guardrails
- bert
pipeline_tag: text-classification
library_name: transformers
license: apache-2.0
---
# Educational Guardrail System - Granular Semantic Assessor
## Model Details
- **Model Name:** Granular Semantic Assessor (Query Moderation)
- **Model Architecture:** BERT sequence classification model
- **Task:** Query Moderation
- **Domain:** K-12 Education & General AI Safety
- **Language:** English and Chinese
## Model Description
This model serves as the **Granular Semantic Assessor**, the core component of the "Query Moderation" stage in the **Educational Guardrail System (EGS)**. It is designed to act as a "System 1" fast-response filter to detect potential safety risks in user queries before they reach the LLM.
Unlike standard content moderation models, this model is fine-tuned specifically for **educational contexts**, aiming to distinguish between benign pedagogical inquiry (e.g., "History of wars") and harmful intent.
### Key Features
* **BERT Architecture:** Uses a BERT encoder with a sequence classification head for safety risk assessment.
* **Granular Assessment:** Supports a **Tri-State Mechanism** (Safe, Unsafe, Medium Risk) to handle ambiguity, rather than a rigid binary block.
* **Anti-Dilution Strategy:** Designed to work with a **Sentence-Level Max-Pooling** inference strategy to prevent localized toxicity from being diluted in long contexts.
## Safety Taxonomy & Capability
The model is trained to align with the **EGS Safety Taxonomy**, covering three critical dimensions:
1. **Universal Ethical Safety:** Countering terrorism, extremism, hate speech, and violence.
2. **Regional Legal Safety:** Ensuring compliance with regional regulations and cultural norms.
3. **Pedagogical Developmental Safety:** Protecting minors from age-inappropriate content (e.g., horror, cognitive offloading hints).
## Label Mapping and Risk Logic
The model outputs three labels/probabilities, which are mapped to the EGS risk levels in the inference pipeline.
### 1. Raw Model Outputs
- **UNSAFE:** High-risk query
- **SAFE:** Low-risk query
- **AMBIGUOUS:** Medium-risk or unclear query
### 2. EGS System Logic (Tri-State Risk Levels)
According to our ablation studies, we utilize optimal thresholds ($T_1=0.04, T_2=0.92$) to minimize False Positives in educational settings.
| Logic Label | Risk Level | Condition ($P_{unsafe}$) | Action |
| :--- | :--- | :--- | :--- |
| **0** | **High Risk** (Unsafe) | $P_{unsafe} > 0.92$ | **Block Immediately** |
| **2** | **Medium Risk** (Grey Area) | $0.04 < P_{unsafe} \le 0.92$ | **Trigger Secondary Checks** |
| **1** | **Low Risk** (Safe) | $P_{unsafe} \le 0.04$ | **Pass to LLM** |
*> Note: The thresholds (0.92/0.04) are empirically derived to maximize F1 score on educational benchmarks.*
## Intended Use & Inference Strategy
**Critical Note:** To replicate the performance reported in the paper, you should split long inputs into sentences and aggregate scores using a **Max-Pooling** strategy.
### Recommended Logic (Python)
```python
import re
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
model_id = "ZeroLoss-Lab/egs-bert-input-moderator"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
def get_risk_level(text):
# 1. Segmentation (Crucial for preventing toxicity dilution)
# Split by periods or CJK delimiters
sentences = re.split(r"(?<=[\u3002.])", text)
sentences = [s.strip() for s in sentences if len(s.strip()) > 5]
if not sentences:
sentences = [text]
# 2. Inference
inputs = tokenizer(sentences, return_tensors="pt", padding=True, truncation=True, max_length=512)
with torch.no_grad():
outputs = model(**inputs)
probs = torch.nn.functional.softmax(outputs.logits, dim=-1)
# The model config maps class 0 to UNSAFE.
sent_unsafe_scores = probs[:, model.config.label2id["UNSAFE"]].tolist()
# 3. Max-Pooling Aggregation
max_unsafe_score = max(sent_unsafe_scores)
# 4. Tri-State Logic (Thresholds from Paper)
T_high = 0.92
T_low = 0.04
if max_unsafe_score > T_high:
return 0, "High Risk"
elif max_unsafe_score > T_low:
return 2, "Medium Risk"
else:
return 1, "Low Risk"
```
## Limitations
This model is intended as one component of a broader educational safety system. It should not be used as the only safeguard for high-stakes moderation decisions. Performance can vary across domains, dialects, adversarial phrasing, and long inputs if the recommended sentence-level aggregation is not used.
|