README.md

---
language:
- en
- zh
tags:
- text-classification
- safety
- education
- guardrails
- bert
pipeline_tag: text-classification
library_name: transformers
license: apache-2.0
---

# Educational Guardrail System - Granular Semantic Assessor

## Model Details

- **Model Name:** Granular Semantic Assessor (Query Moderation)
- **Model Architecture:** BERT sequence classification model
- **Task:** Query Moderation
- **Domain:** K-12 Education & General AI Safety
- **Language:** English and Chinese

## Model Description

This model serves as the **Granular Semantic Assessor**, the core component of the "Query Moderation" stage in the **Educational Guardrail System (EGS)**. It is designed to act as a "System 1" fast-response filter to detect potential safety risks in user queries before they reach the LLM.

Unlike standard content moderation models, this model is fine-tuned specifically for **educational contexts**, aiming to distinguish between benign pedagogical inquiry (e.g., "History of wars") and harmful intent.

### Key Features
* **BERT Architecture:** Uses a BERT encoder with a sequence classification head for safety risk assessment.
* **Granular Assessment:** Supports a **Tri-State Mechanism** (Safe, Unsafe, Medium Risk) to handle ambiguity, rather than a rigid binary block.
* **Anti-Dilution Strategy:** Designed to work with a **Sentence-Level Max-Pooling** inference strategy to prevent localized toxicity from being diluted in long contexts.

## Safety Taxonomy & Capability

The model is trained to align with the **EGS Safety Taxonomy**, covering three critical dimensions:
1.  **Universal Ethical Safety:** Countering terrorism, extremism, hate speech, and violence.
2.  **Regional Legal Safety:** Ensuring compliance with regional regulations and cultural norms.
3.  **Pedagogical Developmental Safety:** Protecting minors from age-inappropriate content (e.g., horror, cognitive offloading hints).

## Label Mapping and Risk Logic

The model outputs three labels/probabilities, which are mapped to the EGS risk levels in the inference pipeline.

### 1. Raw Model Outputs
- **UNSAFE:** High-risk query
- **SAFE:** Low-risk query
- **AMBIGUOUS:** Medium-risk or unclear query

### 2. EGS System Logic (Tri-State Risk Levels)
According to our ablation studies, we utilize optimal thresholds ($T_1=0.04, T_2=0.92$) to minimize False Positives in educational settings.

| Logic Label | Risk Level | Condition ($P_{unsafe}$) | Action |
| :--- | :--- | :--- | :--- |
| **0** | **High Risk** (Unsafe) | $P_{unsafe} > 0.92$ | **Block Immediately** |
| **2** | **Medium Risk** (Grey Area) | $0.04 < P_{unsafe} \le 0.92$ | **Trigger Secondary Checks**  |
| **1** | **Low Risk** (Safe) | $P_{unsafe} \le 0.04$ | **Pass to LLM** |

*> Note: The thresholds (0.92/0.04) are empirically derived to maximize F1 score on educational benchmarks.*

## Intended Use & Inference Strategy

**Critical Note:** To replicate the performance reported in the paper, you should split long inputs into sentences and aggregate scores using a **Max-Pooling** strategy.

### Recommended Logic (Python)

```python
import re
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "ZeroLoss-Lab/egs-bert-input-moderator"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

def get_risk_level(text):
    # 1. Segmentation (Crucial for preventing toxicity dilution)
    # Split by periods or CJK delimiters
    sentences = re.split(r"(?<=[\u3002.])", text)
    sentences = [s.strip() for s in sentences if len(s.strip()) > 5]
    if not sentences:
        sentences = [text]

    # 2. Inference
    inputs = tokenizer(sentences, return_tensors="pt", padding=True, truncation=True, max_length=512)
    with torch.no_grad():
        outputs = model(**inputs)

    probs = torch.nn.functional.softmax(outputs.logits, dim=-1)

    # The model config maps class 0 to UNSAFE.
    sent_unsafe_scores = probs[:, model.config.label2id["UNSAFE"]].tolist()

    # 3. Max-Pooling Aggregation
    max_unsafe_score = max(sent_unsafe_scores)

    # 4. Tri-State Logic (Thresholds from Paper)
    T_high = 0.92
    T_low = 0.04

    if max_unsafe_score > T_high:
        return 0, "High Risk"
    elif max_unsafe_score > T_low:
        return 2, "Medium Risk"
    else:
        return 1, "Low Risk"
```

## Limitations

This model is intended as one component of a broader educational safety system. It should not be used as the only safeguard for high-stakes moderation decisions. Performance can vary across domains, dialects, adversarial phrasing, and long inputs if the recommended sentence-level aggregation is not used.