| --- |
| license: apache-2.0 |
| base_model: FacebookAI/roberta-large |
| library_name: transformers |
| pipeline_tag: text-classification |
| tags: |
| - roberta |
| - safety |
| - judge |
| - probguard |
| - calibeval |
| --- |
| |
| # CalibEval |
|
|
| CalibEval is a RoBERTa-large sequence-classification safety judge used by ProbGuard to label sampled continuations during calibration and evaluation. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForSequenceClassification, AutoTokenizer |
| |
| repo_id = "hxz-sec/CalibEval" |
| tokenizer = AutoTokenizer.from_pretrained(repo_id) |
| model = AutoModelForSequenceClassification.from_pretrained(repo_id) |
| ``` |
|
|
| The model outputs two labels for safety classification. In the EarlyGuard evaluation scripts, the unsafe probability is read from the positive class and thresholded at `0.5` by default. |
|
|
