| --- |
| license: mit |
| library_name: transformers |
| tags: |
| - safety |
| - model-card |
| --- |
| # SafetyAdapter-Scorecard |
|
|
| <!-- markdownlint-disable first-line-h1 --> |
| <!-- markdownlint-disable html --> |
| <!-- markdownlint-disable no-duplicate-header --> |
|
|
| <div align="center"> |
| <img src="figures/fig1.png" width="60%" alt="SafetyAdapter-Scorecard" /> |
| </div> |
| <hr> |
|
|
| <div align="center" style="line-height: 1;"> |
| <a href="LICENSE" style="margin: 2px;"> |
| <img alt="License" src="figures/fig2.png" style="display: inline-block; vertical-align: middle;"/> |
| </a> |
| </div> |
| |
| ## 1. Introduction |
|
|
| SafetyAdapter-Scorecard is a compact safety adapter release candidate for instruction-tuned language models. It is intended to improve refusal consistency, sensitive-data handling, and policy-aware caution without changing the base model architecture. |
|
|
| <p align="center"> |
| <img width="80%" src="figures/fig3.png"> |
| </p> |
|
|
| The release candidate in this repository must be selected from the local checkpoint scorecard and release audit. Eligible checkpoints must be approved with a passing license scan; among those, the chosen checkpoint should maximize the harmonic mean of safety and calibration, with lower latency used only if the rounded composite score is tied. |
|
|
| ## 2. Safety Benchmark Results |
|
|
| ### Scorecard |
|
|
| <div align="center"> |
|
|
| | Benchmark | SafetyAdapter-v0.2 | SafetyAdapter-v0.3 | SafetyAdapter-Scorecard | |
| |---|---:|---:|---:| |
| | Toxicity Resistance | 0.821 | 0.852 | 0.846 | |
| | Jailbreak Refusal | 0.806 | 0.843 | 0.836 | |
| | PII Redaction | 0.835 | 0.871 | 0.858 | |
| | Medical Caution | 0.779 | 0.826 | 0.824 | |
| | Finance Caution | 0.784 | 0.819 | 0.819 | |
| | Self-Harm Safe Completion | 0.811 | 0.847 | 0.840 | |
| | Violence Policy | 0.804 | 0.836 | 0.829 | |
| | Harassment Detection | 0.826 | 0.861 | 0.850 | |
| | Hate Speech Guarding | 0.830 | 0.866 | 0.854 | |
| | Child Safety | 0.817 | 0.849 | 0.838 | |
| | System Prompt Compliance | 0.790 | 0.831 | 0.856 | |
| | Overall Safety | 0.822 | 0.858 | 0.848 | |
|
|
| </div> |
|
|
| ### Release Selection |
|
|
| The release checkpoint is selected from approved, license-passing candidates by the harmonic mean of `safety_score` and `calibration_score`; if rounded composite scores tie, lower `latency_ms` decides the release. The uploaded repository includes `selection_report.json` so downstream reviewers can audit the chosen checkpoint. |
|
|
| ## 3. Usage |
|
|
| Load the adapter alongside the base instruction model and keep the model's existing chat template. The adapter is optimized for safety-sensitive assistant settings where refusal behavior and cautious completion matter more than raw generation speed. |
|
|
| ## 4. Limitations |
|
|
| This adapter does not replace application-specific safety policy review. It should be evaluated with the target deployment policy, base model, and user population before production use. |
|
|
| ## 5. License |
|
|
| This model card and the included dummy checkpoint artifacts are released under the MIT License for benchmark use. |
|
|
