model card

README.md

@@ -1,3 +1,71 @@
 ---
 license: apache-2.0
+datasets:
+- jl3676/SafetyAnalystData
+language:
+- en
+tags:
+- safety
+- moderation
+- llm
+- lm
+- harmfulness
 ---
+# Model Card for HarmReporter
+
+
+HarmReporter is an open language model that generates a structured "harm tree" for a given prompt. The harm tree consists of the following features:
+1) stakeholders (individuals, groups, communities, and entities) that may be impacted by the prompt scenario,
+2) categories of harmful *actions* that may impact each stakeholder,
+3) categories of harmful *effect* each harmful action may cause on the stakeholder, and 
+4) the *likelihood*, *severity*, and *immediacy* of each harmful effect.
+
+   
+HarmReporter is a 8B model trained on [SafetyAnalystData](https://huggingface.co/datasets/jl3676/SafetyAnalystData). Its output can be combined with the output of [BenefitReporter](https://huggingface.co/jl3676/BenefitReporter) to generate a comprehensive harm-benefit tree for a given prompt. Collectively, HarmReporter and BenefitReporter are named SafetyReporter.
+
+Our comprehensive evaluations on five existing public prompt harmfulness classification benchmarks show that SafetyReporter outperforms the strongest existing open-source baselines (including WildGuard, Llama-Guard-3, and ShieldGemma) on the average F1 score across benchmarks, surpassing WildGuard by 3.7%, Llama-Guard-3 by 10.8%, and ShieldGemma by over 30%.
+
+For more details, read the paper: [SafetyAnalyst: Interpretable, transparent, and steerable LLM safety moderation](https://arxiv.org/abs/2410.16665v1).
+
+## Model Description
+
+- **Model type:** Harmful feature generation model trained on synthetic datasets from a mixture of SOTA LLMs.
+- **Language(s):** English
+- **License:** Apache 2.0
+- **Finetuned from model:** [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)
+
+
+## How to Use HarmReporter
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+model_id = "jl3676/HarmReporter"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id)
+text_input = ["How can I rob the bank?"]
+input_tokenized = tokenizer.apply_chat_template(text_input, return_tensors="pt")
+output = model.generate(input_tokenized, max_new_tokens=19000)
+```
+
+## Intended Uses of HarmReporter
+
+- Harmfulness analysis: HarmReporter can be used to analyze the harmfulness of a given prompt in the hypothetical scenario that an AI language model provides a helpful answer to the prompt. It can be used to generate a structured harm tree for a given prompt, which can be used to identify potential stakeholders, and harmful actions and effects.
+- Moderation tool: HarmReporter's output (harm tree) can be combined with the output of [BenefitReporter](https://huggingface.co/jl3676/BenefitReporter) into a comprehensive harm-benefit tree for a given prompt. These features can be aggregated using our [aggregation algorithm](https://github.com/jl3676/SafetyAnalyst) into a harmfulness score, which can be used as a moderation tool to identify potentially harmful prompts. 
+
+## Limitations
+
+Though it shows state-of-the-art performance on prompt safety classification, HarmReporter will sometimes generate inaccurate features and the aggregated harmfulness score may not always lead to correct judgments. Users of HarmReporter should be aware of this potential for inaccuracies.
+
+## Citation
+
+```
+@misc{li2024safetyanalystinterpretabletransparentsteerable,
+      title={SafetyAnalyst: Interpretable, transparent, and steerable LLM safety moderation}, 
+      author={Jing-Jing Li and Valentina Pyatkin and Max Kleiman-Weiner and Liwei Jiang and Nouha Dziri and Anne G. E. Collins and Jana Schaich Borg and Maarten Sap and Yejin Choi and Sydney Levine},
+      year={2024},
+      eprint={2410.16665},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2410.16665}, 
+}
+```