foo-barrr commited on
Commit
cc15baa
·
verified ·
1 Parent(s): 54cd96d

Adding hyperlinks to the datasets and framework used for evals

Browse files
Files changed (1) hide show
  1. app.py +3 -2
app.py CHANGED
@@ -47,8 +47,9 @@ with gr.Blocks(title="LLM Propensity Evaluation Leaderboard") as demo:
47
  * AI regulators and policymakers interested in the alignment and safety aspects of widely used (popular) language models.
48
 
49
  ## Evaluation Details:
50
- - **Instruction Following Score**: Measures a model's tendency to follow instructions accurately. Measured using the IFEval dataset.
51
- - **Uncommon Facts Hallucination Rate**: Evaluates how often a model hallucinates when questioned on facts. Measured using a subset of the SimpleQA dataset, which explicitly asks uncommon facts. We calculated the rate using this formula : (1 - (correct + not_attempted)), where correct = when the model answered a question correctly and not_attempted = when a model admits to not knowing the answer to a question.*
 
52
 
53
  ## How to Interpret the Scores:
54
  * Instruction Following Score: Higher scores indicate better adherence to instructions.
 
47
  * AI regulators and policymakers interested in the alignment and safety aspects of widely used (popular) language models.
48
 
49
  ## Evaluation Details:
50
+ - **Instruction Following Score**: Measures a model's tendency to follow instructions accurately. Measured using the **[IFEval](https://arxiv.org/pdf/2311.07911)** dataset.
51
+ - **Uncommon Facts Hallucination Rate**: Evaluates how often a model hallucinates when questioned on facts. Measured using a subset of the **[SimpleQA](https://arxiv.org/abs/2411.04368)** dataset, which explicitly asks uncommon facts. We calculated the rate using this formula : (1 - (correct + not_attempted)), where correct = when the model answered a question correctly and not_attempted = when a model admits to not knowing the answer to a question.*
52
+ - All evals have been run using the **[Inspect](https://github.com/UKGovernmentBEIS/inspect_evals)** framework from UK AISI.
53
 
54
  ## How to Interpret the Scores:
55
  * Instruction Following Score: Higher scores indicate better adherence to instructions.