Hugging Face's logo

TIGER-Lab
/
TIGERScore-13B

Text Generation
Transformers
Safetensors
llama
text evaluation
metric
llm metric
tigerscore
text2text-generation
Eval Results (legacy)
text-generation-inference
Model card Files
xet
 Community
4

Update README.md

#2
by Kaguya-19 - opened
base: refs/heads/main
←
from: refs/pr/2
Discussion Files changed
+8
-21
Files changed (1) hide show
  1. README.md +8 -21
README.md CHANGED
@@ -5,29 +5,16 @@ license: mit
5
 
6
  ## TIGERScore
7
 
8
- Project Page: [https://tiger-ai-lab.github.io/TIGERScore/](https://tiger-ai-lab.github.io/TIGERScore/)
9
-
10
- Paper: [https://arxiv.org/abs/2310.00752](https://arxiv.org/abs/2310.00752)
11
-
12
- Code: [https://github.com/TIGER-AI-Lab/TIGERScore](https://github.com/TIGER-AI-Lab/TIGERScore)
13
-
14
- Demo: [https://huggingface.co/spaces/TIGER-Lab/TIGERScore](https://huggingface.co/spaces/TIGER-Lab/TIGERScore)
15
 
16
  ## Introduction
17
 
18
  We present TIGERScore, a **T**rained metric that follows **I**nstruction **G**uidance to perform **E**xplainable, and **R**eference-free evaluation over a wide spectrum of text generation tasks. TIGERScore is guided by natural language instruction to provide error analysis to pinpoint the mistakes in the generated text. Our metric is based on LLaMA-2, trained on our meticulously curated instruction-tuning dataset MetricInstruct which covers 6 text generation tasks and 23 text generation datasets. As a reference-free metric, its correlation can even surpass the best existing reference-based metrics. To further qualitatively assess the rationale generated by our metric, we conduct human evaluation on the generated explanations and found that the explanations are 70.8% accurate. Through these experimental results, we believe TIGERScore demonstrates the possibility of building universal explainable metrics to evaluate any text generation task.
19
 
20
- TIGERScore-7B-V1.2: [https://huggingface.co/TIGER-Lab/TIGERScore-7B-V1.2](https://huggingface.co/TIGER-Lab/TIGERScore-7B-V1.2)
21
-
22
- TIGERScore-13B-V1.2: [https://huggingface.co/TIGER-Lab/TIGERScore-13B-V1.2](https://huggingface.co/TIGER-Lab/TIGERScore-13B-V1.2)
23
-
24
- TIGERScore-7B-V1.0: [https://huggingface.co/TIGER-Lab/TIGERScore-7B-V1.0](https://huggingface.co/TIGER-Lab/TIGERScore-7B-V1.0)
25
-
26
- TIGERScore-13B-V1.0: [https://huggingface.co/TIGER-Lab/TIGERScore-13B-V1.0](https://huggingface.co/TIGER-Lab/TIGERScore-13B-V1.0)
27
-
28
  ## Training Data
29
 
30
- The models are trained on the 🤗 [MetricInstruct Dataset](https://huggingface.co/datasets/TIGER-Lab/MetricInstruct), which covers 6 text generation tasks and 23 text generation datasets. Check out the dataset card for more details.
31
 
32
  ## Training Procedure
33
 
@@ -37,9 +24,9 @@ The models are fine-tuned with the MetricInstruct dataset using the original Lla
37
 
38
  TIGERScore significantly surpasses traditional metrics, i.e. BLUE, ROUGE, BARTScore, and BLEURT, and emerging LLM-based metrics as reference-free metrics. Though our dataset was originally sourced from ChatGPT, our distilled model actually outperforms ChatGPT itself, which proves the effectiveness of our filtering strategy. On the unseen task of story generation, TIGERScore also demonstrates reasonable generalization capability.
39
 
40
- | Tasks$\rightarrow$ | Summarization | Translation | Data2Text | Long-form QA | MathQA | Inst-Fol | Story-Gen | Average |
41
  |-------------------------------------------|----------------|----------------|----------------|-----------------|----------------|----------------|----------------|----------------|
42
- | Metrics$\downarrow$ Datasets$\rightarrow$ | SummaEval | WMT22-zh-en | WebNLG2020 | ASQA+ | gsm8k | LIMA+ | ROC | |
43
  | GPT-3.5-turbo (few-shot) | **38.50** | 40.53 | 40.20 | 29.33 | **66.46** | 23.20 | 4.77 | 34.71 |
44
  | GPT-4 (zero-shot) | 36.46 | **43.87** | **44.04** | **48.95** | 51.71 | **58.53** | **32.48** | **45.15** |
45
  | BLEU | 11.98 | 19.73 | 33.29 | 11.38 | 21.12 | **46.61** | -1.17 | 20.42 |
@@ -57,9 +44,9 @@ TIGERScore significantly surpasses traditional metrics, i.e. BLUE, ROUGE, BARTSc
57
  | Llama-2-13b-chat-0-shot | 28.53 | 14.38 | 29.24 | 19.91 | 1.08 | 21.37 | 26.78 | 20.18 |
58
  | COMETKiwi | 16.27 | **48.48** | 27.90 | 18.05 | -11.48 | 34.86 | 18.47 | 21.79 |
59
  | GPTScore-src | 37.41 | 8.90 | 28.82 | 39.48 | 14.25 | 26.46 | 23.91 | 25.61 |
60
- | TIGERScore-7B-V1.2 (ours) | 35.11 | 41.50 | 42.39 | **47.11** | 21.23 | 43.57 | 39.26 | 38.60 |
61
- | TIGERScore-13B-V1.2 (ours) | 36.81 | 44.99 | **45.88** | 46.22 | **23.32** | **47.03** | **46.36** | **41.52** |
62
- | $\Delta$ (ours - best reference-free) | -2 | -3 | +12 | +5 | +9 | +14 | +13 | +16 |
63
 
64
 
65
  ## Formatting
 
5
 
6
  ## TIGERScore
7
 
8
+ [Project Page](https://tiger-ai-lab.github.io/TIGERScore/) | [Paper](https://arxiv.org/abs/2310.00752) | [Code](https://github.com/TIGER-AI-Lab/TIGERScore) | [🤗Demo](https://huggingface.co/spaces/TIGER-Lab/TIGERScore) |
9
+ [🤗TIGERScore-7B](https://huggingface.co/TIGER-Lab/TIGERScore-7B-V1.2) | [🤗TIGERScore-13B](https://huggingface.co/TIGER-Lab/TIGERScore-13B-V1.2)
 
 
 
 
 
10
 
11
  ## Introduction
12
 
13
  We present TIGERScore, a **T**rained metric that follows **I**nstruction **G**uidance to perform **E**xplainable, and **R**eference-free evaluation over a wide spectrum of text generation tasks. TIGERScore is guided by natural language instruction to provide error analysis to pinpoint the mistakes in the generated text. Our metric is based on LLaMA-2, trained on our meticulously curated instruction-tuning dataset MetricInstruct which covers 6 text generation tasks and 23 text generation datasets. As a reference-free metric, its correlation can even surpass the best existing reference-based metrics. To further qualitatively assess the rationale generated by our metric, we conduct human evaluation on the generated explanations and found that the explanations are 70.8% accurate. Through these experimental results, we believe TIGERScore demonstrates the possibility of building universal explainable metrics to evaluate any text generation task.
14
 
 
 
 
 
 
 
 
 
15
  ## Training Data
16
 
17
+ The models are trained on the 🤗 [MetricInstruct Dataset](https://huggingface.co/datasets/TIGER-Lab/MetricInstruct), which covers 6 text generation tasks and 22 text generation datasets. Check out the dataset card for more details.
18
 
19
  ## Training Procedure
20
 
 
24
 
25
  TIGERScore significantly surpasses traditional metrics, i.e. BLUE, ROUGE, BARTScore, and BLEURT, and emerging LLM-based metrics as reference-free metrics. Though our dataset was originally sourced from ChatGPT, our distilled model actually outperforms ChatGPT itself, which proves the effectiveness of our filtering strategy. On the unseen task of story generation, TIGERScore also demonstrates reasonable generalization capability.
26
 
27
+ | Tasks→ | Summarization | Translation | Data2Text | Long-form QA | MathQA | Inst-Fol | Story-Gen | Average |
28
  |-------------------------------------------|----------------|----------------|----------------|-----------------|----------------|----------------|----------------|----------------|
29
+ | Metrics↓ Datasets→ | SummaEval | WMT22-zh-en | WebNLG2020 | ASQA+ | gsm8k | LIMA+ | ROC | |
30
  | GPT-3.5-turbo (few-shot) | **38.50** | 40.53 | 40.20 | 29.33 | **66.46** | 23.20 | 4.77 | 34.71 |
31
  | GPT-4 (zero-shot) | 36.46 | **43.87** | **44.04** | **48.95** | 51.71 | **58.53** | **32.48** | **45.15** |
32
  | BLEU | 11.98 | 19.73 | 33.29 | 11.38 | 21.12 | **46.61** | -1.17 | 20.42 |
 
44
  | Llama-2-13b-chat-0-shot | 28.53 | 14.38 | 29.24 | 19.91 | 1.08 | 21.37 | 26.78 | 20.18 |
45
  | COMETKiwi | 16.27 | **48.48** | 27.90 | 18.05 | -11.48 | 34.86 | 18.47 | 21.79 |
46
  | GPTScore-src | 37.41 | 8.90 | 28.82 | 39.48 | 14.25 | 26.46 | 23.91 | 25.61 |
47
+ | TIGERScore-7B (ours) | 35.11 | 41.50 | 42.39 | **47.11** | 21.23 | 43.57 | 39.26 | 38.60 |
48
+ | TIGERScore-13B (ours) | 36.81 | 44.99 | **45.88** | 46.22 | **23.32** | **47.03** | **46.36** | **41.52** |
49
+ | Δ (ours - best reference-free) | -2 | -3 | +12 | +5 | +9 | +14 | +13 | +16 |
50
 
51
 
52
  ## Formatting