Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,40 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
-
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<div align="center">
|
| 2 |
+
<b style="font-size: 40px;">SummLlama3-8B</b>
|
| 3 |
+
</div>
|
| 4 |
+
|
| 5 |
+
Are you looking for a summarizer that outperforms the nearly 10x larger Llama3-70B-Instruct while offering much faster inference speed?
|
| 6 |
+
|
| 7 |
+
Our **SummLlama3-8B** could be exactly what you need!
|
| 8 |
+
|
| 9 |
+
SummLlama3 is initialized from Llama3-8B-Instruct, with additional training using Direct Preference Optimization (DPO) based on human-like summarization feedback.
|
| 10 |
+
|
| 11 |
+
Please refer to [our paper](link) to catch up how to exploit LLM-generated feedback in the context of text summarization.
|
| 12 |
+
|
| 13 |
+
We also released a larger model, **SummLlama3-70B**. Please go to the [Huggingface link](link) for this model.
|
| 14 |
+
|
| 15 |
---
|
| 16 |
+
|
| 17 |
+
Here is a brief overview of our summarizer:
|
| 18 |
+
|
| 19 |
+
Rather than relying on expensive human feedback, we utilize high-quality, multi-dimensional, and fine-grained feedback generated by large language models (LLMs).
|
| 20 |
+
|
| 21 |
+
This model excels at **faithfulness**, **completeness**, and **conciseness**, which are the three human-preferred aspects to judge what is a good summarizer.
|
| 22 |
+
|
| 23 |
+
- Faithfulness: a summarizer does not manipulate the information in the input text and add any information not directly inferable from the input text.
|
| 24 |
+
- Completeness: a summarizer ensures the inclusion of all key information from the input text in the output summary.
|
| 25 |
+
- Conciseness: a summarizer refrains from incorporating information outside the key information in the output, maintaining a succinct and focused summary.
|
| 26 |
+
|
| 27 |
+
Based on our comprehensive evaluation, which included both human and automated assessments of summary quality, SummLlama3 demonstrated significant improvements over the original Llama3 series.
|
| 28 |
+
|
| 29 |
+
Here is the results:
|
| 30 |
+
|
| 31 |
+
## Human Evaluation
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
## Autoamted Evaluation using [FineSurE](link)
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
|