metadata
base_model:
- meta-llama/Meta-Llama-3-70B-Instruct
pipeline_tag: summarization
SummLlama3-70B
Are you looking for a summarizer that can generate more human-preferred summaries across multiple domains?
Our SummLlama3-70B could be exactly what you need!
SummLlama3-70B is initialized from Llama3-70B-Instruct, with additional training using Direct Preference Optimization (DPO) based on large-scale (over 100K) summarization feedback.
The feedback encompasses a wide range of input documents, from short to lengthy texts, including both dialogue and non-dialogue formats, and spans across seven distinct domains:
- Four non-dialouge domains: News, Lifestyle, Report, Medical
- Three dialogue domains: Daily Life, Interview, Meeting
This is automated evaluation results:
| Config. | Faithfulness | Completeness | Conciseness | Average |
|---|---|---|---|---|
| Llama3-8B-Instruct | 0.864 | 0.583 | 0.450 | 0.632 |
| Llama3-70B-Instruct | 0.931 | 0.596 | 0.487 | 0.671 |
| GPT-4o | 0.940 | 0.657 | 0.437 | 0.678 |
| SummLlama3-70B | 0.950 | 0.632 | 0.754 | 0.779 |
Please refer to our paper to catch up how to exploit LLM-generated feedback in the context of text summarization.