Update README.md
Browse files
README.md
CHANGED
@@ -20,6 +20,14 @@ The feedback encompasses a wide range of input documents, from short to lengthy
|
|
20 |
|
21 |
Surprisingly, it outperforms the nearly 10x larger Llama3-70B-Instruct while offering much faster inference speed.
|
22 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
23 |
Please refer to [our paper](link) to catch up how to exploit LLM-generated feedback in the context of text summarization.
|
24 |
|
25 |
We are plan to release its larger model, **SummLlama3-70B**. Please go to the [Huggingface link](https://huggingface.co/DISLab/SummLlama3-70B) for this model.
|
@@ -69,6 +77,7 @@ Here is the results:
|
|
69 |
| SummLlama3-8B | 0.931 | 0.614 | 0.659 | 0.735 |
|
70 |
| SummLlama3-70B | 0.950 | 0.632 | 0.754 | 0.779 |
|
71 |
|
|
|
72 |
|
73 |
See an example how the summary improved by SummLlama3-8B over Llama3-8/70B-Instruct on the document below:
|
74 |
|
@@ -95,8 +104,6 @@ See an example how the summary improved by SummLlama3-8B over Llama3-8/70B-Instr
|
|
95 |
| Llama3-70B-Instruct | Person 1 invites Person 2 to their house for Thanksgiving dinner on Thursday. Person 2 is grateful for the invitation and explains they can't go to Finland with their parents due to work commitments. Person 1 is happy to have them celebrate with their family, which will consist of 10 people. When Person 2 offers to bring something, Person 1 declines, but eventually agrees to let them bring a bottle of wine. |
|
96 |
| SummLlama3-8B | The conversation is about Thanksgiving dinner arrangements, with Person2 (Paul) confirming his attendance at Person1's (host's) house on Thursday, discussing what to bring and what's already been taken care of, and finalizing the offer to bring wine instead of pie since someone else is handling desserts. |
|
97 |
|
98 |
-
## Example
|
99 |
-
|
100 |
The summary of SummLlama3-8B can be considered a much human-preferred summary for the following reasons:
|
101 |
|
102 |
**Core Focus:** The summary accurately captures the main theme of the conversation, which revolves around the Thanksgiving dinner arrangements. It highlights how the two people confirm plans, discuss what to bring, and finalize the decision for Person2 to bring wine instead of pie. This maintains the core context.
|
|
|
20 |
|
21 |
Surprisingly, it outperforms the nearly 10x larger Llama3-70B-Instruct while offering much faster inference speed.
|
22 |
|
23 |
+
This is human evaluation results:
|
24 |
+
|
25 |
+
| **Config.** | **Faithfulness** | **Completeness** | **Conciseness** | **Average** |
|
26 |
+
|--------------------|------------|-----------|-----------|----------|
|
27 |
+
| Llama3-8B-Instruct | 0.902 | 0.636 | 0.784 | 0.774 |
|
28 |
+
| Llama3-70B-Instruct | 0.953 | 0.659 | 0.792 | 0.801 |
|
29 |
+
| SummLlama3-8B | 0.980 | 0.697 | 0.959 | 0.879 |
|
30 |
+
|
31 |
Please refer to [our paper](link) to catch up how to exploit LLM-generated feedback in the context of text summarization.
|
32 |
|
33 |
We are plan to release its larger model, **SummLlama3-70B**. Please go to the [Huggingface link](https://huggingface.co/DISLab/SummLlama3-70B) for this model.
|
|
|
77 |
| SummLlama3-8B | 0.931 | 0.614 | 0.659 | 0.735 |
|
78 |
| SummLlama3-70B | 0.950 | 0.632 | 0.754 | 0.779 |
|
79 |
|
80 |
+
## Example
|
81 |
|
82 |
See an example how the summary improved by SummLlama3-8B over Llama3-8/70B-Instruct on the document below:
|
83 |
|
|
|
104 |
| Llama3-70B-Instruct | Person 1 invites Person 2 to their house for Thanksgiving dinner on Thursday. Person 2 is grateful for the invitation and explains they can't go to Finland with their parents due to work commitments. Person 1 is happy to have them celebrate with their family, which will consist of 10 people. When Person 2 offers to bring something, Person 1 declines, but eventually agrees to let them bring a bottle of wine. |
|
105 |
| SummLlama3-8B | The conversation is about Thanksgiving dinner arrangements, with Person2 (Paul) confirming his attendance at Person1's (host's) house on Thursday, discussing what to bring and what's already been taken care of, and finalizing the offer to bring wine instead of pie since someone else is handling desserts. |
|
106 |
|
|
|
|
|
107 |
The summary of SummLlama3-8B can be considered a much human-preferred summary for the following reasons:
|
108 |
|
109 |
**Core Focus:** The summary accurately captures the main theme of the conversation, which revolves around the Thanksgiving dinner arrangements. It highlights how the two people confirm plans, discuss what to bring, and finalize the decision for Person2 to bring wine instead of pie. This maintains the core context.
|