Hwanjun commited on
Commit
1f9e699
·
verified ·
1 Parent(s): 669ad08

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +9 -2
README.md CHANGED
@@ -20,6 +20,14 @@ The feedback encompasses a wide range of input documents, from short to lengthy
20
 
21
  Surprisingly, it outperforms the nearly 10x larger Llama3-70B-Instruct while offering much faster inference speed.
22
 
 
 
 
 
 
 
 
 
23
  Please refer to [our paper](link) to catch up how to exploit LLM-generated feedback in the context of text summarization.
24
 
25
  We are plan to release its larger model, **SummLlama3-70B**. Please go to the [Huggingface link](https://huggingface.co/DISLab/SummLlama3-70B) for this model.
@@ -69,6 +77,7 @@ Here is the results:
69
  | SummLlama3-8B | 0.931 | 0.614 | 0.659 | 0.735 |
70
  | SummLlama3-70B | 0.950 | 0.632 | 0.754 | 0.779 |
71
 
 
72
 
73
  See an example how the summary improved by SummLlama3-8B over Llama3-8/70B-Instruct on the document below:
74
 
@@ -95,8 +104,6 @@ See an example how the summary improved by SummLlama3-8B over Llama3-8/70B-Instr
95
  | Llama3-70B-Instruct | Person 1 invites Person 2 to their house for Thanksgiving dinner on Thursday. Person 2 is grateful for the invitation and explains they can't go to Finland with their parents due to work commitments. Person 1 is happy to have them celebrate with their family, which will consist of 10 people. When Person 2 offers to bring something, Person 1 declines, but eventually agrees to let them bring a bottle of wine. |
96
  | SummLlama3-8B | The conversation is about Thanksgiving dinner arrangements, with Person2 (Paul) confirming his attendance at Person1's (host's) house on Thursday, discussing what to bring and what's already been taken care of, and finalizing the offer to bring wine instead of pie since someone else is handling desserts. |
97
 
98
- ## Example
99
-
100
  The summary of SummLlama3-8B can be considered a much human-preferred summary for the following reasons:
101
 
102
  **Core Focus:** The summary accurately captures the main theme of the conversation, which revolves around the Thanksgiving dinner arrangements. It highlights how the two people confirm plans, discuss what to bring, and finalize the decision for Person2 to bring wine instead of pie. This maintains the core context.
 
20
 
21
  Surprisingly, it outperforms the nearly 10x larger Llama3-70B-Instruct while offering much faster inference speed.
22
 
23
+ This is human evaluation results:
24
+
25
+ | **Config.** | **Faithfulness** | **Completeness** | **Conciseness** | **Average** |
26
+ |--------------------|------------|-----------|-----------|----------|
27
+ | Llama3-8B-Instruct | 0.902 | 0.636 | 0.784 | 0.774 |
28
+ | Llama3-70B-Instruct | 0.953 | 0.659 | 0.792 | 0.801 |
29
+ | SummLlama3-8B | 0.980 | 0.697 | 0.959 | 0.879 |
30
+
31
  Please refer to [our paper](link) to catch up how to exploit LLM-generated feedback in the context of text summarization.
32
 
33
  We are plan to release its larger model, **SummLlama3-70B**. Please go to the [Huggingface link](https://huggingface.co/DISLab/SummLlama3-70B) for this model.
 
77
  | SummLlama3-8B | 0.931 | 0.614 | 0.659 | 0.735 |
78
  | SummLlama3-70B | 0.950 | 0.632 | 0.754 | 0.779 |
79
 
80
+ ## Example
81
 
82
  See an example how the summary improved by SummLlama3-8B over Llama3-8/70B-Instruct on the document below:
83
 
 
104
  | Llama3-70B-Instruct | Person 1 invites Person 2 to their house for Thanksgiving dinner on Thursday. Person 2 is grateful for the invitation and explains they can't go to Finland with their parents due to work commitments. Person 1 is happy to have them celebrate with their family, which will consist of 10 people. When Person 2 offers to bring something, Person 1 declines, but eventually agrees to let them bring a bottle of wine. |
105
  | SummLlama3-8B | The conversation is about Thanksgiving dinner arrangements, with Person2 (Paul) confirming his attendance at Person1's (host's) house on Thursday, discussing what to bring and what's already been taken care of, and finalizing the offer to bring wine instead of pie since someone else is handling desserts. |
106
 
 
 
107
  The summary of SummLlama3-8B can be considered a much human-preferred summary for the following reasons:
108
 
109
  **Core Focus:** The summary accurately captures the main theme of the conversation, which revolves around the Thanksgiving dinner arrangements. It highlights how the two people confirm plans, discuss what to bring, and finalize the decision for Person2 to bring wine instead of pie. This maintains the core context.