Spaces:
Running
Running
Commit
·
e185eaa
1
Parent(s):
74c3aa2
Update README.md
Browse files
README.md
CHANGED
@@ -9,4 +9,52 @@ app_file: app.py
|
|
9 |
pinned: false
|
10 |
---
|
11 |
|
12 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
9 |
pinned: false
|
10 |
---
|
11 |
|
12 |
+
|
13 |
+
# Metric Card for RQUGE Score
|
14 |
+
|
15 |
+
## Metric Description
|
16 |
+
|
17 |
+
RQUGE is an evaluation metric designed for assessing the quality of generated questions. RQUGE evaluates the quality of a candidate question without the need to compare
|
18 |
+
it to a reference question. It operates by taking into account the relevant context and answer span and employs a general question-answering module followed by
|
19 |
+
a span scoring mechanism to determine an acceptability score.
|
20 |
+
|
21 |
+
## How to Use
|
22 |
+
|
23 |
+
RQUGE score takes three main inputs; "generated_questions" (list of generated questions), "contexts" (list of related contexts), and "answers" (list of reference answers). Additionally, "qa_model", and "sp_model" are used to provide the path to QA and span scorer modules. "device" is also an optional input.
|
24 |
+
|
25 |
+
```python
|
26 |
+
from evaluate import load
|
27 |
+
generated_questions = ["how is the weather?"]
|
28 |
+
contexts = ["the weather is sunny"]
|
29 |
+
answers = ["sunny"]
|
30 |
+
rqugescore = evaluate.load("rquge_score")
|
31 |
+
results = rquge.compute(generated_questions=generated_questions, contexts=contexts, answers=answers)
|
32 |
+
print([round(v, 2) for v in results["score"]])
|
33 |
+
>>> [5.0]
|
34 |
+
```
|
35 |
+
|
36 |
+
output_dict = {
|
37 |
+
"mean_score": total/len(output),
|
38 |
+
"instance_score": output,
|
39 |
+
}
|
40 |
+
|
41 |
+
## Output Values
|
42 |
+
|
43 |
+
RQUGE score outputs a dictionary with the following values:
|
44 |
+
|
45 |
+
`mean_score`: The average RQUGE score over the input texts, ranging from 1 to 5
|
46 |
+
`instance_score`: Invidivual RQUGE score of each instance in the input, ranging from 1 to 5
|
47 |
+
|
48 |
+
|
49 |
+
## Citation
|
50 |
+
|
51 |
+
```bibtex
|
52 |
+
@misc{mohammadshahi2022rquge,
|
53 |
+
title={RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question},
|
54 |
+
author={Alireza Mohammadshahi and Thomas Scialom and Majid Yazdani and Pouya Yanki and Angela Fan and James Henderson and Marzieh Saeidi},
|
55 |
+
year={2022},
|
56 |
+
eprint={2211.01482},
|
57 |
+
archivePrefix={arXiv},
|
58 |
+
primaryClass={cs.CL}
|
59 |
+
}
|
60 |
+
```
|