alirezamsh commited on
Commit
e185eaa
·
1 Parent(s): 74c3aa2

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +49 -1
README.md CHANGED
@@ -9,4 +9,52 @@ app_file: app.py
9
  pinned: false
10
  ---
11
 
12
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9
  pinned: false
10
  ---
11
 
12
+
13
+ # Metric Card for RQUGE Score
14
+
15
+ ## Metric Description
16
+
17
+ RQUGE is an evaluation metric designed for assessing the quality of generated questions. RQUGE evaluates the quality of a candidate question without the need to compare
18
+ it to a reference question. It operates by taking into account the relevant context and answer span and employs a general question-answering module followed by
19
+ a span scoring mechanism to determine an acceptability score.
20
+
21
+ ## How to Use
22
+
23
+ RQUGE score takes three main inputs; "generated_questions" (list of generated questions), "contexts" (list of related contexts), and "answers" (list of reference answers). Additionally, "qa_model", and "sp_model" are used to provide the path to QA and span scorer modules. "device" is also an optional input.
24
+
25
+ ```python
26
+ from evaluate import load
27
+ generated_questions = ["how is the weather?"]
28
+ contexts = ["the weather is sunny"]
29
+ answers = ["sunny"]
30
+ rqugescore = evaluate.load("rquge_score")
31
+ results = rquge.compute(generated_questions=generated_questions, contexts=contexts, answers=answers)
32
+ print([round(v, 2) for v in results["score"]])
33
+ >>> [5.0]
34
+ ```
35
+
36
+ output_dict = {
37
+ "mean_score": total/len(output),
38
+ "instance_score": output,
39
+ }
40
+
41
+ ## Output Values
42
+
43
+ RQUGE score outputs a dictionary with the following values:
44
+
45
+ `mean_score`: The average RQUGE score over the input texts, ranging from 1 to 5
46
+ `instance_score`: Invidivual RQUGE score of each instance in the input, ranging from 1 to 5
47
+
48
+
49
+ ## Citation
50
+
51
+ ```bibtex
52
+ @misc{mohammadshahi2022rquge,
53
+ title={RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question},
54
+ author={Alireza Mohammadshahi and Thomas Scialom and Majid Yazdani and Pouya Yanki and Angela Fan and James Henderson and Marzieh Saeidi},
55
+ year={2022},
56
+ eprint={2211.01482},
57
+ archivePrefix={arXiv},
58
+ primaryClass={cs.CL}
59
+ }
60
+ ```