maqiuping59 commited on
Commit
8498cce
·
verified ·
1 Parent(s): 150d731

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +116 -0
README.md ADDED
@@ -0,0 +1,116 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: Table Markdown Metrics
3
+ emoji: 📊
4
+ colorFrom: blue
5
+ colorTo: red
6
+ sdk: gradio
7
+ sdk_version: 3.19.1
8
+ app_file: app.py
9
+ pinned: false
10
+ tags:
11
+ - evaluate
12
+ - metric
13
+ - table
14
+ - markdown
15
+ description: >-
16
+ Table evaluation metrics for assessing the matching degree between predicted and reference tables.
17
+ It calculates precision, recall, and F1 score for table data extraction or generation tasks.
18
+ ---
19
+
20
+ # Metric Card for Table Markdown Metrics
21
+
22
+ ## Metric Description
23
+
24
+ This metric evaluates the accuracy of table data extraction or generation by comparing predicted tables with reference tables. It calculates:
25
+
26
+ 1. Precision: The ratio of correctly predicted cells to the total number of cells in the predicted table
27
+ 2. Recall: The ratio of correctly predicted cells to the total number of cells in the reference table
28
+ 3. F1 Score: The harmonic mean of precision and recall
29
+
30
+ ## How to Use
31
+
32
+ This metric requires predictions and references as inputs in Markdown table format.
33
+
34
+ ```python
35
+ >>> table_metric = evaluate.load("table_markdown")
36
+ >>> results = table_metric.compute(
37
+ ... predictions="|A|B|\n|1|2|",
38
+ ... references="|A|B|\n|1|3|"
39
+ ... )
40
+ >>> print(results)
41
+ {'precision': 0.5, 'recall': 0.5, 'f1': 0.5, 'true_positives': 1, 'false_positives': 1, 'false_negatives': 1}
42
+ ```
43
+
44
+ ### Inputs
45
+ - **predictions** (`str`): Predicted table in Markdown format.
46
+ - **references** (`str`): Reference table in Markdown format.
47
+
48
+ ### Output Values
49
+ - **precision** (`float`): Precision score. Range: [0,1]
50
+ - **recall** (`float`): Recall score. Range: [0,1]
51
+ - **f1** (`float`): F1 score. Range: [0,1]
52
+ - **true_positives** (`int`): Number of correctly predicted cells
53
+ - **false_positives** (`int`): Number of incorrectly predicted cells
54
+ - **false_negatives** (`int`): Number of cells that were not predicted
55
+
56
+ ### Examples
57
+
58
+ Example 1 - Simple table comparison:
59
+ ```python
60
+ >>> table_metric = evaluate.load("table_markdown")
61
+ >>> results = table_metric.compute(
62
+ ... predictions="| | lobby | search | band | charge | chain ||--|--|--|--|--|--|| desire | 5 | 8 | 7 | 5 | 9 || wage | 1 | 5 | 3 | 8 | 5 |",
63
+ ... references="| | lobby | search | band | charge | chain ||--|--|--|--|--|--|| desire | 1 | 6 | 7 | 5 | 9 || wage | 1 | 5 | 2 | 8 | 5 |"
64
+ ... )
65
+ >>> print(results)
66
+ {'precision': 0.7, 'recall': 0.7, 'f1': 0.7, 'true_positives': 7, 'false_positives': 3, 'false_negatives': 3}
67
+ ```
68
+
69
+ Example 2 - Complex table comparison:
70
+ ```python
71
+ >>> table_metric = evaluate.load("table_markdown")
72
+ >>> results = table_metric.compute(
73
+ ... predictions="""
74
+ ... | | lobby | search | band |
75
+ ... |--|-------|--------|------|
76
+ ... | desire | 5 | 8 | 7 |
77
+ ... | wage | 1 | 5 | 3 |
78
+ ... """,
79
+ ... references="""
80
+ ... | | lobby | search | band |
81
+ ... |--|-------|--------|------|
82
+ ... | desire | 5 | 8 | 7 |
83
+ ... | wage | 1 | 5 | 3 |
84
+ ... """
85
+ ... )
86
+ >>> print(results)
87
+ {'precision': 1.0, 'recall': 1.0, 'f1': 1.0, 'true_positives': 6, 'false_positives': 0, 'false_negatives': 0}
88
+ ```
89
+
90
+ ## Limitations and Bias
91
+
92
+ 1. The metric assumes that tables are well-formed in Markdown format
93
+ 2. The comparison is case-sensitive
94
+ 3. The metric does not handle merged cells or complex table structures
95
+ 4. The metric treats each cell as a separate unit and does not consider the semantic meaning of the content
96
+
97
+ ## Citation(s)
98
+ ```bibtex
99
+ @article{scikit-learn,
100
+ title={Scikit-learn: Machine Learning in {P}ython},
101
+ author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V.
102
+ and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P.
103
+ and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and
104
+ Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.},
105
+ journal={Journal of Machine Learning Research},
106
+ volume={12},
107
+ pages={2825--2830},
108
+ year={2011}
109
+ }
110
+ ```
111
+
112
+ ## Further References
113
+
114
+ - [Markdown Tables](https://www.markdownguide.org/extended-syntax/#tables)
115
+ - [Table Structure Recognition](https://paperswithcode.com/task/table-structure-recognition)
116
+ - [Table Extraction](https://paperswithcode.com/task/table-extraction)