Hugging Face's logo

UKP-SQuARE
/
tweac_16

Text Classification
Transformers
PyTorch
English
bert
QA
Inference Endpoints
Model card Files Community
1
haritzpuerto commited on
Commit
cd709db
·
1 Parent(s): 80a702f

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +220 -0
README.md ADDED
@@ -0,0 +1,220 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - QA
6
+ license: cc-by-4.0
7
+ datasets:
8
+ - BoolQ
9
+ - CommonSenseQA
10
+ - DROP
11
+ - DuoRC
12
+ - HellaSWAG
13
+ - HotpotQA
14
+ - HybridQA
15
+ - NarrativeQA
16
+ - NaturalQuestionsShort
17
+ - NewsQA
18
+ - QAMR
19
+ - RACE
20
+ - SearchQA
21
+ - SIQA
22
+ - SQuAD
23
+ - TriviaQA-web
24
+ metrics:
25
+ - Accuracy
26
+ - Precision
27
+ - Recall
28
+ - F1
29
+ - MRR
30
+ - R@3
31
+ - R@5
32
+
33
+
34
+ ---
35
+
36
+ BERT for Sequence Classification trained on QA Dataset prediction task.
37
+ - Input: question.
38
+ - Output: dataset from where that question comes from.
39
+ Original paper: TWEAC: Transformer with Extendable QA Agent Classifiers
40
+ https://arxiv.org/abs/2104.07081
41
+
42
+ Datasets used for training:
43
+ ```
44
+ list_datasets = ['BoolQ','CommonSenseQA','DROP','DuoRC','HellaSWAG','HotpotQA','HybridQA','NarrativeQA','NaturalQuestionsShort','NewsQA','QAMR','RACE','SearchQA','SIQA','SQuAD','TriviaQA-web']
45
+ ```
46
+
47
+ Results for all datasets:
48
+ - Accuracy: 0.7919096825783123
49
+ - Precision: 0.731586272892176
50
+ - Recall: 0.7919096825783123
51
+ - F1: 0.7494425609552463
52
+ - MRR: 0.8720871733637521
53
+ - R@3: 0.9438690810655046
54
+ - R@5: 0.9745318608004427
55
+ - Queries/second: 6052.33538824659
56
+
57
+
58
+ Results per dataset:
59
+ ```
60
+ "BoolQ": {
61
+ "accuracy": 0.998776758409786,
62
+ "mrr": 0.999388379204893,
63
+ "r@3": 1.0,
64
+ "r@5": 1.0,
65
+ "query_per_second": 6978.947907596168,
66
+ "precision": 0.8649364406779662,
67
+ "recall": 0.998776758409786,
68
+ "f1": 0.9270508089696281
69
+ },
70
+ "CommonSenseQA": {
71
+ "accuracy": 0.9247135842880524,
72
+ "mrr": 0.9476358338878795,
73
+ "r@3": 0.9705400981996727,
74
+ "r@5": 0.9705400981996727,
75
+ "query_per_second": 5823.984138936813,
76
+ "precision": 0.442443226311668,
77
+ "recall": 0.9247135842880524,
78
+ "f1": 0.5985169491525425
79
+ },
80
+ "DROP": {
81
+ "accuracy": 0.9075083892617449,
82
+ "mrr": 0.9378200367399193,
83
+ "r@3": 0.9609899328859061,
84
+ "r@5": 0.9786073825503355,
85
+ "query_per_second": 6440.988897129248,
86
+ "precision": 0.8636726546906187,
87
+ "recall": 0.9075083892617449,
88
+ "f1": 0.8850480670893842
89
+ },
90
+ "DuoRC": {
91
+ "accuracy": 0.5555803405457654,
92
+ "mrr": 0.7368963429107307,
93
+ "r@3": 0.9092125808610305,
94
+ "r@5": 0.9596996059186557,
95
+ "query_per_second": 6853.643198794893,
96
+ "precision": 0.646814404432133,
97
+ "recall": 0.5555803405457654,
98
+ "f1": 0.5977360905563778
99
+ },
100
+ "HellaSWAG": {
101
+ "accuracy": 0.998406691894045,
102
+ "mrr": 0.9990705702715262,
103
+ "r@3": 1.0,
104
+ "r@5": 1.0,
105
+ "query_per_second": 3091.5012960785157,
106
+ "precision": 0.9974134500596896,
107
+ "recall": 0.998406691894045,
108
+ "f1": 0.9979098238280083
109
+ },
110
+ "HotpotQA": {
111
+ "accuracy": 0.7414435784479837,
112
+ "mrr": 0.8435804344945315,
113
+ "r@3": 0.9325652321247034,
114
+ "r@5": 0.973568281938326,
115
+ "query_per_second": 4972.668019223381,
116
+ "precision": 0.7352150537634409,
117
+ "recall": 0.7414435784479837,
118
+ "f1": 0.7383161801923401
119
+ },
120
+ "HybridQA": {
121
+ "accuracy": 0.7934218118869013,
122
+ "mrr": 0.8806947764680021,
123
+ "r@3": 0.964800923254472,
124
+ "r@5": 0.9930755914598961,
125
+ "query_per_second": 4886.494046259562,
126
+ "precision": 0.7198952879581152,
127
+ "recall": 0.7934218118869013,
128
+ "f1": 0.7548723579467472
129
+ },
130
+ "NarrativeQA": {
131
+ "accuracy": 0.5623756749076442,
132
+ "mrr": 0.7416681781060867,
133
+ "r@3": 0.9011082693947144,
134
+ "r@5": 0.9580373212086767,
135
+ "query_per_second": 7081.067049796865,
136
+ "precision": 0.5623224095472628,
137
+ "recall": 0.5623756749076442,
138
+ "f1": 0.5623490409661377
139
+ },
140
+ "NaturalQuestionsShort": {
141
+ "accuracy": 0.7985353692739171,
142
+ "mrr": 0.8743599435345307,
143
+ "r@3": 0.9439077594266126,
144
+ "r@5": 0.9774072919912745,
145
+ "query_per_second": 7136.590426649795,
146
+ "precision": 0.7963020509633313,
147
+ "recall": 0.7985353692739171,
148
+ "f1": 0.7974171464135678
149
+ },
150
+ "NewsQA": {
151
+ "accuracy": 0.5375118708452041,
152
+ "mrr": 0.71192075967717,
153
+ "r@3": 0.855650522317189,
154
+ "r@5": 0.939696106362773,
155
+ "query_per_second": 7193.851409052092,
156
+ "precision": 0.18757249378624688,
157
+ "recall": 0.5375118708452041,
158
+ "f1": 0.2780985136961061
159
+ },
160
+ "QAMR": {
161
+ "accuracy": 0.6658497602557272,
162
+ "mrr": 0.7969741223377345,
163
+ "r@3": 0.9207778369738945,
164
+ "r@5": 0.973361747469366,
165
+ "query_per_second": 7321.775044800525,
166
+ "precision": 0.8654525309881587,
167
+ "recall": 0.6658497602557272,
168
+ "f1": 0.7526421968624852
169
+ },
170
+ "RACE": {
171
+ "accuracy": 0.8771538617474154,
172
+ "mrr": 0.917901778042666,
173
+ "r@3": 0.9489154672613015,
174
+ "r@5": 0.9693898236367322,
175
+ "query_per_second": 6952.225120744351,
176
+ "precision": 0.8767983789260385,
177
+ "recall": 0.8771538617474154,
178
+ "f1": 0.8769760843129306
179
+ },
180
+ "SearchQA": {
181
+ "accuracy": 0.9762073027090695,
182
+ "mrr": 0.9865069592101393,
183
+ "r@3": 0.9972909305064782,
184
+ "r@5": 0.9984687868080094,
185
+ "query_per_second": 4031.0193826035634,
186
+ "precision": 0.9870191735143503,
187
+ "recall": 0.9762073027090695,
188
+ "f1": 0.9815834665719192
189
+ },
190
+ "SIQA": {
191
+ "accuracy": 0.9969293756397134,
192
+ "mrr": 0.9977823268509042,
193
+ "r@3": 0.9979529170931423,
194
+ "r@5": 1.0,
195
+ "query_per_second": 6711.547709005977,
196
+ "precision": 0.9329501915708812,
197
+ "recall": 0.9969293756397134,
198
+ "f1": 0.9638792676892627
199
+ },
200
+ "SQuAD": {
201
+ "accuracy": 0.550628092881614,
202
+ "mrr": 0.7164538452390565,
203
+ "r@3": 0.8660068519223448,
204
+ "r@5": 0.9366197183098591,
205
+ "query_per_second": 7033.420124363291,
206
+ "precision": 0.48613678373382624,
207
+ "recall": 0.550628092881614,
208
+ "f1": 0.5163766175814368
209
+ },
210
+ "TriviaQA-web": {
211
+ "accuracy": 0.7855124582584125,
212
+ "mrr": 0.8647404868442627,
213
+ "r@3": 0.9321859748266119,
214
+ "r@5": 0.9640380169535063,
215
+ "query_per_second": 4327.642440910395,
216
+ "precision": 0.7404358353510896,
217
+ "recall": 0.7855124582584125,
218
+ "f1": 0.7623083634550667
219
+ },
220
+ ```