Hugging Face's logo

saattrupdan
/
verdict-classifier

Text Classification
Transformers
PyTorch
TensorBoard
Safetensors
xlm-roberta
Generated from Trainer
Inference Endpoints
Model card Files Metrics Community
1
saattrupdan commited on
Commit
f7588be
•
1 Parent(s): 496f7f3

update model card README.md

Browse files
Files changed (1) hide show
  1. README.md +43 -135
README.md CHANGED
@@ -1,97 +1,40 @@
1
  ---
2
  license: mit
3
- language:
4
- - am
5
- - ar
6
- - hy
7
- - eu
8
- - bn
9
- - bs
10
- - bg
11
- - my
12
- - hr
13
- - ca
14
- - cs
15
- - da
16
- - nl
17
- - en
18
- - et
19
- - fi
20
- - fr
21
- - ka
22
- - de
23
- - el
24
- - gu
25
- - ht
26
- - iw
27
- - hi
28
- - hu
29
- - is
30
- - in
31
- - it
32
- - ja
33
- - kn
34
- - km
35
- - ko
36
- - lo
37
- - lv
38
- - lt
39
- - ml
40
- - mr
41
- - ne
42
- - no
43
- - or
44
- - pa
45
- - ps
46
- - fa
47
- - pl
48
- - pt
49
- - ro
50
- - ru
51
- - sr
52
- - zh
53
- - sd
54
- - si
55
- - sk
56
- - sl
57
- - es
58
- - sv
59
- - tl
60
- - ta
61
- - te
62
- - th
63
- - tr
64
- - uk
65
- - ur
66
- - ug
67
- - vi
68
- - cy
69
  tags:
70
  - generated_from_trainer
71
  model-index:
72
- - name: verdict-classifier-en
73
- results:
74
- - task:
75
- type: text-classification
76
- name: Verdict Classification
77
- widget:
78
- - "One might think that this is true, but it's taken out of context."
79
  ---
80
 
81
- # Multilingual Verdict Classifier
 
82
 
83
- This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on 1,500 deduplicated multilingual verdicts from [Google Fact Check Tools API](https://developers.google.com/fact-check/tools/api/reference/rest/v1alpha1/claims/search), translated into 65 languages with the [Google Cloud Translation API](https://cloud.google.com/translate/docs/reference/rest/).
84
- It achieves the following results on the evaluation set, being 1,000 such verdicts, but here including duplicates to represent the true distribution:
85
- - Loss: 0.2245
86
- - F1 Macro: 0.8818
87
- - F1 Misinformation: 0.9842
88
- - F1 Factual: 0.9688
89
- - F1 Other: 0.6923
90
- - Prec Macro: 0.8668
91
- - Prec Misinformation: 0.9887
92
- - Prec Factual: 0.9688
93
- - Prec Other: 0.6429
94
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
95
 
96
  ## Training procedure
97
 
@@ -106,59 +49,24 @@ The following hyperparameters were used during training:
106
  - total_train_batch_size: 32
107
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
108
  - lr_scheduler_type: linear
109
- - lr_scheduler_warmup_steps: 30066
110
  - num_epochs: 1000
111
 
112
  ### Training results
113
 
114
- | Training Loss | Epoch | Step | Validation Loss | F1 Macro | F1 Misinformation | F1 Factual | F1 Other | Prec Macro | Prec Misinformation | Prec Factual | Prec Other |
115
- |:-------------:|:-----:|:-----:|:---------------:|:--------:|:-----------------:|:----------:|:--------:|:----------:|:-------------------:|:------------:|:----------:|
116
- | 1.0833 | 0.13 | 500 | 0.9331 | 0.3136 | 0.9408 | 0.0 | 0.0 | 0.2961 | 0.8882 | 0.0 | 0.0 |
117
- | 0.9697 | 0.27 | 1000 | 0.8839 | 0.3136 | 0.9408 | 0.0 | 0.0 | 0.2961 | 0.8882 | 0.0 | 0.0 |
118
- | 0.9406 | 0.4 | 1500 | 0.7238 | 0.3591 | 0.9394 | 0.0 | 0.1379 | 0.4304 | 0.8911 | 0.0 | 0.4 |
119
- | 0.9183 | 0.53 | 2000 | 0.6494 | 0.3482 | 0.9336 | 0.0 | 0.1111 | 0.3528 | 0.8916 | 0.0 | 0.1667 |
120
- | 0.9103 | 0.67 | 2500 | 0.5686 | 0.3621 | 0.9323 | 0.0 | 0.1538 | 0.3643 | 0.8930 | 0.0 | 0.2 |
121
- | 0.8637 | 0.8 | 3000 | 0.5720 | 0.4233 | 0.9367 | 0.0 | 0.3333 | 0.4036 | 0.9108 | 0.0 | 0.3 |
122
- | 0.7905 | 0.93 | 3500 | 0.3918 | 0.4721 | 0.9418 | 0.0 | 0.4746 | 0.4402 | 0.9206 | 0.0 | 0.4 |
123
- | 0.6625 | 1.06 | 4000 | 0.2771 | 0.8225 | 0.9666 | 0.9375 | 0.5634 | 0.7853 | 0.9929 | 0.9375 | 0.4255 |
124
- | 0.5403 | 1.2 | 4500 | 0.2187 | 0.8083 | 0.9703 | 0.9062 | 0.5484 | 0.7799 | 0.9861 | 0.9062 | 0.4474 |
125
- | 0.4837 | 1.33 | 5000 | 0.1708 | 0.8138 | 0.9728 | 0.9231 | 0.5455 | 0.7916 | 0.9817 | 0.9091 | 0.4839 |
126
- | 0.4411 | 1.46 | 5500 | 0.1734 | 0.7873 | 0.9707 | 0.9206 | 0.4706 | 0.7843 | 0.9729 | 0.9355 | 0.4444 |
127
- | 0.3991 | 1.6 | 6000 | 0.1671 | 0.8131 | 0.9705 | 0.9355 | 0.5333 | 0.7976 | 0.9816 | 0.9667 | 0.4444 |
128
- | 0.344 | 1.73 | 6500 | 0.1719 | 0.7989 | 0.9749 | 0.8955 | 0.5263 | 0.7667 | 0.9885 | 0.8571 | 0.4545 |
129
- | 0.3005 | 1.86 | 7000 | 0.1855 | 0.8052 | 0.9704 | 0.9206 | 0.5246 | 0.7839 | 0.9838 | 0.9355 | 0.4324 |
130
- | 0.2638 | 2.0 | 7500 | 0.1802 | 0.7896 | 0.9752 | 0.9231 | 0.4706 | 0.7777 | 0.9796 | 0.9091 | 0.4444 |
131
- | 0.2362 | 2.13 | 8000 | 0.1752 | 0.7762 | 0.9718 | 0.8986 | 0.4583 | 0.7578 | 0.9773 | 0.8378 | 0.4583 |
132
- | 0.2077 | 2.26 | 8500 | 0.1739 | 0.8101 | 0.9740 | 0.9206 | 0.5357 | 0.7953 | 0.9817 | 0.9355 | 0.4688 |
133
- | 0.1858 | 2.39 | 9000 | 0.1986 | 0.8073 | 0.9716 | 0.9412 | 0.5091 | 0.7748 | 0.9839 | 0.8889 | 0.4516 |
134
- | 0.1755 | 2.53 | 9500 | 0.1945 | 0.7872 | 0.9754 | 0.9180 | 0.4681 | 0.8049 | 0.9710 | 0.9655 | 0.4783 |
135
- | 0.1591 | 2.66 | 10000 | 0.2366 | 0.7880 | 0.9692 | 0.8857 | 0.5091 | 0.7504 | 0.9838 | 0.8158 | 0.4516 |
136
- | 0.1457 | 2.79 | 10500 | 0.2346 | 0.7614 | 0.9671 | 0.8857 | 0.4314 | 0.7334 | 0.9771 | 0.8158 | 0.4074 |
137
- | 0.1376 | 2.93 | 11000 | 0.2361 | 0.8015 | 0.9729 | 0.9118 | 0.52 | 0.7802 | 0.9795 | 0.8611 | 0.5 |
138
- | 0.126 | 3.06 | 11500 | 0.2276 | 0.8331 | 0.9751 | 0.9688 | 0.5556 | 0.8168 | 0.9818 | 0.9688 | 0.5 |
139
- | 0.1133 | 3.19 | 12000 | 0.2972 | 0.8014 | 0.9727 | 0.9231 | 0.5085 | 0.7746 | 0.9861 | 0.9091 | 0.4286 |
140
- | 0.1114 | 3.33 | 12500 | 0.2600 | 0.8038 | 0.9705 | 0.8955 | 0.5455 | 0.7742 | 0.9816 | 0.8571 | 0.4839 |
141
- | 0.1099 | 3.46 | 13000 | 0.3221 | 0.8273 | 0.9738 | 0.9118 | 0.5965 | 0.7882 | 0.9884 | 0.8611 | 0.5152 |
142
- | 0.1116 | 3.59 | 13500 | 0.2277 | 0.8376 | 0.9775 | 0.9231 | 0.6122 | 0.8296 | 0.9797 | 0.9091 | 0.6 |
143
- | 0.106 | 3.73 | 14000 | 0.2347 | 0.8148 | 0.9774 | 0.8955 | 0.5714 | 0.7997 | 0.9819 | 0.8571 | 0.56 |
144
- | 0.098 | 3.86 | 14500 | 0.2337 | 0.8487 | 0.9775 | 0.9688 | 0.6 | 0.8418 | 0.9797 | 0.9688 | 0.5769 |
145
- | 0.0899 | 3.99 | 15000 | 0.2072 | 0.8636 | 0.9820 | 0.9688 | 0.64 | 0.8561 | 0.9842 | 0.9688 | 0.6154 |
146
- | 0.0855 | 4.12 | 15500 | 0.2385 | 0.8409 | 0.9762 | 0.9538 | 0.5926 | 0.8189 | 0.9840 | 0.9394 | 0.5333 |
147
- | 0.0864 | 4.26 | 16000 | 0.2780 | 0.8462 | 0.9774 | 0.9688 | 0.5926 | 0.8287 | 0.9841 | 0.9688 | 0.5333 |
148
- | 0.0784 | 4.39 | 16500 | 0.2668 | 0.8277 | 0.9776 | 0.9524 | 0.5532 | 0.8361 | 0.9754 | 0.9677 | 0.5652 |
149
- | 0.0923 | 4.52 | 17000 | 0.2893 | 0.8399 | 0.9738 | 0.9254 | 0.6207 | 0.8012 | 0.9884 | 0.8857 | 0.5294 |
150
- | 0.0794 | 4.66 | 17500 | 0.3101 | 0.8556 | 0.9773 | 0.9688 | 0.6207 | 0.8289 | 0.9885 | 0.9688 | 0.5294 |
151
- | 0.082 | 4.79 | 18000 | 0.2245 | 0.8818 | 0.9842 | 0.9688 | 0.6923 | 0.8668 | 0.9887 | 0.9688 | 0.6429 |
152
- | 0.084 | 4.92 | 18500 | 0.2771 | 0.8247 | 0.9797 | 0.8986 | 0.5957 | 0.8102 | 0.9841 | 0.8378 | 0.6087 |
153
- | 0.0757 | 5.06 | 19000 | 0.2971 | 0.8594 | 0.9773 | 0.9677 | 0.6333 | 0.8388 | 0.9885 | 1.0 | 0.5278 |
154
- | 0.0709 | 5.19 | 19500 | 0.3601 | 0.8410 | 0.9774 | 0.9688 | 0.5769 | 0.8288 | 0.9819 | 0.9688 | 0.5357 |
155
- | 0.0698 | 5.32 | 20000 | 0.2772 | 0.8333 | 0.9762 | 0.9524 | 0.5714 | 0.8173 | 0.9840 | 0.9677 | 0.5 |
156
- | 0.0652 | 5.45 | 20500 | 0.3397 | 0.8186 | 0.9752 | 0.9524 | 0.5283 | 0.8100 | 0.9796 | 0.9677 | 0.4828 |
157
- | 0.0735 | 5.59 | 21000 | 0.3027 | 0.8412 | 0.9785 | 0.9524 | 0.5926 | 0.8284 | 0.9841 | 0.9677 | 0.5333 |
158
- | 0.0746 | 5.72 | 21500 | 0.3122 | 0.8384 | 0.9751 | 0.9688 | 0.5714 | 0.8176 | 0.9840 | 0.9688 | 0.5 |
159
- | 0.0714 | 5.85 | 22000 | 0.2683 | 0.8381 | 0.9787 | 0.9524 | 0.5833 | 0.8429 | 0.9776 | 0.9677 | 0.5833 |
160
- | 0.073 | 5.99 | 22500 | 0.2436 | 0.8676 | 0.9786 | 0.9841 | 0.64 | 0.8650 | 0.9797 | 1.0 | 0.6154 |
161
- | 0.0653 | 6.12 | 23000 | 0.3380 | 0.8559 | 0.9761 | 0.9688 | 0.6230 | 0.8243 | 0.9907 | 0.9688 | 0.5135 |
162
 
163
 
164
  ### Framework versions
@@ -166,4 +74,4 @@ The following hyperparameters were used during training:
166
  - Transformers 4.11.3
167
  - Pytorch 1.9.0+cu102
168
  - Datasets 1.9.0
169
- - Tokenizers 0.10.2
 
1
  ---
2
  license: mit
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  tags:
4
  - generated_from_trainer
5
  model-index:
6
+ - name: verdict-classifier
7
+ results: []
 
 
 
 
 
8
  ---
9
 
10
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
11
+ should probably proofread and complete it, then remove this comment. -->
12
 
13
+ # verdict-classifier
 
 
 
 
 
 
 
 
 
 
14
 
15
+ This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the None dataset.
16
+ It achieves the following results on the evaluation set:
17
+ - Loss: 1.1573
18
+ - F1 Macro: 0.0550
19
+ - F1 Misinformation: 0.0
20
+ - F1 Factual: 0.1650
21
+ - F1 Other: 0.0
22
+ - Prec Macro: 0.0300
23
+ - Prec Misinformation: 0.0
24
+ - Prec Factual: 0.0899
25
+ - Prec Other: 0.0
26
+
27
+ ## Model description
28
+
29
+ More information needed
30
+
31
+ ## Intended uses & limitations
32
+
33
+ More information needed
34
+
35
+ ## Training and evaluation data
36
+
37
+ More information needed
38
 
39
  ## Training procedure
40
 
 
49
  - total_train_batch_size: 32
50
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
51
  - lr_scheduler_type: linear
52
+ - lr_scheduler_warmup_steps: 162525
53
  - num_epochs: 1000
54
 
55
  ### Training results
56
 
57
+ | Training Loss | Epoch | Step | Validation Loss | F1 Macro | F1 Misinformation | F1 Factual | F1 Other | Prec Macro | Prec Misinformation | Prec Factual | Prec Other |
58
+ |:-------------:|:-----:|:----:|:---------------:|:--------:|:-----------------:|:----------:|:--------:|:----------:|:-------------------:|:------------:|:----------:|
59
+ | 1.2021 | 0.0 | 50 | 1.1573 | 0.0550 | 0.0 | 0.1650 | 0.0 | 0.0300 | 0.0 | 0.0899 | 0.0 |
60
+ | 1.1948 | 0.0 | 100 | 1.1569 | 0.0550 | 0.0 | 0.1650 | 0.0 | 0.0300 | 0.0 | 0.0899 | 0.0 |
61
+ | 1.1968 | 0.01 | 150 | 1.1563 | 0.0550 | 0.0 | 0.1650 | 0.0 | 0.0300 | 0.0 | 0.0899 | 0.0 |
62
+ | 1.1925 | 0.01 | 200 | 1.1554 | 0.0550 | 0.0 | 0.1650 | 0.0 | 0.0300 | 0.0 | 0.0899 | 0.0 |
63
+ | 1.2055 | 0.01 | 250 | 1.1544 | 0.0550 | 0.0 | 0.1650 | 0.0 | 0.0300 | 0.0 | 0.0899 | 0.0 |
64
+ | 1.1927 | 0.01 | 300 | 1.1531 | 0.0550 | 0.0 | 0.1650 | 0.0 | 0.0300 | 0.0 | 0.0899 | 0.0 |
65
+ | 1.1923 | 0.02 | 350 | 1.1515 | 0.0550 | 0.0 | 0.1650 | 0.0 | 0.0300 | 0.0 | 0.0899 | 0.0 |
66
+ | 1.1929 | 0.02 | 400 | 1.1496 | 0.0550 | 0.0 | 0.1650 | 0.0 | 0.0300 | 0.0 | 0.0899 | 0.0 |
67
+ | 1.1924 | 0.02 | 450 | 1.1476 | 0.0550 | 0.0 | 0.1650 | 0.0 | 0.0300 | 0.0 | 0.0899 | 0.0 |
68
+ | 1.1862 | 0.02 | 500 | 1.1454 | 0.0550 | 0.0 | 0.1650 | 0.0 | 0.0300 | 0.0 | 0.0899 | 0.0 |
69
+ | 1.1781 | 0.03 | 550 | 1.1432 | 0.0550 | 0.0 | 0.1650 | 0.0 | 0.0300 | 0.0 | 0.0899 | 0.0 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
70
 
71
 
72
  ### Framework versions
 
74
  - Transformers 4.11.3
75
  - Pytorch 1.9.0+cu102
76
  - Datasets 1.9.0
77
+ - Tokenizers 0.10.2