Commit
·
def38cf
1
Parent(s):
cd68294
Update README.md
Browse files
README.md
CHANGED
@@ -25,12 +25,14 @@ widget:
|
|
25 |
While the original dataset contained tokens denoting speech disfluencies and hesitations (parenthesised) and background noise [bracketed], this model was trained on a version of the dataset where such tokens and duplicates were removed.
|
26 |
As a result, the number of sentences, tokens, and named entities (NEs) in the cleaned dataset changed.
|
27 |
|
|
|
28 |
| Unit | Train | Valid | Test | Total |
|
29 |
| :---: | :---: | :---: | :---: | :---: |
|
30 |
| Sentence | 88,540 (80.00%) | 11,067 (10.00%) | 11,068 (10.00%) | 110,675 (100%) |
|
31 |
| Token | 1,088,461 (80.04%) | 136,021 (10.00%) | 135,426 (9.96%) | 1,359,908 (100%) |
|
32 |
| NE | 106,148 (80.17%) | 13,189 (9.96%) | 13,072 (9.87%) | 132,409 (100%) |
|
33 |
|
|
|
34 |
|Representation| Train | Valid | Test | Total |
|
35 |
| :---: | :---: | :---: | :---: | :---: |
|
36 |
| **AID** | 67,582 (79.99%) | 8,439 (9.99%) | 8,467 (10.02%)| 84,488 (100%) |
|
@@ -40,3 +42,14 @@ As a result, the number of sentences, tokens, and named entities (NEs) in the cl
|
|
40 |
| **EID** | 260 (81.00%) | 27 (8.41%) | 34 (10.59%)| 321 (100%) |
|
41 |
| **FID** | 9 (75.00%) | 1 (8.33%)| 2 (16.67%)| 12 (100%) |
|
42 |
|**Total**| **88,540 (80.00%)** | **11,067 (10.00%)** | **11,068 (10.00%)** | **110,675 (100%)** |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
25 |
While the original dataset contained tokens denoting speech disfluencies and hesitations (parenthesised) and background noise [bracketed], this model was trained on a version of the dataset where such tokens and duplicates were removed.
|
26 |
As a result, the number of sentences, tokens, and named entities (NEs) in the cleaned dataset changed.
|
27 |
|
28 |
+
**Statistics for training (Train), validation (Valid), and test (Test) sets**
|
29 |
| Unit | Train | Valid | Test | Total |
|
30 |
| :---: | :---: | :---: | :---: | :---: |
|
31 |
| Sentence | 88,540 (80.00%) | 11,067 (10.00%) | 11,068 (10.00%) | 110,675 (100%) |
|
32 |
| Token | 1,088,461 (80.04%) | 136,021 (10.00%) | 135,426 (9.96%) | 1,359,908 (100%) |
|
33 |
| NE | 106,148 (80.17%) | 13,189 (9.96%) | 13,072 (9.87%) | 132,409 (100%) |
|
34 |
|
35 |
+
**80 / 10 / 10 split**
|
36 |
|Representation| Train | Valid | Test | Total |
|
37 |
| :---: | :---: | :---: | :---: | :---: |
|
38 |
| **AID** | 67,582 (79.99%) | 8,439 (9.99%) | 8,467 (10.02%)| 84,488 (100%) |
|
|
|
42 |
| **EID** | 260 (81.00%) | 27 (8.41%) | 34 (10.59%)| 321 (100%) |
|
43 |
| **FID** | 9 (75.00%) | 1 (8.33%)| 2 (16.67%)| 12 (100%) |
|
44 |
|**Total**| **88,540 (80.00%)** | **11,067 (10.00%)** | **11,068 (10.00%)** | **110,675 (100%)** |
|
45 |
+
|
46 |
+
**Distribution of representations across sets**
|
47 |
+
|Representation| Train | Valid | Test | Total |
|
48 |
+
| :---: | :---: | :---: | :---: | :---: |
|
49 |
+
| **AID** | 67,582 (76.33%) | 8,439 (76.25%) | 8,467 (76.50%)| 84,488 (76.34%) |
|
50 |
+
| **BID** | 19,006 (21.47%) | 2,380 (21.51%) | 2,338 (21.12%)| 23,724 (21.44%) |
|
51 |
+
| **CID** | 1,050 (1.19%) | 138 (1.25%) | 143 ( 1.29%) | 1,331 (1.20%) |
|
52 |
+
| **DID** | 633 (0.71%) | 82 (0.74%) | 84 (0.76%) | 799 (0.72%) |
|
53 |
+
| **EID** | 260 (0.29%) | 27 (0.24%) | 34 (0.31%)| 321 (0.29%) |
|
54 |
+
| **FID** | 9 (0.01%) | 1 (0.01%)| 2 (0.02%)| 12 (0.01%) |
|
55 |
+
|**Total**| **88,540 (100.00%)** | **11,067 (10.00%)** | **11,068 (10.00%)** | **110,675 (100%)** |
|