ficsort commited on
Commit
1d9e1b6
·
1 Parent(s): a9ed542

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +14 -27
README.md CHANGED
@@ -1,43 +1,27 @@
1
  ---
 
 
 
 
2
  tags:
3
  - generated_from_keras_callback
 
4
  model-index:
5
  - name: hubert-medium-wiki-seq128
6
  results: []
7
  ---
8
 
9
- <!-- This model card has been generated automatically according to the information Keras had access to. You should
10
- probably proofread and complete it, then remove this comment. -->
11
-
12
  # hubert-medium-wiki-seq128
13
 
14
- This model was trained from scratch on an unknown dataset.
15
- It achieves the following results on the evaluation set:
16
-
17
-
18
- ## Model description
19
-
20
- More information needed
21
-
22
- ## Intended uses & limitations
23
-
24
- More information needed
25
 
26
- ## Training and evaluation data
27
-
28
- More information needed
29
-
30
- ## Training procedure
31
-
32
- ### Training hyperparameters
33
-
34
- The following hyperparameters were used during training:
35
- - optimizer: None
36
- - training_precision: float32
37
-
38
- ### Training results
39
 
 
40
 
 
 
 
41
 
42
  ### Framework versions
43
 
@@ -45,3 +29,6 @@ The following hyperparameters were used during training:
45
  - TensorFlow 2.10.0
46
  - Datasets 2.4.0
47
  - Tokenizers 0.12.1
 
 
 
 
1
  ---
2
+ language: hu
3
+ license: apache-2.0
4
+ datasets:
5
+ - wikipedia
6
  tags:
7
  - generated_from_keras_callback
8
+ - hubert
9
  model-index:
10
  - name: hubert-medium-wiki-seq128
11
  results: []
12
  ---
13
 
 
 
 
14
  # hubert-medium-wiki-seq128
15
 
16
+ Fully trained model with the second phase of training is available here: [not available yet](#)
 
 
 
 
 
 
 
 
 
 
17
 
18
+ This model was trained from scratch on the Wikipedia subset of Hungarian Webcorpus 2.0 with MLM and SOP tasks.
 
 
 
 
 
 
 
 
 
 
 
 
19
 
20
+ ### Pre-Training Parameters:
21
 
22
+ - Training steps: 500.000
23
+ - Sequence length: 128 (the model is capable for 512)
24
+ - Batch size: 1024
25
 
26
  ### Framework versions
27
 
 
29
  - TensorFlow 2.10.0
30
  - Datasets 2.4.0
31
  - Tokenizers 0.12.1
32
+
33
+ # Acknowledgement
34
+ [![Artificial Intelligence - National Laboratory - Hungary](https://milab.tk.hu/uploads/images/milab_logo_en.png)](https://mi.nemzetilabor.hu/)