hiroshi-matsuda-rit
commited on
Commit
·
d9559d4
1
Parent(s):
f838186
Update README.md
Browse files
README.md
CHANGED
@@ -48,7 +48,7 @@ This model is trained on the Japanese texts extracted from the [mC4](https://hug
|
|
48 |
We used the [Sudachi](https://github.com/WorksApplications/Sudachi) to split texts into sentences, and also applied a simple rule-based filter to remove nonlinguistic segments of mC4 multilingual corpus.
|
49 |
The extracted texts contains over 600M sentences in total, and we used approximately 200M sentences for pretraining.
|
50 |
|
51 |
-
We used [huggingface/transformers RoBERTa implementation](https://github.com/huggingface/transformers/tree/v4.21.0/src/transformers/models/roberta) for pretraining. The time required for the pretrainig was about
|
52 |
|
53 |
## Licenses
|
54 |
|
|
|
48 |
We used the [Sudachi](https://github.com/WorksApplications/Sudachi) to split texts into sentences, and also applied a simple rule-based filter to remove nonlinguistic segments of mC4 multilingual corpus.
|
49 |
The extracted texts contains over 600M sentences in total, and we used approximately 200M sentences for pretraining.
|
50 |
|
51 |
+
We used [huggingface/transformers RoBERTa implementation](https://github.com/huggingface/transformers/tree/v4.21.0/src/transformers/models/roberta) for pretraining. The time required for the pretrainig was about 700 hours using GCP A100 8gpu instance with enabling Automatic Mixed Precision.
|
52 |
|
53 |
## Licenses
|
54 |
|