Update README.md

README.md

@@ -1,4 +1,71 @@
 ---
-license: odc-by
+language: ja
+license: mit
+datasets:
+- mC4 Japanese
 ---
-Preparing for release
+
+# roberta-long-japanese (jumanpp + sentencepiece, mC4 Japanese)
+
+This is the longer input version of [RoBERTa](https://arxiv.org/abs/1907.11692) Japanese model pretrained on approximately 200M Japanese sentences.
+`max_position_embeddings` has been increased to `1282`, allowing it to handle much longer inputs than the basic `RoBERTa` model.
+
+The tokenization model and logic is completely same as [nlp-waseda/roberta-base-japanese](https://huggingface.co/nlp-waseda/roberta-base-japanese).
+The input text should be pretokenized by [Juman++ v2.0.0-rc3](https://github.com/ku-nlp/jumanpp) and then the [SentencePiece](https://github.com/google/sentencepiece) tokenization will be applied for the whitespace-separated token sequences.
+See `tokenizer_config.json` for details.
+
+## How to use
+
+Please install `Juman++ v2.0.0-rc3` and `SentencePiece` in advance.
+
+- https://github.com/ku-nlp/jumanpp#building-from-a-package
+- https://github.com/google/sentencepiece#python-module
+
+You can load the model and the tokenizer via AutoModel and AutoTokenizer, respectively.
+
+```python
+from transformers import AutoModel, AutoTokenizer
+model = AutoModel.from_pretrained("megagonlabs/roberta-long-japanese")
+tokenizer = AutoTokenizer.from_pretrained("megagonlabs/roberta-long-japanese")
+model(**tokenizer("まさに オール マイ ティー な 商品 だ 。", return_tensors="pt")).last_hidden_state
+tensor([[[ 0.1549, -0.7576,  0.1098,  ...,  0.7124,  0.8062, -0.9880],
+         [-0.6586, -0.6138, -0.5253,  ...,  0.8853,  0.4822, -0.6463],
+         [-0.4502, -1.4675, -0.4095,  ...,  0.9053, -0.2017, -0.7756],
+         ...,
+         [ 0.3505, -1.8235, -0.6019,  ..., -0.0906, -0.5479, -0.6899],
+         [ 1.0524, -0.8609, -0.6029,  ...,  0.1022, -0.6802,  0.0982],
+         [ 0.6519, -0.2042, -0.6205,  ..., -0.0738, -0.0302, -0.1955]]],
+       grad_fn=<NativeLayerNormBackward0>)
+```
+
+## Model architecture
+
+The model architecture is almost the same as [nlp-waseda/roberta-base-japanese](https://huggingface.co/nlp-waseda/roberta-base-japanese) except `max_position_embeddings` has been increased to `1282`; 12 layers, 768 dimensions of hidden states, and 12 attention heads.
+
+## Training data and libraries
+
+This model is trained on the Japanese texts extracted from the [mC4](https://huggingface.co/datasets/mc4) Common Crawl's multilingual web crawl corpus.
+We used the [Sudachi](https://github.com/WorksApplications/Sudachi) to split texts into sentences, and also applied a simple rule-based filter to remove nonlinguistic segments of mC4 multilingual corpus.
+The extracted texts contains over 600M sentences in total, and we used approximately 200M sentences for pretraining.
+
+We used [huggingface/transformers RoBERTa implementation](https://github.com/huggingface/transformers/tree/v4.21.0/src/transformers/models/roberta) for pretraining. The time required for the pretrainig was about 300 hours using GCP A100 8gpu instance with enabling Automatic Mixed Precision.
+
+## Licenses
+
+The pretrained models are distributed under the terms of the [MIT License](https://opensource.org/licenses/mit-license.php).
+
+## Citations
+
+- mC4
+
+Contains information from `mC4` which is made available under the [ODC Attribution License](https://opendatacommons.org/licenses/by/1-0/).
+```
+@article{2019t5,
+    author = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
+    title = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
+    journal = {arXiv e-prints},
+    year = {2019},
+    archivePrefix = {arXiv},
+    eprint = {1910.10683},
+}
+```