Alina Kolesnikova
commited on
Commit
•
fcd48fc
1
Parent(s):
d921e12
Conversational DistilRuBERT added
Browse files- .ipynb_checkpoints/README-checkpoint.md +16 -0
- README.md +16 -0
- config.json +22 -0
- pytorch_model.bin +3 -0
- special_tokens_map.json +1 -0
- tokenizer_config.json +1 -0
- vocab.txt +0 -0
.ipynb_checkpoints/README-checkpoint.md
ADDED
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
language:
|
3 |
+
- ru
|
4 |
+
---
|
5 |
+
# distilrubert-base-cased-conversational
|
6 |
+
Conversational DistilRuBERT \(Russian, cased, 6‑layer, 768‑hidden, 12‑heads, 135.4M parameters\) was trained on OpenSubtitles\[1\], [Dirty](https://d3.ru/), [Pikabu](https://pikabu.ru/), and a Social Media segment of Taiga corpus\[2\] (as [Conversational RuBERT](https://huggingface.co/DeepPavlov/rubert-base-cased-conversational)).
|
7 |
+
|
8 |
+
Our DistilRuBERT was highly inspired by \[3\], \[4\]. Namely, we used
|
9 |
+
* KL loss (between teacher and student output logits)
|
10 |
+
* MLM loss (between tokens labels and student output logits)
|
11 |
+
* Cosine embedding loss between mean of two consecutive hidden states of the teacher and one hidden state of the student
|
12 |
+
|
13 |
+
\[1\]: P. Lison and J. Tiedemann, 2016, OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation \(LREC 2016\)
|
14 |
+
\[2\]: Shavrina T., Shapovalova O. \(2017\) TO THE METHODOLOGY OF CORPUS CONSTRUCTION FOR MACHINE LEARNING: «TAIGA» SYNTAX TREE CORPUS AND PARSER. in proc. of “CORPORA2017”, international conference , Saint-Petersbourg, 2017.
|
15 |
+
\[3\]: Sanh, V., Debut, L., Chaumond, J., & Wolf, T. \(2019\). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
|
16 |
+
\[4\]: <https://github.com/huggingface/transformers/tree/master/examples/research_projects/distillation>
|
README.md
ADDED
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
language:
|
3 |
+
- ru
|
4 |
+
---
|
5 |
+
# distilrubert-base-cased-conversational
|
6 |
+
Conversational DistilRuBERT \(Russian, cased, 6‑layer, 768‑hidden, 12‑heads, 135.4M parameters\) was trained on OpenSubtitles\[1\], [Dirty](https://d3.ru/), [Pikabu](https://pikabu.ru/), and a Social Media segment of Taiga corpus\[2\] (as [Conversational RuBERT](https://huggingface.co/DeepPavlov/rubert-base-cased-conversational)).
|
7 |
+
|
8 |
+
Our DistilRuBERT was highly inspired by \[3\], \[4\]. Namely, we used
|
9 |
+
* KL loss (between teacher and student output logits)
|
10 |
+
* MLM loss (between tokens labels and student output logits)
|
11 |
+
* Cosine embedding loss between mean of two consecutive hidden states of the teacher and one hidden state of the student
|
12 |
+
|
13 |
+
\[1\]: P. Lison and J. Tiedemann, 2016, OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation \(LREC 2016\)
|
14 |
+
\[2\]: Shavrina T., Shapovalova O. \(2017\) TO THE METHODOLOGY OF CORPUS CONSTRUCTION FOR MACHINE LEARNING: «TAIGA» SYNTAX TREE CORPUS AND PARSER. in proc. of “CORPORA2017”, international conference , Saint-Petersbourg, 2017.
|
15 |
+
\[3\]: Sanh, V., Debut, L., Chaumond, J., & Wolf, T. \(2019\). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
|
16 |
+
\[4\]: <https://github.com/huggingface/transformers/tree/master/examples/research_projects/distillation>
|
config.json
ADDED
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"_name_or_path": "distilrubert-base-cased-conversational.pth",
|
3 |
+
"activation": "gelu",
|
4 |
+
"attention_dropout": 0.1,
|
5 |
+
"dim": 768,
|
6 |
+
"dropout": 0.1,
|
7 |
+
"hidden_dim": 3072,
|
8 |
+
"initializer_range": 0.02,
|
9 |
+
"max_position_embeddings": 512,
|
10 |
+
"model_type": "distilbert",
|
11 |
+
"n_heads": 12,
|
12 |
+
"n_layers": 6,
|
13 |
+
"output_attentions": true,
|
14 |
+
"output_hidden_states": true,
|
15 |
+
"output_past": true,
|
16 |
+
"pad_token_id": 0,
|
17 |
+
"qa_dropout": 0.1,
|
18 |
+
"seq_classif_dropout": 0.2,
|
19 |
+
"sinusoidal_pos_embds": false,
|
20 |
+
"tie_weights_": true,
|
21 |
+
"vocab_size": 119547
|
22 |
+
}
|
pytorch_model.bin
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:3351dcc8270fae791b7475f674d0dad3073b5b6958b70c3b9c8aa5fd0e7ad5ae
|
3 |
+
size 541825184
|
special_tokens_map.json
ADDED
@@ -0,0 +1 @@
|
|
|
|
|
1 |
+
{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}
|
tokenizer_config.json
ADDED
@@ -0,0 +1 @@
|
|
|
|
|
1 |
+
{"do_lower_case": false}
|
vocab.txt
ADDED
The diff for this file is too large to render.
See raw diff
|
|