Update README.md
Browse files
README.md
CHANGED
@@ -20,9 +20,9 @@ widget:
|
|
20 |
|
21 |
## 简介 Brief Introduction
|
22 |
|
23 |
-
善于处理NLU任务,采用全词掩码的,中文版的0.97亿参数DeBERTa-v2。
|
24 |
|
25 |
-
Good at solving NLU tasks, adopting Whole Word Masking, Chinese DeBERTa-v2 with 97M parameters.
|
26 |
|
27 |
## 模型分类 Model Taxonomy
|
28 |
|
@@ -35,9 +35,9 @@ Good at solving NLU tasks, adopting Whole Word Masking, Chinese DeBERTa-v2 with
|
|
35 |
|
36 |
参考论文:[DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://readpaper.com/paper/3033187248)
|
37 |
|
38 |
-
为了得到一个中文版的DeBERTa-v2(97M),我们用悟道语料库(180G版本)进行预训练。我们在MLM中使用了全词掩码(wwm)的方式。具体地,我们在预训练阶段中使用了[封神框架](https://github.com/IDEA-CCNL/Fengshenbang-LM/tree/main/fengshen)大概花费了24张A100约7天。
|
39 |
|
40 |
-
To get a Chinese DeBERTa-v2 (97M), we use WuDao Corpora (180 GB version) for pre-training. We employ the Whole Word Masking (wwm) in MLM. Specifically, we use the [fengshen framework](https://github.com/IDEA-CCNL/Fengshenbang-LM/tree/main/fengshen) in the pre-training phase which cost about 7 days with 24 A100 GPUs.
|
41 |
|
42 |
### 下游任务 Performance
|
43 |
|
|
|
20 |
|
21 |
## 简介 Brief Introduction
|
22 |
|
23 |
+
善于处理NLU任务,采用全词掩码的,中文版的0.97亿参数DeBERTa-v2-Base。
|
24 |
|
25 |
+
Good at solving NLU tasks, adopting Whole Word Masking, Chinese DeBERTa-v2-Base with 97M parameters.
|
26 |
|
27 |
## 模型分类 Model Taxonomy
|
28 |
|
|
|
35 |
|
36 |
参考论文:[DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://readpaper.com/paper/3033187248)
|
37 |
|
38 |
+
为了得到一个中文版的DeBERTa-v2-Base(97M),我们用悟道语料库(180G版本)进行预训练。我们在MLM中使用了全词掩码(wwm)的方式。具体地,我们在预训练阶段中使用了[封神框架](https://github.com/IDEA-CCNL/Fengshenbang-LM/tree/main/fengshen)大概花费了24张A100约7天。
|
39 |
|
40 |
+
To get a Chinese DeBERTa-v2-Base (97M), we use WuDao Corpora (180 GB version) for pre-training. We employ the Whole Word Masking (wwm) in MLM. Specifically, we use the [fengshen framework](https://github.com/IDEA-CCNL/Fengshenbang-LM/tree/main/fengshen) in the pre-training phase which cost about 7 days with 24 A100 GPUs.
|
41 |
|
42 |
### 下游任务 Performance
|
43 |
|