Zhuoyang Song
commited on
Commit
•
6df700e
1
Parent(s):
2b5eb6a
edit model card
Browse files
README.md
CHANGED
@@ -22,7 +22,7 @@ Chinese deductive reasoning model based on Transformer-XL.
|
|
22 |
|
23 |
## 模型信息 Model Information
|
24 |
|
25 |
-
|
26 |
|
27 |
* 悟道语料库(280G版本)
|
28 |
* 因果语料库(2.3M个样本):基于悟道语料库(280G版本),通过关联词匹配、人工标注 + [GTSFactory](https://gtsfactory.com/)筛选、数据清洗等步骤获取的具有因果关系的句子对
|
@@ -30,15 +30,13 @@ Chinese deductive reasoning model based on Transformer-XL.
|
|
30 |
* Wudao Corpus (with 280G samples)
|
31 |
* Wudao Causal Corpus (with 2.3 million samples): Based on the Wudao corpus (280G version), sentence pairs with causality were obtained through logic indicator matching, manual annotation + [GTSFactory](https://gtsfactory.com/), and data cleaning.
|
32 |
|
33 |
-
|
34 |
-
### 训练流程 Model Training
|
35 |
1. 在悟道语料库(280G版本)和标注的相似句子对数据集上进行预训练([Randeng-TransformerXL-1.1B-Paraphrasing-Chinese](https://huggingface.co/IDEA-CCNL/Randeng-TransformerXL-1.1B-Paraphrasing-Chinese))
|
36 |
2. 在1.5M因果语料上进行因果生成任务的训练
|
37 |
3. 基于其余0.8M因果语料,协同[Randeng-TransformerXL-5B-Abduction-Chinese](https://huggingface.co/IDEA-CCNL/Randeng-TransformerXL-5B-Abduction-Chinese)和[Erlangshen-Roberta-330M-Causal-Chinese](https://huggingface.co/IDEA-CCNL/Erlangshen-Roberta-330M-Causal-Chinese)进行Self-consistency闭环迭代训练
|
38 |
* 两个生成模型基于核采样和贪心的方式进行因果推理和反绎推理,产生大量伪样本;
|
39 |
* Erlangshen-Roberta-330M-Causal-Chinese模型对伪样本句子对的因果关系进行打分,筛选供自身以及生成模型训练的样本
|
40 |
|
41 |
-
|
42 |
First, the Transformer-XL model was pre-trained on the Wudao Corpus (with 280G samples) and annotated similar-sentence pair dataset (same as [Randeng-TransformerXL-1.1B-Paraphrasing-Chinese](https://huggingface.co/IDEA-CCNL/Randeng-TransformerXL-1.1B-Paraphrasing-Chinese)).
|
43 |
Then, the model was trained on our causal corpus (about 1.5 million samples) for the deductive reasoning task.
|
44 |
At last, based on the remaining 0.8 million samples of the causal corpus, we conducted self-consistent learning on this model, cooperating with [Randeng-TransformerXL-5B-Abduction-Chinese](https://huggingface.co/IDEA-CCNL/Randeng-TransformerXL-5B-Abduction-Chinese) and [Erlangshen-Roberta-330M-Causal-Chinese](https://huggingface.co/IDEA-CCNL/Erlangshen-Roberta-330M-Causal-Chinese).
|
|
|
22 |
|
23 |
## 模型信息 Model Information
|
24 |
|
25 |
+
**数据准备 Corpus Preparation**
|
26 |
|
27 |
* 悟道语料库(280G版本)
|
28 |
* 因果语料库(2.3M个样本):基于悟道语料库(280G版本),通过关联词匹配、人工标注 + [GTSFactory](https://gtsfactory.com/)筛选、数据清洗等步骤获取的具有因果关系的句子对
|
|
|
30 |
* Wudao Corpus (with 280G samples)
|
31 |
* Wudao Causal Corpus (with 2.3 million samples): Based on the Wudao corpus (280G version), sentence pairs with causality were obtained through logic indicator matching, manual annotation + [GTSFactory](https://gtsfactory.com/), and data cleaning.
|
32 |
|
33 |
+
**训练流程 Model Training**
|
|
|
34 |
1. 在悟道语料库(280G版本)和标注的相似句子对数据集上进行预训练([Randeng-TransformerXL-1.1B-Paraphrasing-Chinese](https://huggingface.co/IDEA-CCNL/Randeng-TransformerXL-1.1B-Paraphrasing-Chinese))
|
35 |
2. 在1.5M因果语料上进行因果生成任务的训练
|
36 |
3. 基于其余0.8M因果语料,协同[Randeng-TransformerXL-5B-Abduction-Chinese](https://huggingface.co/IDEA-CCNL/Randeng-TransformerXL-5B-Abduction-Chinese)和[Erlangshen-Roberta-330M-Causal-Chinese](https://huggingface.co/IDEA-CCNL/Erlangshen-Roberta-330M-Causal-Chinese)进行Self-consistency闭环迭代训练
|
37 |
* 两个生成模型基于核采样和贪心的方式进行因果推理和反绎推理,产生大量伪样本;
|
38 |
* Erlangshen-Roberta-330M-Causal-Chinese模型对伪样本句子对的因果关系进行打分,筛选供自身以及生成模型训练的样本
|
39 |
|
|
|
40 |
First, the Transformer-XL model was pre-trained on the Wudao Corpus (with 280G samples) and annotated similar-sentence pair dataset (same as [Randeng-TransformerXL-1.1B-Paraphrasing-Chinese](https://huggingface.co/IDEA-CCNL/Randeng-TransformerXL-1.1B-Paraphrasing-Chinese)).
|
41 |
Then, the model was trained on our causal corpus (about 1.5 million samples) for the deductive reasoning task.
|
42 |
At last, based on the remaining 0.8 million samples of the causal corpus, we conducted self-consistent learning on this model, cooperating with [Randeng-TransformerXL-5B-Abduction-Chinese](https://huggingface.co/IDEA-CCNL/Randeng-TransformerXL-5B-Abduction-Chinese) and [Erlangshen-Roberta-330M-Causal-Chinese](https://huggingface.co/IDEA-CCNL/Erlangshen-Roberta-330M-Causal-Chinese).
|