language:
- zh
tags:
- Question Answering
- Machine Reading
- Text Generation
- Pretrained Chinese T5-Large model
datasets:
- CMRC 2018 dev
metrics:
- RougeL
- BLEU-4
- F1
- EM
- Contain Answer Rate
licence: apache-2.0
T5 for Chinese Question Answering
Randeng-T5-784M-QA-Chinese
Brief Introduction
This T5-Large model, is the first pretrained generative question answering model for Chinese in huggingface. It was pretrained on the Wudao 180G corpus, finetuned on Chinese SQuAD and CMRC2018 dataset. It can produce a fluent and accurate answer given a passage and question.
这是huggingface上首个中文的生成式问答模型。它基于T5-Large结构，使用悟道180G语料进行预训练，之后在翻译的中文SQuAD和CMRC2018两个阅读理解数据集上进行微调。输入一篇文章和一个问题，可以生成准确流畅的回答。
Performance
CMRC 2018 dev (Original span prediction task, we cast it as a generative QA task) CMRC 2018的测试集上的效果（原始任务是一个起始和结束预测问题，这里作为一个生成回答的问题）
| model | F1 | EM | Contain Answer Rate | RougeL | BLEU-4 |
|---|---|---|---|---|---|
| Ours | 77.9 | 57.1 | 76.0 | 82.7 | 61.1 |
| MacBERT-Large(SOTA) | 88.9 | 70.0 | - | - | - |
Our model enjoys a high level of generation quality and accuracy, with 76% of generated answers containing the ground truth, which rivals the EM of span prediction SOTA. Our model has a lower EM because it generates complete sentences while groud truth are segmentations of sentences. The extremely high RougeL and BLEU-4 reveals the overlap between generated results and groud truth. P.S.The SOTA model only predicts the start and end tag as an extractive MRC task.
我们的模型有着极高的生成质量和准确率，76%的回答包含了正确答案(Contain Answer Rate)，和当前最好模型MacBERT-Large想媲美，它70%的起始位置预测和答案精确匹配(EM)。我们的模型EM值较低，因为生成的大部分为完整的句子，而标准答案通常是句子片段。 P.S. SOTA模型只需预测起始和结束位置，这种抽取式阅读理解任务比生成式的简单很多。
Cases
![avatar][cases_t5_cmrc.png]
pred in picture are generated results,target indicates groud truth.
Usage
import numpy as np
from transformers import T5Tokenizer,MT5ForConditionalGeneration
pretrain_path = 'IDEA-CCNL/Randeng-T5-784M-QA-Chinese'
tokenizer=T5Tokenizer.from_pretrained(pretrain_path)
model=MT5ForConditionalGeneration.from_pretrained(pretrain_path)
sample={"context":"","question":"","idx":1}
plain_text='question:'+sample['question']+'knowledge:'+sample['context'][:self.max_knowledge_length]
res_prefix=tokenizer.encode('answer'+'<extra_id_0>',add_special_token=False)
l_rp=len(res_prefix)
tokenized=tokenizer.encode(plain_text,add_special_tokens=False,truncation=True,max_length=self.max_seq_length-2-l_rp)
tokenized+=res_prefix
tokenized.append(EOS_TOKEN_ID)
# Generate answer
pred_ids = model.generate(input_ids=tokenized,max_new_token=self.max_target_length,do_sample=True,top_p=0.9)
tokenizer.batch_decode(pred_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
Citation
You can also cite our website: 欢迎引用我们的网站:
@misc{Fengshenbang-LM,
title={Fengshenbang-LM},
author={IDEA-CCNL},
year={2021},
howpublished={\url{https://github.com/IDEA-CCNL/Fengshenbang-LM}},
}