Hugging Face's logo

scutcyr
/
dstc11-simmc2.1-scut-bds-lab

Model card Files Community
dstc11-simmc2.1-scut-bds-lab / README.md
scutcyr's picture
scutcyr
Update README.md
3af699a
preview code
|
raw
history blame
2.71 kB
metadata
license: apache-2.0

Model Card: dstc11-simmc2.1-iflytek

Recent Update

Overview

The SIMMC2.1 challenge aims to lay the foundations for the real-world assistant agents that can handle multimodal inputs, and perform multimodal actions. It has 4 tasks: Ambiguous Candidate Identification, Multimodal Coreference Resolution, Multimodal Dialog State Tracking, Response Generation. We consider the joint input of textual context, tokenized objects and scene as multi-modal input, as well as compare the performance of single task training and multi task joint training. As to subtask4, we also consider the system belief state (act and slot values) as the prombt for response generation. Non-visual metadata is also considered by adding the embedding to the object.

Model Date

Model was originally released in October 2022.

Model Type

The mt-bart, mt-bart-sys and mt-bart-sys-nvattr have the same model framework (transformer with multi-task head), which are finetuned on SIMMC2.1 based on the pretrained BART-Large model. This repository also contains code to finetune the model.

Results

devtest result

Model Subtask-1 Amb. Candi. F1 Subtask-2 MM Coref F1 Subtask-3 MM DST Slot F1 Subtask-3 MM DST Intent F1 Subtask-4 Response Gen. BLEU-4
mt-bart-ensemble 0.68466 0.77860 0.91816 0.97828 0.34496
mt-bart-sys 0.39064
mt-bart-sys-attr 0.38995

teststd result

The teststd result is provided in the teststd-result. One subfolder corresponds to one model.

Using with Transformers

(1) You should first download the model from huggingface used the scripts:

cd ~
mkdir pretrained_model
cd pretrained_model
git lfs install
git clone https://huggingface.co/scutcyr/dstc11-simmc2.1-iflytek

(2) Then you should clone our code use the follow scripts:

cd ~
git clone https://github.com/scutcyr/dstc11-simmc2.1-iflytek.git

(3) Follow the README to use the model.