README.md

---
license: apache-2.0
---
## Model Card: dstc11-simmc2.1-scut-bds-lab

**Team**: [scut-bds-lab](https://github.com/scut-bds)

## Recent Update
- 👏🏻  2022.10.10: The repository `dstc11-simmc2.1-scut-bds-lab` for [DSTC11 Track1](https://github.com/facebookresearch/simmc2) is created.
- 👏🏻  2022.10.28: The model is public on huggingface, see the link [https://huggingface.co/scutcyr/dstc11-simmc2.1-scut-bds-lab](https://huggingface.co/scutcyr/dstc11-simmc2.1-scut-bds-lab) for detail.

## Overview
The [SIMMC2.1](https://github.com/facebookresearch/simmc2) challenge aims to lay the foundations for the real-world assistant agents that can handle multimodal inputs, and perform multimodal actions. It has 4 tasks: Ambiguous Candidate Identification, Multimodal Coreference Resolution, Multimodal Dialog State Tracking, Response Generation. We consider the joint input of textual context, tokenized objects and scene as multi-modal input, as well as compare the performance of single task training and multi task joint training.
As to subtask4, we also consider the system belief state (act and slot values) as the prombt for response generation. Non-visual metadata is also considered by adding the embedding to the object.

## Model Date
Model was originally released in October 2022.

## Model Type
The **mt-bart**, **mt-bart-sys** and **mt-bart-sys-nvattr** have the same model framework (transformer with multi-task head), which are finetuned on [SIMMC2.1](https://github.com/facebookresearch/simmc2) based on the pretrained [BART-Large](https://huggingface.co/facebook/bart-large) model. This [repository](https://github.com/scutcyr/dstc11-simmc2.1-scut-bds-lab) also contains code to finetune the model.


## Results

### devtest result

| Model | Subtask-1 Amb. Candi. F1 | Subtask-2 MM Coref F1 | Subtask-3 MM DST Slot F1 | Subtask-3 MM DST Intent F1 | Subtask-4 Response Gen. BLEU-4 |
|:----:|:----:|:----:|:----:|:----:|:----:|
| mt-bart-ensemble | 0.68466 | 0.77860 | 0.91816 | 0.97828 | 0.34496 |
| mt-bart-dstcla | 0.67589 | 0.78407 | 0.92013 | 0.97468 |  |
| mt-bart-dstcla-ensemble | 0.67777 | 0.78640 | 0.92055 | 0.97456 |  |
| mt-bart-sys |  |  |  |  | 0.39064 |
| mt-bart-sys-2 |  |  |  |  | 0.3909 |
| mt-bart-sys-ensemble |  |  |  |  | 0.3894 |
| mt-bart-sys-nvattr |  |  |  |  | 0.38995 |

### teststd result
The teststd result is provided in the [teststd-result](https://github.com/scutcyr/dstc11-simmc2.1-iflytek/blob/main/results/teststd-result). One subfolder corresponds to one model.


## Using with Transformers
(1) You should first download the model from huggingface used the scripts:
```bash
cd ~
mkdir pretrained_model
cd pretrained_model
git lfs install
git clone https://huggingface.co/scutcyr/dstc11-simmc2.1-scut-bds-lab
```
(2) Then you should clone our code use the follow scripts:
```bash
cd ~
git clone https://github.com/scutcyr/dstc11-simmc2.1-scut-bds-lab.git
```
(3) Follow the [README](https://github.com/scutcyr/dstc11-simmc2.1-scut-bds-lab#readme) to use the model.


## References

```
@inproceedings{kottur-etal-2021-simmc,
    title = "{SIMMC} 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations",
    author = "Kottur, Satwik  and
      Moon, Seungwhan  and
      Geramifard, Alborz  and
      Damavandi, Babak",
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.401",
    doi = "10.18653/v1/2021.emnlp-main.401",
    pages = "4903--4912",
}

@inproceedings{lee-etal-2022-learning,
    title = "Learning to Embed Multi-Modal Contexts for Situated Conversational Agents",
    author = "Lee, Haeju  and
      Kwon, Oh Joon  and
      Choi, Yunseon  and
      Park, Minho  and
      Han, Ran  and
      Kim, Yoonhyung  and
      Kim, Jinhyeon  and
      Lee, Youngjune  and
      Shin, Haebin  and
      Lee, Kangwook  and
      Kim, Kee-Eung",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2022",
    month = jul,
    year = "2022",
    address = "Seattle, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.findings-naacl.61",
    doi = "10.18653/v1/2022.findings-naacl.61",
    pages = "813--830",
}
```


## Acknowledge
* We would like to express our gratitude to the authors of [Hugging Face's Transformers🤗](https://huggingface.co/) and its open source community for the excellent design on pretrained models usage.
* We would like to express our gratitude to [Meta Research | Facebook AI Research](https://github.com/facebookresearch) for the SIMMC2.1 dataset and the baseline code.
* We would like to express our gratitude to [KAIST-AILab](https://github.com/KAIST-AILab/DSTC10-SIMMC) for the basic research framework on SIMMC2.0.