|
--- |
|
license: apache-2.0 |
|
tags: |
|
- video LLM |
|
--- |
|
|
|
|
|
# Tarsier Model Card |
|
## Model details |
|
**Model type:** |
|
Tarsier-34b is an open-source large-scale video-language models, which is designed to generate high-quality video descriptions, together with good capability of general video understanding (SOTA results on 6 open benchmarks). |
|
|
|
**Model date:** |
|
Tarsier-34b was trained in June 2024. |
|
|
|
**Paper or resources for more information:** |
|
- github repo: https://github.com/bytedance/tarsier |
|
- paper link: https://arxiv.org/abs/2407.00634 |
|
|
|
## License |
|
NousResearch/Nous-Hermes-2-Yi-34B license. |
|
|
|
**Where to send questions or comments about the model:** |
|
https://github.com/bytedance/tarsier/issues |
|
|
|
## Intended use |
|
**Primary intended uses:** |
|
The primary use of Tarsier is research on large multimodal models, especially video description. |
|
|
|
**Primary intended users:** |
|
The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence. |
|
|
|
## Training dataset |
|
Tarsier tasks a two-stage training strategy. |
|
- Stage-1: Multi-task Pre-training on 13M data |
|
- Stage-2: Multi-grained Instruction Tuning on 500K data |
|
|
|
In both stages, we freeze ViT and train all the parameters of projection layer and LLM. |
|
|
|
## Evaluation dataset |
|
- A challenging video desription dataset: [DREAM-1K](https://huggingface.co/datasets/omni-research/DREAM-1K) |
|
- Multi-choice VQA: [MVBench](https://huggingface.co/datasets/OpenGVLab/MVBench), [NeXT-QA](https://github.com/doc-doc/NExT-QA) and [Egoschema](https://drive.google.com/drive/folders/1SS0VVz8rML1e5gWq7D7VtP1oxE2UtmhQ) |
|
- Open-ended VQA: [MSVD-QA](https://opendatalab.com/OpenDataLab/MSVD), [MSR-VTT-QA](https://opendatalab.com/OpenDataLab/MSR-VTT), [ActivityNet-QA](https://github.com/MILVLG/activitynet-qa) and [TGIF-QA](https://opendatalab.com/OpenDataLab/TGIF-QA) |
|
- Video Caption: [MSVD-Caption](https://opendatalab.com/OpenDataLab/MSVD), [MSRVTT-Caption](https://opendatalab.com/OpenDataLab/MSR-VTT), [VATEX](https://eric-xw.github.io/vatex-website/about.html) |
|
|
|
## How to Use |
|
see https://github.com/bytedance/tarsier?tab=readme-ov-file#usage |
|
|
