|
---
|
|
license: apache-2.0
|
|
pipeline_tag: text-generation
|
|
datasets:
|
|
- liuhaotian/LLaVA-Pretrain
|
|
- liuhaotian/LLaVA-Instruct-150K
|
|
---
|
|
# 😈 Imp
|
|
|
|
> A very small man can cast a very large shadow.
|
|
>
|
|
> ——*George R.R. Martin, A Clash of Kings*
|
|
|
|
|
|
\[Technical report (coming soon)\] [[Demo](https://xmbot.net/imp/)\] [[Github](https://github.com/MILVLG/imp)\]
|
|
|
|
## Introduction
|
|
|
|
The Imp project aims to provide a family of a strong multimodal `small` language models (MSLMs). Our `Imp-v1.5-2B-Qwen1.5` is a strong MSLM with only **2B** parameters, which is build upon a small yet powerful SLM [Qwen1.5-1.8B-Chat ](https://huggingface.co/Qwen/Qwen1.5-1.8B-Chat)(1.8B) and a powerful visual encoder [SigLIP ](https://huggingface.co/google/siglip-so400m-patch14-384)(0.4B), and trained on the [LLaVA-v1.5](https://github.com/haotian-liu/LLaVA) training set.
|
|
|
|
As shown in the Table below, `Imp-v1.5-2B-Qwen1.5` significantly outperforms the counterparts of similar model sizes, and even achieves slightly better performance than the strong LLaVA-7B model on various multimodal benchmarks.
|
|
|
|
We release our model weights and provide an example below to run our model . Detailed technical report and corresponding training/evaluation code will be released soon on our [GitHub repo](https://github.com/MILVLG/imp). We will persistently improve our model and release the next versions to further improve model performance :)
|
|
|
|
|
|
## How to use
|
|
|
|
|
|
**Install dependencies**
|
|
```bash
|
|
pip install transformers # latest version is ok, but we recommend v4.31.0
|
|
pip install -q pillow accelerate einops
|
|
```
|
|
|
|
You can use the following code for model inference. The format of text instruction is similar to [LLaVA](https://github.com/haotian-liu/LLaVA). A Colab page to run this example is provided [here](https://colab.research.google.com/drive/1EBYky6xIPjnlPppo2gZaiNK6gEsjXgom?usp=drive_link#scrollTo=2-VpU6QzWCVZ). Note that the example can only be run on GPUs currently.
|
|
|
|
```Python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
from PIL import Image
|
|
|
|
torch.set_default_device("cuda")
|
|
|
|
#Create model
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
"MILVLG/Imp-v1.5-2B-Qwen1.5",
|
|
torch_dtype=torch.float16,
|
|
device_map="auto",
|
|
trust_remote_code=True)
|
|
tokenizer = AutoTokenizer.from_pretrained("MILVLG/Imp-v1.5-2B-Qwen1.5", trust_remote_code=True)
|
|
|
|
#Set inputs
|
|
text = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nWhat are the colors of the bus in the image? ASSISTANT:"
|
|
image = Image.open("images/bus.jpg")
|
|
|
|
input_ids = tokenizer(text, return_tensors='pt').input_ids
|
|
image_tensor = model.image_preprocess(image)
|
|
|
|
#Generate the answer
|
|
output_ids = model.generate(
|
|
input_ids,
|
|
max_new_tokens=100,
|
|
images=image_tensor,
|
|
use_cache=True)[0]
|
|
print(tokenizer.decode(output_ids[input_ids.shape[1]:], skip_special_tokens=True).strip())
|
|
```
|
|
|
|
## Model evaluation
|
|
We conduct evaluation on 9 commonly-used benchmarks, including 5 academic VQA benchmarks and 4 popular MLLM benchmarks, to compare our Imp model with LLaVA (7B) and existing MSLMs of similar model sizes.
|
|
|
|
| Models | Size | VQAv2 | GQA | SQA(IMG) | TextVQA | POPE | MME(P) | MMB |MMBCN |MM-Vet|
|
|
|:--------:|:-----:|:----:|:-------------:|:--------:|:-----:|:----:|:-------:|:-------:|:-------:|:-------:|
|
|
| [LLaVA-v1.5-lora](https://huggingface.co/liuhaotian/llava-v1.5-7b) | 7B |79.10 | 63.00| 68.40 |58.20| 86.40 | 1476.9 | 66.10 |- |30.2|
|
|
| **Imp-v1.5-2B-Qwen1.5** | 3B | **81.18** | **63.54** | **72.78**| **59.84** | **88.87**| **1446.4** | **72.94**| 46.65 |**43.3**|
|
|
|
|
## License
|
|
This project is licensed under the Apache License 2.0 - see the [LICENSE](https://www.apache.org/licenses/LICENSE-2.0) file for details.
|
|
|
|
## About us
|
|
This project is maintained by the [MILVLG](https://github.com/MILVLG)@Hangzhou Dianzi University (HDU) led by Prof. Zhou Yu and Jun Yu, and is mainly developed by Zhenwei Shao and Xuecheng Ouyang. We hope our model may serve as a strong baseline to inspire future research on MSLM, as well as its derivative applications on mobile devices and robots.
|
|
|
|
## Citation
|
|
|
|
If you use our model or refer our work in your studies, please cite:
|
|
|
|
```bibtex
|
|
@misc{imp2024,
|
|
author = {Shao, Zhenwei and Ouyang, Xuecheng and Yu, Zhou and Yu, Jun},
|
|
title = {Imp: An Emprical Study of Multimodal Small Language Models},
|
|
year = {2024},
|
|
url = {https://huggingface.co/MILVLG/imp-v1-3b}
|
|
}
|
|
``` |
