File size: 4,138 Bytes
fd198a3 0237df9 fd198a3 03f5aa0 a134646 1f1bb49 03f5aa0 a134646 cf5e052 a134646 090ee1c a134646 4afa441 a134646 4afa441 a134646 e6f6455 a134646 d6b23f9 a134646 7b03926 a134646 cf5e052 a134646 dc983b7 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 |
---
license: apache-2.0
pipeline_tag: visual-question-answering
---
# 😈Imp
The Imp project aims to provide a family of a strong multimodal `small` language models (MSLMs). Our `imp-v0-3b` is a strong MSLM with only **3B** parameters, which is build upon a small yet powerful SLM [Phi-2 ](https://huggingface.co/microsoft/phi-2)(2.7B) and a powerful visual encoder [SigLIP ](https://huggingface.co/google/siglip-so400m-patch14-384)(0.4B), and trained on the [LLaVA-v1.5](https://github.com/haotian-liu/LLaVA) training set.
As shown in the Table below, `imp-v0-3b` significantly outperforms the counterparts of similar model sizes, and even achieves slightly better performance than the strong LLaVA-7B model on various multimodal benchmarks.
We release our model weights and provide an example below to run our model . Detailed technical report and corresponding training/evaluation code will be released soon on our [GitHub repo](https://github.com/MILVLG/imp). We will persistently improve our model and release the next versions to further improve model performance :)
## How to use
You can use the following code for model inference. We minimize the required dependency libraries that only the `transformers` and `torch` packages are used. The format of text instructions is similar to [LLaVA](https://github.com/haotian-liu/LLaVA).
```Python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from PIL import Image
torch.set_default_device("cuda")
#Create model
model = AutoModelForCausalLM.from_pretrained(
"MILVLG/imp-v0-3b",
torch_dtype=torch.float16,
device_map="auto",
trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("MILVLG/imp-v0-3b", trust_remote_code=True)
#Set inputs
text = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nWhat are the colors of the bus in the image? ASSISTANT:"
image = Image.open("images/bus.jpg")
input_ids = tokenizer(text, return_tensors='pt').input_ids
image_tensor = model.image_preprocess(image)
#Generate the answer
output_ids = model.generate(
input_ids,
max_new_tokens=100,
images=image_tensor,
use_cache=True)[0]
print(tokenizer.decode(output_ids[input_ids.shape[1]:], skip_special_tokens=True).strip())
```
## Model evaluation
We conduct evaluation on 9 commonly-used benchmarks, including 5 academic VQA benchmarks and 4 popular MLLM benchmarks, to compare our Imp model with LLaVA (7B) and existing MSLMs with similar sizes.
| Models | Size | VQAv2 | GQA |VisWiz | SQA (IMG) | TextVQA | POPE | MME | MMB |MM-Vet|
|:--------:|:-----:|:----:|:----:|:-------------:|:--------:|:-----:|:----:|:-------:|:-------:|:-------:|
| [LLaVA-v1.5-lora](https://huggingface.co/liuhaotian/llava-v1.5-7b) | 7B |79.10 | **63.00** |47.80 | 68.40 |58.20| 86.40 | **1476.9** | 66.10 |30.2|
| [TinyGPT-V](https://huggingface.co/Tyrannosaurus/TinyGPT-V) | 3B | - | 33.60 | 24.80 | - | - | -| - | - |-|
| [LLaVA-Phi](https://github.com/zhuyiche/llava-phi) | 3B | 71.40 | - | 35.90 | 68.40 | 48.60 | 85.00 | 1335.1 | 59.80 |28.9|
| [MobileVLM](https://huggingface.co/mtgv/MobileVLM-3B) | 3B | - | 59.00 | - | 61.00 | 47.50 | 84.90 | 1288.9 | 59.60 |-|
| [MC-LLaVA-3b](https://huggingface.co/visheratin/MC-LLaVA-3b) | 3B | 64.24 | 49.6 | 24.88 | - | 38.59 | 80.59 | - | - |-|
| **Imp-v0 (ours)** | 3B | **79.45** | 58.55 | **50.09** |**69.96**| **59.38** | **88.02**| 1434 | **66.49** |**33.1**|
## License
This project is licensed under the Apache License 2.0 - see the [LICENSE](https://www.apache.org/licenses/LICENSE-2.0) file for details.
## About us
This project is maintained by the [MILVLG](https://github.com/MILVLG)@Hangzhou Dianzi University (HDU) led by Prof. Zhou Yu and Jun Yu, and is mainly developed by Zhenwei Shao and Xuecheng Ouyang. We hope our model may serve as a strong baseline to inspire future research on MSLM, as well as its derivative applications on mobile devices and robots. |