update

README.md

@@ -1,3 +1,67 @@
 ---
 license: apache-2.0
+language:
+- en
+pipeline_tag: Multimodal Small Language Model, Phi-2, VQA
 ---
+# :smiling_imp: IMP
+
+The :smiling_imp: IMP project aims to provide a family of  a strong multimodal `small` language models (MSLMs). Our `IMP-v0-3B` model is a strong MSLM with only **3B** parameters, which is build upon a small yet powerful SLM [Phi-2 ](https://huggingface.co/microsoft/phi-2)(2.7B) and a powerful visual encoder [SigLIP ](https://huggingface.co/google/siglip-so400m-patch14-384)(0.4B), and trained on the [LLaVA-v1.5](https://github.com/haotian-liu/LLaVA) training set.  
+
+As shown in the Table below, `IMP-v0-3B` significantly outperforms the counterparts of similar model sizes, and even achieves slightly better performance than the strong LLaVA-7B model on various multimodal benchmarks. 
+
+We release our model weights and provide an example below to run our model . Detailed technical report and corresponding training/evaluation code will be released soon on our [GitHub repo](https://github.com/MILVLG/imp). We will persistently improve our model and release the next versions to further improve model performance :) 
+
+## How to use
+
+You can use the following code for model inference. We minimize the required dependency libraries that only the `transformers` and `torch` packages are used. The format of text instructions is similar to [LLaVA](https://github.com/haotian-liu/LLaVA).
+
+```Python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from PIL import Image
+
+torch.set_default_device("cuda")
+
+#Create model
+model = AutoModelForCausalLM.from_pretrained(
+    "milvlg/imp-v0", 
+    torch_dtype=torch.float16, 
+    device_map="auto",
+    trust_remote_code=True)
+tokenizer = AutoTokenizer.from_pretrained("milvlg/imp-v0", trust_remote_code=True)
+
+#Set inputs
+text = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nWhat are the colors of the bus in the image? ASSISTANT:"
+image = Image.open("images/bus.jpg")
+
+input_ids = tokenizer(text, return_tensors='pt').input_ids
+image_tensor = model.process_images([image])
+
+#Generate the answer
+output_ids = model.generate(
+    input_ids,
+    max_new_tokens=100,
+    images=image_tensor,
+    use_cache=True)[0]
+print(tokenizer.decode(output_ids[input_ids.shape[1]:], skip_special_tokens=True).strip())
+```
+
+## Model evaluation
+We perform evaluation on 8 commonly-used benchmarks to validate the effectiveness of our model, including 5 academic VQA benchmarks and 3 recent MLLM benchmarks.
+
+| Models | Size | VQAv2 | GQA |VisWiz  | SQA (IMG) | TextVQA | POPE |  MME | MMB  |MM-Vet|
+|:--------:|:-----:|:----:|:----:|:-------------:|:--------:|:-----:|:----:|:-------:|:-------:|:-------:|
+| [LLaVA-v1.5-lora](https://huggingface.co/liuhaotian/llava-v1.5-7b) | 7B |79.10 | **63.00** |47.80 |  68.40 |58.20| 86.40 | **1476.9** | 66.10  |30.2|
+| [TinyGPT-V](https://huggingface.co/Tyrannosaurus/TinyGPT-V) | 3B | - | 33.60  | 24.80  |    -   |    -  | -| - | -  |-|
+| [LLaVA-Phi](https://arxiv.org/pdf/2401.02330.pdf) | 3B | 71.40  | - | 35.90 |    68.40   |    48.60  | 85.00 | 1335.1 | 59.80 |28.9|
+| [MobileVLM](https://huggingface.co/mtgv/MobileVLM-3B) | 3B | - | 59.00  | - |    61.00   |    47.50   | 84.90 | 1288.9 | 59.60  |-|
+| [MC-LLaVA-3b](https://huggingface.co/visheratin/MC-LLaVA-3b) | 3B | 64.24 | 49.6  | 24.88 |    -   |    38.59   | 80.59 | - | -  |-|
+| **IMP-v0 (ours)** | 3B | **79.45**  | 58.55 | **50.09** |**69.96**| **59.38** | **88.02**| 1434 | **66.49**  |**33.1**|
+
+
+## License
+This project is licensed under the Apache License 2.0 - see the [LICENSE](https://www.apache.org/licenses/LICENSE-2.0) file for details.
+
+## About us
+Project :smiling_imp: IMP is maintained by the [MILVLG](https://github.com/MILVLG) group led by Prof. Zhou Yu and Jun Yu, and mainly developed by Zhenwei Shao and Xuecheng Ouyang. We hope our model may server as a strong baseline to inspire future research on MSLMs and derivative applications on mobile devices and robotics. 

md.py

@@ -1,25 +0,0 @@
-import torch
-from transformers import AutoModelForCausalLM, AutoTokenizer
-from PIL import Image
-
-torch.set_default_device("cuda")
-
-model = AutoModelForCausalLM.from_pretrained(
-    "../Imp-v0-3b", 
-    torch_dtype=torch.float16, 
-    device_map="auto",
-    trust_remote_code=True)
-tokenizer = AutoTokenizer.from_pretrained("../Imp-v0-3b", trust_remote_code=True)
-
-text = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nWhat are the colors of the bus in the image? ASSISTANT:"
-image = Image.open("images/bus.jpg")
-
-input_ids = tokenizer(text, return_tensors='pt').input_ids
-image_tensor = model.image_preprocess(image)
-
-output_ids = model.generate(
-    input_ids,
-    max_new_tokens=100,
-    images=image_tensor,
-    use_cache=True)[0]
-print(tokenizer.decode(output_ids[input_ids.shape[1]:], skip_special_tokens=True).strip())