File size: 3,556 Bytes
d6340a9 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 |
---
license: apache-2.0
datasets:
- HuggingFaceM4/MMBench
language:
- en
base_model:
- openai/clip-vit-large-patch14-336
- Qwen/Qwen2.5-7B-Instruct
pipeline_tag: image-text-to-text
tags:
- vision-language
- multimodal
---
## POINTS-Qwen-2-5-7B-Chat
### Introduction
We are excited to announce the first version of POINTS, which integrates recent advancement in vision-language model and new techniques proposed by researchers from WeChat AI.
<p align="center">
🏠 <a href="https://github.com/WePOINTS/WePOINTS">Github</a>   |    📑 <a href="https://arxiv.org/abs/2409.04828">Paper</a>    </a>
</p>
### What's new in POINTS?
**Key Innovations**
1. **Strong Baseline**: We integrate the most recent advancement in vision-language model, i.e., CapFusion, Dual Vision Encoder, and
Dynamic High Resolution, into POINTS.
2. **Pre-training Dataset Filtering**: We propose to filter the pre-training dataset using perplexity as a metric. Utilizing this filtering strategy, we can significantly reduce the size of the pre-training dataset and improve the performance of the model.
3. **Model Soup**: We propose to apply model soup to models, fine-tuned with different visual instruction tuning datasets, which can further significantly improve the performance of the model.
<p align="center">
<img src="https://github.com/user-attachments/assets/6af35008-f501-400a-a870-b66a9bf2baab" width="60%"/>
<p>
### How to use POINTS?
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import CLIPImageProcessor
from PIL import Image
import torch
import requests
from io import BytesIO
image_url = 'https://github.com/user-attachments/assets/83258e94-5d61-48ef-a87f-80dd9d895524'
response = requests.get(image_url)
image_data = BytesIO(response.content)
pil_image = Image.open(image_data)
prompt = 'please describe the image in detail'
model_path = 'WePOINTS/POINTS-Qwen-2-5-7B-Chat'
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
model_path, trust_remote_code=True, device_map='cuda').to(torch.bfloat16)
image_processor = CLIPImageProcessor.from_pretrained(model_path)
generation_config = {
'max_new_tokens': 1024,
'temperature': 0.0,
'top_p': 0.0,
'num_beams': 1,
}
res = model.chat(
pil_image,
prompt,
tokenizer,
image_processor,
True,
generation_config
)
print(res)
```
### Evaluation
| Benchmark | InternVL2-8B | LLaVA-OneVision | POINTS |
| :-------: | :----------: | :-------------: | :----: |
| MMBench-dev-en | - | 80.8 | 83.2 |
| MathVista | 58.3 | 62.3 | 63.1 |
| HallucinationBench | 45.0 | 31.6 | 46.0 |
| OCRBench | 79.4 | 62.2 | 72.0 |
| AI2D | 83.6 | 82.4 | 80.9 |
| MMVet | 54.3 | 51.9 | 52.3 |
| MMStar | 61.5 | 61.9 | 61.0 |
| MMMU | 51.2 | 47.9 | 49.4 |
| ScienceQA | 97.1 | 95.4 | - |
| MME | 2215.1 | 1993.6 | 2195.2 |
| RealWorldQA | 64.2 | 69.9 | 67.3 |
| LLaVA-Wild | 73.3 | 81.0 | 71.1 |
### Citation
If you find our work helpful, feel free to cite us:
```
@article{liu2024points,
title={POINTS: Improving Your Vision-language Model with Affordable Strategies},
author={Liu, Yuan and Zhao, Zhongyin and Zhuang, Ziyuan and Tian, Le and Zhou, Xiao and Zhou, Jie},
journal={arXiv preprint arXiv:2409.04828},
year={2024}
}
@article{liu2024rethinking,
title={Rethinking Overlooked Aspects in Vision-Language Models},
author={Liu, Yuan and Tian, Le and Zhou, Xiao and Zhou, Jie},
journal={arXiv preprint arXiv:2405.11850},
year={2024}
}
``` |