---
language:
- en
---
# VeCLIP: Improving CLIP Training via Visual-enriched Captions
* A novel CLIP training scheme that achieves the SoTA performance on zero-shot ImageNet classification and COCO image text retreival using limited visual-enriched captions.* [[Paper](https://arxiv.org/abs/2310.07699)]
[Zhengfeng Lai*](https://zjujefflai.github.io/), [Haotian Zhang*](https://haotian-zhang.github.io/) , [Bowen Zhang](https://zbwglory.github.io/), Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, [Zhe Gan](https://zhegan27.github.io/), Jiulong Shan, [Chen-Nee Chuah](https://www.ece.ucdavis.edu/~chuah/rubinet/people/chuah/bio.html), Yinfei Yang, Meng Cao [*: equal contribution]
<p align="center">
<img src="veclip_diagram.jpg" width="100%"></a> <br>
Diagram of VeCap.
</p>
## Release
- [03/06/2024] 🔥 We released the VeCLIP & VeCap-DFN [checkpoints](#checkpoints).
## Contents
- [Install](#install)
- [Getting Started](#getting-started)
- [Checkpoints](#checkpoints)
## Install
1. Clone this repository
```Shell
git clone https://github.com/apple/ml-veclip
cd ml-veclip
```
2. Create an environment and install related packages
```Shell
conda create -n veclip python=3.9 -y
conda activate veclip
pip install -r requirements.txt
```
## Getting Started
See the [example notebook](https://github.com/apple/ml-veclip/blob/main/load_veclip.ipynb) for details on how to simply load the different checkpoints using HuggingFace transformers.
## Checkpoints
We release the checkpoints for **VeCLIP**, which are trained from scratch on visual-enriched captions VeCap 3M/12M/100M/200M, as reported in the paper. The models are evaluated on COCO/Flickr30k image-text retrieval and ImageNet/ImageNetv2 classification in a zero-shot fashion. Use `wget` or `curl` to download the below checkpoints.
<table>
<thead>
<tr>
<th rowspan="2">Data</th>
<th rowspan="2">Model</th>
<th rowspan="2">Resolution</th>
<th colspan="2">COCO (R@1)</th>
<th colspan="2">Flickr30k (R@1)</th>
<th rowspan="2">ImageNet</th>
<th rowspan="2">ImageNetv2</th>
</tr>
<tr>
<th>I2T</th>
<th>T2I</th>
<th>I2T</th>
<th>T2I</th>
</tr>
</thead>
<tbody>
<tr>
<td rowspan="2">VeCap 3M</td>
<td>CLIP-B/16</td>
<td>224x224</td>
<td>5.46</td>
<td>3.28</td>
<td>12.20</td>
<td>6.36</td>
<td>5.46</td>
<td>7.09</td>
</tr>
<tr>
<td><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/veclip_b16_3m.zip">VeCLIP-B/16</a></td>
<td>224x224</td>
<td>22.30</td>
<td>13.01</td>
<td>40.60</td>
<td>27.58</td>
<td>15.98</td>
<td>13.51</td>
</tr>
<tr>
<td rowspan="2">VeCap 12M</td>
<td>CLIP-B/16</td>
<td>224x224</td>
<td>24.52</td>
<td>14.28</td>
<td>44.70</td>
<td>290.6</td>
<td>31.60</td>
<td>27.03</td>
</tr>
<tr>
<td><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/veclip_b16_12m.zip">VeCLIP-B/16</a></td>
<td>224x224</td>
<td>47.78</td>
<td>31.62</td>
<td>73.90</td>
<td>55.68</td>
<td>38.11</td>
<td>32.53</td>
</tr>
<tr>
<td rowspan="2">VeCap 100M</td>
<td>CLIP-B/16</td>
<td>224x224</td>
<td>47.24</td>
<td>30.61</td>
<td>74.40</td>
<td>57.16</td>
<td>58.64</td>
<td>50.96</td>
</tr>
<tr>
<td><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/veclip_b16_100m.zip">VeCLIP-B/16</a></td>
<td>224x224</td>
<td>64.82</td>
<td>46.12</td>
<td>89.30</td>
<td>73.10</td>
<td>60.77</td>
<td>54.17</td>
</tr>
<tr>
<td rowspan="2">VeCap 200M</td>
<td>CLIP-B/16</td>
<td>224x224</td>
<td>52.20</td>
<td>34.97</td>
<td>80.90</td>
<td>63.26</td>
<td>63.72</td>
<td>56.84</td>
</tr>
<tr>
<td><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/veclip_b16_200m.zip">VeCLIP-B/16</a></td>
<td>224x224</td>
<td>67.20</td>
<td>48.40</td>
<td>91.10</td>
<td>76.32</td>
<td>64.64</td>
<td>57.67</td>
</tr>
</tbody>
</table>
We further found our VeCap can also be complementary to other well-established filtering methods, e.g., [Data Filtering Network (DFN)](https://arxiv.org/abs/2309.17425). We also provide thosse checkpoints (referred to as **VeCap-DFN**) and report their performance below.
<table>
<thead>
<tr>
<th rowspan="2">Backbone</th>
<th rowspan="2">Resolution</th>
<th rowspan="2">Data</th>
<th colspan="2">COCO (R@1)</th>
<th colspan="2">Flickr30k (R@1)</th>
<th rowspan="2">ImageNet</th>
<th rowspan="2">ImageNetV2</th>
</tr>
<tr>
<th>I2T</th>
<th>T2I</th>
<th>I2T</th>
<th>T2I</th>
</tr>
</thead>
<tbody>
<tr>
<td rowspan="3"><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/vecapdfn_clip_b16.zip">VeCap-DFN-B/16</a></td>
<td rowspan="3">224x224</td>
<td>DFN </td>
<td>62.96</td>
<td>43.20</td>
<td>87.10</td>
<td>70.44</td>
<td>76.15</td>
<td>68.19</td>
</tr>
<tr>
<td>VeCap 300M</td>
<td>64.74</td>
<td>44.58</td>
<td>90.10</td>
<td>73.14</td>
<td>46.43</td>
<td>41.15</td>
</tr>
<tr>
<td>DFN + VeCap 300M</td>
<td>66.28</td>
<td>45.12</td>
<td>88.80</td>
<td>73.56</td>
<td>76.19</td>
<td>69.58</td>
</tr>
<tr>
<td rowspan="1"><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/vecapdfn_clip_l14.zip">VeCap-DFN-L/14</a></td>
<td rowspan="1">224x224</td>
<td>DFN + VeCap 300M</td>
<td>71.06</td>
<td>51.13</td>
<td>93.10</td>
<td>80.96</td>
<td>81.95</td>
<td>75.48</td>
</tr>
<tr>
<td rowspan="2"><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/vecapdfn_clip_h14_336.zip">VeCap-DFN-H/14</a></td>
<td rowspan="1">336x336</td>
<td>DFN + VeCap 300M</td>
<td>72.78</td>
<td>52.33</td>
<td>93.60</td>
<td>82.64</td>
<td>83.07</td>
<td>76.37</td>
</tr>
</tbody>
</table>
## Citation
If you find VeCLIP useful, please cite using this BibTeX:
```bibtex
@article{lai2023scarcity,
title={From scarcity to efficiency: Improving clip training via visual-enriched captions},
author={Lai, Zhengfeng and Zhang, Haotian and Zhang, Bowen and Wu, Wentao and Bai, Haoping and Timofeev, Aleksei and Du, Xianzhi and Gan, Zhe and Shan, Jiulong and Chuah, Chen-Nee and Yang, Yinfei and others},
journal={arXiv preprint arXiv:2310.07699},
year={2023}
}
@article{fang2023data,
title={Data filtering networks},
author={Fang, Alex and Jose, Albin Madappally and Jain, Amit and Schmidt, Ludwig and Toshev, Alexander and Shankar, Vaishaal},
journal={arXiv preprint arXiv:2309.17425},
year={2023}
}
```
## Acknowledgement
- [axlearn](https://github.com/apple/axlearn): the codebase we use to train the models.
- [huggingface transformers](https://huggingface.co/docs/transformers/en/index): Transformers provides APIs to load our trained models.