Update README.md (#1)
Browse files- Update README.md (352bc2ae3fcd302d419c0bfc97df3c16276a0039)
README.md
CHANGED
@@ -3,6 +3,32 @@ tags:
|
|
3 |
- image-classification
|
4 |
- timm
|
5 |
library_name: timm
|
6 |
-
license:
|
|
|
|
|
7 |
---
|
8 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3 |
- image-classification
|
4 |
- timm
|
5 |
library_name: timm
|
6 |
+
license: other
|
7 |
+
license_name: apple-ascl
|
8 |
+
license_link: LICENSE
|
9 |
---
|
10 |
+
|
11 |
+
# MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
|
12 |
+
|
13 |
+
MobileCLIP was introduced in [MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
|
14 |
+
](https://arxiv.org/pdf/2311.17049.pdf) (CVPR 2024), by Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel.
|
15 |
+
|
16 |
+
This repository contains the **MobileCLIP-S1** checkpoint for timm.
|
17 |
+
|
18 |
+
![MobileCLIP Performance Figure](fig_accuracy_latency.png)
|
19 |
+
|
20 |
+
### Highlights
|
21 |
+
|
22 |
+
* Our smallest variant `MobileCLIP-S0` obtains similar zero-shot performance as [OpenAI](https://arxiv.org/abs/2103.00020)'s ViT-B/16 model while being 4.8x faster and 2.8x smaller.
|
23 |
+
* `MobileCLIP-S2` obtains better avg zero-shot performance than [SigLIP](https://arxiv.org/abs/2303.15343)'s ViT-B/16 model while being 2.3x faster and 2.1x smaller, and trained with 3x less seen samples.
|
24 |
+
* `MobileCLIP-B`(LT) attains zero-shot ImageNet performance of **77.2%** which is significantly better than recent works like [DFN](https://arxiv.org/abs/2309.17425) and [SigLIP](https://arxiv.org/abs/2303.15343) with similar architectures or even [OpenAI's ViT-L/14@336](https://arxiv.org/abs/2103.00020).
|
25 |
+
|
26 |
+
## Checkpoints
|
27 |
+
|
28 |
+
| Model | # Seen <BR>Samples (B) | # Params (M) <BR> (img + txt) | Latency (ms) <BR> (img + txt) | IN-1k Zero-Shot <BR> Top-1 Acc. (%) | Avg. Perf. (%) <BR> on 38 datasets |
|
29 |
+
|:----------------------------------------------------------|:----------------------:|:-----------------------------:|:-----------------------------:|:-----------------------------------:|:----------------------------------:|
|
30 |
+
| [MobileCLIP-S0](https://hf.co/pcuenq/MobileCLIP-S0) | 13 | 11.4 + 42.4 | 1.5 + 1.6 | 67.8 | 58.1 |
|
31 |
+
| [MobileCLIP-S1](https://hf.co/pcuenq/MobileCLIP-S1) | 13 | 21.5 + 63.4 | 2.5 + 3.3 | 72.6 | 61.3 |
|
32 |
+
| [MobileCLIP-S2](https://hf.co/pcuenq/MobileCLIP-S2) | 13 | 35.7 + 63.4 | 3.6 + 3.3 | 74.4 | 63.7 |
|
33 |
+
| [MobileCLIP-B](https://hf.co/pcuenq/MobileCLIP-B) | 13 | 86.3 + 63.4 | 10.4 + 3.3 | 76.8 | 65.2 |
|
34 |
+
| [MobileCLIP-B (LT)](https://hf.co/pcuenq/MobileCLIP-B-LT) | 36 | 86.3 + 63.4 | 10.4 + 3.3 | 77.2 | 65.8 |
|