JustinLin610
commited on
Commit
•
60d03d8
1
Parent(s):
31aa84f
Update README.md
Browse files
README.md
CHANGED
@@ -5,7 +5,17 @@ license: apache-2.0
|
|
5 |
# OFA-tiny
|
6 |
This is the **tiny** version of OFA pretrained model. OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) to a simple sequence-to-sequence learning framework.
|
7 |
|
8 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
9 |
|
10 |
```
|
11 |
>>> from PIL import Image
|
|
|
5 |
# OFA-tiny
|
6 |
This is the **tiny** version of OFA pretrained model. OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) to a simple sequence-to-sequence learning framework.
|
7 |
|
8 |
+
The directory includes 4 files, namely `config.json` which consists of model configuration, `vocab.json` and `merge.txt` for our OFA tokenizer, and lastly `pytorch_model.bin` which consists of model weights. There is no need to worry about the mismatch between Fairseq and transformers, since we have addressed the issue yet.
|
9 |
+
|
10 |
+
To use it in transformers, please refer to https://github.com/OFA-Sys/OFA/tree/feature/add_transformers. Install the transformers and download the models as shown below.
|
11 |
+
|
12 |
+
```
|
13 |
+
git clone --single-branch --branch feature/add_transformers https://github.com/OFA-Sys/OFA.git
|
14 |
+
pip install OFA/transformers/
|
15 |
+
it clone https://huggingface.co/OFA-Sys/OFA-tiny
|
16 |
+
```
|
17 |
+
|
18 |
+
After, refer the path to OFA-tiny to `ckpt_dir`, and prepare an image for the testing example below. Also, ensure that you have pillow and torchvision in your environment.
|
19 |
|
20 |
```
|
21 |
>>> from PIL import Image
|