Update README.md
Browse files
README.md
CHANGED
@@ -11,11 +11,14 @@ language:
|
|
11 |
- pl
|
12 |
- tr
|
13 |
- fr
|
|
|
14 |
license: mit
|
|
|
15 |
---
|
|
|
16 |
## mmE5-mllama-11b-instruct
|
17 |
|
18 |
-
[mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data](https://arxiv.org/abs/2502.08468.pdf). Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, Zhicheng Dou, arXiv
|
19 |
|
20 |
This model is trained based on [Llama-3.2-11B-Vision](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision).
|
21 |
|
@@ -41,7 +44,7 @@ pip install -r requirements.txt
|
|
41 |
|
42 |
Then you can enter the directory to run the following command.
|
43 |
```python
|
44 |
-
from transformers import MllamaForConditionalGeneration, AutoProcessor
|
45 |
import torch
|
46 |
from PIL import Image
|
47 |
|
@@ -61,17 +64,9 @@ model_name = "intfloat/mmE5-mllama-11b-instruct"
|
|
61 |
|
62 |
# Load Processor and Model
|
63 |
processor = AutoProcessor.from_pretrained(model_name)
|
64 |
-
processor.tokenizer.padding_side = "right"
|
65 |
-
|
66 |
-
config = AutoConfig.from_pretrained(model_name)
|
67 |
-
if hasattr(config, 'use_cache'):
|
68 |
-
config.use_cache = False
|
69 |
-
config.padding_side = "right"
|
70 |
model = MllamaForConditionalGeneration.from_pretrained(
|
71 |
-
model_name,
|
72 |
-
torch_dtype=torch.bfloat16
|
73 |
).to("cuda")
|
74 |
-
model.padding_side = "right"
|
75 |
model.eval()
|
76 |
|
77 |
# Image + Text -> Text
|
@@ -118,4 +113,4 @@ print(string, '=', compute_similarity(qry_output, tgt_output))
|
|
118 |
journal={arXiv preprint arXiv:2502.08468},
|
119 |
year={2025}
|
120 |
}
|
121 |
-
```
|
|
|
11 |
- pl
|
12 |
- tr
|
13 |
- fr
|
14 |
+
library_name: transformers
|
15 |
license: mit
|
16 |
+
pipeline_tag: image-feature-extraction
|
17 |
---
|
18 |
+
|
19 |
## mmE5-mllama-11b-instruct
|
20 |
|
21 |
+
[mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data](https://arxiv.org/abs/2502.08468.pdf). Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, Zhicheng Dou, arXiv 2025
|
22 |
|
23 |
This model is trained based on [Llama-3.2-11B-Vision](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision).
|
24 |
|
|
|
44 |
|
45 |
Then you can enter the directory to run the following command.
|
46 |
```python
|
47 |
+
from transformers import MllamaForConditionalGeneration, AutoProcessor
|
48 |
import torch
|
49 |
from PIL import Image
|
50 |
|
|
|
64 |
|
65 |
# Load Processor and Model
|
66 |
processor = AutoProcessor.from_pretrained(model_name)
|
|
|
|
|
|
|
|
|
|
|
|
|
67 |
model = MllamaForConditionalGeneration.from_pretrained(
|
68 |
+
model_name, torch_dtype=torch.bfloat16
|
|
|
69 |
).to("cuda")
|
|
|
70 |
model.eval()
|
71 |
|
72 |
# Image + Text -> Text
|
|
|
113 |
journal={arXiv preprint arXiv:2502.08468},
|
114 |
year={2025}
|
115 |
}
|
116 |
+
```
|