intfloat

/

mmE5-mllama-11b-instruct

like 12

Hello!

Thanks @Samoed for setting up this work.

Pull Request overview

• Add modules.json to inform Sentence Transformers what modules to use
  • It points to the new MultiModalTransformer module, but also normal Pooling and Normalize modules
• Configure the Pooling layer to use last token pooling
• Add the Sentence Transformers usage to the README

Details

I've updated the usage compared to @Samoed 's suggestion: instead of always specifying both a type and a content, users can now specify a dictionary with text, image, or both. Still, as @intfloat mentioned, you cannot embed both image-text and pure text at the same time, but I think that's not a big problem. As before, just using "A cat and a dog" instead of {"text": "A cat and a dog"} also works, but in the README I want to inform users of {"text": "A cat and a dog"} so they (hopefully) understand how it can be expanded to also include an image in that embedding.

I want to share that I was unable to get 100% correspondance between Transformers and Sentence Transformers, annoyingly enough. The problem is somewhere in the processing or model forward, not the pooling or normalization, and the difference is quite small. For example, the mean and std of the model last hidden states are identical to at least 4 significant digits. The cosine similarity of the Image + Text -> Text README examples is [0.3965, 0.3105] for transformers and [0.3967, 0.3090] for Sentence Transformers. The Text -> Image examples has [0.4219, 0.3887] for transformers and [0.4250, 0.3896] for Sentence Transformers.

• Tom Aarsen