|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
- de |
|
|
- es |
|
|
- fr |
|
|
- it |
|
|
- ja |
|
|
- ko |
|
|
- pl |
|
|
- ru |
|
|
- tr |
|
|
- zh |
|
|
- ar |
|
|
--- |
|
|
<h1 align="center">UForm</h1> |
|
|
<h3 align="center"> |
|
|
Multi-Modal Inference Library<br/> |
|
|
For Semantic Search Applications<br/> |
|
|
</h3> |
|
|
|
|
|
--- |
|
|
|
|
|
UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space! |
|
|
|
|
|
This is the repository of [English](https://huggingface.co/unum-cloud/uform-vl-english/tree/main) and [multilingual](https://huggingface.co/unum-cloud/uform-vl-multilingual) UForm models converted to CoreML MLProgram format. |
|
|
Currently, only __unimodal__ parts of models are converted. |
|
|
|
|
|
## Description |
|
|
|
|
|
Each model is separated into two parts: `image-encoder` and `text-encoder`: |
|
|
|
|
|
* English image-encoder: [english.image-encoder.mlpackage](https://huggingface.co/unum-cloud/uform-coreml/blob/main/english.image-encoder.mlpackage.zip) |
|
|
* English text-encoder: [english.text-encoder.mlpackage](https://huggingface.co/unum-cloud/uform-coreml/blob/main/english.text-encoder.mlpackage.zip) |
|
|
* Multilingual image-encoder: [multilingual.image-encoder.mlpackage](https://huggingface.co/unum-cloud/uform-coreml/blob/main/multilingual.image-encoder.mlpackage.zip) |
|
|
* Multilingual text-encoder: [multilingual.text-encoder.mlpackage](https://huggingface.co/unum-cloud/uform-coreml/blob/main/multilingual.text-encoder.mlpackage.zip) |
|
|
* Multilingual-v2 image-encoder: [multilingual-v2.image-encoder.mlpackage](https://huggingface.co/unum-cloud/uform-coreml/blob/main/multilingual-v2.image-encoder.mlpackage.zip) |
|
|
* Multilingual-v2 text-encoder: [multilingual-v2.text-encoder.mlpackage](https://huggingface.co/unum-cloud/uform-coreml/blob/main/multilingual.text-encoder.mlpackage.zip) |
|
|
* Onnx Multilingual image-encoder: [multilingual.image-encoder.onnx](https://huggingface.co/unum-cloud/uform-coreml/blob/main/multilingual.image-encoder.onnx) |
|
|
* Onnx Multilingual text-encoder: [multilingual.text-encoder.onnx](https://huggingface.co/unum-cloud/uform-coreml/blob/main/multilingual.text-encoder.onnx) |
|
|
|
|
|
Each checkpoint is a zip archive with an MLProgram of the corresponding encoder. |
|
|
|
|
|
Text encoders have the following input fields: |
|
|
|
|
|
* `input_ids`: int32 |
|
|
* `attention_mask`: int32 |
|
|
|
|
|
and support flexible batch size. |
|
|
|
|
|
Image encoders has a single input field `image`: float32 and support only batch of single image (due to CoreML bug). |
|
|
|
|
|
Both encoders return: |
|
|
|
|
|
* `features`: float32 |
|
|
* `embeddings`: float32 |
|
|
|
|
|
If you want to convert a model with other parameters (i.e fp16 precision or other batch size range), you can use [convert.py](https://huggingface.co/unum-cloud/uform-coreml/blob/main/convert_model.py). |
|
|
|
