Integration into ollama possible?

#4
by hheimel - opened

Dear Ovis developers,

I'm fairly new to integrating models provided on HuggingFace in my open-source framework using ollama and I'm uncertain whether it is my inexperience or a general incompatibility that I haven't managed to integrate Ovis2 in ollama yet.
I would love to try out your model over ollama and followed the section "Importing a model from Safetensors weights" from this ollama guidline:

https://github.com/ollama/ollama/blob/main/docs/import.md

in an attempt to integrate it as new model in ollama after downloading it from hugging face with:

"git clone https://huggingface.co/AIDC-AI/Ovis2-34B
"
Unfortunately I get the ollama error: Error: unsupported architecture.

Is this already wrong and i would have needed to follow the section "Importing a fine tuned adapter from Safetensors weights
" instead?

After some search I found this post:

https://github.com/ollama/ollama/issues/6231

stating that "Qwen2ForCausalLM" (your base model architecture) is not supported yet in the quantization method used by ollama which is based upon an older version of llama.cpp. However I tried the integration without any quantization options (assuming the default means no quantization) using only ollama create "name of modelfile". Is this still the reason for the architecture incompatibility?

Ollama provides several qwen LLMs presumably based on the same architecture which furthers my confusion.

Is it possible to circumvent this issue by doing a conversion (and potentially quantization) from safetensors to GGUF outside of ollama first by using a newer version of llama.cpp like it is demonstrated here:

https://github.com/ggml-org/llama.cpp/discussions/7927

or will the resulting gguf, once I try to integrate it following the section "Importing a GGUF based model or adapter" of the first mentioned link cause issues because the quantization is not understood by the older llama.cpp version ollama is built upon?

AIDC-AI org

Hi, thank you for your inquiry.

Ollama does not currently support the Ovis architecture. If you're interested in using the model, please refer to the README for instructions on deploying it with Transformers.

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment