Llama.cpp server support

by vigneshR - opened May 31

May 31

Thanks for the excellent work. I am trying to see if it is possible to run the gguf version as a Llama CPP http server? Could you give me an example of how I can run it please?

Thanks

miaoxinxu803

Jun 4

I have the same problem. Otherwise, the efficiency is too low.

tc-mb

OpenBMB org Jun 4

Hi
we are glad that our work can provide some help to the open source community.
However, we have limited manpower and may continue to support it for some time in the future, but we are not yet able to give any clear time point.

BingoBird

Jun 7

I can run it in normal (text) mode with a recent build of llama.cpp main and server programs.
Image loading not working yet.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment