minimum vram?
not very familiar with moe models. does it require 685GB or 37GB vram?
need a100 x 10
@CHNtentes it needs about 1tb vram
What if you have a single GPU with 48GB VRAM and 1tb ordinary system RAM? Someone told me that it's possible to separate the layers so that only the active expert (37GB if using a Q8) is in VRAM at any given time, and the rest is in system RAM...
I have no doubt this is possible to do - but would the performance be even close to usable??
What if you have a single GPU with 48GB VRAM and 1tb ordinary system RAM? Someone told me that it's possible to separate the layers so that only the active expert (37GB if using a Q8) is in VRAM at any given time, and the rest is in system RAM...
I have no doubt this is possible to do - but would the performance be even close to usable??
you could try with vLLM as it has CPU offloading with--cpu-offload-gb 900