2024-03-14

This is a 3-bit exl2 (Exllamav2) conversion of MistralAI's Mixtral-8x7B-v0.1

This takes about 17GB on disk, so it should load on consumer cards with 24GB of VRAM.

More on ExLlamav2 -> https://github.com/turboderp/exllamav2 ( works with Oobabooga's textgen web ui )
Original Model -> https://huggingface.co/mistralai/Mixtral-8x7B-v0.1 ( Original files are ~87GB on disk )

Downloads last month: 10

Inference Examples

This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.