2024-03-14

This is a 3-bit exl2 (Exllamav2) conversion of MistralAI's Mistral-7B-Instruct-v0.2

This takes about 2.8G on disk, so it should load on consumer cards.

More on ExLlamav2 -> https://github.com/turboderp/exllamav2 ( works with Oobabooga's textgen web ui )
Original Model -> https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 ( Original files are ~17GB on disk )

Downloads last month: 5

Inference Examples

This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.