Hugging Face's logo

TheBloke
/
WizardLM-13B-V1.1-GGML

Model card Files Community
3
WizardLM-13B-V1.1-GGML / README.md
TheBloke's picture
TheBloke
Update README.md
1657a99
preview code
|
raw
history blame
8.28 kB
metadata
inference: false
license: other
TheBlokeAI

WizardLM's WizardLM 13B V1.1 GGML

These files are GGML format model files for WizardLM's WizardLM 13B V1.1.

GGML files are for CPU + GPU inference using llama.cpp and libraries and UIs which support this format, such as:

Repositories available

Prompt template: Vicuna 

A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
USER: prompt
ASSISTANT:

Compatibility

Original llama.cpp quant methods: q4_0, q4_1, q5_0, q5_1, q8_0

I have quantized these 'original' quantisation methods using an older version of llama.cpp so that they remain compatible with llama.cpp as of May 19th, commit 2d5db48.

These are guaranteed to be compatbile with any UIs, tools and libraries released since late May.

New k-quant methods: not supported at the moment due to model's vocab size

Unfortunately it is not possible to make the new k-quant format quantisations for this model at this time.

This is because the model uses a non-standard vocab size of 32,001, which is not divisible by 256.

This is being investigated by the llama.cpp team and may be fixed in future. You can read more about that here: https://github.com/ggerganov/llama.cpp/issues/1919

Provided files

Name Quant method Bits Size Max RAM required Use case
wizardlm-13b-v1.1.ggmlv3.q4_0.bin q4_0 4 7.32 GB 9.82 GB Original llama.cpp quant method, 4-bit.
wizardlm-13b-v1.1.ggmlv3.q4_1.bin q4_1 4 8.14 GB 10.64 GB Original llama.cpp quant method, 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models.
wizardlm-13b-v1.1.ggmlv3.q5_0.bin q5_0 5 8.95 GB 11.45 GB Original llama.cpp quant method, 5-bit. Higher accuracy, higher resource usage and slower inference.
wizardlm-13b-v1.1.ggmlv3.q5_1.bin q5_1 5 9.76 GB 12.26 GB Original llama.cpp quant method, 5-bit. Even higher accuracy, resource usage and slower inference.
wizardlm-13b-v1.1.ggmlv3.q8_0.bin q8_0 8 13.83 GB 16.33 GB Original llama.cpp quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users.

Note: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.

How to run in llama.cpp

I use the following command line; adjust for your tastes and needs:

./main -t 10 -ngl 32 -m wizardlm-13b-v1.1.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:"

If you're able to use full GPU offloading, you should use -t 1 to get best performance.

If not able to fully offload to GPU, you should use more cores. Change -t 10 to the number of physical CPU cores you have, or a lower number depending on what gives best performance.

Change -ngl 32 to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.

If you want to have a chat-style conversation, replace the -p <PROMPT> argument with -i -ins

How to run in text-generation-webui

Further instructions here: text-generation-webui/docs/llama.cpp-models.md.

Discord

For further support, and discussions on these models and AI in general, join us at:

TheBloke AI's Discord server

Thanks, and how to contribute.

Thanks to the chirper.ai team!

I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.

If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.

Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.

Special thanks to: Luke from CarbonQuill, Aemon Algiz.

Patreon special mentions: RoA, Lone Striker, Gabriel Puliatti, Derek Yates, Randy H, Jonathan Leane, Eugene Pentland, Karl Bernard, Viktor Bowallius, senxiiz, Daniel P. Andersen, Pierre Kircher, Deep Realms, Cory Kujawski, Oscar Rangel, Fen Risland, Ajan Kanaga, LangChain4j, webtim, Nikolai Manek, Trenton Dambrowitz, Raven Klaugh, Kalila, Khalefa Al-Ahmad, Chris McCloskey, Luke @flexchar, Ai Maven, Dave, Asp the Wyvern, Sean Connelly, Imad Khwaja, Space Cruiser, Rainer Wilmers, subjectnull, Alps Aficionado, Willian Hasse, Fred von Graf, Artur Olbinski, Johann-Peter Hartmann, WelcomeToTheClub, Willem Michiel, Michael Levine, Iucharbius , Spiking Neurons AB, K, biorpg, John Villwock, Pyrater, Greatston Gnanesh, Mano Prime, Junyu Yang, Stephen Murray, John Detwiler, Luke Pendergrass, terasurfer , Pieter, zynix , Edmond Seymore, theTransient, Nathan LeClaire, vamX, Kevin Schuppel, Preetika Verma, ya boyyy, Alex , SuperWojo, Ghost , Joseph William Delisle, Matthew Berman, Talal Aujan, chris gileta, Illia Dulskyi.

Thank you to all my generous patrons and donaters!

Original model card: WizardLM's WizardLM 13B V1.1

This is the Full-Weight of WizardLM-13B V1.1 model.

Repository: https://github.com/nlpxucan/WizardLM

Twitter: https://twitter.com/WizardLM_AI/status/1677282955490918401