eachadea
/

ggml-vicuna-13b-1.1

Document Question Answering

Model card Files Files and versions Community

eachadea commited on Apr 19, 2023

Commit

bb35084

•

1 Parent(s): 6b1bec2

Update README.md

Files changed (1) hide show

README.md +1 -0

README.md CHANGED Viewed

@@ -9,6 +9,7 @@ inference: false
 - 4-bit quantized
 - Based on version 1.1
 - Used PR "More accurate Q4_0 and Q4_1 quantizations #896" (should be closer in quality to unquantized)
 - Choosing between q4_0 and q4_1, the logic of higher number \= better does not apply. If you are confused, stick with q4_0.
   - If you have performance to spare, it might be worth getting the q4_1. It's ~20% slower and requires 1GB more RAM, but has a ~5% lower perplexity, which is good for generation quality. You're not gonna notice it though.
   - If you have *lots* of performance to spare, [TheBloke's conversion](https://huggingface.co/TheBloke/vicuna-13B-1.1-GPTQ-4bit-128g-GGML) is maybe ~7% better in perplexity but ~50% slower and requires 2GB more RAM.

 - 4-bit quantized
 - Based on version 1.1
 - Used PR "More accurate Q4_0 and Q4_1 quantizations #896" (should be closer in quality to unquantized)
+- For q4_2, "Q4_2 ARM #1046" was used. Will update regularly if new changes are made.
 - Choosing between q4_0 and q4_1, the logic of higher number \= better does not apply. If you are confused, stick with q4_0.
   - If you have performance to spare, it might be worth getting the q4_1. It's ~20% slower and requires 1GB more RAM, but has a ~5% lower perplexity, which is good for generation quality. You're not gonna notice it though.
   - If you have *lots* of performance to spare, [TheBloke's conversion](https://huggingface.co/TheBloke/vicuna-13B-1.1-GPTQ-4bit-128g-GGML) is maybe ~7% better in perplexity but ~50% slower and requires 2GB more RAM.