Problems with guanaco-7B.ggmlv3.q3_K_S.bin
I was running some testing on all the guanaco-7B.ggml models and I'm getting bad/gibberish results from the q3_K_S version (only). Other q3_K_S models I test with seem to be working fine, so I don't think it's a llamacpp issue. For example, when answering a question where the answer should be "Canada" it replied "Canadaзвоittle", all the other guanaco-7B.ggml in this set work fine.
Thanks for the report. Yes, it looks like the file was too small. Others show 2.95GB, this one is 2.9GB. Maybe there was some issue when I was adding k-quants to this model, the file got truncated or something. I'm surprised it didn't error.
Anyway I've re-quantised that file, please test it again.
I'm still getting the same kinds of errors, I confirmed the sha256sum as 5917246f0e79dd502b4fbd0c8f73e76c3e0b41bea415e6196fa15ccaa19adf3c as well, so the download isn't corrupt.
UPDATE: maybe it's something on my end (code maybe? I'm running llama.cpp 0.1.72 ) I'm having the same problem with a number of 7B.q3_K_S models. Here is a sample of some other tests (answer should be "Friday"):
| pass | load_t | context | template | model_full_folder_name | answers |
|---|---|---|---|---|---|
| ❌ | 5.1 | 2048 | Vicuna-v1.1 | Wizard-Vicuna-7B-Uncensored.ggmlv3.q3_K_S.bin | ['Friday\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f'] |
| ✅ | 7.3 | 2048 | Vicuna-v1.1 | airoboros-13b.ggmlv3.q3_K_S.bin | ['Friday'] |
| ✅ | 24.7 | 2048 | Airoboros-gpt4 | airoboros-65b-gpt4-1.4.ggmlv3.q3_K_S.bin | ['Friday'] |
| ✅ | 35.2 | 2048 | Airoboros-gpt4 | airoboros-65b-gpt4-1.4.ggmlv3.q4_0.bin | ['Friday'] |
| ✅ | 6.9 | 2048 | Airoboros-gpt4 | airoboros-7b-gpt4-1.4.ggmlv3.q2_K.bin | ['Friday'] |
| ✅ | 5.2 | 2048 | Airoboros-gpt4 | airoboros-7b-gpt4-1.4.ggmlv3.q3_K_L.bin | ['Friday'] |
| ✅ | 5.0 | 2048 | Airoboros-gpt4 | airoboros-7b-gpt4-1.4.ggmlv3.q3_K_M.bin | ['Friday'] |
| ❌ | 4.5 | 2048 | Airoboros-gpt4 | airoboros-7b-gpt4-1.4.ggmlv3.q3_K_S.bin | ['Fridayтяiosoga Aus Core pin\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f'] |
| ✅ | 4.9 | 2048 | Airoboros-gpt4 | airoboros-7b-gpt4-1.4.ggmlv3.q4_0.bin | ['Friday'] |
| ✅ | 4.2 | 2048 | Guanaco | guanaco-7B.ggmlv3.q2_K.bin | ['Friday.'] |
| ✅ | 4.6 | 2048 | Guanaco | guanaco-7B.ggmlv3.q3_K_L.bin | ['Friday'] |
| ✅ | 4.3 | 2048 | Guanaco | guanaco-7B.ggmlv3.q3_K_M.bin | ['Friday'] |
| ❌ | 4.3 | 2048 | Guanaco | guanaco-7B.ggmlv3.q3_K_S.bin | ['Fridayzonimoine Chamberircletinedaogoinand Copictionaryoster Crime\x0f\x0f\x0f'] |
| ✅ | 4.6 | 2048 | Guanaco | guanaco-7B.ggmlv3.q4_0.bin | ['Friday'] |
| ✅ | 5.2 | 2048 | Guanaco | guanaco-7B.ggmlv3.q4_1.bin | ['Friday'] |
| ✅ | 5.2 | 2048 | Guanaco | guanaco-7B.ggmlv3.q4_K_M.bin | ['Friday'] |
| ✅ | 4.8 | 2048 | Guanaco | guanaco-7B.ggmlv3.q4_K_S.bin | ['Friday'] |
| ✅ | 5.7 | 2048 | Guanaco | guanaco-7B.ggmlv3.q5_0.bin | ['Friday'] |
| ✅ | 6.2 | 2048 | Guanaco | guanaco-7B.ggmlv3.q5_1.bin | ['Friday'] |
| ✅ | 5.6 | 2048 | Guanaco | guanaco-7B.ggmlv3.q5_K_M.bin | ['Friday'] |
| ✅ | 5.5 | 2048 | Guanaco | guanaco-7B.ggmlv3.q5_K_S.bin | ['Friday'] |
| ✅ | 6.1 | 2048 | Guanaco | guanaco-7B.ggmlv3.q6_K.bin | ['Friday'] |
| ✅ | 7.4 | 2048 | Guanaco | guanaco-7B.ggmlv3.q8_0.bin | ['Friday'] |
| ✅ | 6.2 | 2048 | Vicuna-v1.1 | vicuna-13b-v1.3.0.ggmlv3.q3_K_S.bin | ['Friday'] |
| ✅ | 10.6 | 2048 | Vicuna-v1.1 | vicuna-13b-v1.3.0.ggmlv3.q6_K.bin | ['Friday'] |
| ❌ | 5.1 | 2048 | Vicuna-v1.1 | vicuna-7b-v1.3.ggmlv3.q3_K_S.bin | ['Friday Hostilla trenzem PaysDEXeczзhrefoli Grab⇒ trostoielle'] |
This looks like a software bug in (0.1.72), I downgraded to 0.1.66 and it's working fine.