Hugging Face's logo

hierholzer
/
Llama-3.1-70B-Instruct-GGUF

Text Generation
Transformers
GGUF
PyTorch
English
llama
meta
llama-3.1
llama-3.1-instruct
ollama
Text-generation-webui
instruct
llama_3.1
conversational
Model card Files Community
1
hierholzer commited on
Commit
3d2a534
·
verified ·
1 Parent(s): 6c01945

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +40 -35
README.md CHANGED
@@ -1,13 +1,12 @@
1
- ---
2
- license: apache-2.0
3
- language:
4
- - en
5
- ---
6
 
 
7
 
8
- ---
9
-
10
- # Model<br>
11
 
12
 
13
  Here is a Quantized version of Llama-3.1-70B-Instruct using GGUF<br>
@@ -19,39 +18,45 @@ Models initially developed in frameworks like PyTorch can be converted to GGUF f
19
 
20
  ## Uploaded Quantization Types<br>
21
 
22
- Currently, I have uploaded 2 quantized versions:<br>
23
-
24
- Q5_K_M : - large, very low quality loss<br>
25
-
26
- and<br>
27
 
28
- Q8_0 : - very large, extremely low quality loss<br>
 
 
29
 
30
  ### All Quantization Types Possible
31
 
32
  Here are all of the Quantization Types that are Possible. Let me know if you need any other versions
33
 
34
- | 2 | or | Q4_0 | : | small, very high quality loss - legacy, prefer using Q3_K_M |<br>
35
- | 3 | or | Q4_1 | : | small, substantial quality loss - legacy, prefer using Q3_K_L |<br>
36
- | 8 | or | Q5_0 | : | medium, balanced quality - legacy, prefer using Q4_K_M |<br>
37
- | 9 | or | Q5_1 | : | medium, low quality loss - legacy, prefer using Q5_K_M |<br>
38
- | 10 | or | Q2_K | : | smallest, extreme quality loss - not recommended |<br>
39
- | 12 | or | Q3_K | : | alias for Q3_K_M |<br>
40
- | 11 | or | Q3_K_S | : | very small, very high quality loss |<br>
41
- | 12 | or | Q3_K_M | : | very small, very high quality loss |<br>
42
- | 13 | or | Q3_K_L | : | small, substantial quality loss |<br>
43
- | 15 | or | Q4_K | : | alias for Q4_K_M |<br>
44
- | 14 | or | Q4_K_S | : | small, significant quality loss |<br>
45
- | 15 | or | Q4_K_M | : | medium, balanced quality - *recommended* |<br>
46
- | 17 | or | Q5_K | : | alias for Q5_K_M |<br>
47
- | 16 | or | Q5_K_S | : | large, low quality loss - *recommended* |<br>
48
- | 17 | or | Q5_K_M | : | large, very low quality loss - *recommended* |<br>
49
- | 18 | or | Q6_K | : | very large, very low quality loss |<br>
50
- | 7 | or | Q8_0 | : | very large, extremely low quality loss |<br>
51
- | 1 | or | F16 | : | extremely large, virtually no quality loss - not recommended |<br>
52
- | 0 | or | F32 | : | absolutely huge, lossless - not recommended |<br>
53
-
 
54
 
55
  ## Uses
56
 
57
- By using the GGUF version of Llama-3.1-70B-Instruct, you will be able to run this LLM while having to use significantly less resources than you would using the non quantized version.
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ language:
4
+ - en
5
+ ---
6
 
7
+ [![Hierholzer Banner](https://tvtime.us/static/images/LLAMA3.1.jpg)](#)
8
 
9
+ # Model
 
 
10
 
11
 
12
  Here is a Quantized version of Llama-3.1-70B-Instruct using GGUF<br>
 
18
 
19
  ## Uploaded Quantization Types<br>
20
 
21
+ Currently, I have uploaded 2 quantized versions:
 
 
 
 
22
 
23
+ - [x] Q5_K_M ~ *Recommended*
24
+ - [x] Q8_0
25
+ - [ ] Q4_K_M
26
 
27
  ### All Quantization Types Possible
28
 
29
  Here are all of the Quantization Types that are Possible. Let me know if you need any other versions
30
 
31
+ | **#** | **or** | **Q#** | **:** | _Description Of Quantization Types_ |
32
+ |-------|:------:|:------:|:-----:|----------------------------------------------------------------|
33
+ | 2 | or | Q4_0 | : | small, very high quality loss - legacy, prefer using Q3_K_M |
34
+ | 3 | or | Q4_1 | : | small, substantial quality loss - legacy, prefer using Q3_K_L |
35
+ | 8 | or | Q5_0 | : | medium, balanced quality - legacy, prefer using Q4_K_M |
36
+ | 9 | or | Q5_1 | : | medium, low quality loss - legacy, prefer using Q5_K_M |
37
+ | 10 | or | Q2_K | : | smallest, extreme quality loss - *NOT Recommended* |
38
+ | 12 | or | Q3_K | : | alias for Q3_K_M |
39
+ | 11 | or | Q3_K_S | : | very small, very high quality loss |
40
+ | 12 | or | Q3_K_M | : | very small, high quality loss |
41
+ | 13 | or | Q3_K_L | : | small, high quality loss |
42
+ | 15 | or | Q4_K | : | alias for Q4_K_M |
43
+ | 14 | or | Q4_K_S | : | small, some quality loss |
44
+ | 15 | or | Q4_K_M | : | medium, balanced quality - *Recommended* |
45
+ | 17 | or | Q5_K | : | alias for Q5_K_M |
46
+ | 16 | or | Q5_K_S | : | large, low quality loss - *Recommended* |
47
+ | 17 | or | Q5_K_M | : | large, very low quality loss - *Recommended* |
48
+ | 18 | or | Q6_K | : | very large, very low quality loss |
49
+ | 7 | or | Q8_0 | : | very large, extremely low quality loss |
50
+ | 1 | or | F16 | : | extremely large, virtually no quality loss - *NOT Recommended* |
51
+ | 0 | or | F32 | : | absolutely huge, lossless - *NOT Recommended* |
52
 
53
  ## Uses
54
 
55
+ By using the GGUF version of Llama-3.1-70B-Instruct, you will be able to run this LLM while having to use significantly less resources than you would using the non quantized version.
56
+
57
+ [![Hugging Face](https://img.shields.io/badge/Hugging%20Face-FFD21E?logo=huggingface&logoColor=000)](#)
58
+ [![OS](https://img.shields.io/badge/OS-linux%2C%20windows%2C%20macOS-0078D4)](https://docs.abblix.com/docs/technical-requirements)
59
+ [![CPU](https://img.shields.io/badge/CPU-x86%2C%20x64%2C%20ARM%2C%20ARM64-FF8C00)](https://docs.abblix.com/docs/technical-requirements)
60
+ [![forthebadge](https://forthebadge.com/images/badges/license-mit.svg)](https://forthebadge.com)
61
+ [![forthebadge](https://forthebadge.com/images/badges/made-with-python.svg)](https://forthebadge.com)
62
+ [![forthebadge](https://forthebadge.com/images/badges/powered-by-electricity.svg)](https://forthebadge.com)