Hugging Face's logo

second-state
/
Yi-34Bx2-MoE-60B-GGUF

Text Generation
Transformers
GGUF
mixtral
Mixture of Experts
Inference Endpoints
conversational
Model card Files Community
1
apepkuss79 commited on
Commit
88ff284
•
1 Parent(s): 8646192

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +24 -2
README.md CHANGED
@@ -48,13 +48,22 @@ tags:
48
  - Run as LlamaEdge service
49
 
50
  ```bash
51
- wasmedge --dir .:. --nn-preload default:GGML:AUTO:Yi-34Bx2-MoE-60B-Q5_K_M.gguf llama-api-server.wasm -p chatml -r '<|im_end|>'
 
 
 
 
 
52
  ```
53
 
54
  - Run as LlamaEdge command app
55
 
56
  ```bash
57
- wasmedge --dir .:. --nn-preload default:GGML:AUTO:Yi-34Bx2-MoE-60B-Q5_K_M.gguf llama-chat.wasm -p chatml -r '<|im_end|>'
 
 
 
 
58
  ```
59
 
60
  ## Quantized GGUF Models
@@ -72,3 +81,16 @@ tags:
72
  | [Yi-34Bx2-MoE-60B-Q5_K_M.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-Q5_K_M.gguf) | Q5_K_M | 5 | 43.1 GB| large, very low quality loss - recommended |
73
  | [Yi-34Bx2-MoE-60B-Q5_K_S.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-Q5_K_S.gguf) | Q5_K_S | 5 | 41.9 GB| large, low quality loss - recommended |
74
  | [Yi-34Bx2-MoE-60B-Q6_K.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-Q6_K.gguf) | Q6_K | 6 | 49.9 GB| very large, extremely low quality loss |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
48
  - Run as LlamaEdge service
49
 
50
  ```bash
51
+ wasmedge --dir .:. --nn-preload default:GGML:AUTO:Yi-34Bx2-MoE-60B-Q5_K_M.gguf \
52
+ llama-api-server.wasm \
53
+ --prompt-template chatml \
54
+ --reverse-prompt '<|im_end|>' \
55
+ --ctx-size 7168 \
56
+ --model-name Yi-34Bx2-MoE-60B
57
  ```
58
 
59
  - Run as LlamaEdge command app
60
 
61
  ```bash
62
+ wasmedge --dir .:. --nn-preload default:GGML:AUTO:Yi-34Bx2-MoE-60B-Q5_K_M.gguf \
63
+ llama-chat.wasm \
64
+ --prompt-template chatml \
65
+ --reverse-prompt '<|im_end|>' \
66
+ --ctx-size 7168
67
  ```
68
 
69
  ## Quantized GGUF Models
 
81
  | [Yi-34Bx2-MoE-60B-Q5_K_M.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-Q5_K_M.gguf) | Q5_K_M | 5 | 43.1 GB| large, very low quality loss - recommended |
82
  | [Yi-34Bx2-MoE-60B-Q5_K_S.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-Q5_K_S.gguf) | Q5_K_S | 5 | 41.9 GB| large, low quality loss - recommended |
83
  | [Yi-34Bx2-MoE-60B-Q6_K.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-Q6_K.gguf) | Q6_K | 6 | 49.9 GB| very large, extremely low quality loss |
84
+ | [Yi-34Bx2-MoE-60B-Q8_0-00001-of-00003.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-Q8_0-00001-of-00003.gguf) | Q8_0 | 8 | 32.2 GB| very large, extremely low quality loss - not recommended |
85
+ | [Yi-34Bx2-MoE-60B-Q8_0-00002-of-00003.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-Q8_0-00002-of-00003.gguf) | Q8_0 | 8 | 32.1 GB| very large, extremely low quality loss - not recommended |
86
+ | [Yi-34Bx2-MoE-60B-Q8_0-00001-of-00003.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-Q8_0-00003-of-00003.gguf) | Q8_0 | 8 | 312 MB| very large, extremely low quality loss - not recommended |
87
+ | [Yi-34Bx2-MoE-60B-f16-00001-of-00008.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-f16-00001-of-00008.gguf) | f16 | 16 | 31.9 GB| |
88
+ | [Yi-34Bx2-MoE-60B-f16-00002-of-00008.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-f16-00002-of-00008.gguf) | f16 | 16 | 31.7 GB| |
89
+ | [Yi-34Bx2-MoE-60B-f16-00003-of-00008.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-f16-00003-of-00008.gguf) | f16 | 16 | 31.7 GB| |
90
+ | [Yi-34Bx2-MoE-60B-f16-00004-of-00008.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-f16-00004-of-00008.gguf) | f16 | 16 | 31.7 GB| |
91
+ | [Yi-34Bx2-MoE-60B-f16-00005-of-00008.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-f16-00005-of-00008.gguf) | f16 | 16 | 31.7 GB| |
92
+ | [Yi-34Bx2-MoE-60B-f16-00006-of-00008.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-f16-00006-of-00008.gguf) | f16 | 16 | 31.7 GB| |
93
+ | [Yi-34Bx2-MoE-60B-f16-00007-of-00008.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-f16-00007-of-00008.gguf) | f16 | 16 | 31.7 GB| |
94
+ | [Yi-34Bx2-MoE-60B-f16-00008-of-00008.gguf](https://huggingface.co/second-state/Yi-34Bx2-MoE-60B-GGUF/blob/main/Yi-34Bx2-MoE-60B-f16-00008-of-00008.gguf) | f16 | 16 | 21.1 GB| |
95
+
96
+ *Quantized with llama.cpp b2734*