|
--- |
|
base_model: google/madlad400-10b-mt |
|
inference: false |
|
license: apache-2.0 |
|
model_name: madlad400-10b-mt-gguf |
|
pipeline_tag: translation |
|
--- |
|
|
|
# MADLAD-400-10B-MT - GGUF |
|
|
|
- Original model: [MADLAD-400-10B-MT](https://huggingface.co/google/madlad400-10b-mt) |
|
|
|
## Description |
|
|
|
This repo contains GGUF format model files for [MADLAD-400-10B-MT](https://huggingface.co/google/madlad400-10b-mt) for |
|
use with [llama.cpp](https://github.com/ggerganov/llama.cpp) and compatible software. |
|
|
|
Converted to gguf using llama.cpp [convert_hf_to_gguf.py](https://github.com/ggerganov/llama.cpp/blob/master/convert_hf_to_gguf.py) |
|
and quantized using llama.cpp llama-quantize, llama.cpp version [b3325](https://github.com/ggerganov/llama.cpp/commits/b3325). |
|
|
|
|
|
## Provided files |
|
|
|
| Name | Quant method | Bits | Size | VRAM required | |
|
| ---- | ---- | ---- | ---- | ---- | |
|
| [model-q3_k_m.gguf](https://huggingface.co/thirteenbit/madlad400-10b-mt-gguf/blob/main/model-q3_k_m.gguf) | Q3_K_M | 3 | 4.9 GB| 5.7 GB | |
|
| [model-q4_k_m.gguf](https://huggingface.co/thirteenbit/madlad400-10b-mt-gguf/blob/main/model-q4_k_m.gguf) | Q4_K_M | 4 | 6.3 GB| 7.1 GB | |
|
| [model-q5_k_m.gguf](https://huggingface.co/thirteenbit/madlad400-10b-mt-gguf/blob/main/model-q5_k_m.gguf) | Q5_K_M | 5 | 7.2 GB| 7.9 GB | |
|
| [model-q6_k.gguf](https://huggingface.co/thirteenbit/madlad400-10b-mt-gguf/blob/main/model-q6_k.gguf) | Q6_K | 6 | 8.2 GB| 8.9 GB | |
|
| [model-q8_0.gguf](https://huggingface.co/thirteenbit/madlad400-10b-mt-gguf/blob/main/model-q8_0.gguf) | Q8_0 | 8 | 11 GB| 11.3 GB | |
|
|
|
**Note**: the above VRAM usage figures are observed with all layers GPU offloading, on Linux with NVIDIA GPU. |
|
|
|
|
