FP8 LLMs for vLLM - a neuralmagic Collection

neuralmagic 's Collections

DeepSeek-R1-Distill Quantized

Granite 3.1 Quantization

Sparse-Llama-3.1-2of4

Vision Language Models Quantization

FP8 LLMs for vLLM

Llama-3.2 Quantization

Llama-3.1 Quantization

INT8 LLMs for vLLM

INT4 LLMs for vLLM

Sparse Foundational Llama 2 Models

Compression Papers

DeepSparse Sparse LLMs

Sparse Finetuning MPT

Compressed LLMs from the Community

FP8 LLMs for vLLM

updated Oct 17, 2024

Accurate FP8 quantized models by Neural Magic, ready for use with vLLM!

neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8

Text Generation • Updated Oct 9, 2024 • 4.27k • 31
neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8

Text Generation • Updated Oct 9, 2024 • 167k • 38
neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8

Text Generation • Updated 16 days ago • 109k • 41
neuralmagic/Mistral-Nemo-Instruct-2407-FP8

Text Generation • Updated Jul 19, 2024 • 36.4k • 18
neuralmagic/Meta-Llama-3-8B-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 9.66k • 22
neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8-dynamic

Text Generation • Updated Oct 19, 2024 • 285 • 14
neuralmagic/Meta-Llama-3-70B-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 36k • 11
neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8

Text Generation • Updated Jul 18, 2024 • 1.44k • 3
neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV

Text Generation • Updated Jun 19, 2024 • 5.7k • 7
neuralmagic/Meta-Llama-3-70B-Instruct-FP8-KV

Text Generation • Updated Jun 26, 2024 • 954 • 2
neuralmagic/Mixtral-8x22B-Instruct-v0.1-AutoFP8

Text Generation • Updated Aug 12, 2024 • 893 • 3
neuralmagic/Qwen2-72B-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 301 • 12
neuralmagic/Qwen2-7B-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 6.15k • 1
neuralmagic/Qwen2-1.5B-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 2.71k
neuralmagic/Qwen2-0.5B-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 1.36k • 3
neuralmagic/Mistral-7B-Instruct-v0.3-FP8

Text Generation • Updated Jul 18, 2024 • 440 • 2
neuralmagic/Llama-2-7b-chat-hf-FP8

Text Generation • Updated Jul 18, 2024 • 270
neuralmagic/gemma-2-9b-it-FP8

Text Generation • Updated Jul 18, 2024 • 588 • 6
neuralmagic/Qwen2-57B-A14B-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 263 • 1
neuralmagic/DeepSeek-Coder-V2-Lite-Instruct-FP8

Text Generation • Updated Jul 18, 2024 • 4.87k • 6
neuralmagic/DeepSeek-Coder-V2-Lite-Base-FP8

Text Generation • Updated Jul 18, 2024 • 35
neuralmagic/DeepSeek-Coder-V2-Base-FP8

Text Generation • Updated Jul 22, 2024 • 89
neuralmagic/DeepSeek-Coder-V2-Instruct-FP8

Text Generation • Updated Jul 22, 2024 • 438 • 7
neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic

Text Generation • Updated Oct 19, 2024 • 8.04k • 5
neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8-dynamic

Text Generation • Updated Oct 19, 2024 • 3.75k • 6
neuralmagic/Meta-Llama-3.1-8B-FP8

Text Generation • Updated Oct 9, 2024 • 2.92k • 6
neuralmagic/Meta-Llama-3.1-70B-FP8

Text Generation • Updated Oct 9, 2024 • 388 • 1
neuralmagic/starcoder2-15b-FP8

Text Generation • Updated Oct 9, 2024 • 130
neuralmagic/starcoder2-3b-FP8

Text Generation • Updated Oct 9, 2024 • 154
neuralmagic/starcoder2-7b-FP8

Text Generation • Updated Oct 9, 2024 • 83
neuralmagic/Meta-Llama-3.1-405B-FP8

Text Generation • Updated Oct 9, 2024 • 189
neuralmagic/gemma-2-2b-it-FP8

Updated Oct 9, 2024 • 137 • 1
neuralmagic/Llama-3.2-1B-Instruct-FP8-dynamic

Text Generation • Updated Oct 9, 2024 • 19.7k • 2
neuralmagic/Llama-3.2-3B-Instruct-FP8-dynamic

Text Generation • Updated Oct 9, 2024 • 2.93k • 2
neuralmagic/Llama-3.2-3B-Instruct-FP8

Text Generation • Updated Oct 16, 2024 • 21.8k • 4
neuralmagic/Llama-3.2-1B-Instruct-FP8

Text Generation • Updated Oct 16, 2024 • 10.3k • 2
neuralmagic/Llama-3.2-1B-FP8

Updated Oct 9, 2024 • 134
neuralmagic/Phi-3.5-mini-instruct-FP8-KV

Text Generation • Updated Oct 1, 2024 • 237 • 2
neuralmagic/Llama-3.1-Nemotron-70B-Instruct-HF-FP8-dynamic

Text Generation • Updated Oct 17, 2024 • 1.69k • 14