|
--- |
|
tags: |
|
- fp8 |
|
- vllm |
|
language: |
|
- en |
|
- de |
|
- fr |
|
- it |
|
- pt |
|
- hi |
|
- es |
|
- th |
|
pipeline_tag: image-text-to-text |
|
license: apache-2.0 |
|
library_name: vllm |
|
base_model: |
|
- mistral-community/pixtral-12b |
|
- mistralai/Pixtral-12B-2409 |
|
base_model_relation: quantized |
|
datasets: |
|
- HuggingFaceH4/ultrachat_200k |
|
--- |
|
|
|
# Pixtral-12B-2409: FP8 Dynamic Quant + FP8 KV Cache |
|
|
|
Quant of [mistral-community/pixtral-12b](https://huggingface.co/mistral-community/pixtral-12b) using [LLM Compressor](https://github.com/vllm-project/llm-compressor) for optimised inference on VLLM. |
|
|
|
FP8 dynamic quant on language model, and FP8 quant of KV cache. multi_modal_projector and vision_tower left in FP16 since it's a small part of the model. |
|
|
|
Calibrated on 2048 ultrachat samples. |
