meta-llama/Llama-3.2-11B-Vision-Instruct, UQFF quantization

Run with mistral.rs. Documentation: UQFF docs.

  1. Flexible πŸŒ€: Multiple quantization formats in one file format with one framework to run them all.
  2. Reliable πŸ”’: Compatibility ensured with embedded and checked semantic versioning information from day 1.
  3. Easy πŸ€—: Download UQFF models easily and quickly from Hugging Face, or use a local file.
  4. Customizable πŸ› οΈ: Make and publish your own UQFF files in minutes.

Examples

Quantization type(s) Example
FP8 ./mistralrs-server -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF -a vllama --from-uqff llama3.2-vision-instruct-f8e4m3.uqff
HQQ4 ./mistralrs-server -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF -a vllama --from-uqff llama3.2-vision-instruct-hqq4.uqff
HQQ8 ./mistralrs-server -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF -a vllama --from-uqff llama3.2-vision-instruct-hqq8.uqff
Q3K ./mistralrs-server -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF -a vllama --from-uqff llama3.2-vision-instruct-q3k.uqff
Q4K ./mistralrs-server -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF -a vllama --from-uqff llama3.2-vision-instruct-q4k.uqff
Q5K ./mistralrs-server -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF -a vllama --from-uqff llama3.2-vision-instruct-q5k.uqff
Q8_0 ./mistralrs-server -i vision-plain -m EricB/Llama-3.2-11B-Vision-Instruct-UQFF -a vllama --from-uqff llama3.2-vision-instruct-q8_0.uqff
Downloads last month
23
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for EricB/Llama-3.2-11B-Vision-Instruct-UQFF

Quantized
(17)
this model

Collection including EricB/Llama-3.2-11B-Vision-Instruct-UQFF