---
base_model:
- unsloth/Mistral-Nemo-Instruct-2407
---
Note: This model is no longer the optimal W8A8 quantization, please consider using a better quantization model I made later:
noneUsername/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token-better

My first quantization uses the quantization method provided by vllm:

https://docs.vllm.ai/en/latest/quantization/int8.html

NUM_CALIBRATION_SAMPLES = 2048

MAX_SEQUENCE_LENGTH = 8192

smoothing_strength=0.8

I will verify the validity of the model and update the readme as soon as possible.

edit: The performance in my ERP test was comparable to Mistral-Nemo-Instruct-2407-GPTQ-INT8, which I consider a successful quantization.

vllm (pretrained=/root/autodl-tmp/Mistral-Nemo-Instruct-2407,add_bos_token=true,tensor_parallel_size=2,max_model_len=4096,gpu_memory_utilization=0.85,swap_space=0), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.800|±  |0.0253|
|     |       |strict-match    |     5|exact_match|↑  |0.784|±  |0.0261|


lm_eval --model vllm   --model_args pretrained="/mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token",add_bos_token=true,dtype=half,tensor_parallel_size=2,max_model_len=4096,gpu_memory_utilization=0.85,swap_space=0   --tasks gsm8k   --num_fewshot 5   --limit 250   --batch_size 1
vllm (pretrained=/mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token,add_bos_token=true,dtype=half,tensor_parallel_size=2,max_model_len=4096,gpu_memory_utilization=0.85,swap_space=0), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.784|±  |0.0261|
|     |       |strict-match    |     5|exact_match|↑  |0.768|±  |0.0268|
In gsm8k, still a bit worse than the original...


lm_eval --model vllm \
l_args >   --model_args pretrained="/mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token",add_bos_token=true,dtype=half,tensor_parallel_size=2,max_model_len=4096,gpu_memory_utilization=0.85,swap_space=0 \
ks hellaswag \
 >   --tasks hellaswag \
>    --limit 150 \
>   --num_fewshot 10 \
--batch_size 1

vllm (pretrained=/mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token,add_bos_token=true,dtype=half,tensor_parallel_size=2,max_model_len=4096,gpu_memory_utilization=0.85,swap_space=0), gen_kwargs: (None), limit: 150.0, num_fewshot: 10, batch_size: 1
|  Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|---------|------:|------|-----:|--------|---|-----:|---|-----:|
|hellaswag|      1|none  |    10|acc     |↑  |0.5800|±  |0.0404|
|         |       |none  |    10|acc_norm|↑  |0.7533|±  |0.0353|