Update README.md

README.md

@@ -12,6 +12,12 @@ I will verify the validity of the model and update the readme as soon as possibl
 
 edit: The performance in my ERP test was comparable to Mistral-Nemo-Instruct-2407-GPTQ-INT8, which I consider a successful quantization.
 
+vllm (pretrained=/root/autodl-tmp/Mistral-Nemo-Instruct-2407,add_bos_token=true,tensor_parallel_size=2,max_model_len=4096,gpu_memory_utilization=0.85,swap_space=0), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
+|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
+|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
+|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.800|±  |0.0253|
+|     |       |strict-match    |     5|exact_match|↑  |0.784|±  |0.0261|
+
 
 lm_eval --model vllm   --model_args pretrained="/mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token",add_bos_token=true,dtype=half,tensor_parallel_size=2,max_model_len=4096,gpu_memory_utilization=0.85,swap_space=0   --tasks gsm8k   --num_fewshot 5   --limit 250   --batch_size 1
 vllm (pretrained=/mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-Token,add_bos_token=true,dtype=half,tensor_parallel_size=2,max_model_len=4096,gpu_memory_utilization=0.85,swap_space=0), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: 1
@@ -19,7 +25,7 @@ vllm (pretrained=/mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8-Dynamic-Per-
 |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
 |gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.784|±  |0.0261|
 |     |       |strict-match    |     5|exact_match|↑  |0.768|±  |0.0268|
-
+In gsm8k, still a bit worse than the original...
 
 
 lm_eval --model vllm \