xmadai
/

Mistral-Small-Instruct-2409-xMADai-INT4

Text Generation

Inference Endpoints

4-bit precision

Model card Files Files and versions Community

JonahYixMAD commited on 3 days ago

Commit

191f8fb

•

1 Parent(s): 006e173

Update README.md

Files changed (1) hide show

README.md +4 -8

README.md CHANGED Viewed

@@ -15,14 +15,10 @@ This repository contains [`mistralai/Mistral-Small-Instruct-2409`](https://huggi
 2. **Accuracy:** This xMADified model preserves the quality of the full-precision model. In the table below, we present the zero-shot accuracy on popular benchmarks of this xMADified model against the [GPTQ](https://github.com/AutoGPTQ/AutoGPTQ)-quantized model (both w4g128 for a fair comparison). GPTQ fails on the difficult **MMLU** task, while the xMADai model offers significantly higher accuracy.
-| Benchmark | xMADai Mistral-Small-Instruct-2409<br>(compared to GPTQ Mistral-Small-Instruct-2409) |
-|---|---|
-| **MMLU** | 49.45 → **68.59** |
-| **Arc Challenge** | 56.14 → **57.51** |
-| **Arc Easy** | 80.64 → **82.83** |
-| **LAMBADA** | 75.1 → **77.74** |
-| **WinoGrande** | 77.74 → **79.56** |
-| **PIQA** | 77.48 → **81.34** |
 # How to Run Model

 2. **Accuracy:** This xMADified model preserves the quality of the full-precision model. In the table below, we present the zero-shot accuracy on popular benchmarks of this xMADified model against the [GPTQ](https://github.com/AutoGPTQ/AutoGPTQ)-quantized model (both w4g128 for a fair comparison). GPTQ fails on the difficult **MMLU** task, while the xMADai model offers significantly higher accuracy.
+| Model | MMLU | Arc Challenge | Arc Easy | LAMBADA | WinoGrande | PIQA |
+|---|---|---|---|---|---|---|
+| GPTQ Mistral-Small-Instruct-2409 | 49.45 | 56.14 | 80.64 | 75.1 | 77.74 | 77.48 |
+| xMADai Mistral-Small-Instruct-2409 | **68.59** | **57.51** | **82.83** | **77.74** | **79.56** | **81.34** |
 # How to Run Model