Updated README

README.md

@@ -14,14 +14,15 @@ base_model:
 ---
 
 
-# Atlas-Chat Model Card
+# JAIS Intiative: Atlas-Chat Models
 
 
 ## Model Overview
 
-Atlas-Chat is a family of open models instruction-tuned for Darija, the colloquial Arabic of Morocco, developed as part of the [Jais](https://arxiv.org/abs/2308.16149) project for standard Arabic and its extentions to dialectal Arabic. These models are designed for language generation and excel in various applications such as question answering, summarization, and translation. Thanks to their compact size, Atlas-Chat models can be deployed in resource-constrained environments like laptops, desktops, or personal cloud setups, making advanced AI accessible to Darija speakers and promoting widespread innovation. Two versions are available:
+Atlas-Chat is a family of open models instruction-tuned for Darija, the colloquial Arabic of Morocco, developed as part of the [Jais](https://arxiv.org/abs/2308.16149) project for standard Arabic and its extentions to dialectal Arabic. These models are designed for language generation and excel in various applications such as question answering, summarization, and translation. Thanks to their compact size, Atlas-Chat models can be deployed in resource-constrained environments like laptops, desktops, or personal cloud setups, making advanced AI accessible to Darija speakers and promoting widespread innovation. Three sizes are available:
 * [Atlas-Chat-2B](https://huggingface.co/MBZUAI-Paris/Atlas-Chat-2B): A small-sized version with 2 billion parameters, capable of generating fluent Moroccan Darija text while maintaining efficiency.
-* [Atlas-Chat-9B](https://huggingface.co/MBZUAI-Paris/Atlas-Chat-9B): A larger version with 9 billion parameters, providing more nuanced, contextually rich language generation for complex tasks.
+* [Atlas-Chat-9B](https://huggingface.co/MBZUAI-Paris/Atlas-Chat-9B): A medium-sized with 9 billion parameters, providing more nuanced, contextually rich language generation for complex tasks.
+* [Atlas-Chat-27B](https://huggingface.co/MBZUAI-Paris/Atlas-Chat-27B): A large-sized version with 27 billion parameters, offering even more advanced capabilities for complex tasks and nuanced language generation compared to the 2B and 9B versions.
 
 The models are designed to assist with:
 
@@ -352,6 +353,7 @@ Our training dataset [Darija-SFT-Mixture](https://huggingface.co/datasets/MBZUAI
 Atlas-Chat models are based on Gemma 2 models. The Atlas-Chat models were trained using 8 Nvidia's A100 80 GB GPUs in parallel using FSDP on AWS Sagemaker. The model is trained using HuggingFace transformers and parameter-efficient fine-tuning with LoRA rank of 256.
 
 
+<!--
 ## Evaluation
 The Atlas-Chat models were evaluated on a comprehensive suite of tasks using various datasets and benchmarks to assess their performance across multiple dimensions. These included tasks such as:
 
@@ -360,6 +362,7 @@ The Atlas-Chat models were evaluated on a comprehensive suite of tasks using var
 * **Belebele Ary_Arab:** Belebele is a multiple-choice machine reading comprehension dataset published by Facebook spanning 122 language variants. The Evaluation is done on the Ary_Arab part of Belebele that refers to Darija.
 * **Sentiment Analysis.**
 * **Translation:** Including six directions and four languages: Darija, MSA, English and French.
+* **Transliteration:** Transforming a sentence from Darija (written in Arabic characters) to Arabizi (Written in Latin characters) and vice-versa.
 * **Summarization.**
 
 The models were compared against a collection of existing open-source Arabic models to gauge their effectiveness, with a particular focus on performance in Darija. All scores are based on zero-shot performance. The prompts are written mainly in Darija. The metric used for DarijaMMLU, DarijaHellaSwag, Belebele Ary and Sentiment Analysis is the normalized accuracy. We used [Language Model Evaluation Harness](https://github.com/MBZUAI-Paris/lm-evaluation-harness-atlas-chat) to conduct these evaluations.
@@ -371,12 +374,24 @@ The models were compared against a collection of existing open-source Arabic mod
         <td rowspan="2"><a href="MBZUAI-Paris/DarijaHellaSwag" target="_blank">DarijaHellaSwag</a></td>
         <td rowspan="2"><a href="https://huggingface.co/datasets/facebook/belebele/viewer/ary_Arab" target="_blank">Belebele Ary</a></td>
         <td rowspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">Sentiment Analysis</a></td>
-        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">DoDa-10k (Translation)</a></td>
+        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">DODa-10k (Translation)</a></td>
+        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">MADAR (Translation)</a></td>
+        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">FLORES+ (Translation)</a></td>
+        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">NLLB-Seed (Translation)</a></td>
+        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">DODa-10k (Transliteration)</a></td>
         <td rowspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">MArSum (Summarization)</a><br/>(LLM as a judge)</td>
     </tr>
     <tr>
         <td>BLEU</td>
         <td>chrF</td>
+        <td>BLEU</td>
+        <td>chrF</td>
+        <td>BLEU</td>
+        <td>chrF</td>
+        <td>BLEU</td>
+        <td>chrF</td>
+        <td>BLEU</td>
+        <td>chrF</td>
     </tr>
     <tr>
         <td><a href="https://huggingface.co/inceptionai/jais-family-1p3b-chat" target="_blank">jais-family-1p3b-chat</a></td>
@@ -387,6 +402,14 @@ The models were compared against a collection of existing open-source Arabic mod
         <td>00.13</td>
         <td>06.18</td>
         <td>00.50</td>
+        <td>15.43</td>
+        <td>02.44</td>
+        <td>19.14</td>
+        <td>01.99</td>
+        <td>12.60</td>
+        <td>00.01</td>
+        <td>03.01</td>
+        <td>00.50</td>
     </tr>
     <tr>
         <td><a href="https://huggingface.co/inceptionai/jais-family-2p7b-chat" target="_blank">jais-family-2p7b-chat</a></td>
@@ -396,6 +419,14 @@ The models were compared against a collection of existing open-source Arabic mod
         <td>51.56</td>
         <td>00.25</td>
         <td>07.46</td>
+        <td>00.62</td>
+        <td>16.36</td>
+        <td>04.25</td>
+        <td>18.22</td>
+        <td>03.10</td>
+        <td>08.19</td>
+        <td>00.01</td>
+        <td>03.27</td>
         <td>00.90</td>
     </tr>
     <tr>
@@ -406,8 +437,52 @@ The models were compared against a collection of existing open-source Arabic mod
         <td>53.36</td>
         <td>00.10</td>
         <td>04.96</td>
+        <td>00.12</td>
+        <td>06.66</td>
+        <td>01.55</td>
+        <td>18.59</td>
+        <td>02.78</td>
+        <td>23.69</td>
+        <td>00.01</td>
+        <td>02.08</td>
         <td>06.80</td>
     </tr>
+    <tr>
+        <td><a href="meta-llama/Llama-3.2-1B-Instruct" target="_blank">Llama-3.2-1B-Instruct</a></td>
+        <td>27.66</td>
+        <td>26.88</td>
+        <td>28.89</td>
+        <td>46.27</td>
+        <td>00.07</td>
+        <td>05.95</td>
+        <td>00.80</td>
+        <td>18.71</td>
+        <td>04.53</td>
+        <td>18.39</td>
+        <td>04.52</td>
+        <td>17.06</td>
+        <td>00.02</td>
+        <td>03.74</td>
+        <td>08.23</td>
+    </tr>
+    <tr>
+        <td><a href="meta-llama/Llama-3.2-3B-Instruct" target="_blank">Llama-3.2-3B-Instruct</a></td>
+        <td>32.60</td>
+        <td>28.33</td>
+        <td>38.00</td>
+        <td>49.20</td>
+        <td>00.62</td>
+        <td>13.67</td>
+        <td>01.18</td>
+        <td>22.12</td>
+        <td>08.59</td>
+        <td>35.21</td>
+        <td>13.75</td>
+        <td>43.63</td>
+        <td>00.21</td>
+        <td>09.68</td>
+        <td>08.23</td>
+    </tr>
     <tr>
         <td><strong><a href="https://huggingface.co/MBZUAI-Paris/Atlas-Chat-2B" target="_blank">Atlas-Chat-2B</a></strong></td>
         <td><b>44.97</td>
@@ -416,6 +491,14 @@ The models were compared against a collection of existing open-source Arabic mod
         <td><b>73.99</td>
         <td><b>22.76</td>
         <td><b>44.86</td>
+        <td><b>16.67</td>
+        <td><b>41.64</td>
+        <td><b>14.92</td>
+        <td><b>43.03</td>
+        <td><b>23.88</td>
+        <td><b>52.19</td>
+        <td><b>08.18</td>
+        <td><b>21.54</td>
         <td><b>55.22</td>
     </tr>
     <tr style="border-top: 4px solid;"></tr>
@@ -427,6 +510,14 @@ The models were compared against a collection of existing open-source Arabic mod
         <td>56.78</td>
         <td>00.73</td>
         <td>11.85</td>
+        <td>01.88</td>
+        <td>23.22</td>
+        <td>04.25</td>
+        <td>18.22</td>
+        <td>04.62</td>
+        <td>20.22</td>
+        <td>00.02</td>
+        <td>03.79</td>
         <td>03.02</td>
     </tr>
     <tr>
@@ -437,6 +528,14 @@ The models were compared against a collection of existing open-source Arabic mod
         <td>52.72</td>
         <td>00.60</td>
         <td>09.43</td>
+        <td>03.45</td>
+        <td>25.88</td>
+        <td>07.25</td>
+        <td>23.21</td>
+        <td>01.25</td>
+        <td>02.22</td>
+        <td>00.04</td>
+        <td>03.24</td>
         <td>02.82</td>
     </tr>
     <tr>
@@ -447,6 +546,14 @@ The models were compared against a collection of existing open-source Arabic mod
         <td>41.73</td>
         <td>00.92</td>
         <td>11.71</td>
+        <td>04.01</td>
+        <td>28.48</td>
+        <td>05.70</td>
+        <td>27.24</td>
+        <td>04.50</td>
+        <td>22.56</td>
+        <td>00.03</td>
+        <td>03.57</td>
         <td>01.77</td>
     </tr>
     <tr>
@@ -457,6 +564,14 @@ The models were compared against a collection of existing open-source Arabic mod
         <td>66.68</td>
         <td>00.87</td>
         <td>10.52</td>
+        <td>04.02</td>
+        <td>25.29</td>
+        <td>06.66</td>
+        <td>23.46</td>
+        <td>20.14</td>
+        <td>47.87</td>
+        <td>0.04</td>
+        <td>04.77</td>
         <td>01.92</td>
     </tr>
     <tr>
@@ -467,6 +582,14 @@ The models were compared against a collection of existing open-source Arabic mod
         <td>40.23</td>
         <td>00.44</td>
         <td>11.33</td>
+        <td>01.05</td>
+        <td>19.24</td>
+        <td>06.92</td>
+        <td>36.03</td>
+        <td>11.05</td>
+        <td>44.55</td>
+        <td>00.06</td>
+        <td>04.74</td>
         <td>02.28</td>
     </tr>
     <tr>
@@ -477,6 +600,14 @@ The models were compared against a collection of existing open-source Arabic mod
         <td>59.58</td>
         <td>00.98</td>
         <td>16.70</td>
+        <td>00.81</td>
+        <td>20.23</td>
+        <td>08.73</td>
+        <td>40.76</td>
+        <td>14.02</td>
+        <td>48.28</td>
+        <td>00.12</td>
+        <td>06.32</td>
         <td>02.80</td>
     </tr>
     <tr>
@@ -487,6 +618,14 @@ The models were compared against a collection of existing open-source Arabic mod
         <td>59.87</td>
         <td>03.10</td>
         <td>19.16</td>
+        <td>01.72</td>
+        <td>24.35</td>
+        <td>05.18</td>
+        <td>36.96</td>
+        <td>08.23</td>
+        <td>43.57</td>
+        <td>00.17</td>
+        <td>09.14</td>
         <td>13.81</td>
     </tr>
     <tr>
@@ -497,6 +636,14 @@ The models were compared against a collection of existing open-source Arabic mod
         <td>44.08</td>
         <td>00.92</td>
         <td>14.19</td>
+        <td>01.46</td>
+        <td>23.82</td>
+        <td>08.89</td>
+        <td>33.08</td>
+        <td>11.85</td>
+        <td>35.51</td>
+        <td>00.11</td>
+        <td>06.02</td>
         <td>01.28</td>
     </tr>
     <tr>
@@ -507,13 +654,530 @@ The models were compared against a collection of existing open-source Arabic mod
         <td><b>81.89</td>
         <td><b>28.08</td>
         <td><b>50.48</td>
+        <td><b>18.16</td>
+        <td><b>43.91</td>
+        <td><b>18.63</td>
+        <td><b>47.53</td>
+        <td><b>29.98</td>
+        <td><b>58.26</td>
+        <td><b>22.08</td>
+        <td><b>34.17</td>
         <td><b>59.76</td>
     </tr>
+    <tr style="border-top: 4px solid;"></tr>
+    <tr>
+        <td><a href="https://huggingface.co/inceptionai/jais-family-30b-8k-chat" target="_blank">jais-family-30b-8k-chat</a></td>
+        <td>51.88</td>
+        <td>35.61</td>
+        <td>65.67</td>
+        <td>56.73</td>
+        <td>01.10</td>
+        <td>14.40</td>
+        <td>01.67</td>
+        <td>23.37</td>
+        <td>08.52</td>
+        <td>35.41</td>
+        <td>13.71</td>
+        <td>41.33</td>
+        <td>00.05</td>
+        <td>04.48</td>
+        <td>00.46</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/google/gemma-2-27b-it" target="_blank">gemma-2-27b-it</a></td>
+        <td>36.47</td>
+        <td>37.04</td>
+        <td>35.78</td>
+        <td>57.59</td>
+        <td>00.67</td>
+        <td>13.04</td>
+        <td>01.74</td>
+        <td>24.63</td>
+        <td>05.17</td>
+        <td>37.08</td>
+        <td>07.36</td>
+        <td>42.49</td>
+        <td>00.03</td>
+        <td>04.94</td>
+        <td>11.10</td>
+    </tr>
+    <tr>
+        <td><strong><a href="https://huggingface.co/MBZUAI-Paris/Atlas-Chat-27B" target="_blank">Atlas-Chat-27B</a></strong></td>
+        <td><b>61.95</td>
+        <td><b>48.37</td>
+        <td><b>75.67</td>
+        <td>73.00</td>
+        <td><b>29.55</td>
+        <td><b>51.74</td>
+        <td><b>19.66</td>
+        <td><b>45.65</td>
+        <td><b>20.34</td>
+        <td><b>49.19</td>
+        <td><b>31.61</td>
+        <td><b>59.37</td>
+        <td><b>33.03</td>
+        <td><b>40.95</td>
+        <td><b>60.70</td>
+    </tr>
 
 
     
 </table>
+-->
+
+## Evaluation
+The Atlas-Chat models were evaluated on a comprehensive suite of tasks using various datasets and benchmarks to assess their performance across multiple dimensions. These included tasks such as:
+
+* **DarijaMMLU:** A Darija version of ArabicMMLU and MMLU benchmarks translated from MSA and English respectively.
+* **DarijaHellaSwag:** A Darija version of HellaSwag.
+* **Belebele Ary_Arab:** Belebele is a multiple-choice machine reading comprehension dataset published by Facebook spanning 122 language variants. The Evaluation is done on the Ary_Arab part of Belebele that refers to Darija.
+* **DarijaAlpacaEval:** A Darija version of AlpacaEval translated to Darija and adapted to the Moroccan culture.
+* **Sentiment Analysis.**
+* **Translation:** Including six directions and four languages: Darija, MSA, English and French.
+* **Transliteration:** Transforming a sentence from Darija (written in Arabic characters) to Arabizi (Written in Latin characters) and vice-versa.
+* **Summarization.**
 
+The models were compared against a collection of existing open-source Arabic models to gauge their effectiveness, with a particular focus on performance in Darija. All scores are based on zero-shot performance. The prompts are written mainly in Darija. The metric used for DarijaMMLU, DarijaHellaSwag, Belebele Ary and Sentiment Analysis is the normalized accuracy. We used [Language Model Evaluation Harness](https://github.com/MBZUAI-Paris/lm-evaluation-harness-atlas-chat) to conduct these evaluations.
+
+**LLMs Benchmarks:**
+<table>
+    <tr>
+        <td>Model</td>
+        <td><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaMMLU" target="_blank">DarijaMMLU</a></td>
+        <td><a href="MBZUAI-Paris/DarijaHellaSwag" target="_blank">DarijaHellaSwag</a></td>
+        <td ><a href="https://huggingface.co/datasets/facebook/belebele/viewer/ary_Arab" target="_blank">Belebele Ary</a></td>
+        <td ><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaAlpacaEval" target="_blank">DarijaAlpacaEval</a></td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/inceptionai/jais-family-1p3b-chat" target="_blank">jais-family-1p3b-chat</a></td>
+        <td>35.39</td>
+        <td>32.51</td>
+        <td>38.33</td>
+        <td>35.56</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/inceptionai/jais-family-2p7b-chat" target="_blank">jais-family-2p7b-chat</a></td>
+        <td>37.44</td>
+        <td>34.49</td>
+        <td>44.11</td>
+        <td>52.97</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/google/gemma-2-2b-it" target="_blank">gemma-2-2b-it</a></td>
+        <td>28.58</td>
+        <td>32.42</td>
+        <td>25.22</td>
+        <td>58.67</td>
+    </tr>
+    <tr>
+        <td><a href="meta-llama/Llama-3.2-1B-Instruct" target="_blank">Llama-3.2-1B-Instruct</a></td>
+        <td>27.66</td>
+        <td>26.88</td>
+        <td>28.89</td>
+        <td>23.57</td>
+    </tr>
+    <tr>
+        <td><a href="meta-llama/Llama-3.2-3B-Instruct" target="_blank">Llama-3.2-3B-Instruct</a></td>
+        <td>32.60</td>
+        <td>28.33</td>
+        <td>38.00</td>
+        <td>47.62</td>
+    </tr>
+    <tr>
+        <td><strong><a href="https://huggingface.co/MBZUAI-Paris/Atlas-Chat-2B" target="_blank">Atlas-Chat-2B</a></strong></td>
+        <td><b>44.97</b></td>
+        <td><b>41.48</b></td>
+        <td><b>53.89</b></td>
+        <td><b>92.31</b></td>
+    </tr>
+    <tr style="border-top: 4px solid;"></tr>
+    <tr>
+        <td><a href="https://huggingface.co/inceptionai/jais-family-6p7b-chat" target="_blank">jais-family-6p7b-chat</a></td>
+        <td>39.96</td>
+        <td>41.57</td>
+        <td>51.22</td>
+        <td>65.18</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/inceptionai/jais-adapted-7b-chat" target="_blank">jais-adapted-7b-chat</a></td>
+        <td>39.30</td>
+        <td>35.19</td>
+        <td>43.67</td>
+        <td>61.84</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/inceptionai/jais-family-13b-chat" target="_blank">jais-family-13b-chat</a></td>
+        <td>45.11</td>
+        <td>43.90</td>
+        <td>58.67</td>
+        <td>69.93</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/inceptionai/jais-adapted-13b-chat" target="_blank">jais-adapted-13b-chat</a></td>
+        <td>45.20</td>
+        <td>40.65</td>
+        <td>49.67</td>
+        <td>77.52</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/FreedomIntelligence/AceGPT-7B-chat" target="_blank">AceGPT-7b-chat</a></td>
+        <td>35.98</td>
+        <td>36.57</td>
+        <td>30.11</td>
+        <td>47.31</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/FreedomIntelligence/AceGPT-13B-chat" target="_blank">AceGPT-13b-chat</a></td>
+        <td>41.09</td>
+        <td>38.35</td>
+        <td>33.11</td>
+        <td>52.79</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/google/gemma-2-9b-it" target="_blank">gemma-2-9b-it</a></td>
+        <td>35.91</td>
+        <td>42.43</td>
+        <td>31.00</td>
+        <td>90.86</td>
+    </tr>
+    <tr>
+        <td><a href="meta-llama/Meta-Llama-3.1-8B-Instruct" target="_blank">Llama-3.1-8B-Instruct</a></td>
+        <td>44.13</td>
+        <td>38.24</td>
+        <td>47.00</td>
+        <td>78.08</td>
+    </tr>
+    <tr>
+        <td><strong><a href="https://huggingface.co/MBZUAI-Paris/Atlas-Chat-9B" target="_blank">Atlas-Chat-9B</a></strong></td>
+        <td><b>58.23</b></td>
+        <td><b>57.75</b></td>
+        <td><b>74.56</b></td>
+        <td><b>95.62</b></td>
+    </tr>
+    <tr style="border-top: 4px solid;"></tr>
+    <tr>
+        <td><a href="https://huggingface.co/inceptionai/jais-family-30b-8k-chat" target="_blank">jais-family-30b-8k-chat</a></td>
+        <td>51.88</td>
+        <td>35.61</td>
+        <td>65.67</td>
+        <td>24.64</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/google/gemma-2-27b-it" target="_blank">gemma-2-27b-it</a></td>
+        <td>36.47</td>
+        <td>37.04</td>
+        <td>35.78</td>
+        <td>95.07</td>
+    </tr>
+    <tr>
+        <td><strong><a href="https://huggingface.co/MBZUAI-Paris/Atlas-Chat-27B" target="_blank">Atlas-Chat-27B</a></strong></td>
+        <td><b>61.95</b></td>
+        <td><b>48.37</b></td>
+        <td><b>75.67</b></td>
+        <td><b>96.58</b></td>
+    </tr>
+</table>
+
+**Standard NLP Tasks:**
+<table>
+    <tr>
+        <td rowspan="2">Model</td>
+        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">DODa-10k (Translation)</a></td>
+        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">MADAR (Translation)</a></td>
+        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">FLORES+ (Translation)</a></td>
+        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">NLLB-Seed (Translation)</a></td>
+        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">DODa-10k (Transliteration)</a></td>
+        <td rowspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">MArSum (Summarization)</a><br/>(LLM as a judge)</td>
+        <td rowspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">Sentiment Analysis</a></td>
+    </tr>
+    <tr>
+        <td>BLEU</td>
+        <td>chrF</td>
+        <td>BLEU</td>
+        <td>chrF</td>
+        <td>BLEU</td>
+        <td>chrF</td>
+        <td>BLEU</td>
+        <td>chrF</td>
+        <td>BLEU</td>
+        <td>chrF</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/inceptionai/jais-family-1p3b-chat" target="_blank">jais-family-1p3b-chat</a></td>
+        <td>00.13</td>
+        <td>06.18</td>
+        <td>00.50</td>
+        <td>15.43</td>
+        <td>02.44</td>
+        <td>19.14</td>
+        <td>01.99</td>
+        <td>12.60</td>
+        <td>00.01</td>
+        <td>03.01</td>
+        <td>00.50</td>
+        <td>45.29</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/inceptionai/jais-family-2p7b-chat" target="_blank">jais-family-2p7b-chat</a></td>
+        <td>00.25</td>
+        <td>07.46</td>
+        <td>00.62</td>
+        <td>16.36</td>
+        <td>04.25</td>
+        <td>18.22</td>
+        <td>03.10</td>
+        <td>08.19</td>
+        <td>00.01</td>
+        <td>03.27</td>
+        <td>00.90</td>
+        <td>51.56</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/google/gemma-2-2b-it" target="_blank">gemma-2-2b-it</a></td>
+        <td>00.10</td>
+        <td>04.96</td>
+        <td>00.12</td>
+        <td>06.66</td>
+        <td>01.55</td>
+        <td>18.59</td>
+        <td>02.78</td>
+        <td>23.69</td>
+        <td>00.01</td>
+        <td>02.08</td>
+        <td>06.80</td>
+        <td>53.36</td>
+    </tr>
+    <tr>
+        <td><a href="meta-llama/Llama-3.2-1B-Instruct" target="_blank">Llama-3.2-1B-Instruct</a></td>
+        <td>00.07</td>
+        <td>05.95</td>
+        <td>00.80</td>
+        <td>18.71</td>
+        <td>04.53</td>
+        <td>18.39</td>
+        <td>04.52</td>
+        <td>17.06</td>
+        <td>00.02</td>
+        <td>03.74</td>
+        <td>08.23</td>
+        <td>46.27</td>
+    </tr>
+    <tr>
+        <td><a href="meta-llama/Llama-3.2-3B-Instruct" target="_blank">Llama-3.2-3B-Instruct</a></td>
+        <td>00.62</td>
+        <td>13.67</td>
+        <td>01.18</td>
+        <td>22.12</td>
+        <td>08.59</td>
+        <td>35.21</td>
+        <td>13.75</td>
+        <td>43.63</td>
+        <td>00.21</td>
+        <td>09.68</td>
+        <td>08.23</td>
+        <td>49.20</td>
+    </tr>
+    <tr>
+        <td><strong><a href="https://huggingface.co/MBZUAI-Paris/Atlas-Chat-2B" target="_blank">Atlas-Chat-2B</a></strong></td>
+        <td><b>22.76</td>
+        <td><b>44.86</td>
+        <td><b>16.67</td>
+        <td><b>41.64</td>
+        <td><b>14.92</td>
+        <td><b>43.03</td>
+        <td><b>23.88</td>
+        <td><b>52.19</td>
+        <td><b>08.18</td>
+        <td><b>21.54</td>
+        <td><b>55.22</td>
+        <td><b>73.99</td>
+    </tr>
+    <tr style="border-top: 4px solid;"></tr>
+    <tr>
+        <td><a href="https://huggingface.co/inceptionai/jais-family-6p7b-chat" target="_blank">jais-family-6p7b-chat</a></td>
+        <td>00.73</td>
+        <td>11.85</td>
+        <td>01.88</td>
+        <td>23.22</td>
+        <td>04.25</td>
+        <td>18.22</td>
+        <td>04.62</td>
+        <td>20.22</td>
+        <td>00.02</td>
+        <td>03.79</td>
+        <td>03.02</td>
+        <td>56.78</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/inceptionai/jais-adapted-7b-chat" target="_blank">jais-adapted-7b-chat</a></td>
+        <td>00.60</td>
+        <td>09.43</td>
+        <td>03.45</td>
+        <td>25.88</td>
+        <td>07.25</td>
+        <td>23.21</td>
+        <td>01.25</td>
+        <td>02.22</td>
+        <td>00.04</td>
+        <td>03.24</td>
+        <td>02.82</td>
+        <td>52.72</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/inceptionai/jais-family-13b-chat" target="_blank">jais-family-13b-chat</a></td>
+        <td>00.92</td>
+        <td>11.71</td>
+        <td>04.01</td>
+        <td>28.48</td>
+        <td>05.70</td>
+        <td>27.24</td>
+        <td>04.50</td>
+        <td>22.56</td>
+        <td>00.03</td>
+        <td>03.57</td>
+        <td>01.77</td>
+        <td>41.73</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/inceptionai/jais-adapted-13b-chat" target="_blank">jais-adapted-13b-chat</a></td>
+        <td>00.87</td>
+        <td>10.52</td>
+        <td>04.02</td>
+        <td>25.29</td>
+        <td>06.66</td>
+        <td>23.46</td>
+        <td>20.14</td>
+        <td>47.87</td>
+        <td>0.04</td>
+        <td>04.77</td>
+        <td>01.92</td>
+        <td>66.68</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/FreedomIntelligence/AceGPT-7B-chat" target="_blank">AceGPT-7b-chat</a></td>
+        <td>00.44</td>
+        <td>11.33</td>
+        <td>01.05</td>
+        <td>19.24</td>
+        <td>06.92</td>
+        <td>36.03</td>
+        <td>11.05</td>
+        <td>44.55</td>
+        <td>00.06</td>
+        <td>04.74</td>
+        <td>02.28</td>
+        <td>40.23</td>
+    </tr> 
+    <tr>
+        <td><a href="https://huggingface.co/FreedomIntelligence/AceGPT-13B-chat" target="_blank">AceGPT-13b-chat</a></td>
+        <td>00.98</td>
+        <td>16.70</td>
+        <td>00.81</td>
+        <td>20.23</td>
+        <td>08.73</td>
+        <td>40.76</td>
+        <td>14.02</td>
+        <td>48.28</td>
+        <td>00.12</td>
+        <td>06.32</td>
+        <td>02.80</td>
+        <td>59.58</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/google/gemma-2-9b-it" target="_blank">gemma-2-9b-it</a></td>
+        <td>03.10</td>
+        <td>19.16</td>
+        <td>01.72</td>
+        <td>24.35</td>
+        <td>05.18</td>
+        <td>36.96</td>
+        <td>08.23</td>
+        <td>43.57</td>
+        <td>00.17</td>
+        <td>09.14</td>
+        <td>13.81</td>
+            <td>59.87</td>
+    </tr>
+    <tr>
+        <td><a href="meta-llama/Meta-Llama-3.1-8B-Instruct" target="_blank">Llama-3.1-8B-Instruct</a></td>
+        <td>00.92</td>
+        <td>14.19</td>
+        <td>01.46</td>
+        <td>23.82</td>
+        <td>08.89</td>
+        <td>33.08</td>
+        <td>11.85</td>
+        <td>35.51</td>
+        <td>00.11</td>
+        <td>06.02</td>
+        <td>16.14</td>
+        <td>44.08</td>
+    </tr>
+    <tr>
+        <td><strong><a href="https://huggingface.co/MBZUAI-Paris/Atlas-Chat-9B" target="_blank">Atlas-Chat-9B</a></strong></td>
+        <td><b>28.08</td>
+        <td><b>50.48</td>
+        <td><b>18.16</td>
+        <td><b>43.91</td>
+        <td><b>18.63</td>
+        <td><b>47.53</td>
+        <td><b>29.98</td>
+        <td><b>58.26</td>
+        <td><b>22.08</td>
+        <td><b>34.17</td>
+        <td><b>59.76</td>
+        <td><b>81.89</td>
+    </tr>
+    <tr style="border-top: 4px solid;"></tr>
+    <tr>
+        <td><a href="https://huggingface.co/inceptionai/jais-family-30b-8k-chat" target="_blank">jais-family-30b-8k-chat</a></td>
+        <td>01.10</td>
+        <td>14.40</td>
+        <td>01.67</td>
+        <td>23.37</td>
+        <td>08.52</td>
+        <td>35.41</td>
+        <td>13.71</td>
+        <td>41.33</td>
+        <td>00.05</td>
+        <td>04.48</td>
+        <td>00.46</td>
+        <td>56.73</td>
+    </tr>
+    <tr>
+        <td><a href="https://huggingface.co/google/gemma-2-27b-it" target="_blank">gemma-2-27b-it</a></td>
+        <td>00.67</td>
+        <td>13.04</td>
+        <td>01.74</td>
+        <td>24.63</td>
+        <td>05.17</td>
+        <td>37.08</td>
+        <td>07.36</td>
+        <td>42.49</td>
+        <td>00.03</td>
+        <td>04.94</td>
+        <td>11.10</td>
+        <td>57.59</td>
+    </tr>
+    <tr>
+        <td><strong><a href="https://huggingface.co/MBZUAI-Paris/Atlas-Chat-27B" target="_blank">Atlas-Chat-27B</a></strong></td>
+        <td><b>29.55</td>
+        <td><b>51.74</td>
+        <td><b>19.66</td>
+        <td><b>45.65</td>
+        <td><b>20.34</td>
+        <td><b>49.19</td>
+        <td><b>31.61</td>
+        <td><b>59.37</td>
+        <td><b>33.03</td>
+        <td><b>40.95</td>
+        <td><b>60.70</td>
+        <td>73.00</td>
+    </tr>
+
+
+    
+</table>
 
 ## Usage and Limitations