text: 383,0,1249779748961767425,2785337469,Sebastian Ruder,"I'm excited to announce XTREME, a new benchmark that covers 9 tasks and 40 typologically diverse languages. Paper: <LINK> Blog post: <LINK> Code: <LINK> <LINK> XTREME evaluates models on their capability to do zero-shot cross-lingual transfer when fine-tuned on English. In the paper, we conduct experiments with several state-of-the-art models that have been pre-trained on large multilingual corpora. <LINK> Overall, we find that there is a large gap to English and human performance and a lot of potential for improvement, particularly on syntactic tasks. See below for instance for the performance of XLM-R across tasks and languages. <LINK> This work would have not been possible without the contributions of many amazing people including @JunjieHu12 @orf_bnw @gneubig Melvin, Aditya @dhgarrette @JonClarkSeattle and others. @gena_d Argh. The last letter of the website got lost during copy-pasting. 😅 Here's the correct link: <LINK> @boknilev Yes. For all but TyDiQA-GoldP, human performance is based on English. That comes with the obvious caveat that performance will differ across languages. IMO for datasets that are derived using translation (XNLI, PAWS-X, XQuAD), human perf should be very similar across languages.",https://arxiv.org/abs/2003.11080,"Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing. To this end, we introduce the Cross-lingual TRansfer Evaluation of Multilingual Encoders XTREME benchmark, a multi-task benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models, particularly on syntactic and sentence retrieval tasks. There is also a wide spread of results across languages. We release the benchmark to encourage research on cross-lingual learning methods that transfer linguistic knowledge across a diverse and representative set of languages and tasks. ","XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating