text: 6,115,1437655382994956292,795620317532143616,Jonas Pfeiffer,"In our paper xGQA: Cross-Lingual Visual Question Answering We propose a new multilingual multimodal benchmark, covering 7 new typologically diverse languages 📃 <LINK> 🌐 <LINK> @GregorGeigle @ashkamath20 @jmsteitz @stefanroth @licwu @IGurevych <LINK> We further propose new adapter-based approaches to adapt multimodal transformer-based models to become multilingual, and—vice versa—multilingual models to become multimodal. <LINK> While our adapter-based architecture outperforms the SotA M3P model in cross-lingual zero-shot scenarios, the overall transfer performance remains low across the board, with an average drop of around 38 accuracy points across target languages. <LINK> This demonstrates the inherent difficulty of the task, even though the corresponding questions are ar- guably simple, containing only 8.5 words on average. In few-shot scenarios we find that utilizing an increasing amount of data instances in the target language consistently improves accuracy, culminating in an improvement of up to 20 accuracy points when specializing the model with only 48 images in the target language. <LINK> Our results suggest that simple cross-lingual transfer of multimodal models yields latent multilingual multimodal misalignment, which can only be partially recovered through few-shot learning, calling for more sophisticated methods for vision and multilingual language modeling. @srchvrs I agree that it’s surprising that it works well for text-only tasks—which has been throughly investigated in the past (i.e. <LINK> @PDufter)—however, what surprised me even more, was this doesn’t seem to translate to scenarios where we add an additional modality.",https://arxiv.org/abs/2109.06082,"Recent advances in multimodal vision and language modeling have predominantly focused on the English language, mostly due to the lack of multilingual multimodal datasets to steer modeling efforts. In this work, we address this gap and provide xGQA, a new multilingual evaluation benchmark for the visual question answering task. We extend the established English GQA dataset to 7 typologically diverse languages, enabling us to detect and explore crucial challenges in cross-lingual visual question answering. We further propose new adapter-based approaches to adapt multimodal transformer-based models to become multilingual, and -- vice versa -- multilingual models to become multimodal. Our proposed methods outperform current state-of-the-art multilingual multimodal models (e.g., M3P) in zero-shot cross-lingual settings, but the accuracy remains low across the board; a performance drop of around 38 accuracy points in target languages showcases the difficulty of zero-shot cross-lingual transfer for this task. Our results suggest that simple cross-lingual transfer of multimodal models yields latent multilingual multimodal misalignment, calling for more sophisticated methods for vision and multilingual language modeling. ",xGQA: Cross-Lingual Visual Question Answering,7,"['In our paper\n\nxGQA: Cross-Lingual Visual Question Answering\n\nWe propose a new multilingual multimodal benchmark, covering 7 new typologically diverse languages\n📃 <LINK>\n🌐 <LINK>\n@GregorGeigle @ashkamath20 @jmsteitz @stefanroth @licwu @IGurevych <LINK>', 'We further propose new adapter-based approaches to adapt multimodal transformer-based models to become multilingual, and—vice versa—multilingual models to become multimodal. https://t.co/omuxSK3E17', 'While our adapter-based architecture outperforms the SotA  M3P model in cross-lingual zero-shot scenarios, the overall transfer performance remains low across the board, with an average drop of around 38 accuracy points across target languages. https://t.co/EIjLKsQ01e', 'This demonstrates the inherent difficulty of the task, even though the corresponding questions are ar- guably simple, containing only 8.5 words on average.', 'In few-shot scenarios we find that utilizing an increasing amount of data instances in the target language consistently improves accuracy, culminating in an improvement of up to 20 accuracy points when specializing the model with only 48 images in the target language. https://t.co/PL68ttBlgT', 'Our results suggest that simple cross-lingual transfer of multimodal models yields latent multilingual multimodal misalignment, which can only be partially recovered through few-shot learning, calling for more sophisticated methods for vision and multilingual language modeling.', '@srchvrs I agree that it’s surprising that it works well for text-only tasks—which has been throughly investigated in the past (i.e. https://t.co/gDLr8gFZeK @PDufter)—however, what surprised me even more, was this doesn’t seem to translate to scenarios where we add an additional modality.']",21,09,1665