text: 196,18,1034363142757908480,809336237194706944,Rico Sennrich,"Why Self-Attention? New #EMNLP2018 paper doing targeted evaluation of different NMT architectures. Analysis shows that word sense disambiguation is a big strength of Transformer, but not long-distance dependencies, as speculated by Vaswani et al. (2017). <LINK>",https://arxiv.org/abs/1808.08946,"Recently, non-recurrent architectures (convolutional, self-attentional) have outperformed RNNs in neural machine translation. CNNs and self-attentional networks can connect distant words via shorter network paths than RNNs, and it has been speculated that this improves their ability to model long-range dependencies. However, this theoretical argument has not been tested empirically, nor have alternative explanations for their strong performance been explored in-depth. We hypothesize that the strong performance of CNNs and self-attentional networks could also be due to their ability to extract semantic features from the source text, and we evaluate RNNs, CNNs and self-attention networks on two tasks: subject-verb agreement (where capturing long-range dependencies is required) and word sense disambiguation (where semantic feature extraction is required). Our experimental results show that: 1) self-attentional networks and CNNs do not outperform RNNs in modeling subject-verb agreement over long distances; 2) self-attentional networks perform distinctly better than RNNs and CNNs on word sense disambiguation. ","Why Self-Attention? A Targeted Evaluation of Neural Machine Translation