text: 190,313,1316291822952484866,272682658,Robert Hawkins,"🔥 more EMNLP content for your news feed🔥 we were interested in how recent neural language models integrate lexical/semantic information into their knowledge of grammatical constructions. obviously, we needed to study verb biases.  link: <LINK> <LINK> in collab with the brilliant @TakaYamakoshi, @adelegoldberg1, and Tom Griffiths, we present the DAIS corpus of 50K acceptability ratings for 5K sentence pairs in the dative alternation, containing 200 different verbs: <LINK> e.g. how much better is A than B here? A: Ava gave her friend a book. B: Ava gave a book to her friend. what about with a different verb? A: Ava explained her friend a problem. B: Ava explained a problem to her friend. This preference varies widely across verbs! <LINK> It turns out that larger models account for these verb-specific preferences better than smaller models, and transformer architectures (e.g. GPT-2) do so better than recurrent architectures (e.g. LSTMs), even with comparable param numbers. <LINK> To begin to understand why, we probed the hidden states of these models as they proceed through the sentence. Immediately upon seeing the verb, GPT2's hidden layers already contains a surprising amount of information about human ratings, and this improved after seeing the 1st arg <LINK> as a side-note, we also analyzed how these representations are organized as a function of layer depth in GPT2 -- at the verb, human ratings are more decodable at lower layers (corresponding to lexical info?), while later in the sentence, it's only decodable at intermediate layers <LINK> After a thoughtful suggestion from a reviewer, we also compared these models on an older natural speech corpus (assembled by the inimitable Joan Bresnan) and found similar rankings (vs. the roughly 92% previously achieved from hand-annotated features) <LINK> <LINK> We speculate that transformers may be able to learn a better layer-wise pipeline for integrating lexically specific information with representations of higher-level syntactic constructions like the dative & double-object, but further work is needed! We hope our dataset will be useful for future work in both psycholinguistics and NLP, as we push our models to account for subtler phenomena and understand how they deviate from human expectations.",https://arxiv.org/abs/2010.02375,"Languages typically provide more than one grammatical construction to express certain types of messages. A speaker's choice of construction is known to depend on multiple factors, including the choice of main verb -- a phenomenon known as \emph{verb bias}. Here we introduce DAIS, a large benchmark dataset containing 50K human judgments for 5K distinct sentence pairs in the English dative alternation. This dataset includes 200 unique verbs and systematically varies the definiteness and length of arguments. We use this dataset, as well as an existing corpus of naturally occurring data, to evaluate how well recent neural language models capture human preferences. Results show that larger models perform better than smaller models, and transformer architectures (e.g. GPT-2) tend to out-perform recurrent architectures (e.g. LSTMs) even under comparable parameter and training settings. Additional analyses of internal feature representations suggest that transformers may better integrate specific lexical information with grammatical constructions. ",Investigating representations of verb bias in neural language models,9,"['🔥 more EMNLP content for your news feed🔥\n\nwe were interested in how recent neural language models integrate lexical/semantic information into their knowledge of grammatical constructions. obviously, we needed to study verb biases. \n\nlink: <LINK> <LINK>', 'in collab with the brilliant @TakaYamakoshi, @adelegoldberg1, and Tom Griffiths, we present the DAIS corpus of 50K acceptability ratings for 5K sentence pairs in the dative alternation, containing 200 different verbs:\n\nhttps://t.co/k51FOANwoI', 'e.g. how much better is A than B here?\n\nA: Ava gave her friend a book.\nB: Ava gave a book to her friend.\n\nwhat about with a different verb?\n\nA: Ava explained her friend a problem.\nB: Ava explained a problem to her friend.\n\nThis preference varies widely across verbs! https://t.co/vPPQ2vqknW', 'It turns out that larger models account for these verb-specific preferences better than smaller models, and transformer architectures (e.g. GPT-2) do so better than recurrent architectures (e.g. LSTMs), even with comparable param numbers. https://t.co/RBBm9Li0gJ', ""To begin to understand why, we probed the hidden states of these models as they proceed through the sentence. Immediately upon seeing the verb, GPT2's hidden layers already contains a surprising amount of information about human ratings, and this improved after seeing the 1st arg https://t.co/XBIan8K0JB"", ""as a side-note, we also analyzed how these representations are organized as a function of layer depth in GPT2 -- at the verb, human ratings are more decodable at lower layers (corresponding to lexical info?), while later in the sentence, it's only decodable at intermediate layers https://t.co/AEAAOsYqVW"", 'After a thoughtful suggestion from a reviewer, we also compared these models on an older natural speech corpus (assembled by the inimitable Joan Bresnan) and found similar rankings (vs. the roughly 92% previously achieved from hand-annotated features) https://t.co/TMQUg74IMl https://t.co/oWRFPlgxag', 'We speculate that transformers may be able to learn a better layer-wise pipeline for integrating lexically specific information with representations of higher-level syntactic constructions like the dative &amp; double-object, but further work is needed!', 'We hope our dataset will be useful for future work in both psycholinguistics and NLP, as we push our models to account for subtler phenomena and understand how they deviate from human expectations.']",20,10,2281