text: 296,79,1426119061671276549,15068044,Daniel Beck,"So we have a new paper, spearheaded by Joe Han, who just finished his Masters under mine and @trevorcohn's supervision. If you're looking into new ways of evaluating diversity in generation you should definitely take a look on what we propose here. (1/n) <LINK> @trevorcohn This will appear at INLG 2021. We propose a new way to evaluate diversity by grounding it on quality. Our rationale is that a diverse set of generated sentences is only good if it's also of good quality. How to evaluate both jointly? Use multiple references. (2/n) @trevorcohn We assume the ""gold standard"" of diversity is reflected in multiple references. So the goal of diversity is to ""cover"" the reference set. Assuming a sentence-level metric, we turn this into a maximum matching problem. (3/n) @trevorcohn This approach has a number of perks. For instance, a perfect model that generates the reference set exactly will achieve maximum score. But a model that generates N good sentences that are just slight variations of each other will be penalised. (4/n) It is also completely agnostic of whatever sentence-level metric is used, as long as it is bounded. So this can be used for a range of generation tasks. (5/n) One limitation is the reliance on the reference set: if it's not diverse enough then the method will not give preference to more diverse models, even if they give good quality sentences. (6/n) We also don't really test this against an implicit human evaluation of diversity (as quality metrics do). This is something we certainly would love to do in the future. However, it's not clear to us how to even define diversity intrinsically... (7/n) This is why we focus on the reference set. Our intuition is that it gives an ""extrinsic"" measure of diversity. It is not without its drawbacks (as I mention above) but it give us more information compared to other diversity metrics we know about. (8/n) Anyways, check our paper for more info. Happy to discuss more about diversity in generation if you have ideas =). (9/9) PS: yes, we know we messed up the references in the current arXiv version... we will update it shortly... =S",https://arxiv.org/abs/2108.05659,"Text generation from semantic graphs is traditionally performed with deterministic methods, which generate a unique description given an input graph. However, the generation problem admits a range of acceptable textual outputs, exhibiting lexical, syntactic and semantic variation. To address this disconnect, we present two main contributions. First, we propose a stochastic graph-to-text model, incorporating a latent variable in an encoder-decoder model, and its use in an ensemble. Second, to assess the diversity of the generated sentences, we propose a new automatic evaluation metric which jointly evaluates output diversity and quality in a multi-reference setting. We evaluate the models on WebNLG datasets in English and Russian, and show an ensemble of stochastic models produces diverse sets of generated sentences, while retaining similar quality to state-of-the-art models. ",Generating Diverse Descriptions from Semantic Graphs,10,"[""So we have a new paper, spearheaded by Joe Han, who just finished his Masters under mine and @trevorcohn's supervision.\n\nIf you're looking into new ways of evaluating diversity in generation you should definitely take a look on what we propose here. (1/n)\n<LINK>"", ""@trevorcohn This will appear at INLG 2021.\n\nWe propose a new way to evaluate diversity by grounding it on quality. Our rationale is that a diverse set of generated sentences is only good if it's also of good quality.\n\nHow to evaluate both jointly? Use multiple references. (2/n)"", '@trevorcohn We assume the ""gold standard"" of diversity is reflected in multiple references. So the goal of diversity is to ""cover"" the reference set. Assuming a sentence-level metric, we turn this into a maximum matching problem. (3/n)', '@trevorcohn This approach has a number of perks. For instance, a perfect model that generates the reference set exactly will achieve maximum score. But a model that generates N good sentences that are just slight variations of each other will be penalised. (4/n)', 'It is also completely agnostic of whatever sentence-level metric is used, as long as it is bounded. So this can be used for a range of generation tasks. (5/n)', ""One limitation is the reliance on the reference set: if it's not diverse enough then the method will not give preference to more diverse models, even if they give good quality sentences. (6/n)"", ""We also don't really test this against an implicit human evaluation of diversity (as quality metrics do). This is something we certainly would love to do in the future. However, it's not clear to us how to even define diversity intrinsically... (7/n)"", 'This is why we focus on the reference set. Our intuition is that it gives an ""extrinsic"" measure of diversity. It is not without its drawbacks (as I mention above) but it give us more information compared to other diversity metrics we know about. (8/n)', 'Anyways, check our paper for more info. Happy to discuss more about diversity in generation if you have ideas =). (9/9)', 'PS: yes, we know we messed up the references in the current arXiv version... we will update it shortly... =S']",21,08,2122