text: 318,19,1354837390188109831,1515424688,Armen Aghajanyan,"I'm happy to present our new paper MUPPET (<LINK>), arguing for an additional stage between pre-training and fine-tuning, called pre-finetuning which uses massively multi-task learning (~50 tasks) to further refine representations. Recent work has shown gains from MTL/multi-stage fine-tuning, but it can be hard to know which intermediate tasks will best transfer. We show that multi-task supervised tuning is effective if done at scale (# tasks), removing the need to pre-select the best intermediate tasks. Our multi-task set up consists of 46 datasets across 4 task types with close to 5 million supervised samples, using a classification head for each classification dataset, and unified heads for MRC and CommonSense tasks. <LINK> After solving practical problems (loss scaling, heterogeneous batches, etc) we pre-finetune RoBERTa variants and BART. We first look at fine-tuning MUPPET over datasets available in the pre-finetuning regime. MUPPET unanimously improves over it’s base model for GLUE/SQuAD. <LINK> MUPPET also improves over it’s base model for sentence prediction, commonsense, and summarization tasks. <LINK> To show that MUPPET representations are more generalizable, we also measure MUPPET performance over datasets not available in the pre-finetuning regime. MUPPET once again improves consistently over it’s only pre-trained counterparts. <LINK> MTL historically has given inconclusive results. So why does MUPPET work? Turns out scale is fundamental for MTL. There exists a critical # of tasks (~15) under which pre-finetuning degrades representations. But over this critical point linearly improves representations. <LINK> As another side-effect, MUPPET variants of pre-trained models show much better data-efficiency for downstream fine-tuning. This was joint work with great authors Anchit Gupta, @AkshatS07, Xilun Chen, @LukeZettlemoyer, @sonalsgupta",https://arxiv.org/abs/2101.11038,"We propose pre-finetuning, an additional large-scale learning stage between language model pre-training and fine-tuning. Pre-finetuning is massively multi-task learning (around 50 datasets, over 4.8 million total labeled examples), and is designed to encourage learning of representations that generalize better to many different tasks. We show that pre-finetuning consistently improves performance for pretrained discriminators (e.g.~RoBERTa) and generation models (e.g.~BART) on a wide range of tasks (sentence prediction, commonsense reasoning, MRC, etc.), while also significantly improving sample efficiency during fine-tuning. We also show that large-scale multi-tasking is crucial; pre-finetuning can hurt performance when few tasks are used up until a critical point (usually above 15) after which performance improves linearly in the number of tasks. ",Muppet: Massive Multi-task Representations with Pre-Finetuning,9,"[""I'm happy to present our new paper MUPPET (<LINK>), arguing for an additional stage between pre-training and fine-tuning, called pre-finetuning which uses massively multi-task learning (~50 tasks) to further refine representations."", 'Recent work has shown gains from MTL/multi-stage fine-tuning, but it can be hard to know which intermediate tasks will best transfer. We show that multi-task supervised tuning is effective if done at scale (# tasks), removing the need to pre-select the best intermediate tasks.', 'Our multi-task set up consists of 46 datasets across 4 task types with close to 5 million supervised samples, using a classification head for each classification dataset, and unified heads for MRC and CommonSense tasks. https://t.co/4Nw4oQlbRo', 'After solving practical problems (loss scaling, heterogeneous batches, etc) we pre-finetune RoBERTa variants and BART. We first look at fine-tuning MUPPET over datasets available in the pre-finetuning regime. MUPPET unanimously improves over it’s base model for GLUE/SQuAD. https://t.co/UtrMe18vnW', 'MUPPET also improves over it’s base model for sentence prediction, commonsense, and summarization tasks. https://t.co/wTNwum3BB5', 'To show that MUPPET representations are more generalizable, we also measure MUPPET performance over datasets not available in the pre-finetuning regime. MUPPET once again improves consistently over it’s only pre-trained counterparts. https://t.co/lrYvGnZtBD', 'MTL historically has given inconclusive results. So why does MUPPET work? Turns out scale is fundamental for MTL. There exists a critical # of tasks (~15) under which pre-finetuning degrades representations. But over this critical point linearly improves representations. https://t.co/AJmjtfHvxp', 'As another side-effect, MUPPET variants of pre-trained models show much better data-efficiency for downstream fine-tuning.', 'This was joint work with great authors Anchit Gupta, @AkshatS07, Xilun Chen, @LukeZettlemoyer, @sonalsgupta']",21,01,1880