text: 237,123,1369688157558431749,1253758756304809984,Igor Mordatch,"What are the limits to the generalization of large pretrained transformer models? We find minimal fine-tuning (~0.1% of params) performs as well as training from scratch on a completely new modality! with @_kevinlu, @adityagrover_, @pabbeel paper: <LINK> 1/8 We take pretrained GPT-2 and freeze the attention & FF layers to obtain core of Frozen Pretrained Transformer (FPT). To adapt to new modality & task, we init *linear* input and output layers. Despite only training .1% of params, FPT matches performance of full transformer! 2/8 <LINK> We visualize the attention maps of the frozen transformer. Despite not finetuning the self-attention layers on the new modality, FPT is able to learn to attend to the relevant bits to compute an elementwise XOR with perfect accuracy for sequences of length up to 256. 3/8 <LINK> Compared to a randomly initialized frozen transformer, pretraining with language (FPT) yields large compute benefits, showing that, much like common practice for in-modality finetuning, we can save computation by starting from a pretrained model: 4/8 <LINK> What enables this transfer? We find that simply using a randomly initialized frozen transformer already greatly outperforms a randomly initialized frozen LSTM: 5/8 <LINK> Additionally, by incorporating various sources of pretraining supervision, even a little pretraining, for example learning to memorize bits, can help transfer: 6/8 <LINK> As we move from small specialist models to large generalist models, we’re excited by the potential for pretraining regimes that could train a universal computation engine. Simply adding more parameters and using a larger model already improves performance: 7/8 <LINK> For more details, see our paper on arXiv or play our demo/code on Github! arXiv: <LINK> Github: <LINK> 8/8 @rndmcnlly @_kevinlu @adityagrover_ @pabbeel Indeed Adam! That was our starting hypothesis. One thing we're still not quite clear on is whether this FPT interpreter largely performs generic input summarization or more extensive computation.",http://arxiv.org/abs/2103.05247,"We investigate the capability of a transformer pretrained on natural language to generalize to other modalities with minimal finetuning -- in particular, without finetuning of the self-attention and feedforward layers of the residual blocks. We consider such a model, which we call a Frozen Pretrained Transformer (FPT), and study finetuning it on a variety of sequence classification tasks spanning numerical computation, vision, and protein fold prediction. In contrast to prior works which investigate finetuning on the same modality as the pretraining dataset, we show that pretraining on natural language can improve performance and compute efficiency on non-language downstream tasks. Additionally, we perform an analysis of the architecture, comparing the performance of a random initialized transformer to a random LSTM. Combining the two insights, we find language-pretrained transformers can obtain strong performance on a variety of non-language tasks. ",Pretrained Transformers as Universal Computation Engines,9,"['What are the limits to the generalization of large pretrained transformer models?\n\nWe find minimal fine-tuning (~0.1% of params) performs as well as training from scratch on a completely new modality!\n\nwith @_kevinlu, @adityagrover_, @pabbeel\npaper: <LINK>\n\n1/8', 'We take pretrained GPT-2 and freeze the attention &amp; FF layers to obtain core of Frozen Pretrained Transformer (FPT).\n\nTo adapt to new modality &amp; task, we init *linear* input and output layers.\n\nDespite only training .1% of params, FPT matches performance of full transformer!\n\n2/8 https://t.co/1zjfCKm9rK', 'We visualize the attention maps of the frozen transformer.\n\nDespite not finetuning the self-attention layers on the new modality, FPT is able to learn to attend to the relevant bits to compute an elementwise XOR with perfect accuracy for sequences of length up to 256.\n\n3/8 https://t.co/tC1USSV4rS', 'Compared to a randomly initialized frozen transformer, pretraining with language (FPT) yields large compute benefits, showing that, much like common practice for in-modality finetuning, we can save computation by starting from a pretrained model:\n\n4/8 https://t.co/vmJlrBsfhg', 'What enables this transfer? We find that simply using a randomly initialized frozen transformer already greatly outperforms a randomly initialized frozen LSTM:\n\n5/8 https://t.co/17uMUZ1YO5', 'Additionally, by incorporating various sources of pretraining supervision, even a little pretraining, for example learning to memorize bits, can help transfer:\n\n6/8 https://t.co/KqOxlBgTY9', 'As we move from small specialist models to large generalist models, we’re excited by the potential for pretraining regimes that could train a universal computation engine.\n\nSimply adding more parameters and using a larger model already improves performance:\n\n7/8 https://t.co/sxtgfCUrGw', 'For more details, see our paper on arXiv or play our demo/code on Github!\n\narXiv: https://t.co/DtWGJ0Afh7\n\nGithub: https://t.co/9ts1FlqHFw\n\n8/8', ""@rndmcnlly @_kevinlu @adityagrover_ @pabbeel Indeed Adam! That was our starting hypothesis. One thing we're still not quite clear on is whether this FPT interpreter largely performs generic input summarization or more extensive computation.""]",21,03,2038