text: 163,180,1441074229366059008,2550133394,Mostafa Dehghani,"Check out our new paper, presenting insights on scaling Transformers/T5 within the context of the pretraining-finetuning setup.  <LINK> With @YiTayML, @Jeffy_Sailing, @LiamFedus, @samiraabnar, @hwchung27, @sharan0909, @DaniYogatama, @ashVaswani, and @metzlerd. <LINK> In search of the best scaling recipe, we ran an extensive search over different knobs and evaluated our modes w.r.t not only upstream performance (perplexity) but also the performance on different downstream tasks/benchmarks after finetuning. A few cool take-home messages: (1) If you, by default, take the T5-Base for your next amazing project to build on top of it, you may want to reconsider that. Transformer/T5 Base is not necessarily the Pareto-efficient configuration. We present some better alternatives. (2) Although model size is a key factor determining the scaling behavior of Transformers in pretraining, the model shape matters a lot for fine-tuning. So blindly scaling up may look good upstream, but can be disappointing when you finetune your model on a downstream task. (3) ""How to scale up?"" is pretty much depending on the region you're in. If you're working with Tiny models and want to go to Base size, the best recipe differs from the case of going from Large to XL. We have many more observations and insights. You should check the paper. Oh, btw, we are releasing 100+ pretrained checkpoints from our experiments, which will be an amazing playground for doing further analysis and interesting research :) You should also check @YiTayML's great thread that highlights more points from the paper: <LINK>",https://arxiv.org/abs/2109.10686,"There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come with an associated computational cost which have both financial and/or environmental impact. The goal of this paper is to present scaling insights from pretraining and finetuning Transformers. While Kaplan et al. presents a comprehensive study of the scaling behaviour of Transformer language models, the scope is only on the upstream (pretraining) loss. Therefore, it is still unclear if these set of findings transfer to downstream task within the context of the pretrain-finetune paradigm. The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient. To this end, we present improved scaling protocols whereby our redesigned models achieve similar downstream fine-tuning quality while having 50\% fewer parameters and training 40\% faster compared to the widely adopted T5-base model. We publicly release over 100 pretrained checkpoints of different T5 configurations to facilitate future research and analysis. ","Scale Efficiently: Insights from Pre-training and Fine-tuning