text: 231,226,1445794870354870278,3119778197,Hanie Sedghi,"🆕📰“Exploring the limits of large scale pre-training” <LINK> We systematically study the effect of scaling up data, model size and training time in image recognition on a wide range of downstream tasks, pinpoint the limits, the reasons & provide guidelines. 🧵 <LINK> We establish that scaling doesn’t lead to a one-model-fits-all solution. As US acc. improves, DS acc. saturates to values &lt;&lt; 100%. This gap depends on the relationship between US & DS tasks.+in set of models with similar US accuracy, the best model for different DS tasks varies. We investigate more than 4800 experiments on Vision Transformers, MLP-Mixers & ResNets with No. of parameters ranging from ten million to ten billion, trained on the largest scale of available image data (JFT, ImageNet21K) & evaluated on &gt; 20 downstream image recognition tasks <LINK> We propose a model for downstream performance that reflects the saturation phenomena & captures the nonlinear relationship in upstream and downstream performance. The model is fitted to the upper hull of trained networks & is robust to sample size variations & sampling biases. We study how scaling up the model size, data size, and compute affects DS performance and show that these parameters impact DS performance mainly through the US performance. 5/10 <LINK> Delving deeper to understand the reasons that give rise to these phenomena, we show that the saturation behavior we observe is closely related to the way that representations evolve through the layers of the Models. 6/10 <LINK> We showcase an even more extreme scenario where performance on upstream and downstream are at odds with each other: in order to have a better DS performance, we need to hurt US accuracy.+the optimal hyper-parameters for the head used in pre-training are different for US and DS. <LINK> The reason behind this discrepancy: by changing head hyper-parameters such as WD & LR, we push the information compressed in the head down to lower layers which leads to performance degradation on US and performance improvement on DS tasks that are related to the US task. 8/10 Our observations are robust to several choices: US data size, no. of shots, transfer vs few-shot setting & architecture. We assert that we should make design choices that improve performance on a breadth of DS tasks +when investing on scaling look at an extra axis: data diversity with my awesome collaborators @bneyshabur @samiraabnar @m__dehghani 10/10",https://arxiv.org/abs/2110.02095,"Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might observe that improvements in pre-training would transfer favorably to most downstream tasks. In this work, we systematically study this phenomena and establish that, as we increase the upstream accuracy, the performance of downstream tasks saturates. In particular, we investigate more than 4800 experiments on Vision Transformers, MLP-Mixers and ResNets with number of parameters ranging from ten million to ten billion, trained on the largest scale of available image data (JFT, ImageNet21K) and evaluated on more than 20 downstream image recognition tasks. We propose a model for downstream performance that reflects the saturation phenomena and captures the nonlinear relationship in performance of upstream and downstream tasks. Delving deeper to understand the reasons that give rise to these phenomena, we show that the saturation behavior we observe is closely related to the way that representations evolve through the layers of the models. We showcase an even more extreme scenario where performance on upstream and downstream are at odds with each other. That is, to have a better downstream performance, we need to hurt upstream accuracy. ",Exploring the Limits of Large Scale Pre-training,10,"['🆕📰“Exploring the limits of large scale pre-training”\n\n<LINK>\n\nWe systematically study the effect of scaling\nup data, model size and training time in image recognition on a wide range of downstream tasks, pinpoint the limits, the reasons &amp; provide guidelines.\n🧵 <LINK>', 'We establish that scaling doesn’t lead to a one-model-fits-all solution. As US acc. improves, DS acc. saturates to values &lt;&lt; 100%. This gap depends on the relationship between US &amp; DS tasks.+in set of models with similar US accuracy, the best model for different DS tasks varies.', 'We investigate more than 4800 experiments on Vision Transformers,\nMLP-Mixers &amp; ResNets with No. of parameters ranging from ten million to\nten billion, trained on the largest scale of available image data (JFT,\nImageNet21K) &amp; evaluated on &gt; 20 downstream image recognition tasks https://t.co/137zv2J22B', 'We propose a model for downstream performance that reflects the saturation\nphenomena &amp; captures the nonlinear relationship in upstream and downstream performance. The model is fitted to the upper hull of trained networks &amp; is robust to sample size variations &amp; sampling biases.', 'We study how scaling up the model size, data size, and compute affects DS performance and show that these parameters impact DS performance mainly through the US performance. 5/10 https://t.co/UrTy1jqHJv', 'Delving deeper to understand the reasons that give rise to\nthese phenomena, we show that the saturation behavior we observe is closely\nrelated to the way that representations evolve through the layers of the\nModels. 6/10 https://t.co/iJ6FA4Y0nT', 'We showcase an even more extreme scenario where performance on upstream and\ndownstream are at odds with each other: in order to have a better\nDS performance, we need to hurt US accuracy.+the optimal hyper-parameters for the head used in pre-training are different for US and DS. https://t.co/puKEwVkFso', 'The reason behind this discrepancy: by changing head hyper-parameters such as WD &amp; LR, we push the information compressed in the head down to lower layers which leads to performance degradation on US and performance improvement on DS tasks that are related to the US task. 8/10', 'Our observations are robust to several choices: US data size, no. of shots, transfer vs few-shot setting &amp; architecture. We assert that we should make design choices that improve performance on a breadth of DS tasks +when investing on scaling look at an extra axis: data diversity', 'with my awesome collaborators @bneyshabur @samiraabnar @m__dehghani  10/10']",21,10,2450