text: 463,32,1397908669501632519,1220766363549126656,Alain Rossier,"New paper ""Scaling Properties of Deep Residual Networks"" has been accepted for publication as a conference paper at ICML 2021! @alainsamjr @icmlconf Collaboration with @instadeepai <LINK> 1/n <LINK> As you may know, residual network (ResNet) is a rockstar among Deep Learning architectures. It leads to huge improvements over feedforward networks for computer vision and speech recognition. 2/n <LINK> The forward rule of a ResNet (top) looks oddly familiar to the Euler scheme of a differential equation (bottom). This insight culminated in the now famous Neural ODE model @RickyTQChen that uses various ODE schemes to train their model. 3/n <LINK> For those of you familiar with numerical scheme of ODEs, the link between the two equations only holds under the particular conditions below... and they may not be satisfied in practice. 4/n <LINK> We introduce a more general framework for the trained weights of ResNets that admits the Neural ODE assumptions as a special case. We test the framework on synthetic and standardized datasets (coming at you, CIFAR 10). 5/n We found that depending on the activation function, the shape of the trained weights for an absurdly deep network (L ~ 10'000) looks drastically different! left: sigma=tanh; right: sigma=relu 6/n <LINK> For those of you who know about stochastic calculus, the image on the right looks pretty much like white noise. So let's integrate it... We get a process that looks oddly familiar with a diffusion process (think: Brownian motion)! 7/n <LINK> In our paper, we lay down two general assumptions that describe the behaviour of trained weights of ResNets. We prove the convergence of their limiting equations, which can be either the Neural ODE, a linear ODE or a linear-quadratic SDE (the interesting stuff below). 8/n <LINK> What’s the point of all this? Well, knowing the scaling limit of trained weights will help you choose your initialization strategy, as well as directly train the scaling limit in the Neural ODE fashion. 9/n Is this the end of the story? Of course not! Our results hold for fully-connected networks, but we observed that convolutional networks have different scaling behaviour (sparse limit?)... stay tuned for the next episode! <LINK> 10/10, like this thread",https://arxiv.org/abs/2105.12245,"Residual networks (ResNets) have displayed impressive results in pattern recognition and, recently, have garnered considerable theoretical interest due to a perceived link with neural ordinary differential equations (neural ODEs). This link relies on the convergence of network weights to a smooth function as the number of layers increases. We investigate the properties of weights trained by stochastic gradient descent and their scaling with network depth through detailed numerical experiments. We observe the existence of scaling regimes markedly different from those assumed in neural ODE literature. Depending on certain features of the network architecture, such as the smoothness of the activation function, one may obtain an alternative ODE limit, a stochastic differential equation or neither of these. These findings cast doubts on the validity of the neural ODE model as an adequate asymptotic description of deep ResNets and point to an alternative class of differential equations as a better description of the deep network limit. ",Scaling Properties of Deep Residual Networks,10,"['New paper ""Scaling Properties of Deep Residual Networks"" has been accepted for publication as a conference paper at ICML 2021! @alainsamjr @icmlconf \nCollaboration with @instadeepai \n<LINK>\n\n1/n <LINK>', 'As you may know, residual network (ResNet) is a rockstar among Deep Learning architectures. It leads to huge improvements over feedforward networks for computer vision and speech recognition.\n\n2/n https://t.co/B5yKkhLeDh', 'The forward rule of a ResNet (top) looks oddly familiar to the Euler scheme of a differential equation (bottom). This insight culminated in the now famous Neural ODE model @RickyTQChen that uses various ODE schemes to train their model.\n\n3/n https://t.co/bbMbF8Gawp', 'For those of you familiar with numerical scheme of ODEs, the link between the two equations only holds under the particular conditions below... and they may not be satisfied in practice.\n\n4/n https://t.co/Sx6VEtn26E', 'We introduce a more general framework for the trained weights of ResNets that admits the Neural ODE assumptions as a special case. We test the framework on synthetic and standardized datasets (coming at you, CIFAR 10).\n\n5/n', ""We found that depending on the activation function, the shape of the trained weights for an absurdly deep network (L ~ 10'000) looks drastically different!\nleft: sigma=tanh;  right: sigma=relu\n\n6/n https://t.co/otS7NZlW2R"", ""For those of you who know about stochastic calculus, the image on the right looks pretty much like white noise. So let's integrate it... We get a process that looks oddly familiar with a diffusion process (think: Brownian motion)!\n\n7/n https://t.co/2OAta45u0R"", 'In our paper, we lay down two general assumptions that describe the behaviour of trained weights of ResNets. We prove the convergence of their limiting equations, which can be either the Neural ODE, a linear ODE or a linear-quadratic SDE (the interesting stuff below).\n\n8/n https://t.co/Lg0W74Xs86', 'What’s the point of all this? Well, knowing the scaling limit of trained weights will help you choose your initialization strategy, as well as directly train the scaling limit in the Neural ODE fashion.\n\n9/n', 'Is this the end of the story? Of course not! Our results hold for fully-connected networks, but we observed that convolutional networks have different scaling behaviour (sparse limit?)... stay tuned for the next episode!\nhttps://t.co/9PXGsuObLp\n\n10/10, like this thread']",21,05,2254