text: 338,23,1057077610549821440,3301643341,Roger Grosse,"Think you understand weight decay? Think again. We found three distinct mechanisms by which it achieves a regularization effect, depending on the architecture and optimization algorithm. New paper w/ @Guodzh, Chaoqi Wang, and Bowen Xu. <LINK> It's tempting in deep learning to just do what works, but it's important to sometimes dig deeper to track down what's really happening. In this case, the answer often had to do with the nuts and bolts of the scales of parameters and how this interacts with optimization hyperparameters. Not the stuff we usually write papers about, but it seems to matter a lot. @ogrisel @Guodzh Each K-FAC update is about 50% more expensive than SGD or Adam. So you can mentally adjust the plots based on that. But note that our hyperparameters were chosen based on final validation accuracy, so they might not be optimal for earlier in training. @AmirRosenfeld @Guodzh Yeah, I guess one way to think about it is that learning rate schedules have a huge impact on final performance, and with BN, the WD parameter gives you a knob with which to tune the effective learning rate decay schedule.",https://arxiv.org/abs/1810.12281,"Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization. Literal weight decay has been shown to outperform $L_2$ regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures. We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks. ",Three Mechanisms of Weight Decay Regularization,5,"['Think you understand weight decay?  Think again. We found three distinct mechanisms by which it achieves a regularization effect, depending on the architecture and optimization algorithm. New paper w/ @Guodzh, Chaoqi Wang, and Bowen Xu.\n\n<LINK>', ""It's tempting in deep learning to just do what works, but it's important to sometimes dig deeper to track down what's really happening."", 'In this case, the answer often had to do with the nuts and bolts of the scales of parameters and how this interacts with optimization hyperparameters. Not the stuff we usually write papers about, but it seems to matter a lot.', '@ogrisel @Guodzh Each K-FAC update is about 50% more expensive than SGD or Adam. So you can mentally adjust the plots based on that. But note that our hyperparameters were chosen based on final validation accuracy, so they might not be optimal for earlier in training.', '@AmirRosenfeld @Guodzh Yeah, I guess one way to think about it is that learning rate schedules have a huge impact on final performance, and with BN, the WD parameter gives you a knob with which to tune the effective learning rate decay schedule.']",18,10,1119