text: 216,194,1357364064154968066,1062061528931819523,Kale-ab Tessera,"Excited to present my first preprint 🥳🎉 - ""Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization"" , with @sarahookr and @BenjaminRosman . 📜 <LINK> . We use Gradient Flow (GF) to study sparse network optimization. <LINK> That is to say, we use GF to study how sparse networks are affected by different optimizers, activation functions, architectures, learning rates and regularizers.  This follows on promising GF work - <LINK> (by @utkuevci) and <LINK> . 1. Firstly, to study sparse networks, we propose a simple, empirical framework - Same Capacity Sparse vs Dense Comparison (SC-SDC). The key idea is to compare sparse networks to their equivalent dense counterparts (same number of connections and same initial weight init). <LINK> 2. Secondly, we also propose a new, normalized, layerwise measure of gradient flow, Effective Gradient Flow (EGF). EGF is normalized by the number of active weights and distributed evenly across all the layers. We show EGF correlates better, than other GF measures, to ... <LINK> performance in sparse networks and hence it is a good formulation for studying the training dynamics of sparse networks. <LINK> 3.1 Using EGF and SC-SDC, we show that BatchNorm is more important for sparse networks than it is for dense networks (the result is statistically significant), which suggests that gradient instability is a key obstacle to starting sparse. 3.2 We show that optimizers that use an exponentially weighted moving average (EWMA) to obtain an estimate of the variance of the gradient, such as Adam and RMSProp, are sensitive to higher gradient flow. This could explain why these methods are more sensitive to L2 and data aug. 3.3 Finally, we show that Swish and PReLU (when using SGD) are promising activation functions, especially for sparse networks. For the Swish result, we suggest this could be due to Swish’s non-monotonic formulation, that allows for negative gradient flow, which helps with 3.2. <LINK> We also extend some of these results from MLPs -&gt; CNNs and from random, fixed sparse networks -&gt; magnitude pruned networks. <LINK> In conclusion, our work agrees with and contributes to the literature that emphasizes that initialization is only one piece of the puzzle and taking a wider view of tailoring optimization to sparse networks yields promising results. Thanks for reading. Also, please let us know if we missed any related work/results and if you have any feedback 🙏:)",https://arxiv.org/abs/2102.01670,"Training sparse networks to converge to the same performance as dense neural architectures has proven to be elusive. Recent work suggests that initialization is the key. However, while this direction of research has had some success, focusing on initialization alone appears to be inadequate. In this paper, we take a broader view of training sparse networks and consider the role of regularization, optimization, and architecture choices on sparse models. We propose a simple experimental framework, Same Capacity Sparse vs Dense Comparison (SC-SDC), that allows for a fair comparison of sparse and dense networks. Furthermore, we propose a new measure of gradient flow, Effective Gradient Flow (EGF), that better correlates to performance in sparse networks. Using top-line metrics, SC-SDC and EGF, we show that default choices of optimizers, activation functions and regularizers used for dense networks can disadvantage sparse networks. Based upon these findings, we show that gradient flow in sparse networks can be improved by reconsidering aspects of the architecture design and the training regime. Our work suggests that initialization is only one piece of the puzzle and taking a wider view of tailoring optimization to sparse networks yields promising results. ","Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network