text: 214,57,1140559642663239680,259050097,Namhoon Lee,"1/4 A Signal Propagation Perspective for Pruning Neural Networks at Initialization: our new work on pruning neural networks at initialization is available now. paper: <LINK> with @tha_ajanthan, Stephen Gould, Philip Torr. 2/4 In SNIP (ICLR19), we showed that pruning can be done on a randomly initialized network without pretraining: ""pruning at initialization"". However, it's unclear exactly why pruning untrained networks is effective, how it should be understood, whether it can be extended further. 3/4 This work provides a signal propagation perspective based on dynamical isometry and mean field theory to pruning at initialization. We improve performance by orthogonal initialization, present unsupervised pruning, and bring forth the notion of neural architecture sculpting. 4/4 We are really excited about what this work could possibly bring us even more, so please stay tuned for our future endeavours! @_onionesque Thanks :))) Yeah I realized he had SNIPER paper a little while after I submitted SNIP; and of course I had chat with him about it :) @_onionesque Maybe I should name my future work SNIPER, aiming at the lottery ticket lol @_onionesque preempted here thanks to your help lol",https://arxiv.org/abs/1906.06307,"Network pruning is a promising avenue for compressing deep neural networks. A typical approach to pruning starts by training a model and then removing redundant parameters while minimizing the impact on what is learned. Alternatively, a recent approach shows that pruning can be done at initialization prior to training, based on a saliency criterion called connection sensitivity. However, it remains unclear exactly why pruning an untrained, randomly initialized neural network is effective. In this work, by noting connection sensitivity as a form of gradient, we formally characterize initialization conditions to ensure reliable connection sensitivity measurements, which in turn yields effective pruning results. Moreover, we analyze the signal propagation properties of the resulting pruned networks and introduce a simple, data-free method to improve their trainability. Our modifications to the existing pruning at initialization method lead to improved results on all tested network models for image classification tasks. Furthermore, we empirically study the effect of supervision for pruning and demonstrate that our signal propagation perspective, combined with unsupervised pruning, can be useful in various scenarios where pruning is applied to non-standard arbitrarily-designed architectures. ","A Signal Propagation Perspective for Pruning Neural Networks at