text: 117,111,1391829045889863683,32713500,Rob Nowak,"What kinds of functions do deep neural networks learn? Now we know! <LINK> Our paper presents a new representer theorem for deep ReLU networks and provides theoretical insights into weight decay, sparsity, skip connections, and low-rank weight matrices. @roydanroy The main result says that a solution to a variational problem in a certain Banach space (compositional Radon BV) is a deep ReLU network having widths proportional to sample size, and that this optimization is equivalent to one over NNs that is min loss+ridge penalty on weights. @roydanroy It really falls together seamlessly. I think these are the “right” spaces to be working with, at least for multilayer relu nets.",https://arxiv.org/abs/2105.03361,"We develop a variational framework to understand the properties of functions learned by fitting deep neural networks with rectified linear unit activations to data. We propose a new function space, which is reminiscent of classical bounded variation-type spaces, that captures the compositional structure associated with deep neural networks. We derive a representer theorem showing that deep ReLU networks are solutions to regularized data fitting problems over functions from this space. The function space consists of compositions of functions from the Banach spaces of second-order bounded variation in the Radon domain. These are Banach spaces with sparsity-promoting norms, giving insight into the role of sparsity in deep neural networks. The neural network solutions have skip connections and rank bounded weight matrices, providing new theoretical support for these common architectural choices. The variational problem we study can be recast as a finite-dimensional neural network training problem with regularization schemes related to the notions of weight decay and path-norm regularization. Finally, our analysis builds on techniques from variational spline theory, providing new connections between deep neural networks and splines. ","What Kinds of Functions do Deep Neural Networks Learn? Insights from